We are getting closer to the official launch of TorchTPU.
ALERT🚨🚨: On apples-to-apples InferenceX Comparisons, TPUv7 Ironwood achieves 50% better perf per dollar than Blackwell Ultra on the new TorchTPU external inference stack!
TorchTPU brings native PyTorch to TPUs, & along with Google open-sourcing a bunch of their Pallas inference kernels, Google has laid out a solid foundation for TPU to rapidly externalize. We at SemiAnalysis strongly believe that TPU externalization is heading in the right direction and moving full steam ahead.
Readers added context they thought people might want to know
The 50% perf/dollar claim is from SemiAnalysis' own modeled InferenceX preview using TCO assumptions, not independent/audited benchmarks. TorchTPU stack is in private beta; attached video is AI-generated.
newsletter.semianalysis.com/p/tpu-inferenc…
inferencex.semianalysis.com
siliconreport.com/best-ai-chips-…
x.com/Xxi5olc/status…
Watch this space - SGLang on TorchTPU (developers.googleblog.com/to…) coming soon.
RadixArk and Google Cloud are joining forces with the SGLang community to make TPU a drop-in, cost-efficient path to frontier inference.
SGL-JAX already serves the major open model families on the latest TPU generations: Gemma, Qwen, DeepSeek, GLM, Kimi, Ling, MiniMax, MiMo, Grok, plus Wan and Flux for video and image generation.
With this partnership, TPU unlocks SGLang's production feature set: 5D parallelism, Radix Cache, HiCache, quantization, speculative decoding, on TPU Pallas kernels co-built by Google, RadixArk, and the SGLang community. SGL-torchtpu opens a PyTorch-native path to all of it.
SGLang on TPU comes with the same API and the same features, so for teams already running it, getting started is simple. Going forward, TPU support for new open models will land alongside the rest, and that carries forward to each new TPU generation.
Full announcement and SGL-JAX repo 👇
Lee Howes retweeted
A new PyTorch-native backend is coming to unlock the power of Google TPUs:
✨ Run existing PyTorch with minimal code changes.
✨ Get a 50-100%+ performance boost with Fused Eager mode.
Read the engineering deep dive here: goo.gle/4vbTQQl
#TorchTPU #PyTorch #MLOps #AI
ALT The Google Cloud and PyTorch logos appear with text reading "Google TPU" and "PyTorch" over a background image of a green circuit board. The board features several large chips labeled "Ironwood" and rows of silver heatsinks.
Lee Howes retweeted
Career advice for an ambitious infrastructure engineer like me: find the world's largest Rust codebase and solve their hard problems.
My team is hiring a senior engineer to work closely with me. In simple form, the job is: show up to an office in NYC, Seattle, Bellevue, or Menlo Park at least 3 days per week and make better developer infrastructure for a medium-sized Rust codebase (10 million lines) so that it can become a large Rust codebase (200 million lines) sooner.
This is highly leveraged work that directly impacts the productivity, reliability, and performance for hundreds of teams building distributed systems of all kinds, developer tools, mobile apps with billions of users, data pipelines, AI training, network infrastructure, privacy infrastructure, consumer devices, and more in Rust.
Lee Howes retweeted
I am looking to recruit a PhD student (fully funded at UK home tuition rate) to work on automated testing and verification of machine learning compilers and runtimes! Deadline: 30th April. Please spread the word! Details here: doc.ic.ac.uk/~afd/PhD-Advert…
Lee Howes retweeted
Optimizing Linux kernel with BOLT: github.com/llvm/llvm-project…
> By improving the code layout, BOLT can boost the kernel's performance by up to 5%.
Lee Howes retweeted
Meta is hiring compilers, low-level, and performance engineers. Especially engineers w/ expertise in CPU and GPU performance, CUDA, C++, x86, and LLVM. The roles are located in the SF Bay Area, Seattle, and NY. A U.S. work permit is required.
Lee Howes retweeted
There is a perception that the sender async model is hard to understand.
The underlying concepts are actually very simple.
These five interfaces are the sum total of the sender model. Everything else is window dressing.
Lee Howes retweeted
Wisdom, from @kirkshoop:
ALT no-detached-work is religious in the same way that no-raw-new is religious and goto-is-harmful is religious. the common theme in each one is that structure (async-scope, RAII, if/for) produces better code.
Lee Howes retweeted
Hey! You!
I wrote a blog post!
Come read about why I'm unreasonably excited about senders.
It's Monday morning. Work can wait.
ericniebler.com/2024/02/04/w…
Getting on the threads bandwagon (threads.net/@xrikcus) as one does.
Lee Howes retweeted
$META "A study concluded that every dollar spent on our ads drives, on average, $3.31 in revenues for our advertisers in the U.S. Over $400 billion in economic activity annually is linked to our platforms, supporting more than 3 million jobs."
nitter.cf/UPHOLDINGS/status/1489…
Before you call $FB evil, you should talk to small businesses who have leveraged their platform to attract customers (many without paying a dime!).
2nd order effects.
Lee Howes retweeted
Build faster with #Buck2! Our #opensource build system is designed to make your build experience better, including:
- Increased parallelism
- Integration with remote executing & virtual file systems
- Redesigned console output
Learn how: ow.ly/lWNg50Nzoxm
Lee Howes retweeted
So it begins...
isocpp.org/files/papers/P276…
ALT Screenshot of a WG21 paper entitled Sender/Receiver Interface for Networking
Lee Howes retweeted
Read about how NVIDIA is using portable asynchrony to accelerate standard C++.
My team and I published an article about the new stdexec library -- based on P2300 `std::execution` -- that NVIDIA recently shipped.
hpcwire.com/2022/12/05/new-c…
#cpp #hpc #nvidia @NVIDIAHPCDev
Lee Howes retweeted
New blog post: Introducing `cxx-async`! pcwalton.github.io/_posts/20…
It's a new crate I've published that allows easy interoperability between C++20 coroutines and Rust async I/O. You can await/co_await on both sides and it "just works".
Lee Howes retweeted
Rust is now a first-class server-side language at Meta. Supporting a programming language at Meta is a long and careful process, and including Rust is a testament to the qualities of the language. I am proud of our team that made this possible.
engineering.fb.com/2022/07/2…
Earlier this morning Facebook announced that we are joining the Rust foundation: developers.facebook.com/blog…
Rust has grown in importance to our developers and we recently started a full time team focused on supporting Rust developers inside and outside the company. My team and I put together a blog post with some of the history: engineering.fb.com/2021/04/2…
Recently Rust continued its growth to become a "supported language" within Meta. Today my colleague Eric Garcia published a post describing what that means: engineering.fb.com/2022/07/2…
Lee Howes retweeted
Just sent out a proposal for ClangIR, a new MLIR based IR for Clang. Very excited with the new diagnostics possibilities for C++. Check out the RFC for github links and more information!
discourse.llvm.org/t/rfc-an-…
![[RFC] An MLIR based Clang IR (CIR)](https://pbs.twimg.com/card_img/2116062151085637632/Qk5NtUY2?format=jpg&name=280x280_2)