Ze-Wei (Johnny) Liou (@zwljohnny) profile picture

Second year PhD @Princeton advised by @tri_dao working on ML Systems

Princeton, NJ
Joined February 2026
Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)Photo by Ze-Wei (Johnny) Liou (@zwljohnny)
Ze-Wei (Johnny) Liou retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
261
825
288
7,379
1,045,457
Ze-Wei (Johnny) Liou retweeted
TPUs have different bottlenecks from GPUs: few cores so you don’t need yo think about parallelism as much, large VMEM so easier to fuse kernels, but vector units have lower throughput so anything non matmul require much more care
Releasing FLAT: Flash Linear Attention on TPUs 🚀 Pallas training kernels for Linear Attention, Mamba-2 and GLA. On one TPU v6e, forward + backward is up to 3x faster than FLA on an H100 for GLA, and 1.5x faster for Mamba-2. 🧵
14
24
367
16,935
Ze-Wei (Johnny) Liou retweeted
Releasing FLAT: Flash Linear Attention on TPUs 🚀 Pallas training kernels for Linear Attention, Mamba-2 and GLA. On one TPU v6e, forward + backward is up to 3x faster than FLA on an H100 for GLA, and 1.5x faster for Mamba-2. 🧵
14
45
8
262
48,090
ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang! It speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x), and now also enables a low-precision SSM state whose rounding error never propagates through the recurrence. Building on our earlier ReplaySSM work, we collaborated with @NVIDIA to pin down what the method really unlocks. Caching SSM inputs instead of the state is what ReplaySSM does. What it changes is that computing the output and updating the state can be separated during decoding. All three gains follow from that separation. Joint blog post, paper, and code with @NVIDIA and @tri_dao 👇
20
77
13
455
72,728
[5/N] The KL gap shows up in accuracy, and it shows up where you would expect: long generations. On GPQA-Diamond with Nemotron-3-Super-NVFP4, split by generation length: a shared NVFP4 state matches FP32 on short answers and falls away as generations grow, down to 23% versus 51% in the longest quartile. Separate precisions for the two jobs stay within a few points of FP32 throughout.
2
2
10
981
[6/N] The speed side is already in serving engines. ReplaySSM is integrated in TensorRT-LLM, vLLM, FlashInfer, and SGLang, in production in GLM-5.3-Flash and Kimi-K3. At serving batch sizes on one B300: standard decoding 1.2 to 1.47x faster across four hybrid models, and speculative decoding 1.85x and 2.1x the throughput of vLLM's spec decoding on the two 120B-class models. Together with @NVIDIA, we contributed the integration to these open-source serving systems: - TensorRT-LLM: github.com/NVIDIA/TensorRT-L… - vLLM: github.com/vllm-project/vllm…, github.com/vllm-project/vllm…, github.com/vllm-project/vllm… - FlashInfer: github.com/flashinfer-ai/fla…, github.com/flashinfer-ai/fla…
2
11
892
Ze-Wei (Johnny) Liou retweeted
🎉 Thrilled to share that our paper, “Thought-Level Beam Search for Reasoning,” has been accepted to COLM 2026! Different from Self-Consistency and Pruning-only approaches, Gambit actively reallocates test-time compute while generation is in flight — pruning low-scoring reasoning traces and branching from the strongest prefixes. See thread 🧵
11
24
3
118
10,866
Ze-Wei (Johnny) Liou retweeted
Robot policies can move but can't think. LLMs can think but can't move. So we connected them. Real robot: 16.7% → 97.3% Sim (LIBERO-PRO): 12.8% → 53.3%
23
53
9
303
96,710
Why do we store the SSM state at all? More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters. We only write it back every step so the next step can read it. ReplaySSM caches the recent inputs instead and rebuilds the state on the fly. Same outputs, half the memory traffic → ~2x on spec decode at large batch sizes, which barely even helped SSMs before → up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B) Work with @tri_dao Blog + Code👇
13
24
3
166
47,085
[5/N] ReplaySSM also changes what must be produced. Before, states and outputs were both needed. Now, most steps only need the outputs. So we compute the output directly from the buffer and never materialize the state at all, which is what the figure below shows for a single decode step. Speculative decoding is where this really pays off, since the same output-only form removes the serial state dependence that made SSMs hard to parallelize and turns the whole draft verification into GEMMs.
1
1
8
937
[6/N] ReplaySSM works for both standard and speculative decoding, and it generalizes to GDN. We built it on vLLM and tested at serving batch sizes, 4B to 550B. Standard decoding gets up to 1.48x, and speculative decoding hits 1.96x. The speculative decoding result is particularly important because vLLM's current SSM speculative decode is actually slower than just decoding token-by-token.
2
6
757