10X Builder | AI Performance Engineer | Co-founded @mybridgecard(YC S22)
Making GPUs Go Brrr
Joined July 2018
- Tweets3.8K
- Following1.7K
- Followers568
- Likes5.9K
Pinned Tweet
I recently got a PR merged into @nvidia's Dynamo (A Datacenter Scale Distributed Inference Serving Framework) and the process was a great A/B test of KV data transfer over TCP vs KV data transfer over RDMA.
PR link: github.com/ai-dynamo/dynamo/… 🧵
Festus retweeted
The sessions from SysConf 2026 have been uploaded on YouTube: youtube.com/playlist?list=PL….
Watch, re-watch and share because we think they contain a lot of value.
#SysConf26
Many of us have used inference engines like @vllm_project and @sgl_project , but what actually happens under the hood?
@charles_irl from @modal gives a great deep dive into how modern LLM inference engines work.
youtu.be/woIYJYd_etI
Fun fact the details is arguably much simpler than databases and web servers.
I recommend pairing this talk with the OG blog post by @gordic_aleksa:
aleksagordic.com/blog/vllm
It goes much deeper into the code and implementation details of @vllm_project.
As we push LLMs towards trillion-parameter scale and increasingly long agentic workloads, we face challenges along three major axes: model depth, model width, and context length.
While depth and width drive model size and compute, longer contexts introduce another challenge: attention complexity and KV-cache memory.
There have been two interesting approaches to this:
- @deepseek_ai’s Multi-head Latent Attention (MLA) compresses keys and values into a shared low-rank latent representation, significantly reducing KV-cache memory.
- @Kimi_Moonshot’s Kimi Delta Attention (KDA) rethinks attention itself, using recurrent state updates to achieve linear rather than quadratic scaling with sequence length in its attention layers.
This video by @baseten does a deep dive into the latter using Kimi K3 as a case study.
Really enjoyed watching it and absolutely recommend!
youtu.be/a58ldGhR5wM?is=Vetw…
Huge!!!
DeepSeek just open sourced its high-performance, large-scale inference framework DeepInfer in Apache-2.0 license.
github.com/deepseek-ai/DeepI…
Supported features:
- Optimized kernels, model and sampler runtime
- Parallelism: Attention DP / balanced CP, MoE EP + EPLB
- DSpark and confidence-scheduled verification
- SWA bounded replay
- Asynchronous scheduling (zero-overhead batch scheduler)
- Batch-invariant inference
- Deterministic sampling
- Structured output
One thing I've found really interesting about efficient LLM serving is how many of the optimizations we use today build on ideas that have existed in computer science for decades.
Some examples:
1. PagedAttention and Virtual Memory
2. Pipeline Parallelism and CPU Instruction Pipelining
3. Speculative Decoding and Speculative Execution
🧵
A smaller draft model generates several candidate tokens, and the larger target model verifies them in a single forward pass.
If enough of those tokens are accepted, we can advance multiple tokens without repeatedly running the expensive target model for each one.
We often talk about scaling AI infrastructure as if it just means adding more GPUs.
This conversation between @dylan522p and @sorcer, CTO of @CoreWeave, was a great reminder of how much more there is underneath.
At large bare-metal scale, you’re thinking about hardware failures and automated recovery, security and isolation, networking, power infrastructure, and liquid cooling.
I loved the discussion around how CoreWeave approaches full liquid cooling and how former crypto-mining sites have become valuable for AI infrastructure because much of the hard electrical work is already there: large grid connections, substations and power distribution.
Really great conversation on what it actually takes to operate AI infrastructure at scale.
youtu.be/5kerW51jdgk?is=zKx4…
I'll be speaking at 15:40 GMT+1 today. Reaally excited to be giving this talk.
Tune in. #SysConf #SysConf26
We are excited to announce Festus Owumi (@_enfinity) as a speaker at SysConf 2026.
His talk is titled “Distributed LLM Inference From First Principles: When One GPU Isn’t Enough.”
Distributed inference solves at least two important problems: memory capacity limits on GPUs and a need for higher serving throughput. But how do we build distributed inference?
In this talk, Festus walks us through building a distributed inference runtime from first principles, based on a learning project. It starts by reviewing the “boring” case of one model, one process and one [CPU] device. Then the talk will introduce ranks, world size and point-to-point communication.
After the above foundation, Festus will introduce more complex concepts to distribute the model such as Pipeline Parallelism, Tensor Parallelism and the techniques used to achieve them including partitioning and collectives such as AllGather or AllReduce.
#SysConf #SysConf26
ALT “Distributed LLM Inference From First Principles: When One GPU Isn’t Enough” is the title of Festus Owumi’s talk at SysConf 2026.
ALT Festus Owumi is an engineering leader and ML systems engineer building production distributed systems and cloud-native infrastructure, with a focus on LLM inference and GPU serving. He's contributed to NVIDIA Dynamo, an open-source inference framework.