Aayush Karan

So how do you make an efficient inference chip? Like most systems problems, it comes down to getting the most useful work out of a fixed power budget Maia 300 is a cool example of getting significant wins in tokens/W by fitting to the compute constraints of inference. Unlike training and high-throughput inference, interactive inference is mostly memory bound. This enables chip designers to cut systolic flops (saving both power and area) while designing the architecture around inference dataflow to limit data movement Looking at the specs, Maia is eerily similar to OpenAI’s Jalapeño. While both orgs clearly did some very cool things in the architecture, microarchitecture, and networking, the main win is HBM4 (which gives >2x bandwidth over HBM3) and better provisioning of resources If you look at the FLOPs/W and bandwidth/W figure I scrapped together, Rubins will still be the best for compute-bound workloads (and combined with Nvidia’s superior software stack and reliability, they will likely maintain their monopoly on training) However, if we estimate the ceiling of Maia 300 and Jalapeño at ~40% more power efficient than a Rubin on inference (in large part because they have similar HBM4 bandwidth but spend way less power on FLOPs) and they can bring up a reasonable software stack, there is a strong case to use Maia/Jalapeno-class chips for more power efficient memory-bound inference Another side note for anyone interested in Groq and Cerebras chips who also claim inference-specialization, they actually cut out HBM altogether and provision much more area for SRAM, which hyper-optimizes for ultra low batch, low latency inference. They are moving in the direction of heterogenous disaggregation by using GPUs for prefill and even decode attention, while keeping their SRAM chips for exclusively decode. However, because of memory capacity limitations, these approaches struggle to keep up with growing model sizes and longer context lengths I think we’ll see more chips along the Maia, Jalapeño, TPU v8i route that still uses HBM to keep weights and KV cache in a more power efficient way than SRAM, but provision chips correctly to not waste power on training optimizations Takeaway - inference has become sufficiently different from training (and has carved out a majority of demand) that we are fundamentally rethinking our software stacks, chip architectures, microarchitectures, and resource provisioning. Its exciting to see the wins starting to pay off with meaning power efficiency gains!