Bobby Costin

Read the full post here: tilderesearch.com/blog/momoe Mo' Money, Mo(E)' Problems

~2/8~ Despite their promise, existing open-source MoE kernels/implementations often fail to deliver on performance (particularly during training and fine-tuning). They are hindered by Python control flow, inefficient token batching, and suboptimal memory access patterns.

Mixture‑of‑Experts (MoE) powers many frontier models like R1, K2, & Qwen3 ⚡️ To make frontier-scale MoE models accessible to train, we open-source MoMoE, a hyper-performant MoE implementation built for training and inference, outpacing the fastest existing ones by up to: - 70% in inference throughput - 20% in training throughput - 90% in memory savings - and more! A 🧵 (1/8)

We’re excited to announce that Tilde completed an $8M seed round earlier this year, led by Khosla Ventures. Understanding model intelligence is the most important problem in the world, and the key to actualizing the promise that ASI can offer. 🧵 A thread on our approach:

Sparse attention (MoBA/NSA) trains faster & beats full attention in key tasks. But we’ve had no idea how they truly work…until now. 🔍 We reverse-engineered them to uncover: - Novel attention patterns - Hidden "attention sinks" - Better performance - And more A 🧵… ~1/8~