Pinned Tweet
(1/9) 🐍 Mamba Recall Scaling Laws: accepted to #NeurIPS2026! 🎉
How many facts can Mamba actually store and recall?
By reverse-engineering Mamba models, we reveal an interpretable hash-based recall mechanism, and theoretically derive precise scaling laws - predicting recall accuracy directly from model dimensions. Here's the story. 🧵👇
📄 Paper: arxiv.org/abs/2609.07681
💻 Code: github.com/yuvalko1/mamba-re…
(8/9) The bottom line:
💡 To recall N_f facts over a vocabulary of size V, Mamba's dimensions must satisfy N·D ~ N_f·log V - state memory linear in the number of facts, and logarithmic in the vocabulary size.
Interestingly, this is optimal: an information-theoretic argument shows that any fixed-state recurrent model needs at least N_f·log V bits of state memory for reliable recall - for Mamba, that reads N·D = Ω(N_f·log V). The hashing mechanism Mamba learns is essentially the best possible use of its state.
(9/9) For full details, proofs, and many more results, see:
📄 Paper: arxiv.org/abs/2609.07681
💻 Code: github.com/yuvalko1/mamba-re…
Finally, huge thanks to my amazing collaborators @abk_tau, @RGiryes, and @ItamarZimerman (who initiated and led this work), and to my advisor @liorwolf - for the invaluable guidance and support that made this work possible, while also making the process meaningful and enjoyable! ✨

