swe @databricks by day pretraining research by night
Joined October 2025
- Tweets33
- Following720
- Followers74
- Likes3.4K
previously i thought linear attention was about trading off some long-context performance for major efficiency/computational gains. turns out a mix of linear attn and mla can outperform pure mla on long context
one intuition behind this is that linear attn can act as a learned, data-dependent positional bias. in fact they conclude that rope is not only redundant, but hurts the performance
paper: kimi linear (arxiv.org/abs/2510.26692)
Marko Šišović retweeted
New NanoGPT Speedrun WR at 79.2 (-0.5s) from @sisovicm, by moving the MLP up projection to FP8 on the forward pass, and increasing step count by 5 to offset the loss increase. The march towards lower precision continues! github.com/KellerJordan/modd…
Marko Šišović retweeted
New NanoGPT Speedrun WR at 89.1 (-0.7s) from @sisovicm , with a technique called partitioned hyperconnections. The learned weights reveal that the final attn modules prefer to ignore the prediction vectors generated by the final MLPs, and instead query representations from slightly earlier layers. github.com/KellerJordan/modd…
