Lisen (@Lisennlp) profile picture

Lisen

@Lisennlp
 Vancouver
Joined October 2022
  • Tweets3
  • Following13
  • Followers72
New NanoGPT Speedrun WR at 76.3 (-2.9s) from @Lisennlp , with a radical boost to the final attention layer that drops step count by 7%, called lightweight Dynamically Composable MHA. A secondary local window (112 tokens) is recomputed on the same q/k. After softmax it mixes the per-head attn maps into one unified map, then applies the shared map to each head's values with a per-head scale. Every mix and scale operation is computed per-token from a MUDD MLP. All in a custom kernel. Takes inspiration from Dynamically Composable MHA arxiv.org/abs/2405.08553. github.com/KellerJordan/modd…
Improving Transformers with Dynamically Composable Multi-Head Attention

Improving Transformers with Dynamically Composable Multi-Head Attention

Multi-Head Attention (MHA) is a key component of Transformer. In MHA, attention heads work independently, causing problems such as low-rank bottleneck of attention score matrices and head...

arxiv.org
6
28
2
300
36,384
New NanoGPT Speedrun WR at 81.8 (-2.6s) from @.Lisennlp on Github with MUDD skip connections, an expressive and efficient mechanism for data dependent skips! Instead of a learned scalar or sigmoid(linear) gate, MUDD uses a 64 neuron 'MLP' to generate the coefficients. The key efficiency is in reusing the same input projection for up to 14 coefficients at once. github.com/KellerJordan/modd…
Image from Larry Dial (@classiclarryd)
8
29
2
176
13,951
Load more