Vincent Abbott (@vtabbott_) header
Vincent Abbott (@vtabbott_) profile picture

Maker of *those* diagrams for deep learning algorithms | @mit @mitlids PhD student under @GioeleZardini

Boston
Joined July 2022
Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)Photo by Vincent Abbott (@vtabbott_)
agi is upon us
5
422
Did I ever post about how we got backpropagation into the diagrams? Aim is to develop *the* mathematical tools models will use to understand themselves.
3
14
1
161
5,521
Something nice about abstract representations is we can just *derive* where the KV-cache should go to get the inference form. There is an expressible relationship between prefill and inference! The little symbols in the top right switch between modes.
2
3
2
59
3,535
Posted Mixtral here cause it fits, but check out other models here ~~ zardini.mit.edu/diagrams/mix…
1
2
419
Kimi-K3 and MiMo-V2.6-Pro added to zardini dot mit dot edu slash diagrams ~ enjoy !
3
2
40
2,436
Doing an SF trip from Thursday Oct 1st to Tuesday Oct 6th! DM if you’re around and want to talk about GPU software/hardware :)
2
9
1,157
The updated diagrams page with @GioeleZardini lets us "see like a GPU". We can see how attention is usually expressed, but this doesn't give much info. How are we applying the Matmul? Over which axis is the SoftMax operating? (1/4)
3
6
1
116
4,734
But we can do even better! GPUs see everything in terms of axes - they are what we batch, parallelize, stream, and reduce over. Notice how the q-axis is parallelized in each expression? Well, by drawing the axes of arrays, we can immediately see this pattern. (3/4)
1
1
11
444
All of these diagrams can be found on our updated, interactive website. Switch between the forms at the top of the page to start seeing like a GPU yourself! (4/4) zardini.mit.edu/diagrams/att…
1
6
301
Big update to the diagrams page :) will make a series of posts with @GioeleZardini zardini.mit.edu/diagrams/att…
2
9
2
155
7,111
Here's *one* of the new features I really like: comparing standard graphs to broadcasted diagrams! This should build on intuition better than previous diagrams, building up from simple arrows-and-boxes up to fully broadcasted forms. The wires represent axes!
2
469
Vincent Abbott retweeted
Check this out!!! 🚀
At long last, the interactive DeepSeek-v4.1-Flash diagram is up ~ vtabbott.io/diagrams/deepsee…
1
3
901
At long last, the interactive DeepSeek-v4.1-Flash diagram is up ~ vtabbott.io/diagrams/deepsee…
6
38
3
295
12,441
damn deepseek v4.1 flash engram got hands, at least at the end of the interactive deepseek v4.1 flash project I'll have an understanding of the entire architecture
18
1,189
busy cooking
7
9
206
11,459
FlashAttention on an actual napkin: very Boston-coded experience with @vtabbott_ arxiv.org/pdf/2412.03317
2
44
2,172
Working on a diagram for DeepSeek V4.1-Flash. Some really interesting features that I'll write about, the KV compression is next level.
17
24
4
424
17,080
Fable is past the critical point where I can interact with my codebase via plain English commands. The modern gen of models is crazy. Like, we’re at takeoff now. This is it.
2
1
10
1,166
The critical point is a competence + reliability. It needs to not make mistakes, not make a mess, and parse instructions well. Opus doesn’t quite make the cut, and Astra is ime VERY spikey. I’m curious about others’ experience though.
1
5
576
I cannot recommend highly enough making visuals for what you’re working on. The model => human bit rate is a major workflow bottleneck and humans have a very very good visual reasoning stack that you should use.
8
239