Sangdoo Yun

a few thoughts on "recurrent depth" transformers the main question: why recurrent depth instead of just scaling depth? it's not faster at inference or training* since you still go through the full "effective depth", the advantage is storage (for kv cache storage btw you could do kv sharing) and a few other things i tried to list below 1) could be better in undertrained regimes (too few data or compute constraints) but i don't think any frontier model is in this regime? could also help convergence somehow, maybe it's just not possible to train a 10T effective param model (so maybe 2Tx5) without tricks like this, but i doubt it.. 2) one positive aspect could be better adaptive depth/compute (which is a good direction imo): depending on the token, you skip some recurrent blocks. but you could also just skip layers (or skip moe routing like in longcat flash) and get something quite similar you could also imagine co-training models this way, sol with 8 blocks, terra with 4, luna with 2 or something like that 3) another interesting thing (to verify?): you could simulate higher batch size at inference by overlapping the computation of depth_block=1 of request 1 and depth_block=t of request 2 on the same gpus, so effective batch size = batch size * number of recurrent blocks (and you could even overlap across models if you do the co-training i mentioned above!), see the picture attached for an illustration of what i mean still quite skeptical overall, i think scaling depth is just more expressive when you are not compute/data bound but excited to see a potential new direction *on "faster to train", could be if you only backprop through the last few loops or something like that