is the Anthropic IPO chart a better LR schedule than warmup cosine decay?
let's find out! @mridul__agarwal@ananthsiyer and i are running 4 identical pretraining runs, with the LR driven live by ANT, SPCX, SPY, and NVDA.
watch the models train at stockasticdescent.live