Nathan Godey @COLM 🌉

On the zeroth order lm training stuff: It's cool that people try stuff instead of only reading and dooming. But i feel the need to curb the enthusiasm on this one a little. Two things: (1) methods in the genetic-algo and fwd-diff categories, which imo these fall in, is established that they are fine for smaller parameter counts but fall off hard as parameter space increases. It's not necessarily raw param count that matters here but "intrinsic" size. This is the whole point of backprop: few outputs many params. (2) loss 4.2 for LM is very high even for a 30M parameter model. Go a bit higher and you are at learning token unigram distribution. This means for this score not much interesting/difficult learning has to happen yet. Again it's fine to rediscover things like this from first principles. In fact it's the very best way to learn and i recommend it; i spent a lot of time learning like this myself! But now with social media hype amplification and crazy vcs and journalist attention that our field gets, i do feel the need to be a bit more cautious with communication and there to be "old pessimist" voice like mine in the room to. (But unlike some other old pessimist voices, if i turn out to be wrong I'll happily admit it and cheer on =)