Ivan retweeted
rick beato is jow part of the local inference open weights underground.
this timeline is killing me.
youtu.be/YTLnnoZPALI?is=SDn-…
Ivan retweeted
Two announcement events recently that I simply don't get.
Amp (a coding agent tool) declaring coding agents as dead.
Linear (a issue tracking tool) declaring issue tracking as dead.
Ivan retweeted
This is more evidence that current frontier models remain completely reliant on content-level memorization, as opposed to higher-level generalizable knowledge (such as metalearning knowledge, problem-solving strategies...)
Ivan retweeted
Replying to @AnnaLeptikon
Computers used to be unforgiving. I wonder what will happen if the next generation of computer scientists does not grow up with “syntax error in line 20” but with “you are absolutely right, let me try…”
Ivan retweeted
I give it less than 6 months before Garry stops preaching LOC and starts preaching maintainable code bases.
And with that one move he will go from junior engineer to a bit more senior.
We watching his Eng journey live 🍿
unavailable
It’s quite unfortunate that GEPA Optimize Anything didn’t get enough traction, while very, very similar ideas promoted by Karpathy’s autoresearch + Lütke’s pi-autoresearch - got so much traction, despite being less general
Hey @AnthropicAI folks, I’m on $100 Claude subscription, and I just started my 5hour window but already seeing “Claude Opus 4 limit reached, now using Sonnet 4.” Had a similar experience yesterday. I understand that Opus 4 counts 5x in terms of consumption, but it’s clearly off.
1
321
Ivan retweeted
I don't have too too much to add on top of this earlier post on V3 and I think it applies to R1 too (which is the more recent, thinking equivalent).
I will say that Deep Learning has a legendary ravenous appetite for compute, like no other algorithm that has ever been developed in AI. You may not always be utilizing it fully but I would never bet against compute as the upper bound for achievable intelligence in the long run. Not just for an individual final training run, but also for the entire innovation / experimentation engine that silently underlies all the algorithmic innovations.
Data has historically been seen as a separate category from compute, but even data is downstream of compute to a large extent - you can spend compute to create data. Tons of it. You've heard this called synthetic data generation, but less obviously, there is a very deep connection (equivalence even) between "synthetic data generation" and "reinforcement learning". In the trial-and-error learning process in RL, the "trial" is model generating (synthetic) data, which it then learns from based on the "error" (/reward). Conversely, when you generate synthetic data and then rank or filter it in any way, your filter is straight up equivalent to a 0-1 advantage function - congrats you're doing crappy RL.
Last thought. Not sure if this is obvious. There are two major types of learning, in both children and in deep learning. There is 1) imitation learning (watch and repeat, i.e. pretraining, supervised finetuning), and 2) trial-and-error learning (reinforcement learning). My favorite simple example is AlphaGo - 1) is learning by imitating expert players, 2) is reinforcement learning to win the game. Almost every single shocking result of deep learning, and the source of all *magic* is always 2. 2 is significantly significantly more powerful. 2 is what surprises you. 2 is when the paddle learns to hit the ball behind the blocks in Breakout. 2 is when AlphaGo beats even Lee Sedol. And 2 is the "aha moment" when the DeepSeek (or o1 etc.) discovers that it works well to re-evaluate your assumptions, backtrack, try something else, etc. It's the solving strategies you see this model use in its chain of thought. It's how it goes back and forth thinking to itself. These thoughts are *emergent* (!!!) and this is actually seriously incredible, impressive and new (as in publicly available and documented etc.). The model could never learn this with 1 (by imitation), because the cognition of the model and the cognition of the human labeler is different. The human would never know to correctly annotate these kinds of solving strategies and what they should even look like. They have to be discovered during reinforcement learning as empirically and statistically useful towards a final outcome.
(Last last thought/reference this time for real is that RL is powerful but RLHF is not. RLHF is not RL. I have a separate rant on that in an earlier tweet
nitter.cf/karpathy/status/182127…)
DeepSeek (Chinese AI co) making it look easy today with an open weights release of a frontier-grade LLM trained on a joke of a budget (2048 GPUs for 2 months, $6M).
For reference, this level of capability is supposed to require clusters of closer to 16K GPUs, the ones being brought up today are more around 100K GPUs. E.g. Llama 3 405B used 30.8M GPU-hours, while DeepSeek-V3 looks to be a stronger model at only 2.8M GPU-hours (~11X less compute). If the model also passes vibe checks (e.g. LLM arena rankings are ongoing, my few quick tests went well so far) it will be a highly impressive display of research and engineering under resource constraints.
Does this mean you don't need large GPU clusters for frontier LLMs? No but you have to ensure that you're not wasteful with what you have, and this looks like a nice demonstration that there's still a lot to get through with both data and algorithms.
Very nice & detailed tech report too, reading through.
Ivan retweeted
Been diving into some great AI/ML blogs lately:
- Blog by @chipro
- Blog by @eugeneyan
(links below)
I'm curious to find new gems. What are you reading these days?
Ivan retweeted
That’s a true masterclass in russian imperialism, when you seamlessly believe that Ukrainians have to forgive russians in the name of some (imaginary) love, all while struggling to understand why Ukrainians keep fighting so fiercely to protect home and families they actually love
Ivan retweeted
The inspiration from @ncooper57 to get tmux to do the heavy lifting to make ShellSage work is genius! Thanks to this, ShellSage is <100 lines of code (plus comments and prompts).
But the result is amazing. Try it out and tell us how you go. Details here:
answer.ai/posts/2024-12-05-i…
Ivan retweeted
If you haven’t tried Atuin yet, you should! Has become a daily-use tool for me in the last couple of years.
This quoted post is unavailable.
Ivan retweeted
★ Dynamicland's new website documents ten years of progress toward a humane dynamic medium. dynamicland.org
Ivan retweeted
Big news: I'm helping with @martinkl with a second edition of Designing Data-Intensive Applications! An early release of the first 3 chapters is now available (O'Reilly Learning subscribers only at this point) and we're hoping to finish it next year.
learning.oreilly.com/library…
Ivan retweeted
I want @cursor_ai but in @pycharm . I keep switching between the two!





















