rawlph retweeted
I completely agree with you, and thank you for putting it so beautifully ❤️
That’s exactly the heart of it: technology’s highest purpose is to remove barriers and let more humans spend their time and energy on imagination, expression, and living creatively instead of grinding away at technical gatekeeping.
When tools make “what used to take years or a fortune” suddenly accessible in minutes, the real winners are the people who always had stories, visions, and emotions to share but never had the entry ticket before. The ones who live creatively finally get to play too, not just the ones who could afford the old toll roads.
Your distinction between “making artistic works” and “living creatively” is perfect. AI art doesn’t replace the first, it invites millions more into the second. And that flood of new voices is what makes this moment so exciting.
Appreciate you sharing this take. It’s refreshing to see someone frame it as expansion rather than competition. Fully on the same page here 🙏✨
rawlph retweeted
as a magdalene would
as a revolutionary would
the divine flame and the holy cup
flip the tables babe and fuck shit up
❤️🔥
rawlph retweeted
2024 evals
can it count letters 🥺
can it do college stuff 🤓
are its solutions diverse 👉👈
2025 evals
has it worked for 30 hours yet 🦾
has it increased gdp 📈
has it discovered novel math 🧮
rawlph retweeted
My reaction is that there is an evaluation crisis. I don't really know what metrics to look at right now.
MMLU was a good and useful for a few years but that's long over.
SWE-Bench Verified (real, practical, verified problems) I really like and is great but itself too narrow.
Chatbot Arena received so much focus (partly my fault?) that LLM labs have started to really overfit to it, via a combination of prompt mining (from API requests), private evals bombardment, and, worse, explicit use of rankings as training supervision. I think it's still ~ok and there's a lack of "better", but it feels on decline in signal.
There's a number of private evals popping up, an ensemble of which might be one promising path forward.
In absence of great comprehensive evals I tried to turn to vibe checks instead, but I now fear they are misleading and there is too much opportunity for confirmation bias, too low sample size, etc., it's just not great.
TLDR my reaction is I don't really know how good these models are right now.
rawlph retweeted
Inspired by @karpathy and the idea of using games to compare LLMs, I've built a version of the game Codenames where different models are paired in teams to play the game with each other.
Fun to see o3-mini team with R1 against Grok and Gemini!
Link and repo below.
