Building the future of evaluations @cognition • Open Sourcerer (@unifiedjs, @tus_io, @uppy_io) • Dabbing in the mainframe
Amsterdam
Joined October 2015
- Tweets535
- Following911
- Followers254
- Likes3.4K
For a long time I thought it's over for @MistralAI, but they're so back.
FrontierCode 2.0 is a massive leap in benchmark quality over 1.1, reflecting real-world usage a lot more.
Let's see if the vibes confirm it. We desperately need some AI relevance in Europe.
The arena skill from pstack, 3 subagents attempt the same thing and a judge mixes the best of all, has been surprisingly helpful
Merlijn Vos retweeted
Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.
Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
The result is a benchmark that is far from saturated, with the top model scoring just 45%.
Relatable. Also reminds me of this from @KentBeck
"I'm not a great programmer; I'm just a good programmer with great habits"
next windows volume control is just copilot asking how loud you're feeling today
I don't know how to measure when things went wrong at Microsoft, but I think removing the audio mixer is a solid landmark
Readers added context they thought people might want to know
Recent versions of Windows have a volume mixer.
x.com/shanselman/sta…
agi is going to boot up and its first words will be 'who did this pipe work'
tomshardware.com/tech-indust…
netlify's q4 icp: guys who've never heard of cron
How do you run an AI workflow every morning without leaving your laptop open?
@mr_rtl wanted to publish a document automatically at 6 a.m., so he used Netlify Agent Runners to create a scheduled serverless function.
He'd never even heard of a cron job before.
reading the docs and learning something is now a documentation attack 🚨
the fact that big ai labs called it a "distillation attack" has caused so much confusion among people
DISTILLATION IS NOT A FORM OF "ATTACK"
distillation is an ML technique used to transfer knowledge from a larger model into a smaller model
the only reason labs suffix it with "attack" is because of some misguided idea that no one should use data from their models (that people pay for) to train their own models. it has literally nothing to do with cybersecurity and everything to do with labs playing dirty tricks to keep their competitive edge just a little bit longer
distillation is not something you do to hack into someone's servers.
I'm in @bot unification hell
- already have X premium via company
- in X I choose the new unified sub
- confusing that I go to Cursor initially to login? I used my account previously via GitHub but since I have X premium via business, I logged in with X
- tried to setup mobile app, can't get into the same account. Logging in via X is new account, bots gone.
- logged in again on Cursor web to reproduce but no login method leads to the old account. GitHub, X, and direct email login are directed to the same new account
- no idea how to get back into the initial account with my bots and connectors
- In Grok Bot macOS (new account), it says "premium" under usage but also call out to unify account, even though I already did that on X web initially.
- On cursor.com/unify/accounts I can connect my cursor, grok, and X successfully. Low key surprising, since my X is already connected to that other account I can no longer access?
- The next steps makes me pay for the $30 plan, but I already have it. It even recognizes that my X account has premium, but now I need a personal credit card even though premium comes with my X Business affiliation
end state: other account lost even though it was literally the same email that was associated with it. unifying seemingly not possible without paying twice (company + personal)
@samsokolin @larsencc @parkersmith @krgeorge @pengzheng_ @lingxi @aryamankhawow @roshan_s
my therapist says it's not ai psychosis if the PR got merged
Truly the best of all worlds
- Keep your heavily subsidized OpenAI tokens
- Run ALL sessions in the cloud for a flat $20/month. No unpredictable compute costs.
- Use Devin Fusion (e.g SWE-2 + Astra) for the same intelligence while burning 40% less of your ChatGPT sub
- Devin Review on your PRs + Security scans
Love how minimal this tweet is.
Someone next week on LinkedIn:
🚨 FACTORED RSA-260.
35 years. One impossible problem. Zero excuses.
Here are 7 lessons it taught me about startups, leadership & AI 🧵👇
1. Stay curious
2. Move fast
3. Believe in yourself
Agree? ♻️
Installed @OmarchyLinux in almost 1 minute. Seeing these install times online had me assuming you need €3K hardware for it, but on my €750 GMKtec mini pc I'm still clocking in fast 🔥
