Labs really need to start considering real-world task time next to token efficiency in their RL pipeline. The amount of times I've had models trigger 60s or 5 minute timeouts to wait for a result when the action takes <10s is infuriating.
There's this very specific AI use case at work where LLMs/VLMs have constantly failed at... until Opus 5.5, which does it almost flawlessly. Kind of a humbling experience.
Really curious how @Mistral Large 4 would perform on Kernelbench @elliotarledge, any chance there will be a run? 😃
First the issues with 1019, now 1024. Bruh Nvidia is vibe coding these kernels..
Heads-up DGX Spark owners: kernel 7.0.0-1024-nvidia (latest update) made both my Sparks hard-reset every few minutes. 31 resets in under 90 min, even at idle. No panic, nothing in the logs. Rolled back to 7.0.0-1019: stable. Hold off on
1024 for now. #DGXSpark
Opus is so strange. It's capable of creating beautiful, tasteful web design, but only if you first let it indulge in some slop and then correct it.
Latest @deepseek_ai 4.1 Flash distilled this annoying behaviour from Claude lol. Its at ~100k context,
>New benchmark drops
>A GLM model does exceptionally well for it’s size
@Zai_org is just built different
Introducing sudo L7: Can Coding Agents Act Like Staff Engineers?
Coding agents are getting very good at solving well-defined problems. But staff engineers do much more than write correct code.
They understand unfamiliar systems, make architectural decisions, spot risks the ticket never mentioned, and know when passing the tests isn't enough.
sudo L7 measures that judgment.
Our new software engineering benchmark contains 60 expert-authored tasks, primarily in private production codebases from real companies, complete with legacy code, technical debt, and messy dependencies.
We evaluate eight dimensions of engineering behavior, from functional correctness and engineering craft to practical judgment, thought partnership, and knowing when to push back.
The best-performing agents score under ~45%.
Coding is only part of the job. See you next cycle.
Explore sudo L7: surgehq.ai/blog/sudo-l7
Got curious after seeing the Micro Centers in the US running out of DGX Sparks, I can find and buy at least 120 shown in stock in just Dutch stores. Really strange how Europe seems to have so much more Sparks available than the US. They are around 7.5k USD incl. VAT though.
Ok Opus 5.5 makes me want to get a Claude sub again, but I just *know* that in 2-3 weeks it's going to be quanted to uselessness again.
