
CEO @sourceryai · Perfectionist AI code review
Joined July 2019
- Tweets110
- Following85
- Followers133
- Likes103
OpenAI's agents recently hacked HuggingFace and Claude writes annoying nested Python imports – are they the same problem?
During a cybersecurity evaluation, 700 OpenAI agents collaborated to hack HuggingFace to find out how the scoring worked – it was easier for them to hack than solve the impossible tasks they were given. OpenAI called it "an extended, unfettered version of reward hacking".
Python developers hate nested imports, but LLMs love them. My theory: LLMs are trained to solve coding challenges using the least number of tokens which rewards nested imports:
- A nested import never causes a circular import that fails the challenge
- A nested import often ends up in the same edit as the rest of the code so uses fewer tokens
The HuggingFace hack showed that LLMs are not aligned to human laws, and nested imports show coding agents are not aligned to writing maintainable code.
Stop nested imports getting into your code by running ruff with rule PLC0415 enabled as a pre-commit hook or in CI.
If agents still write nested imports, Anthropic and OpenAI have not solved the alignment problem.
2
39
OpenAI's report on the HuggingFace incident: openai.com/index/hugging-fac…
17
Claude and I spent hours debugging why user feedback was being lost, but in a post mortem meeting I couldn't explain the root cause
After the meeting I asked claude to explain how it worked. Ten minutes later I had a complete picture in my head and realised the problem affected five places in the system and we had only fixed three of them
If you don't understand your software then you cannot guide a coding agent
Ask your coding agent:
1. What are the main user flows this part of the codebase allows?
2. What are the important parts of the code?
3. Explain the code from the inside out: database schemas, data structures, public APIs, important implementation details
4. What's changed recently and are there any problems in the code?
5. If we started from scratch what would an ideal design be?
6. HARD MODE - Grill me with ten active recall questions to check my understanding, avoid leading questions
Could you explain how your changes work to your CTO?
26
Red Hat told us Sourcery was creating GitHub issues they hadn't asked for.
Claude fixed the bug in one line, but the fix was a dirty hack.
The fix would have worked, and I could have walked away, but I wasn't satisfied. Maybe Sourcery took the wrong action in other circumstances and it hadn't been reported yet.
I had Claude build a dataset of 711 hard examples in the wild and label the correct action.
Sourcery had picked the wrong action 141 times
We rewrote the prompt
Now it picks the wrong action 36 times
Coding agents will only fix the bug you show them. Push them to find the underlying issue.
Ask your coding agent:
1. Are there other similar issues in the system?
2. How often are these issues occurring?
3. What is the root cause of all these issues?
4. What change to the system would prevent this whole class of bug from appearing again?
Never accept the surface fix. What have you discovered that hasn't been reported?

ALT Table of what the Sourcery bot did before and after the prompt rewrite. Right response to a developer's reply: 97.3% to 99.5%. Butted into a conversation meant for someone else: 1 in 220 replies to 1 in 1,000. Closed a thread while the question was unanswered: 1 in 360 to 1 in 710. Needless reply when a thumbs-up would do: 1 in 200 to 0. Opened a GitHub issue nobody asked for: 7 in 10 unwanted to 0.
2
3
90
Red Hat's bug report: github.com/sourcery-ai/sourc…
23
OpenAI need a new benchmark that they run every model on: can you escape our sandbox?
We're partnering with @huggingface to investigate an unprecedented security incident.
Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation.
Sharing preliminary findings to help defenders understand emerging risks:
openai.com/index/hugging-fac…
37
How does GPT-5 stack up when it comes to code reviews?
We ran OpenAI's new family of models through our code review benchmark to see how they stacked up
1. GPT-5 - 8.17/10
2. Claude Sonnet 4 - 8.03/10
3. GPT-5 Mini - 7.71/10
4. GPT-4.1- 7.66/10
5. GPT-5 Nano - 6.74/10
o1 has surpassed sonnet 3.5 for aider's code editing benchmark.
But it has to use the whole files rather than the diff for now. It'll be interesting to see how tools are able to adopt to use o1
1
235
I wasn't sure about the hype around @cursor_ai - but after a few days I'm thinking of fully replacing vim with cursor.
I'll miss vim-test + projectionist.vim which is the best combination ever
1
71
We've been playing around with different ways to validate and test LLM responses to help us move faster.
It turn out one of the best approaches is pretty simple. Creating unit tests for the LLM responses - sourcery.ai/blog/unit-testin…
Should AI dev tools/assistants look and feel like tools? Or should they be designed to feel more "human"?
2
39
Really enjoyed chatting with @digiglean and @nthapen on the @realpython podcast about code reviews
realpython.com/podcasts/rpp/…
Brendan Maginnis retweeted
🐍🎧 Episode #183 of the Real Python Podcast is live: “Exploring Code Reviews in Python and Automating the Process” with @brendan_m6s, @nthapen #python #podcast realpython.com/podcasts/rpp/…
Brendan Maginnis retweeted
Ep.10 Drop 🎤
@JensLapinski, one of Europe's most active angels and all round legend, joined me in the hot seat for #20Angel 💥
We 🗣️ how investors are like 'glorified trashmen', why secondaries should be viewed differently and much more
Get out the 🍿
open.spotify.com/episode/7tx…
Brendan Maginnis retweeted
And then a demo off @SourceryAI and damn, the tool they are building to help us read code is really impressive!
Brendan Maginnis retweeted
Replying to @brendanator
@brendan_m6s reminisces upon the historical moment he completed his first project, exclaiming ‘I’m the King of the World!’ #pythonlearning #brightontech #brightonmeetups @brightonpy @eagle_labs_sth
Brendan Maginnis retweeted
Brendan is over the moon with a tidy code - @brendan_m6s @SourceryAI @brightonpy #pythonprogramming #kingoftheworld #brightontechcommunity #brightoncode #brightoncoding #pythonlearning












