Compulsive learner, eater and powerlifter. ML Infra Engineer. Prev: Netflix, Scala Center (EPFL).
San Francisco
Joined May 2014
- Tweets2.2K
- Following498
- Followers1.6K
- Likes5.4K
Locate me! Was a fun talk at the SF Lean meetup about Bend2. So glad @moxspace is hosting these talks.
Jorge Vicente Cantero retweeted
We’re releasing our Blackwell kernel verifier, building on our recently released work on "AI as a Compiler." In this approach, an LLM generates optimized PTX for a given workload, and our verifier checks it for correctness. By skipping intermediate compiler layers, we enable finer-grained control over the hardware at the PTX level, paving the way for more efficient kernels and models.
The verifier checks several properties, including symbolic correctness, race conditions, and synchronization issues.
You can get access here:
Volta Blackwell Verifier: github.com/francois141/ptx_v…
Triton AI Compiler: github.com/francois141/trito…
And here is the paper:
AI as a Compiler: arxiv.org/abs/2609.36800
Motivated by growing industry interest, we're excited to share our initial efforts toward stronger correctness guarantees for AI compilation and GPU kernel generation!
Our verifier, Volta Blackwell, analyzes NVIDIA Blackwell ML kernels by extracting a happens-before relationship graph alongside symbolic expressions representing their computations. By performing correctness checks over these representations, it can detect race conditions, synchronization issues, and incorrect expressions that conventional kernel testing might miss.
Using this approach, Volta Blackwell identified subtle fence-related issues in our Triton AI compilation pipeline that would not have been caught by the standard correctness tests proposed by KernelBench. This highlights the limitations of test-based validation and the importance of formal reasoning about GPU kernel execution.
Developing comprehensive and sound verifiers is a critical next step for AI-driven kernel generation. Subtle synchronization, memory-ordering, and computational correctness bugs remain major challenges, and no existing verifier yet provides the end-to-end guarantees required for fully trusted AI compilation and GPU kernel generation for ML workloads.
💻 Volta Blackwell: github.com/francois141/ptx_v…
💻 Triton AI Compiler: github.com/francois141/trito…
📄 AI as a Compiler: arxiv.org/abs/2609.36800
The key insight in the blog post is thought-provoking and interesting! But:
- Grocery spend depends on quality expectations... it's more flexible than it seems
- Childcare cost isn't fixed either, can be mitigated through family/friends
- Transportation and healthcare costs are very variable as well family to family
So the survival bar might not be as high?
This is what I’ve been doing personally to learn hard topics out of my wheelhouse. The workflow works surprisingly well while still keeping you in the loop for everything.
Equivalent to you “becoming a junior” on the job while still being able to shape requirements.
I cannot emphasize enough how much I prefer this "tutorial doc + build-it-yourself" coding workflow to the typical "ugh" feeling of reviewing huge agent PRs. You can try it right now and see for yourself:
1) Instead of having Claude Code make a PR, ask it to output a Markdown doc with a deep, detailed guide for a dev to implement the change, including teaching you background and context about the broader surrounding system, and rationale behind why we're doing things a certain way. Tell it to be very exhaustive and ultrathink. (The result should be go way more in-depth than a typical plan mode output, it may take a while so I tee this up well in advance)
2) Read the resulting doc. And then code it yourself! Use Cursor to go faster: tab complete is your friend. You can also use the Composer model in Cursor Agent to fast-forward thru tedious bits -- but avoid using slower models or doing too much AI work at one time. You are staying connected to the work and going step by step. Think about it as you go, correct course as needed. Go off-script from the tutorial doc whenever you want.
What I find when I do this is that I'm able to reactivate the "building things manually" parts of my brain which help me make good decisions and understand what's going on, but I'm also able to go super fast with the AI tools.
I still think a big failure mode for all of these frontier LLMs is that they don't ask enough questions by default. Don't assume, simply ask me. Currently, they branch out in different directions, and force you to decide.
This is matching what I'm observing in my recent Gemini 3 chats very well.
How good is Gemini 3? From our internal testing at @elicitorg, it seems to be sacrificing calibration & carefulness in exchange for higher average accuracy
@PradyuPrasad tested Gemini 3 Pro for writing Elicit reports. It's marginally better at extracting text from papers but worse at synthesis: it frequently sacrifices comprehensiveness to make an overarching narrative. For systematic reviews, that's the wrong tradeoff!
On our internal benchmark of question answering from papers, Gemini gets about 95% (compared to 90% for our internal baseline) - but it also hallucinates that answers aren't available in papers 6% of the time, compared to our internal model which doesn't do that at all
On sample user queries we checked, Gemini often gives answers that are much less comprehensive. For example, on hydrocortisone for septic shock where two major trials contradict each other, our current model dives deep into the contradiction, whereas Gemini just mentions both trials without explaining why they differ
As usual, maybe all of this can be addressed with careful prompting - but evals are hard, and many people (and orgs are people too) use models out of the box. And in that setting we see multiple data points that suggest a trend towards narrative coherence at the expense of nuance
I'm surprised but I'm enjoying ChatGPT 5.1 a lot more than Gemini 3 for writing. It's not even a competition. G3's fact-oriented tendencies make it great for any technical question or explanation. But for document writing, it's significantly inferior, even stubborn sometimes.
Gavin is arguing Kimi K2 thinking and other lesser models are overfitting on benchmarks and don't generalize as well. I'm a bit skeptical about that, I think people mostly don't use these models mainly because of speed issues, lack of integrations, and a missing track record.
On speed, I was expecting them to be much faster than they actually are. Tool calling still has problems. I got used to Claude Code/Codex, the gain of swapping them out isn't worth it.
For those that don't know: the legend documentarist Ken Burns is soon releasing a new documentary: The American Revolution on 11/16. I'm so looking forward to watching it.
Just watched "Where is the friend's house" by Abbas Kiarostami. What a masterpiece. From beginning to end, it's the kind of simple movie that gets you hooked and marvel at the story and feelings it causes on you. Highly recommended.
Incredibly interesting that Codex just leaked some thoughts on the actual answer he gave me. First time I've caught him thinking out loud in the last part of the answer. Has anyone experienced the same?
There's something that Codex gets pretty right compared to Claude Code: very succinct output. Claude Code is overly verbose by default, it takes quite a lot of prompting to prevent that. Codex is action-focused, especially when asked for advice or explanations.
In my experience, this benefit is a huge inconvenience when Codex implements things. It dives too eagerly into action, doesn't check in with you, often goes the wrong route. I can see why Claude Code leaned on verbosity to fix that, feels more aligned at all times.