Code Pretraining for Nemotron, NVIDIA; **Views are my own** Previous: Head of Research for Code@Essential AI + YC (W15, S18) + PhD + Postdoc in Code Synthesis
San Francisco, CA
Joined November 2008
- Tweets261
- Following1.5K
- Followers1.1K
- Likes260
Saurabh Srivastava retweeted
The largest math proof in human history is now computer-verified.
Today, Axiom announces a complete Lean auto-formalization of the Classification of Finite Simple Groups, the "periodic table of symmetry."
About 15,000 pages, built by over 100 mathematicians across generations.
Saurabh Srivastava retweeted
Up until last year, all these results would be papers separated by years, waiting for peer review for months, communicated in arxiv papers first, published after a year in some journal.
Now, all that research using AI and formal tools can be compressed into hours.
I've just made a live tracker for the community effort for optimizing the "Integer multiplication below n log n" mathematical problem, so everybody can see the progress on it: beyond-n-log-n.netlify.app
Saurabh Srivastava retweeted
Well worth reading: proofsandprompts.com/2026/10…
Saurabh Srivastava retweeted
Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry.
Artificial intelligence is extraordinary technology that will advance discovery, productivity, security, health, and prosperity for generations to come.
But its full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility.
This is bigger than a single product. It's the beginning of an open ecosystem to build the trust layer for safe agent systems.
Together, we are building the foundation of the AI economy.
Trust and innovation are not in conflict. Safety is how trust is earned. We must build not only the most capable AI, but the most trusted AI, so that this extraordinary technology can realize its enormous promise for the world. nvda.ws/4hcoq7m
Watch this.
The craft: your decisions, learnings, and what you build. Not how. I started coding with basic, assembly, C. “Dry ran” my first program with pen/paper. Then vim with no auto-complete from 1998-2025. Copilot for half of 2025; and now agent swarms. The craft has changed.
We’re living the sci-fi future we were promised. When coding for research, insightful decisions used to be interspersed with days of manual coding. Now I get to make 10s of those research decisions/day. and when I am running 6-9 workstreams in parallel, we get a quarter’s worth of learnings in a day.
I will genuinely miss the borrow checker of Rust, playing with memory layouts in C, higher order programming in OCaml and Py, manually debugging SMT formulae, and the joy of hand drawn LaTeX and XFig. Guessing math, biology, algorithms, material sciences, have similar feelings. My 2006 self is looking at my 2026 workflow with insane jealousy.
Saurabh Srivastava retweeted
I think one of the reasons people in the AI field — especially in the labs — tend to be more “AGI-pilled” than people outside is that we get to observe AI succeed at tasks that we know for a fact we didn’t explicitly train it for.
Whereas if you’re outside the labs, then in theory (stretching the imagination), every AI success so far could be downstream of some kind of costly explicit training effort by the lab insiders (which is partly but not completely true).
In other words, being a lab insider gives you a direct observation of the “magical” properties of this synthetic-brain technology. Whereas for outsiders, AI’s abilities can always plausibly seem artificial/non-magical/man-made.
Saurabh Srivastava retweeted
I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions. Video attached.
TLA+ also works well. I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt.
I don't know either language well, but Claude is excellent at both. This approach is super useful for formally modeling your code and finding bugs that a human probably wouldn't have spotted.
Is formal verification the future of coding (or at least, bug finding)?
So, a) did I feel successful in exploring a research hypothesis; and b) was there significant course correction; and c) did the interaction feel like independent research thinking? a: Yes, b: Yes and c: No.
I did get a nice hypothesis explored in my spare time: What if we don't make the model write proofs, but make it set up a search whose output is a proof. Use search (Z3 + Lean discharges) to sidestep hurdles models face in full native reasoning about code proofs.
It is a functioning "automatic code-synthesize-and-verify harness". Give it a problem statement, the model guesses a control-flow template, and it interactively leverages Lean+Z3 to produce a program together with an inductive invariant, and a termination-ranking function.
The synthesizer is the artifact. The transcripts are the findings.
Public repo: github.com/saurabh-srivastav…
More important, private version with dev-transcripts/ (DM me for access): github.com/saurabh-srivastav…
Open the repo in Claude Code. Describe a program need in english, and the framework will guide the search for the program and code.
The output is a triple: The synthesized source (Python, C, or Rust), the discovered inductive invariant and ranking function, and the Lean proof artifacts you can re-check independently.
In the repo:
* Framework: an agent that consumes an english description of what a program should do and produces provably correct code with Lean + Z3 artifacts. Look at the benchmarks checked in.
* Session transcripts (in the private repo): the unedited raw interactions for what using Claude in a research process looks like, in the `dev-transcripts` folder.
* Companion audit runs for english->code+proof: 9 rounds of automated audits, 25+ agent-authored benchmarks with formal proofs.
Now to the subjective part of the thread:
Limitations to being RA-ready:
a) Hard: Lack of independent hypothesis generation.
b) Soft: In-session long-horizon coherence lacking over the 100M-400M token session.
c) Soft: Reverting to default biases rather than adapting with learnings as the project evolved, e.g., failing to remember that efficiency is a wall rather than a good-to-have in this research.
The first is a model limitation. The latter two are solvable with on-disk memory and skills/directives/readmes; and continual learning solutions should directly address "c)".
To make the model graduate from RPs -> RAs, we also need:
1. Self-skepticism: A distinction between "I asserted this" and "I verified this." An internal audit loop.
2. Long-horizon "learnings" instead of just long-horizon "memory": Not just retrieved facts, but sustained hypotheses and working approaches learnt during the process.
3. Evaluating viability of options in weight/token-space: The ability to say "of these ten options, this is the right one, for these reasons", instead of the laundry list.
4. Knowing what is workable given prior experience: Recognizing when a hypothesis is not testable, and saying so. As a corollary, limiting focus to testable hypotheses.
5. Don't assume your PI is right: Real pushback on the PI's assumptions--not "great question, here are considerations."
None of these look implausible from where we are today. Some might just need training-time data, or maybe a few architectural tweaks + data that leverages that tweak.
Phase 1 — reproduce the papers.
Claude was excellent here. It translated paper prose into working code faithfully. It found real bugs in the original approach (some my fault, some just terminology drift over 15 years). The core algorithm was rebuilt and its benchmarks verified in a single session that lasted ~1 week.
I gave it two important papers and a deprecated old codebase with partial implementation as a start. It easily got to a functioning system with minimal guidance, excising unneeded directions, setting up success/failure test cases.
The most important failure cases: Opus did create valid repro code, but its ability to notice runtime efficiency of the synthesizer was lacking. In particular, in this case where we use Z3 and Lean to tackle harder than NP-complete problems, efficiency reasoning can be critical to building a functioning system.
Verdict: strong at faithful reproduction from a clear specification. This alone would accelerate a lot of research.
Phase 4 — validate against a related external benchmark.
VERINA (arXiv:2505.23135) provides 189 Lean verification tasks. 108 "basic" and 81 "advanced". When published, 22.2% of the basic tier could be solved by the then frontier.
Of the 108 basic, 72 are expressible in our current IR -- and the system produced a machine-checkable (code, proof) for 66. That would provide some evidence for proof as search being more viable.
The caveat, up front: this is not a side-by-side. VERINA has the model write the Lean proof, i.e., proof as a generation task. This synthesizer treats proof as a search task, i.e., the model proposes a template + predicate space, and a solver finds the invariant and discharges the obligations by delegating to Lean or Z3 as required. We synthesize code *to be provable*, compared to the alternative of "write the code" > "prove the code in Lean".
We did 72, but most of the 36 basic tasks are encodable as simple IR extensions to get similar successes over all basic tasks. I do not yet know whether advanced are similarly solvable or they are outside the scope of this synthesis system.
Phase 3 — try things not in any paper.
I ran 9 rounds of "community-validation audits": spawn fresh Claude agents with no prior context, have them author brand-new verification benchmarks in this "harness". Rounds 5 through 9 hit 100% verified, without any intervention from me.
The corpus now includes 25+ agent-authored benchmarks with formalization that is likely interesting: two-pointer palindromes, self-referential array recurrences, multiplicative-index UF axioms, 2D matrix transpose with proofs, and more.
Every one of them comes with an inductive invariant, a termination measure, and a Lean-verified proof.
Phase 2 — add 15 years of improvements.
The design space has changed over the past 15 years. We do not need a _mechanical formal_ code+proof synthesizer. An LLM can make guesses and use Z3+Lean to discharge their correctness. Collaboratively, Opus and I added many features: Lean 4 as a dual verifier alongside Z3; chain-aware translators for multi-loop patterns; helper-citation codegen for axiom-heavy benchmarks; native 2D array support; resource-bound (cost) invariants.
Some of these Claude drove more than I did; others I steered. The pattern was consistent: the clearer I was, the faster the research progress. When I was vague, what happened is best described as "research idea thrashing".
Research feels 10-100x faster now, given enough GPUs and tokens--yet I still have to give fine-grained supervision throughout. How do we square that with results like the Navier-Stokes blowup?
Three differences: 1) a massive compute swarm, 2) most daily research isn't cleanly verifiable, 3) a human still states the hypothesis.
NS had all three: a swarm hammering a human-posed hypothesis on a verifiable problem, checked in Lean. Swarms are inevitable. Verifiability will be case-by-case.
Hypothesis generation isn't a given, but doesn't feel implausible within the year.
Why this particular "english -> correct code" as a test-bed?
A: It is an open problem we care about: models are prone to writing code with subtle errors. Programmers do not like writing proofs--feels like double work. But models don't mind that. Z3/Lean is just another programming language as shown by Anthropic's FLT translation into Lean.
B: I already knew some candidate directions that would work. I could evaluate Claude's actual contributions in two phases: repro prior research, extend beyond current. Every claim about "Claude helped here" is falsifiable against what I would have built as a human researcher.
RPs are great for idea exploration and refutation. Not for independent hypothesis generation.
Claude can execute a research plan competently. It can spot flaws in your reasoning if you tell it what to check. It can implement and stress-test a design faithfully.
But: Its planning is more mechanical than independent research. Two examples of non-mechanical planning one needs: a) Independently surface questions whose answer will invalidate the current direction--i.e., be your own devil's advocate, b) Independently ask questions whose answer will re-prioritize the options--i.e., turn what looks like non-greedy "research intuition" into a greedy pick by getting evidence that makes the counter-intuitive choice the greedy choice based on acquired evidence.
What it does do well: When the hypothesis is posed, it will comprehensively explore its scope. Which is useful but different from identifying the research choices. This is a tall ask though. Good human RAs should be decent at "b)", and might even be more blind-spott-ed than models when it comes to "a)".
This project can be viewed as a manually constructed baseline for "Fall'26 (RP:yes, RA:no)". If a model can single-shot long-horizon (weeks?) go from "here are some old papers" TO "system for provably correct code", I would happily say RA:yes. I don't have the setup to test Sol/Astra/Fable/... Maybe someone in the community does.
If I coded this manually, it would have been a months-long exercise. Instead I wanted to advise a frontier model in the way I would collaborate with a PhD student. This session (with compactions) might have consumed 100M-400M tokens.
It was productive--but I did not see evidence of independent hypothesis generation yet, which is what would graduate an RP into an RA.
My bet: we should have full RAs by next summer.
Reading Kevin @kliu128's post on research acceleration makes me wonder if the "research intern" objective is conservative--feels more RP than RA: humans inject intuition, Astra is the RP. @merettm's essay feels more bullish.
A ladder: AR (automated researcher) > RA (research assistant) > RP (research programmer).
In 2026, we're at the RP-stage--not RA. The missing step to full RA: hypothesis generation.
A few months ago I wondered if models were already good RAs? The setting: Build a system to generate provably correct code from english. Input = natural language. Output = code + proof. A decade ago, I had spent many years of my PhD on this, so I had context. The system is good, but the Human+AI process leading to it is arguably more valuable. In a single session spanning many weeks, I logged my experience.
In this thread: a) a harness that treats proof as *search*, not generation--english -> code + proof (Lean + Z3), correctness machine-checked, b) transcripts of a long-horizon session (100M-400M tokens pre-compaction?), and c) a subjective assessment of whether current models are good RAs or good RPs.