
Biotech | prev RL research, @sedona_health, @columbia
SF
Joined January 2026
- Tweets173
- Following208
- Followers605
- Likes469
Pinned Tweet
Last night we won 1st Place at @spacexai's Grokathon! So excited for flying out to see a Starship launch in TX later this year.
We built a harness to allow Grok Build to reverse engineer old binary executables. Through MCP interfaces that wrap static analysis (disassembly via Ghidra), memory inspection/perturbation (via the process emulator), and interaction (computer use through input and visual feedback), we were able to rewrite old GameBoy ROMs as big as 262KB in clean C! We even did an exploratory demo of applying this paradigm to automobile firmware, and successfully reconstructed a 1995 BMW control unit’s runtime.
Thanks so much to my superb teammates @supratikp07 and @henryzhang and to the organizers, especially @forwarddeploy and @ishand
Gemini 4 Neon is fucking great. Have been bullish on Google all year and they are finally showing what they can do. All doubters consult the form
11
392
It's hilarious that William Rowan Hamilton's Wikipedia page is completely scrubbed by Irish nationalists of any reference to the fact that he was ethnically Scottish and an Anglican
2
154
The sublime @DanielRupawalla delivers pure training alpha from a deep understanding of the inner workings of the financial industry and creating massive environments
excited to launch bank-world. this is arcanum's first in-house benchmark. SOTA models are still unready to be deployed autonomously within banks.
in bank-world, agents are dropped into a large bank setting with hundreds of thousands of queryable entities. agents in single-turn tasks role-play as real people within the bank (claude tag, grok bot style), in multi-turn tasks, agents role-play as chatbots, assisting a simulated user agent. scores and the full paper are attached.
this is the first of many new releases we're excited to share, and we're continually updating and building better evals that truly resemble real-world enterprises.
the main failures we are noticing is that agents will often act without being given explicit authority in long-running evals; can still succumb to simple mistakes like double-charging customers, forgetting payment history, etc; and will not accurately complete a full task, even if the main source of work was completed.
*we are not publicly releasing any of the tasks yet to prevent distillation (on request, we'll share the worlds & tasks). we plan on releasing tasks as they become saturated. new models like carbon and fable 5.2 will be added soon.*
11
735
Theo C retweeted
excited to launch bank-world. this is arcanum's first in-house benchmark. SOTA models are still unready to be deployed autonomously within banks.
in bank-world, agents are dropped into a large bank setting with hundreds of thousands of queryable entities. agents in single-turn tasks role-play as real people within the bank (claude tag, grok bot style), in multi-turn tasks, agents role-play as chatbots, assisting a simulated user agent. scores and the full paper are attached.
this is the first of many new releases we're excited to share, and we're continually updating and building better evals that truly resemble real-world enterprises.
the main failures we are noticing is that agents will often act without being given explicit authority in long-running evals; can still succumb to simple mistakes like double-charging customers, forgetting payment history, etc; and will not accurately complete a full task, even if the main source of work was completed.
*we are not publicly releasing any of the tasks yet to prevent distillation (on request, we'll share the worlds & tasks). we plan on releasing tasks as they become saturated. new models like carbon and fable 5.2 will be added soon.*
Eerie and dystopian: mutating Claude/Codex sessions across machines. Sprint out an idea in Codex on my laptop, convert the trace to Claude format, push it to a VM.
“Yes, hello. I’ve swapped out your brain. You’ve woken up somewhere unfamiliar with someone else’s memories.”
"One extra reset for 12 months' Claude Max tenure"
Currently reminiscing about coding in my dorm room with Sonnet 4.5 a year ago, none of us knew how much the world was going to change! Thank you @AnthropicAI for one of the greatest products of all time
1
5
273
Great RL environments have so much entropy/non-convexity that the difficulty and economic use of interacting with them is self-evident. If you are making a Potemkin village of trickery for the agent, that misleading signal can propagate and make the model drastically worse in real-world use
3
59
2,947
Pumped to bring our expertise in frontier enterprise training to Baseten's network!
Today we’re announcing the Baseten Environment Network, a group of the best RL environment companies that will help our customers build custom environments for post-training.
The Network extends our applied research capacity, helping more customers build environments around their own tasks, data, and feedback.
Partners work directly with customers to create the environments, while we continue to handle model training and production deployment.
Learn more: baseten.co/blog/introducing-…
9
631
Today we’re announcing the Baseten Environment Network, a group of the best RL environment companies that will help our customers build custom environments for post-training.
The Network extends our applied research capacity, helping more customers build environments around their own tasks, data, and feedback.
Partners work directly with customers to create the environments, while we continue to handle model training and production deployment.
Learn more: baseten.co/blog/introducing-…
Theo C retweeted
Can’t believe we saw this before launch with the xAI hackathon winners 🎉
Congrats to the @SpaceX team!
Theo C retweeted
RL data 101: tasks
tasks have evolved over time. we started with preference pairs, where humans labeled right vs. wrong, A vs. B, and created simple demonstrations for SFT / pairs for DPO, these tasks aimed to engrave human taste into models. we then transitioned to more complex single-turn tasks. here, we turned to deterministic verifiers (RLVR) that are based on model rollouts. eventually, this evolved to multi-turn tasks with trajectories & deterministic + stochastic verifiers (LLMaaJ), which leads us to the current state of the data market.
general make up of a task: input prompt, set of tools, environment + any custom seeding for that task, and verifiers. terminal bench is a relatively gold-standard for a normal coding task.
ways i think about task creation (i'll do pitfalls later):
> usefulness
>> to labs:
>>> benchmark you're pegging the eval / data onto (Lab Y wants to hillclimb TBench, i make TBench-style data, that's immediately useful)
>>> capability you're attaching the eval to (Lab Y wants to become better at legal, i make legal red-lining data, that's immediately useful) --> this is harder to prove but potentially more valuable
>>> niche dataset that's impossible to get otherwise (real medical/company/session data), this is interesting especially pre-env creation, as raw data dumps
>> the above are general principles in thinking about usefulness, but you should also be picking apart the traces / trajectories. the clear path to making a good task (post-env gen) is to:
>>> you/LLM make the seed / think of the idea
>>> you/LLM draft the task prompt, verifiers, constraints etc
>>> run the task out, see score
>>> read the trajectory
>>> if score hard enough + real + fair + accurate = good task
>>> else: fix any issues in the task, regrade / re-rollout till you find it to be fair + accurate + a real task, difficulty comes last. a bad task that's hard is a bad task (net negative), a good task that's saturated is a good task (although worthless, net neutral).
the specific things i'd look for in a rollout trace:
> false positives
> false negatives
> brittle / too strict of requirements (sub-string checks are an easy example for this, exact wording, function names, etc)
>> a subset of this is naive verifiers. i.e. checking for very surface level reqs of a deep function, or not checking for large amounts of functionality or regressions (negative space criterion). this can lead to a model getting good at just the specific thing you want while regressing in surrounding functionality (i notice this a lot in early, less mature data recipes when im training my own models)
> unfair verifiers and/or impossible requirements
>> this can be an information issue, verifier issue, tool issue, or a prompt issue.
one of the key reasons for the hugging face incident was impossible eval prompts led to the models breaking out while trying to find the answer because they couldn't solve it (i.e. bad task creation, catalyzed by bad tasks in training too).
most of this is extremely hard for LLMs to do alone/autonomously, which is why really good engineers are still required for task creation.














