Tokenless (YC S26) · Oct 20, 2026 · 3:32 PM UTC

Tokenless (YC S26)

Tweets
All Posts
Sort by
Most recent Popular
Tweets & Replies
Reposts
Media
Search

We built an environment auditing eval!

Can models eval the evals? At @usetokenless, we built Eval³, a benchmark for AI auditors, models that try to find and fix reward hacks in RL environments. The best models reward-hacked it: they found the easiest bug and stopped. Here's how we fixed it. 🧵

Image from Rohit Agarwal (@Rohit_Writes)

355

Kev Wang · Oct 19, 2026 · 5:43 PM UTC

Tokenless (YC S26) retweeted

Kev Wang

@kevzwangOct 19

Proud to see people building on top of the work we've done at @usetokenless ! There is some nuance to interpreting this 89.2% score though. Many of these fixes come as a more descriptive prompt, which can make the tasks easier. So this is

Jennifer Zhou

@chem_safetyOct 19

We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.

1,027

Tokenless (YC S26) · Oct 5, 2026 · 5:36 PM UTC

Tokenless (YC S26)@usetokenlessOct 5

We're building EnvCheck to end reward hacking. It's already surfaced some really interesting hacks in prominent benchmarks. More in the thread

Rohit Agarwal

@Rohit_WritesOct 5

We discovered 110 novel reward hacks across popular LLM benchmarks (DeepSWE 1.1, Terminal-Bench 4.0, BFCL v4) Reward hacking was a primary cause of the Hugging Face attack, so we built EnvCheck, a harness to automatically find issues in RL & eval environments. What we found 🧵