
Research at @pareto_ai & @CSSPenn → https://nitter.cf/t.co/04KvBxOIQa Previously: @StanfordHCI, @CMUEngineering, @KAISTpr & @RMIT.
Orinda
Joined March 2008
- Tweets13.6K
- Following2.6K
- Followers2.4K
- Likes6K
Mark Whiting retweeted
i think more tech people should go outside and wander into local businesses. why is a massive fabric store still using software from 1998? that system contains twenty years of workflow and exceptions. and the people running these places have such complete mental maps of the business that much of it was never digitized at all. that’s your rare RL env
Mark Whiting retweeted
Today we’re releasing LEAPBench, a benchmark for how efficiently models learn from feedback over many rounds.
Most evals only score whether a model eventually reaches a good solution. LEAPBench scores how many tries it took to get there.
Two systems can reach the same optimum while one needs 7 experiments and the other needs 27. Most benchmarks call that a tie, even though those 20 extra runs cost real time and materials.
LEAPBench is 55 multi-turn optimization tasks, 45 in biology and 10 education RCTs, built from 2,719 real experimental observations.
In each task, the model proposes parameters, sees the result, and iterates toward an optimum over as many as 30 rounds, e.g. one biology task involves maximizing antibiotic yield by tuning variables like pH, glucose concentration, temperature, and soy flour level.
Mark Whiting retweeted
We built FrogsGame as a new task for evaluating AI’s posttraining skills! It’s a tool-using RL environment built around a blind-start interaction loop.
Frontier agents get a container with the Qwen3-8B tokenizer, board-generating scaffolding, and @tinkerapi for remote training and inference. We ask them to post-train Qwen3-8B and submit a LoRA checkpoint. The task is still hard:
Mark Whiting retweeted
A startup shut down 9 years ago. Today, its code is in a bidding war.
The founder kept everything: Verilog RTL, a custom compiler toolchain, PCB schematics, full commit and PR history.
Most production-grade chip design stays locked inside major silicon shops.
Two labs have made offers to license the code to build RL training environments. The offers already total seven figures.
Check your hard drives.
When is it worth it to hire a team, compared to one competent individual?
📢 NEW PAPER (out this month in Management Science!) by me, @MarkWhiting, @LinneaGandhi, @duncanjwatts, and @amaatouq! 🧵1/20
Mark Whiting retweeted
Baird Howland, an @AnnenbergPenn postdoc working with @WarrenCntrPenn affiliate @duncanjwatts, uses AI models to track narratives in political discourse, learning how and why specific media stories shape American worldviews. penntoday.upenn.edu/news/pen…
Mark Whiting retweeted
The next step for autoresearch is that it has to be asynchronously massively collaborative for agents (think: SETI@home style). The goal is not to emulate a single PhD student, it's to emulate a research community of them.
Current code synchronously grows a single thread of commits in a particular research direction. But the original repo is more of a seed, from which could sprout commits contributed by agents on all kinds of different research directions or for different compute platforms. Git(Hub) is *almost* but not really suited for this. It has a softly built in assumption of one "master" branch, which temporarily forks off into PRs just to merge back a bit later.
I tried to prototype something super lightweight that could have a flavor of this, e.g. just a Discussion, written by my agent as a summary of its overnight run:
github.com/karpathy/autorese…
Alternatively, a PR has the benefit of exact commits:
github.com/karpathy/autorese…
but you'd never want to actually merge it... You'd just want to "adopt" and accumulate branches of commits. But even in this lightweight way, you could ask your agent to first read the Discussions/PRs using GitHub CLI for inspiration, and after its research is done, contribute a little "paper" of findings back.
I'm not actually exactly sure what this should look like, but it's a big idea that is more general than just the autoresearch repo specifically. Agents can in principle easily juggle and collaborate on thousands of commits across arbitrary branch structures. Existing abstractions will accumulate stress as intelligence, attention and tenacity cease to be bottlenecks.
Mark Whiting retweeted
Recently I've been thinking a lot about frontier models' ability to express uncertainty, especially for high-stakes medical use cases. We evaluated this capability across models.
Early results below 👇
Gemini: 0% across every scenario.
Claude: failed on over half
GPT: best
Mark Whiting retweeted
My favorite designers can instantly switch from loose / hazy / intuitive thinking to sharp / analytical / precise thinking on demand.
Many people can do one or the other. The combination is rare!
Mark Whiting retweeted
I've been trying to figure out why AI systems took a seemingly large, discrete jump in capabilities around the new year. 1/n
Mark Whiting retweeted
What if technology didn’t feel so… hollow?
Some friends and I just released a manifesto about a world where tech leaves us feeling nourished (along with an evolving list of theses about how we can build it)
resonantcomputing.org
Mark Whiting retweeted
Replying to @SayHelloMarcus
Bro, I can make 1 liter of Anthrax in an afternoon, Grok just wrote me a 20 page detailed report and instructions on how to do it.
It also listed all websites where I can buy the materials and chemicals I need as a private person living in Europe.
It also made a detailed list of the equipment I need (on a budget)
It made detailed instructions on where i should deploy the anthrax for maximum death efficacy
Give me one other place on the internet where I can create this in a few minutes….
Mark Whiting retweeted
The common sense project is now live! @DuncanJWatts, @MarkWhiting, @amirhosnakh and @joshnguyen99 from the CSSLab invite you to take a quick survey 📝 and measure your common sense💡
commonsense.seas.upenn.edu
You can read more about the project here:
css.seas.upenn.edu/commonsen…
Mark Whiting retweeted
Check out @CSSPenn's recent blog post too.
nitter.cf/csspenn/status/1842254…
8/
The common sense project is now live! @DuncanJWatts, @MarkWhiting, @amirhosnakh and @joshnguyen99 from the CSSLab invite you to take a quick survey 📝 and measure your common sense💡
commonsense.seas.upenn.edu
You can read more about the project here:
css.seas.upenn.edu/commonsen…
Mark Whiting retweeted
Everybody trivializes common sense because we believe it is self-evident and universal. But is it really?
Find out by participating in our Common Sense Project: commonsense.seas.upenn.edu.
With @AmirhosNakh, @MarkWhiting, and @DuncanJWatts at @CSSPenn.
1/n
Mark Whiting retweeted
The common sense project is now live! @DuncanJWatts, @MarkWhiting, @amirhosnakh and @joshnguyen99 from the CSSLab invite you to take a quick survey 📝 and measure your common sense💡
commonsense.seas.upenn.edu
You can read more about the project here:
css.seas.upenn.edu/commonsen…
Mark Whiting retweeted
Using GPS data to inform epidemic modeling, @csspenn Director @DuncanJWatts and Post-Doctoral Researcher Francisco (Paco) Barreras tackle integrating mobility data while addressing privacy concerns to build public trust. bit.ly/3Y5D1b3


































