Joe Fenton (@JoeFenton) header
Joe Fenton (@JoeFenton) profile picture

Pretraining data and evals // prev: founding team @inflectionai, PM @googledeepmind, VC at Invoke Capital and astrophysicist

Joined March 2009
Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)Photo by Joe Fenton (@JoeFenton)
“All that matters for anyone in life is their family, their health and that’s always the same for everyone” — Mike Lynch bbc.co.uk/sounds/play/p0jkc9…
4
1,743
As models get smarter, the tasks they can tackle take longer, so I run more projects in parallel. However, I've maxed out the number of projects my brain can manage. To get another step change in productivity, faster inference is more important to me than more intelligence.
5
6
424
Vulnerabilities tracked in NIST’s National Vulnerability Database, over time - shows the trend even more visibly
Mandatory cyberattack reports on Swiss critical infrastructure are up 85% year-on-year and 58% month-on-month
2
155
Mandatory cyberattack reports on Swiss critical infrastructure are up 85% year-on-year and 58% month-on-month
1
2
306
Easter egg for you all
beautiful scaling plot from MAI 😍
1
5
278
…or just talk with your kids like a human being
cool use case of chatgpt work i heard last night: connect your family calendars and explain your kids' interests. every morning for the drive to school, have it make a podcast that talks about one kid's soccer game that afternoon, one kid's upcoming birthday, some news, etc.
1
8
629
The market is underestimating the fragility of Anthropic and OpenAI.
17
7
173
29,621
Business idea: > Free holiday in Maldives for mathematicians > In exchange for 10 math problem per day > Sell the problems to AI labs > Profit Who's in? First 50 people only have to produce 8 problems/day. Inspired by Benchmarks in Leipzig arxiv.org/abs/2606.05818
1
4
868
Positive reaction to the mai-thinking-1 tech report is more than I imagined. Some nice write-ups from the open research community
2
11
1,060
MAI-Thinking-1 by Microsoft looks to be approaching sonnet level model, the 109 page tech report is gold they got 29T unique tokens without any synthetic tokens for pretraining which is exact opposite of what they were doing with phi models !! so many counter intuitive decisions but the best part is they talk a lot about data.. this is a must must read
1
275
"MAI-Thinking-1: Building a Hill-Climbing Machine" Microsoft just did something almost no frontier AI lab has done before They shared how they engineered the data behind a frontier-scale model in unusual depth. From data collection and eval decontamination, to data mix scaling, this paper lays out how they managed 30T pretraining tokens plus 3.55T midtraining tokens Surprisingly, they also used no third-party distillation and no open-source training datasets The model itself is not a jaw-dropping release, but the paper might be the best open look yet at a frontier-scale data factory and hill-climbing loop.
1
149
Super detailed tech report for MAI-Thinking-1, with a ton of info on all stages of the pipeline. I'm surprised so much of this info is released :) Super long thread on my notes:
102
> buy truckloads of good books > remove unspeakable amounts of slop from web data > build a shedload of held-out evals that was my work on mai-thinking-1 the model gets 97% AIME and I can speak for hours about ISBNs read the tech report: microsoft.ai/wp-content/uplo…
1
6
349
Anthropic achieves escape velocity - question is, who will be next...
1
1
3
539
Qualitatively observed the same among AI researchers. The most successful are often exceptionally strong in seemingly orthogonal areas. Stay general kids…
2
384
Wondering if OpenAI falls into this category…
"Invest in companies that would be happy to see a 100x improvement in foundation models" -- paraphrasing Sam Altman
381