Chris Flath retweeted
1/8 There is lots of noise in the AI debates. One thing those debates do not need is pejorative framing: labels like “AI slop” and “plagiarism machine” that invite dismissal before evaluation. Academics need to have higher standards.
Chris Flath retweeted
I've plotted the most expensive McDonald's burger and the least expensive MacBook over time. This analysis projects that the most expensive burger will be more expensive than the cheapest laptop as soon as 2081
Chris Flath retweeted
micropolis.c-ehrlich.dev
(not mobile friendly for now)
Excellent!
How much does intelligence cost? How concentrated is the AI market and is it winner take all? When prices fall, how does demand change and is there a Jevons effect? These questions matter, but actual market data has been hard to come by. We use data from Microsoft Azure and OpenRouter to find out.
Details below.
118
Chris Flath retweeted
Something I realized we should have really emphasized more is that we released the entire training pipeline open-source. None of the current SOTA pathology FMs do that.
From the public dataset used to training code to model weights, this release is truly open-source.
At NeurIPS a pathology AI researcher said he tells other students and researchers about OpenMidnight for this very reason, making our model easy to get started with. We're glad to see our model getting good reception :)
Excited to share our latest @SophontAI release 🥳
"How to Train a State-of-the-Art Pathology Foundation Model with $1.6k"
We present OpenMidnight, our first pathology foundation model!
It has SOTA perf. despite being only trained on 12k whole slide images w/ $1.6k compute!
Chris Flath retweeted
late 1980s, @ylecun and @LeonBottou used amiga 1000 and a bespoke modem to implement and research artificial neural nets using SN-1. the legend was born.
Chris Flath retweeted
Checkmate! Did you know LaTeX can typeset chess games? We've written this comprehensive guide to get you started. overleaf.com/learn/latex/Che…
Chris Flath retweeted
When over 500 players set up the pieces at the same time!
#FreestyleChess #GrenkeChess
Chris Flath retweeted
Why it is important to know your data well
Ht @MTabarrok
Chris Flath retweeted
Man muss kein Wähler der Grünen sein, man muss sie nicht mal mögen, um erkennen zu können, dass Katharina Dröge hier genau den Nerv trifft und zu 100 % sowohl in Richting CDU/ CSU, SPD und die Linken wesentliche und wahre Punkte nennt
Chris Flath retweeted
Here’s a behind-the-scenes look at our recent @QJEHarvard paper—a project that began in one of my PhD classes at @EconUniMannheim and evolved into a full-scale field experiment on LinkedIn.
Recently accepted by #QJE, “LinkedOut? A Field Experiment on Discrimination in Job Network Formation,” by Evsyukova (@yu_evsyukova), Rusche (@felix_rusche), and Mill (@Econ_Mill): doi.org/10.1093/qje/qjae035
Chris Flath retweeted
Wish I could take this course at Stanford on AI & Simulation: joonspk-research.github.io/c… Looks fantastic...
Chris Flath retweeted
🚀 The latest edition of the “Connect the Dots” newsletter features:
• A Call for Papers for a Strategic Management Journal Special Issue on chance, luck, and serendipity (@Strategic_Mgmt)
• Some key takeaways from the Academy of Management (@AOMConnect) Annual Meeting
• A new video on the science of serendipity, highlighting actionable insights
• Resources for educators, including complimentary copies of CONNECT THE DOTS.
Wishing you many serendipitous moments ahead!
bit.ly/connect_the_dots_octo…
@AOM_STR @AOM_OMT @AOM_TIM @ENT_Div_AoM @USCMarshall
Chris Flath retweeted
New research by Simon Masuch, Jan Landwehr, @chris_flath and @FThiesse explores the impact of short delivery times on product returns in online commerce. Quick deliveries attract customers, but also lead to increased return costs
doi.org/10.1016/j.jretai.202…
#journalofretailing
Wenn jetzt wieder alle fragen was gute Sportförderung ist habe ich seit vielen Jahren die Antwort - Sport attraktiv mit vollwertigem Studium kombinieren anstatt Kadersportler als Sportsoldaten ohne Karriereperspektive zu parken.
Athletes from four California universities won 89 Olympic medals. (The United States won 126 total).
Athletes from Stanford University alone won more medals than all but seven countries in the world.
🥇🥈🥉
1
278
Chris Flath retweeted
Kein Elfer. 🤡🤡🤡 #suiger
Chris Flath retweeted
📽️ New 4 hour (lol) video lecture on YouTube:
"Let’s reproduce GPT-2 (124M)"
youtu.be/l8pRSuU81PU
The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model:
- first we build the GPT-2 network
- then we optimize it to train very fast
- then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers
- then we bring up model evaluation, and
- then cross our fingers and go to sleep.
In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.
Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step.
github.com/karpathy/build-na…
Chapters.
On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail:
00:00:00 intro: Let’s reproduce GPT-2 (124M)
00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint
00:13:47 SECTION 1: implementing the GPT-2 nn.Module
00:28:08 loading the huggingface/GPT-2 parameters
00:31:00 implementing the forward pass to get logits
00:33:31 sampling init, prefix tokens, tokenization
00:37:02 sampling loop
00:41:47 sample, auto-detect the device
00:45:50 let’s train: data batches (B,T) → logits (B,T,C)
00:52:53 cross entropy loss
00:56:42 optimization loop: overfit a single batch
01:02:00 data loader lite
01:06:14 parameter sharing wte and lm_head
01:13:47 model initialization: std 0.02, residual init
01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms
01:39:38 float16, gradient scalers, bfloat16, 300ms
01:48:15 torch.compile, Python overhead, kernel fusion, 130ms
02:00:18 flash attention, 96ms
02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
02:34:09 gradient accumulation
02:46:52 distributed data parallel (DDP)
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
03:23:10 validation data split, validation loss, sampling revive
03:28:23 evaluation: HellaSwag, starting the run
03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro
03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA
03:59:39 summary, phew, build-nanogpt github repo
Chris Flath retweeted
A student in my class shared this FT article on transformers & I thought it was very nicely done: ig.ft.com/generative-ai/




































