Songrun He (@hesongrun) profile picture

Financial economist. Markets, AI, asset pricing. Mostly harmless.

Joined October 2019
🚀 Introducing ChronoBERT: A Chronologically Consistent Language Model ⏳📜 Excited to share ChronoBERT, a joint work with @LinyingLyu , @AsafManela , and @jimmywucm ! Our new pre-trained language model ensures no lookahead bias and strong language understanding! 🧠✨
2
4
1
11
3,036
Songrun He retweeted
With job market season in full swing, I thought I’d share the latest CFAR rankings of finance research departments to give applicants more information. Unlike other rankings, these are per capita, so you can see how many faculty are actually load-bearing.
1
2
7
1,391
New slide deck on the biggest open problems in post-training and how the field got to where it is today. Largely, this is a statement of my goals with Trillium Labs (not that we can do every project at once). Stay in touch if you're working on any of these in the open! Opens with the evolution of post-training and ends with a debate on how RSI is changing this type of research. The research areas on my mind: 1. Harness specialization and design – How do we design environments for context management that enable new paradigms of using LLMs? 2. Character training – How do we create the personality of models and how does this impact safety, trust, and diffusion of LLMs? 3. Open-source RL environments – How do we create scalable, open data methods? 4. Reasoning efficiency in RL – How do we control and manipulate the reasoning efficiency of LLMs? 5. Scaling laws for RL – How do we design principled experimentation methodology for large-scale post-training? 6. Multi-agent scaling laws – What are the main axes of inquiry into multi-LLM systems and collaborations? Re-recording video soon :) was from a recent visit to Cornell
22
193
23
1,630
147,702
Songrun He retweeted
Hi everyone! I'm Chris Hyland, an Oxford PhD candidate on the job market. In the past two decades, we've seen multiple tail-risk events: the GFC, COVID-19, and recent geopolitical tensions. My paper asks: how should monetary policy respond to the risk of such disasters? 🧵 1/15
5
24
2
113
9,477
Songrun He retweeted
Some of you have been asking very politely. Some of you have just been asking. GPT-6.2 Astra and Sol are now available. Super efficient and built to perform where it counts—the real world. openai.com/index/introducing…
2,302
3,614
3,734
36,062
13,831,579
Songrun He retweeted
Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen
61,462
121,822
24,530
1,175,787
59,730,758
good read
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.) Instead: constant LR + averaging the last N checkpoints (or a running weight average). I guess that makes it easier in practice because you don't have to commit to a total training length upfront. A little meta analysis based on those who share training details: - Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4 - WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2 - No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
75
Songrun He retweeted
there's something immensely satisfying about the way 42 walks its way into the sticks, then sinking into the landing rails and swaying
Views from Mechazilla of the first ever Starship catch
367
1,275
131
12,856
1,521,930
Songrun He retweeted
Starship Flight 15 mission in one timelapse Launched from the tower → around Earth 14 times → caught by the tower
229
1,929
167
13,214
971,282
Songrun He retweeted
For all but a vanishingly brief instant near the dawn of history, the word ‘ship’ will mean ‘spaceship.’ - Arthur C. Clarke
148
1,114
156
12,175
3,362,170
Songrun He retweeted
In case you missed it, we are on the rookie market this and looking to hire excellent assistant professors of finance passionate about AI. Apply at apply.interfolio.com/196794
3
15
1,852
Songrun He retweeted
what objective is reasoning solving? i've been trying to better understand some weird behaviors in RL/reasoning (especially latent reasoning), so i went back to basics to correct some misunderstandings i had new post on reasoning as pathfinding in token space:
34
102
7
1,403
77,286
Anthropic 路演视频太炸裂了吧!谁说 AI 有泡沫的? 开场,Dario Amodei 预言:癌症、阿尔茨海默病、心脏病、精神疾病,"不到十年"会成为过去。 路演里 CFO 给出的数字: · Q3 营收近 170 亿美元,同比 12 倍 · 今年前三季度毛利率 85%,和美光最新一季的 86.8% 基本持平,而美光是靠这轮存储涨价才冲到这个水平的 · 工程师人均代码量是 2024 年的 45 倍 个人看法:码农真的是夕阳产业,大科技公司还有很大的裁员空间 · 人均营收从 140 万美元升到 660 万美元以上 · 超过一半的财富 500 强年投入超过 10 万美元,但平均只占它们总支出的 0.1% 不到 按路演的说法,现在赚的钱只是冰山一角 · 年投入超过 10 万美元的客户 11,900 家,比年初增长近 8 倍 客户:BMS 预计临床试验时间缩短 20–30%;埃森哲的安全扫描从几天缩短到不到一小时。 结尾,Dario 说"如果我们做对了",就能开创一个"奇迹时代"。 不构成投资建议。 $ANT
58
77
6
530
145,790
机器人世界模型赛道完蛋了,Fable 5.2 已经用基模给搞定了 The bitter lesson 又来一遍
Replying to @claudeai
Fable 5.2 shows substantially improved skills in controlling physical robots, even on tasks it has never explicitly been trained to do. On RoboDojo, a benchmark with a simulated two-arm robot, it outperformed other models and most AI systems built to control robots.
2
4
110
23,666
Songrun He retweeted
i repeat, specialist models are a dead end
Replying to @claudeai
Fable 5.2 shows substantially improved skills in controlling physical robots, even on tasks it has never explicitly been trained to do. On RoboDojo, a benchmark with a simulated two-arm robot, it outperformed other models and most AI systems built to control robots.
67
43
6
850
51,648
Songrun He retweeted
In one of my favorite experiments, I told Fable 5.2 to build an endless, hands-off journey through 3D worlds linked by portals. 25,000 agents created over 450 worlds inspired by books, art, places, all in code. Play: portals-endless-worlds.netli… Github: github.com/emollick/ai-porta…
226
178
78
1,938
111,409
Songrun He retweeted
The becoming begins.
1,729
1,468
1,379
18,740
2,916,990
Songrun He retweeted
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: huggingface.co/spaces/huggin…
139
625
82
4,570
281,152
Songrun He retweeted
📖 The Ultimate MoE Training Handbook 📖 We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training. huggingface.co/spaces/huggin…
33
244
20
1,595
112,712
Don't get too confident about prediction markets! MIT finance department PhD job-market candidate working on prediction market shows miscalibration. The long-shot bias exists and exploiting has a historical Sharpe ratio in the neighborhood of 2! jiagengliu.com/
5
21
1
219
15,498
Computer use in Codex is indeed amazing
The biggest gap between Claude Code and Codex now is computer use. Codex is just miles ahead here. I literally can't live without computer use.
1
104