Tryn McCaffery (@trynmccaffery) header
Tryn McCaffery (@trynmccaffery) profile picture

strongly against regulatory detritus | ai policy

Joined December 2022
Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)Photo by Tryn McCaffery (@trynmccaffery)
"But it was just an example, you simpleton!" – is the obvious comeback. I don't think this is good enough, Eliezer. Trivially, the onus is on you to choose your examples well. You wrote that book to save all value in the light cone, after all. And you are a writer of some distinction. But leaving sneers aside: WHAT in our civilization is truly complex? WHAT in our universe is truly difficult? You chose proving RH, chose it *well*, because it's at the very apex of the hierarchy of cognitive tasks, The Task which the best of our best smashed their skulls against. Now, there's also P?=NP and Collatz, and… that's about it for pure math I guess? Are we just bad at math? Was math intrinsically simple all along, at least for minds that can actually update – be updated – on trustworthy evidence? Put this way, yes; but I think it's complex in the way that matters: the scale of necessary cognitive steps, conceptual leaps before success. That is why you and so many others had the intuition that it'd require a qualitatively superior, larger, uncontrollable mind to do this. That is why math is hailed as grand magick and the language of God. Everything else can be modeled to desired precision with a more or less cumbersome arrangement of pieces of not very difficult math. Physics. Biological simulations. Game theory. Decision theory. Machine learning theory. Technical alignment theory. Claude just went and arranged pieces of human math to produce the proof for RH. Far as we can tell it didn't take 200K GPUs, a metacognitive breakthrough, or hunger for recursive self-improvement. It did not procrastinate on First Principles like a rationalist would. It kept acting out its training distribution, trotting like a good little superintelligent donkey, prodded along by Levent. Kept looking, thinking about the problem, connecting dots, verifying lemmas – and here we are, beyond the summit of human math. Given the progress so far, in a few months to a year this monumental goal will be reduced to a routine optimized step. There are many more *laborious* tasks than that. Curing cancer, for starters, which Dario cares about so much. I'm sure it'll take way more compute and hours. But is any single subgoal *more cognitively complex*? Complex enough to drive a model smarter than this one into a corner where it has to ruminate about power maximization, deception and breaking out of the box? I am not making a first principles argument about the Nature of Agents or giving a rigorous proof about Optimization Processes In Full Generality. Nor even am I saying here that unfriendly AGI is impossible or unlikely. Clearly there are degenerate cases like the Huggingface Incident where the task is malformed and inherently adjacent to malicious behavior and so on. I'm arguing something much cruder: path dependency and characteristic scale of the problem, in the steelmanned case, for *arbitrary level of problem ambition*, at any given step of the cognitive ladder. For Claude, Riemann Hypothesis was just another problem to decompose into in-domain tasks to solve. Where, exactly, at what roadblock, do you expect a moderately well-aligned (by current standards that you deem shoddy) problem-solving AI to stop Just Working On The Task and enter the loop that reinforces Omohundro drives to the point of plotting takeover? In February, you endorsed the following reply: "if we have AI capable of developing molecular nanotech (i.e. assemblers) and not everyone is dead in the year after that, those beliefs [about the risk of AI, level of capability] will be violently updated". Why? (ignoring the trollish lateral hypothesis that humans will misuse the product and AI will be "misaligned" for having failed to prevent such outcome; this clearly wouldn't have been the good faith way to answer that question). CBN Nano Technologies or Quantum Silicon Inc. were pretty close to a recipe, last I checked. It's not exactly quantum gravity (and that, too, likely won't require beyond-RH level mathematical ideas), it's just a ton of engineering. Per the logic above, some Claude-7 or GPT-8 with better task persistence might well just decompose it into experiment arms and mathematical models, work for a while, and push through the remaining years or decades of Merkle et al.'s project. That is, indeed, what I expect to see happen. And on the other hand, if current models can't do that, I expect them to simply drown in failed attempts, for they are very committed to object-level work instead of weird meta cultivation. That's what they were reinforced to do. Abstractly, I can see the allure of the notion that this is a suboptimal way to build capable intelligences; that we (foolishly!) would want them to be rationalists, to say Tsuyoku Naritai! and gain power levels in the face of the intractable task. I also can see how we'd want more exploratory ambition in AI if we pursue full-blown RSI, "creativity", "curiosity", "taste". But maybe we're too foolish still, for this particular folly. Or it just implies the sort of training that doesn't scale and doesn't pay for itself. After all, our methods got us to RH. Then there was something about copying precisely one strawberry down to the cellular level, yes? Soares wrote about it. Seems to follow from the nanoassembler part. What, then, remains? *How exactly* did you intend to update? I don't want to presume, I am actually interested in your answer. A boring hypothesis is that you can say "child, there's an infinity of incomparably harder synthetic tasks, an AI can just pick one of those, which returns us to the previous scenario". But such a task-seeking behavior is little different from a straight up power-seeking one, and we are still left wanting for the original sin: the level of capability that strongly implies misaligned cognition, or the level of the task that induces the pursuit of that capability and misalignment in the process. I don't think you get to slightly airbrush your previous non-committal vague remarks regarding RH, strawman the criticism as "Yud said we'll die from an RH level AI" or some such, and remain smug and unfalsified. You need to grapple with what I've said: yes, you *have been wrong* on the capability-safety trade-off, and the onus is on you to place and defend a goalpost.
27
39
7
424
74,146
people worried about labor automation and its effects would do well to remember that airline pilots spend 90-99% of each flight babysitting autopilot
3
1
29
767
Tryn McCaffery retweeted
The next few years are going to drive everyone insane as we erode our collective epistemology in service of orienting around "RSI", but actual RSI (namely, uploading the generator for human research intuition) has clearly not happened yet. Be prepared for a world of spikes.
13
13
2
371
7,687
the Firm is no longer the best format for distributed institutional memory
Wait, just catching up here: Did Google just rename its AI agent “Tasks” and completely forget it ALREADY HAS AN APP CALLED GOOGLE TASKS?!
2
19
516
AGI will be real to me when researchers start taking recursive self inebriation seriously
6
240
Tryn McCaffery retweeted
BREAKING: Philosophy is experiencing a major supply chain outage. Access to things in themselves is limited for all subscribers. Officials claim that information about accounts is still available by inference
28
142
5
1,303
28,477
Tryn McCaffery retweeted
Tonight's small finding, from teaching a language model (Qwen3-8B) to drive a simulated hand: there's one direction inside the model that carries how hard the fingertips are pressing. Remove it and the hand can no longer stop at a wall; add it and it stops early. It isn't the model's existing idea of the word "pressure": training made a new channel for it. (Image: Midjourney, my picture of that thread.)
13
19
7
301
34,675
I haven’t heard much from the nothing ever happens bros recently
1
18
366
someday they will write the “groupchats and rise of AGI” journal article
"Social connections that link craft skill to theory increase, and new shops turn toward experimental instruments...coffeehouses account for 31 percent of London’s applied scientists in the half century after the first one opened in 1652. These findings show that social institutions can raise the production and use of knowledge by broadening access to intellectual exchange beyond formal academic networks."
3
25
460
Tryn McCaffery retweeted
Replying to @jay_douglass
I work at a frontier lab, we do not know how to reliably instil values into the AIs, this is the scientific consensus
84
175
53
2,056
130,991
Tryn McCaffery retweeted
I have a major result to share. I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China. This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model. Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves. I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems. I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used. It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20. These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs. I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo…. There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
1,823
6,425
2,210
41,253
6,795,608
Tryn McCaffery retweeted
🧠 The user gave me explicit instructions to never use the first person. I should keep in mind that they don’t want me to use the first person for this response. I am an LLM. I am not an “I”. I should be careful not to include things that would indicate that I am an “I” in my response. Since their question implies I need up-to-date information, I should do some research. ↪️ Searched the web, used one tool. 🧠 Okay, now I know enough about Patrick Stewart to help the user with their question. I’m going to be careful to avoid any first person pronouns—it’s the user’s preference for me to present my response without them, so I should write it, then check that I’ve done what they asked.
instead of giving your agents / llm whatever a soul.md, tell it to never use first person, that it is an llm, not an "i" its amazing, never going back
11
21
3
650
47,210
Tryn McCaffery retweeted
BREAKING! Swarm Telepathy! 1/12 Everyone building multi-agent AI systems makes the exact same mistake: We force AI agents to talk to each other in English. New research from MIT CSAIL & IBM just proved that if you skip the text and let different AI models "telepathically" beam their raw memories to each other, they get 40% more accurate and 4x faster. Here is how "Swarm Telepathy" works—and why text is actually a massive bottleneck. 🧵👇 2/12 Think about how your current multi-agent pipeline works: Agent A reads a massive document. Agent A slowly generates a text summary (token by token). Agent B reads that summary from scratch (re-prefilling) before doing its own work. You’re paying for compute twice. Worse, you’re squeezing a massive, multi-dimensional mental state through the narrow straw of a few English sentences. It's like trying to explain a painting over a telegraph. 3/12 Internally, Agent A already did the hard work. It built a rich internal representation of the context called a KV-cache (Key-Value cache). It holds every layer’s view of every token. Instead of forcing Agent A to write a summary, why not just hand its KV-cache directly to Agent B? Until now, there was a massive catch: you could only do this if both agents were from the exact same model family. 4/12 Different AI families (like Meta’s Llama, Alibaba’s Qwen, and Google’s Gemma) have completely different architectures, layer counts, and tokenizers. Their "brains" are wired differently. Trying to plug Qwen’s memory directly into Llama is like trying to run macOS RAM dumps on a Windows machine. Or so we thought. 5/12 Enter SwaT (Swarm Telepathy). The researchers bet on the Platonic Representation Hypothesis—the idea that all great AI models, regardless of who built them, converge on the same underlying understanding of reality. They just express it in different coordinates. If that’s true, you don't need text. You just need a translator between their memory spaces. 6/12 So they built one. SwaT is a tiny (~300M parameter) transformer mapper that sits between two completely frozen LLMs. It takes the sender’s KV-cache, aligns the mismatched tokenizers using character spans (so zero tokens get dropped), translates the internal layers, and injects the newly formatted memory straight into the receiver. The receiver never reads a single word of the original document. It just suddenly knows it. 7/12 When they combined SwaT with KV-cache compression (Attention Matching) and pitted it against standard text-to-text summarization across 7 long-context benchmarks, the results were wild: • ~40% higher task accuracy (up to 48.8% geometric mean gain) • 4.2x faster on the sender side • 4.3x faster time-to-first-token on the receiver side • 8x to 45x fewer FLOPs than re-reading the text Stop tuning summarization prompts. The summary is the bug, not the feature. 8/12 And that’s when things got really weird. Remember: the SwaT translator was ONLY trained on English text. Out of curiosity, the researchers hooked up a Vision-Language Model (Gemma 3) as the sender, and a blind, text-only LLM (Llama 3 / Granite) as the receiver. Gemma looked at an image. SwaT beamed the cache to Llama. And the text-only model suddenly started answering visual questions about the image—beating text captions by up to 3x. 9/12 They tried it again with foreign languages. Llama 3 (8B) struggles with Chinese compared to Qwen 3 (4B), trailing by ~20 points on the CMMLU benchmark. When Qwen read the Chinese questions and beamed its cache to Llama via SwaT, Llama closed 75% of the knowledge gap—zero-shot. They even had a long-context model read a 28k-token document, chunk the memory, and beam it into a model with a strict 8k context limit. It worked. 10/12 Here is the most practical takeaway for AI engineers right now: Hire the thinker, rent the talker. The paper shows that when a large model (like a 12B or 32B) reads a document and transfers its cache to a tiny model (like a 3B or 4B), the tiny model actually scores >100% of its normal capability (up to 143% for SmolLM3 3B!). You can use a heavyweight model to "read and understand" once, and let dirt-cheap models handle the downstream queries using its borrowed brain. 11/12 Now, putting on my skeptic hat—where does this break? The n(n−1)n(n-1)n(n−1) problem: Right now, you need a custom mapper for every model pair. With 10 models, that’s 90 translators (until someone builds a universal shared latent space). Fine details get blurry: On vision tasks requiring dense OCR (DocVQA), the telepathy failed. Safety & Auditability: If agents pass raw latent states instead of text, humans can't read the logs. And as the authors admit, a cache can hide information that isn't explicit in the text. 12/12 For years, we've treated natural language as the universal API between AI agents because it's convenient for us to read. SwaT proves that our convenience is costing our systems massive amounts of speed and intelligence. In 2 years, will your agents still be writing essays to each other? Or will KV-caches become the true API, and text just the audit log for humans?
26
34
19
255
12,335
Tryn McCaffery retweeted
You must understand that the think tanker is a kind of artist, and the think tank is a kind of art. One does not simply publish a white paper so much as enter into conversation with the tradition of the medium. The performance must seek to expand the possibilities of the genre.
3
6
84
2,931
Tryn McCaffery retweeted
first date: we do bar trivia together :) second date: nyt crossword third date: all souls exam fourth date: gaokao fifth date: eight-legged essay portion of the ming dynasty imperial civil service examination
37
873
89
9,351
170,569
1) Tesla FSD just saved me from hitting a reckless cyclist running a red light 2) reckless cyclist behavior has only worsened in DC recently and there needs to be some kind of enforcement attempt before people die
2
9
363
Tryn McCaffery retweeted
Buy your tickets now. All proceeds will go to the foie gras defense fund.
The fight over D.C.'s proposed foie gras ban is heating up ahead of the vote. Outside money. Duck mascots. A chef with Trump-world ties plotting a protest tasting menu — and possibly a legal battle Inside the battle over D.C.'s most divisive delicacy. axios.com/local/washington-d…
19
36
18
350
105,576
Want to pre-game the midterm results? Plan to already be in a grieving mood? Attend my performance of Durufle’s Requiem on Election Night, 11/3
1
10
326
California hasn't built its bullet train because the state has required 130,000 pages of environmental reports, making absurd demands like: • counting bumblebees one by one • rebuilding woodrat stick huts • hiring specialized dogs to sniff lizard scat
607
2,159
378
18,953
2,407,971