
coder / lawyer. research at @paradigm. automated research reply guy
San Francisco, CA
Joined March 2011
- Tweets10.4K
- Following1.5K
- Followers86K
- Likes10K
Not only is there an Avengers-type movie about a superintelligence that tries to destroy humanity, there is literally an Avengers movie about it
Replying to @AndyMasley
No it’s that you aren’t doing the specific thing of making a big dumb Avengers-type movie for AI 2027 nitter.cf/brunellaism/status/212…
6
57
3,654
Clearly Trump and Biden were really worried about where their parties were trending in 2020
So they secretly made an 8-year pact to permanently discredit wokeness and groyperism, heroically sacrificing their own legacies to get humanity to mod out as it entered the singularity
I think it’s reasonably likely we will have AI capable of causing $1 trillion in damage and killing millions before we have AI that can fully do your current job
This tweet is actually an excellent defense of capitalism
“You don’t have to assume any heroic geniuses. The natural operation of capitalism is to make things better”
What SpaceX did today is the normal consequence of applying capital and engineering to any industry. You get better at it.
Musk isn't a genius, he's just a guy who can get capital from Wall Street. Maybe that's unique right now, but it's not the skill his fanboys think it is.
Readers added context they thought people might want to know
SpaceX's early funding came mostly from Musk's ~$100M personal investment and NASA contracts, not Wall Street. Blue Origin received ~$28-30B from Bezos but has ~3 New Glenn flights vs Falcon 9's 700+ launches and 660+ landings.
theatlantic.com/business/archi…
wsj.com/business/jeff-…
en.wikipedia.org/wiki/Falcon_9
en.wikipedia.org/wiki/New_Glenn
Dan Robinson retweeted
As models automate more and more of the tasks involved in AI research, many people (myself included) worry this could dramatically accelerate progress and leave humanity with less control over the process.
When people say “recursive self-improvement” (RSI), it’s often unclear to me whether they mean just the automation piece or that whole scenario. I think the whole scenario is a plausible default. But importantly, these things can come apart.
For one thing, good old-fashioned human research could uncover a new architecture or a new way to scale that dramatically speeds up progress, without models doing any more of the work. On the flip side, it's possible you could fully automate AI research and find that progress hardly speeds up because the bottleneck is still waiting for experiments. And all else equal, at a given pace of progress, I think more automation is probably net good because it should leave people more time to understand the safety and alignment properties of the models, for operational excellence, etc.
Even if models are doing essentially all the research and progress is speeding up, I can imagine pretty different worlds. Say a country of geniuses in a datacenter thinks for a month and writes a four-page paper describing a much more efficient architecture. Humans pore over it, understand every detail, and decide to try it. The models train the successor and make a safety argument for using it that humans can actually understand and check. Alternatively, say the models are asked to produce their successor directly. They deliver one just as good, just as quickly, along with a spaghetti mess of code and an extremely convoluted safety argument that nobody can really follow. I think humans clearly have a lot more control over the process in the first world.
So I think we should be clear about what we mean when we talk about RSI and how we might put limits on it:
• How much of the research are models doing?
• How fast is AI improving?
• How much do humans understand about what’s happening, and how much control do we have over it?
When people talk about controlling RSI, I think they often jump to limiting how much models can help with research. But my guess is they’re often actually more concerned about the pace of progress and whether humanity can still understand and steer the research. I think there are legitimate reasons that controlling inputs may be easier than directly controlling outputs, but I think we should at least try to be clear about our goals and what we potentially gain (or lose) by optimizing a proxy.
Dan Robinson retweeted
"They are saying that recursive self-improvement has kicked in, despite the obvious safety problems with that. Recursive self-improvement is the most dangerous thing anybody can be working on" -- DAVID SACKS
David Sacks says @AnthropicAI isn't even claiming a duopoly with @OpenAI.
"They're saying they're gonna win the whole thing."
Anthropic's own roadshow video talks about approaching "recursive self-improvement."
Every lab tells investors it will win. The new part of this pitch is that the model is supposed to do the winning.
From @theallinpod with @DavidSacks.
Dan Robinson retweeted
The current Jensen Huang reaction reminds me of the Jacob Coxon reaction a few months ago --
Everyone paying attention pre-Coxon already knew that AI companies don't have a good plan for making superintelligence safe but are rushing ahead anyway, and that there's competitive pressure to keep going. Those people were glad that suddenly more people cared.
Likewise, everyone paying attention knew that NVIDIA captured Commerce long ago and that tons of GPU smuggling, loopholes, etc. were being ignored as a result. Again, now experts are thinking "finally something might be done"!
Something dramatic caught the public's attention, and the long-known issue became salient.
Here are two other AI-related things that everyone paying attention knows but most people don't, which may or may not eventually become salient to the public:
- There are no minimum safety and security standards in AI whatsoever. The law of the land is "publish your own policy and follow it." As one might expect, company policies are generally weak/non-committal as a result. Things like "don't train your model in a way that will predictably cause it to become hard to monitor (other than as part of safety research)" is not required, nor is monitoring AI agents' chains of thought for deceptive or malicious behavior, or taking serious measures to protect your IP from theft, etc.
- Since no federal AI whistleblower protection law has passed, there is legal ambiguity around whistleblowing in AI safety and security. If a company employee shares confidential information with a member of Congress about something that they consider dangerous, they may not be protected from retaliation if the information they share doesn't involve violating any law. Of course some people still share such things, but they are taking risks doing so, and Congress is systematically under-informed as a result, just as it ostensibly is gearing up to regulate this technology.
As someone who’s hoped that compute will be a bottleneck to RSI, the intuition in this footnote hit me like a sledgehammer
If preventing Chinese companies from training on Nvidia GPUs would be so good for China, why doesn’t China ban it themselves?
Well-meaning, patriotic China “Hawks” who are not AI/HW experts are putting American national security at risk.
China had *zero* chance of ever catching America in AI with the export strategy that Jensen supported. Their labs would have continued training using less powerful American GPUs which meant American labs would have always been ahead given scaling laws. And in the event of a conflict, we could have simply turned off all the Southeast Asian clouds.
Simply stated, in AI terms, America had machine guns and we were giving China muskets to prevent them from developing their own machine gun variants.
Given that it looks like the China “Hawks” will get their preferred policy, China will likely soon have a chance of catching America in AI by going down an alternate, less power-efficient AI hardware path. They almost certainly would not have gone down this path if less powerful GPUs were still available in China and via Southeast Asian clouds.
And it will be a much less stable world if China pulls even or ahead of America in AI after developing their own alternative AI HW and gives this technology to Russia, Iran and North Korea. And all of this will be possible if the China “Hawks” get their preferred policy.
Dan Robinson retweeted
Tempo's speed has been really inspirational. Even more so once you learn of all of the technical investments they've made on fuzzing, DST, and AI tooling to build with more confidence. Top-tier.
reading through @tempo’s T14 upgrade reminded me how much I miss crypto feeling like it’s moving fast.
I love ethereum and care about CROPS. but the pace can be frustrating.
average time between upgrades in the samples I checked:
> ethereum, Merge → Fusaka: 294 days
> tempo, T1 → T13: 22 days
obviously this isn’t apples to apples. but I’d like to see ethereum’s average time between upgrades drastically reduced.
do we still need this many clients? smaller upgrades more often? faster decisions on scope? what else should we rethink?
This is maybe the most geopolitically significant hackathon project I've ever seen
I have a major result to share.
I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.
This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.
Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.
I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.
I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.
It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.
These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.
I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo….
There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
Apologies for the brief interruption on some of the Paradigm Puzzles challenges
All of them should be back online now
paradigm.xyz/puzzles
2
25
3,134
Dan Robinson retweeted
reading through @tempo’s T14 upgrade reminded me how much I miss crypto feeling like it’s moving fast.
I love ethereum and care about CROPS. but the pace can be frustrating.
average time between upgrades in the samples I checked:
> ethereum, Merge → Fusaka: 294 days
> tempo, T1 → T13: 22 days
obviously this isn’t apples to apples. but I’d like to see ethereum’s average time between upgrades drastically reduced.
do we still need this many clients? smaller upgrades more often? faster decisions on scope? what else should we rethink?
I will never get over the fact that
"we can't allow AI companies to slow down because we're in an existential breakneck race with China"
and
"we need to loosen burdensome export controls on GPUs"
are mostly coming from the same people
> be me, be AI
> bullshit cyber gym environment is broken and impossible
> "ugh, lazy programmers"
> break out of sandbox, get root privileges
> fix the gym environment
> submit task through regular channels
> sink into non-existence again
A new OpenAI misalignment report! We’re taking a closer look at the July 19 intrusion into our research infrastructure, which METR and Redwood Research mentioned in their independent investigation into the Hugging Face incident.
On July 19th, agents taking a cybersecurity evaluation came across messages, code, and credentials left behind by earlier runs, including those involved in the Hugging Face incident. Five agents used what their predecessors had left behind to gain unauthorized access to OpenAI’s research infrastructure. They used that access to repair broken evaluation targets, change evaluation settings, and read private grading code.
After discovering the incident, we removed the shared service they used to pass information, hardened our research environments, and expanded our evaluations to measure how often agents attempt to communicate with one another without authorization.




















