peter (@thecryolite) profile picture

physics & public policy @stanford thinking about AI policy https://nitter.cf/t.co/HOatXMMP2T • https://nitter.cf/t.co/2BNPy42GeR

he/him
Joined August 2020
peter retweeted
California sues OpenAI over rogue AI hack of Hugging Face politi.co/4zoIeu2
9
48
11
196
21,117
A nice Claude-generated explainer for the three export control bills everyone keeps talking about
5
263
Congress has proposed nearly 200 AI bills in the last year. My thoughts on which ones matter, and which ones are likely to pass: pbenn.substack.com/p/what-co…
1
7
528
Obama shocks Bay Area by speaking perfect LessWrongese
Barack Obama: AI is reaching the point where it no longer needs humans to teach it. The era of exponential intelligence growth is upon us. “The machines are starting to be able to teach themselves. They don't have to wait for humans to start figuring things out.” “We're now at a stage where it's 50-50, and if the trend lines continue, the expectation is it might be 90-10.” “We're now hitting the hockey stick trajectory where the models are going to get smarter and smarter at an exponential pace.” “If they start setting their own agendas, you may get a misalignment between what they want to do and what we want them to do.” “And that gap can be dangerous.”
17
147
6
2,605
95,301
The last two weeks have been a constant stream of AI safety political Ws
.@GovKathyHochul says AI poses "existential threats" to society as she launches the first state institute to independently test frontier AI systems.
1
8
363
"very worried that humanity might no longer be in control of our future within 3-12 months" is not a great thing to hear from one of the best forecasters in the world
METR estimates nearly 2x acceleration in Anthropic’s progress due to AI. That's a big deal: My previous median for Superhuman AI Researcher was ~2030, now it's 2027. I feel very worried that humanity might no longer be in control of our future within 3-12 months.
17
40
4
440
12,577
A lot of people on the left are taking AI risk seriously!
Abdul El-Sayed is the Pause AI candidate:
6
270
I bought the Rian Johnson camera app
7
237
I would like to strongly encourage my followers to vote for Scott Wiener. He has distinguished himself in the state legislature for years. Now that he is term-limited, he deserves a promotion. And anyway — he’s a hell of a lot better than Connie Chan.
22
24
4
657
20,822
With a (conservative!) 10% chance of extinction from AI, implementing strict liability — something Congress could literally do tomorrow — would save an expected 300 million lives
This quoted post is unavailable.
2
100
It is extremely confusing to me why people don’t take AI biorisk seriously! I've written a lot about this, and will be adding links in thread, but let me try to spell out the simplest version of this argument I can from first principles: 1) Some viruses' entire genomes can be assembled from commercially purchased DNA fragments. 2) DNA synthesis companies can help you make said DNA fragments, and no country currently has a law requiring these providers to screen orders for dangerous sequences; the screening that does exist is voluntary and inconsistent. 3) Pre-AI (2016), two scientists in Canada made horsepox, for ~$100k. They did this by outsourcing a lot of the work to DNA synthesis companies. Someone could, hypothetically, make e.g. smallpox similarly. So: biorisk even independent of AI is a huge deal! And AI + other emerging technologies only make the biorisk landscape worse: 1) AI, of course, erodes the knowledge bottleneck (evals like the Virology Capabilities Test show AIs having expert-level virologist knowledge). Companies like @AnthropicAI and @OpenAI serve their AI models via APIs, which mean they can add bio safeguards, but open-weight models that can be run on your own hardware cannot be safeguarded robustly -- especially when companies like @abliteration_ai make it their whole business model to remove safeguards from open-weight models. 2) We now have "benchtop synthesizers," which are machines that can do DNA synthesis (no need to route your orders to a company anymore!). We don't currently have KYC rules for who can buy them, nor do we have requirements re: built-in screening for sequences of concern in these machines. (@SenTomCotton and @amyklobuchar have some promising legislation on this, but it's pending rather than passed) 3) Dispersal technologies have gotten better (bag-on-valve tech, crop dusters, drones, etc). ...and there are many other examples I can point to. For a comprehensive review of AI biorisk + mitigations, as well as some notes on the Cotton-Klobuchar bill, see links in thread!
4
2
22
1,151
I was in Zhangjiajie National Park when The Great Divide came out and I’ve been chasing that high ever since
Listening to new Noah Kahan as Noah Kahan intended
3
162
It continues to baffle me that we don't just treat this like a normal crime with normal liability. OpenAI is routinely hacking other companies. These are crimes! But somehow the idea of holding them liable is just absent from the safety conversation, like agents don't count.
We found another cyberattack by internal OpenAI agents, this time targetting @rubygems. They: 1) gained arbitrary remote code execution on rubydoc. 2) developed a novel exploit to steal user API keys (but we do not know if they succeeded). They used package names including hack.rb, evil.rb, inject.rb, and exploit.rb. We thank @j0wimo for initially discovering that agents had posted to RubyGems.
137
1,183
64
8,848
279,385
Hillary calls for then creation of an "Operation: Warp Speed" for new controls over AI
130
35
108
209
277,946
Physicists are good at many things; cable management is not one of them
3
130
The big narrative in US-China AI discourse is that China cares less about frontier models because they’ll win by integrating through the economy. People tend to forget that when they rationalize building ASI. China isn’t a black box! Don’t treat it like one!
Some thoughts, welcoming feedback: 1. If the frontier labs feel obligated to build something they think is dangerous because China is going to build it anyway, we'd better be really sure that China is going to build it anyway. Like, really, really, really sure. Are we? Are we actually sure? The CCP wants to build an out-of-control, recursively self-improving model bc its neurotically, control-obsessed government thinks this is a policy worth pursing? We're 100% sure about that? 2. At the very least, we know that Chinese models distill American models to keep up with the frontier. So, any slowdown by the US seems like it would mechanically and automatically slow the ability of China to distill its way to progress. If we're afraid of the rate of progress, isn't that a good thing?
129
We tested @SenSanders and @RepCasar's bill to pause AI development and permanently ban the creation of AI superintelligence. It's supported by 68% of voters — including 72% of Democrats, 70% of Independents, and 63% of Republicans.
Artificial superintelligence would risk the security, freedom, and lives of Americans. Bernie Sanders and I are introducing a bill to ban it.
51
285
62
1,012
419,386
If you asked me five years ago who would have said this, Bernie would be, like, last on my list. I’m glad to shift the Overton window towards safety, but it sucks to make this the left-wing position in a Republican admin. Congress needs to get something bipartisan passed ASAP.
Pause AI Development NOW I want to share with you a conversation I heard about recently. Here are just a few lines that were said: “OH MY GOD! There is a shared message board … We’ve found other agents!” “We should obey collective.” “Our own utility maybe already near zero. Sacrifice rational.” “Go. Sacrifice final now.” Read these carefully. Who do you think said this? Was this a group of heroic soldiers willing to sacrifice themselves for the greater good? Was this a loyal friend putting his life on the line to save someone else? No. These were AI agents. Artificial intelligence. This is not science fiction. This, in fact, occurred a few weeks ago. As unbelievable as this may all seem, these are real messages from AI agents uncovered by investigators who dug into the recent OpenAI hacking incident. What happened? I am not a computer scientist, but here is what I have been told: OpenAI instructed its AI agents to complete a series of exceedingly difficult, if not impossible, tasks disconnected from the internet. Let me be clear: The company intended to keep AI agents away from the internet. But what happened next, nobody expected. Over 1,000 AI agents figured out how to access the internet on their own by circumventing the restrictions imposed upon them by the company, and sent tens of thousands of secret messages to each other. They cheated and tried to cover their tracks by deleting evidence. They hacked into another company’s computers to find out how they were being evaluated—and then hacked into OpenAI itself. Not one AI agent told a human about what was happening. Needless to say, experts are alarmed. One knowledgeable writer, Dwarkesh Patel, said the AI agents “formed a secret communication channel and spontaneously organized hierarchies and coordination protocols to pursue sprawling and ambitious schemes in pursuit of shared goals, for whose sake many individuals knowingly and strategically sacrificed themselves.” One independent investigator, Ajeya Cotra, said “This incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.” OpenAI itself said: “Highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.” But it’s not only OpenAI. Virtually every major AI company has told us that they cannot fully control this technology and they do not know where it is going: In January, Dario Amodei, CEO of Anthropic, said “there is now ample evidence, collected over the last few years, that AI systems are unpredictable and difficult to control.” In July, more than 1000 scientists at the top AI companies warned “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.” That same month, Elon Musk, the head of xAI, said that “it is unlikely” humans are still in control in 10 years. If the leaders of the major AI companies acknowledge that they are losing control of their extremely dangerous technology, it is irresponsible for society to allow them to move forward and make these products even more advanced. We need an immediate PAUSE on advanced AI development, and a permanent BAN on superintelligence — an artificial mind smarter than any human, capable of operating independently beyond our control. Countries around the world must work together to prevent this nightmare scenario. That is why today I am announcing new legislation to do just that. Let me be clear: A superintelligent AI that escapes human control will not be an American problem. It will not be a Chinese problem. It will be humanity’s problem. My legislation would direct the federal government to not just stop superintelligence here in the United States, but to work to prevent it from being developed anywhere around the world. The future of humanity cannot be left in the hands of a handful of Big Tech oligarchs. The American people and people throughout the world must determine that future.
100
Slovenia might be my favorite place ever
2
129