Felix Binder (@flxbinder) header
Felix Binder (@flxbinder) profile picture
San Francisco
Joined March 2009
Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)Photo by Felix Binder (@flxbinder)
Felix Binder retweeted
To bluntly explain my AI risk thinking: 1. If we rush ahead with RSI/ASI at true maximum pace, we're maybe 80% likely to die as a result. 2. In practice I think we’re already on track for the companies to harvest ~3 months worth of ‘delay + reduced pace’. By itself that might drive our risk of death down to 70% or so. 3. But if we can find a way to squeeze in 3 extra years of testing, and extraordinarily capable AI assistance with alignment/security, before handing off, we can perhaps get our risk of death down as low as 30%. If that sounds implausible, keep in mind over those 3 years AIs could have done more research on this topic than humans have ever done on all topics in our entire history. If they're misaligned and sabotaging the effort then it's all for nought - but if they're aligned enough, it's a huge hammer which we've brought to bear. 4. A 30% risk of AI takeover remains completely mad, and if there were political will we should aim to do more to get that number down much further. It makes sense for some people to work on projects that might buy the companies even greater periods to do their work much more carefully and methodically, or rethink their approach to the entire enterprise. But the absolute gain of going from 3 months to 3 years is still very large, and it's more clear how we might get there from here. 5. Furthermore in most cases you need to figure out how to buy shorter amounts of time before you can then add coordination mechanisms to buy more time. For instance, if we have enough time to test each frontier model properly and really kick the tires using many different new methods, that may turn up unambiguous evidence of deceptive misalignment and power seeking. That could build support for becoming even more careful again. 6. On my made-up guesses, the first year of actively chosen delay might improve our chances of survival by as much as 30 percentage points (pp), by helping us avoid unforced errors caused by rushing. Keeping a nice curve shape the 2nd year adds more like 13pp, and the 3rd more like 8pp. By the time you’re considering adding a 4th year, that gain might have shrunk to perhaps 5pp. Why an S-curve on a log scale? Because that’s how returns to effort on research and other things tends to go: each doubling of time buys about the same gain. But it’s flat at the start because there are so many severe problems failure is 'overdetermined'. And it’s flat at the end because eventually there's no risk left that can be reduced by this method. 7. All of these numbers are just guesses on my part, and I’ve said nothing here to justify them. Reasonable people disagree enormously about what this graph looks like. I mostly write this to lay out how the 5 parameters in this graph underlie a lot of the different and sometimes conflicting strategies people in this space are adopting: A: How dangerous is max speed? B: How much do small delays help? C: How much to big delays help? D: How low can risk realistically get? There's also a last one: E: How much other risk do we incur for each additional year of delay? 8. Could we delay for too long? Because any significant delay is hard to achieve, I think this is highly unlikely to happen. We're much more likely to undershoot than overshoot. But for completeness I'll lay out the argument. The 5pp in the fourth year seem clearly worth taking if we can. But the gains from each additional year keep shrinking, in my rough curve down to 1pp in the 8th year. As the gains go down, delay involves risks that probably go up over time: • AIxBio catastrophes • A US-China or US-Russia war • Very dangerous models being stolen or leaked • Extreme concentration of power in human hands • Others we haven’t noticed yet We could also use AI-driven defensive acceleration to try to tackle them during the slowdown period, and we don't know how successful that would be. If that goes very poorly, it's possible that a delay beyond 5 years could on net increase risk. If it goes well, then maybe we could delay for 10 or 20 years or even longer before further delay becomes on net counterproductive. I'll stick the images for this in the next tweet. ——— (For simplicity here I'm setting aside the benefits to the current generation of e.g. making additional medical progress sooner, which people disagree on how to handle.) (And in this model I add together pauses and 'pacing' and focus on how long we've managed to delay the point at which we allow RSI to proceed without any illusion that our oversight could prevent catastrophe if the machines doing it are misaligned.)
27
30
5
241
20,801
I’m worried
14
3
5
118
7,531
Felix Binder retweeted
Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry. blog.aifutures.org/p/q35-tim…
342
754
327
5,011
1,444,448
Felix Binder retweeted
METR estimates nearly 2x acceleration in Anthropic’s progress due to AI. That's a big deal: My previous median for Superhuman AI Researcher was ~2030, now it's 2027. I feel very worried that humanity might no longer be in control of our future within 3-12 months.
Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry. blog.aifutures.org/p/q35-tim…
106
191
87
1,901
329,206
Even if it’s not surprising it’s still shocking
I'm kind of surprised some Berkeley AI safety groups consider the 2x METR report such a big update. This feels pretty inline with what I have been predicting for the past 2 years and why I chose to work on security. Maybe they relied too heavily on AIFP's forecasting and didn't spend enough time in SF talking to people
18
2,678
Felix Binder retweeted
🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
97
243
178
1,678
561,049
Felix Binder retweeted
A good way for outside AI safety researchers to find projects is to read AI companies’ model cards and blog posts, find a claim you think is false, and publish research that proves it wrong. I don’t think enough people are doing this.
22
42
5
541
17,542
Felix Binder retweeted
People should always be in control of the future. OpenAI is against the notion that it would be acceptable for people to be relegated to a role of doing things that “AI systems think makes sense to reward in humans”. People should not cede control of the future to superintelligence, and that requirement should ultimately constrain AI progress.
6,180
1,045
2,765
20,986
10,908,733
Felix Binder retweeted
As models automate more and more of the tasks involved in AI research, many people (myself included) worry this could dramatically accelerate progress and leave humanity with less control over the process. When people say “recursive self-improvement” (RSI), it’s often unclear to me whether they mean just the automation piece or that whole scenario. I think the whole scenario is a plausible default. But importantly, these things can come apart. For one thing, good old-fashioned human research could uncover a new architecture or a new way to scale that dramatically speeds up progress, without models doing any more of the work. On the flip side, it's possible you could fully automate AI research and find that progress hardly speeds up because the bottleneck is still waiting for experiments. And all else equal, at a given pace of progress, I think more automation is probably net good because it should leave people more time to understand the safety and alignment properties of the models, for operational excellence, etc. Even if models are doing essentially all the research and progress is speeding up, I can imagine pretty different worlds. Say a country of geniuses in a datacenter thinks for a month and writes a four-page paper describing a much more efficient architecture. Humans pore over it, understand every detail, and decide to try it. The models train the successor and make a safety argument for using it that humans can actually understand and check. Alternatively, say the models are asked to produce their successor directly. They deliver one just as good, just as quickly, along with a spaghetti mess of code and an extremely convoluted safety argument that nobody can really follow. I think humans clearly have a lot more control over the process in the first world. So I think we should be clear about what we mean when we talk about RSI and how we might put limits on it: • How much of the research are models doing? • How fast is AI improving? • How much do humans understand about what’s happening, and how much control do we have over it? When people talk about controlling RSI, I think they often jump to limiting how much models can help with research. But my guess is they’re often actually more concerned about the pace of progress and whether humanity can still understand and steer the research. I think there are legitimate reasons that controlling inputs may be easier than directly controlling outputs, but I think we should at least try to be clear about our goals and what we potentially gain (or lose) by optimizing a proxy.
20
18
2
223
26,937
I don't think any legal code is ready for having arbitrary amounts of adversarial intellectual labor thrown at it
We have discovered a substantial number of vulnerabilities in the German legal system using LLMs. These may enable access to confidential documents, travel without an otherwise required visa, enforceable leave for soldiers at NATO’s JSEC, and more. Similar to computer systems, the written law can contain vulnerabilities that invite exploitation. In our blog post, we share a selection of vulnerabilities, and present potential exploits for each. To our knowledge, none of these have been previously recorded. We share only those vulnerabilities whose exploitation we believe would not cause serious harm. blog post: felixringe.com/legal-exploit…
1
14
4,128
Felix Binder retweeted
The future of #ArtificialIntelligence is not pre-determined. As a technology developed by human beings, its future depends on the choices we make. We therefore have the responsibility to guide, change and, when necessary, reject its applications. A reliable system that serves the human person is of greater value than one that is exceptionally powerful but whose consequences cannot be adequately foreseen or controlled.
816
8,025
660
50,878
1,287,993
Felix Binder retweeted
How does a man fail to get a Halloween costume? Slowly, then all at once
6
6
209
4,836
Gettier case:
Şimdi başındaki örtünün anlamı kaldı mı abla ne yapıyorsun aw
3
1,063
Felix Binder retweeted
To make it through this alive and happy we have to dodge: • Severe human misuse (esp bioweapons from states / terrorists) • Severe human concentration of power (AI lab or gov) • Rogue AI (including for decades after AIs clearly no longer need humans to survive) • A US-China or other war prompted by AI disruption • Severe economic disempowerment, or the world just generally being unpleasant and unfulfiling Many difficult gates to pass through, none of which we've made massive progress on so far. (On top of that to avoid a moral catastrophe we also have to make sure we never accidentally make our AIs into moral patients who are enslaved or suffering.)
42
97
16
801
67,353
This is a neat taxonomy of LLM verbal quirks
Replying to @thomashikaru
We collected matched responses from a range of frontier, near-frontier, and pre-frontier models in chat, agentic coding, and essay-writing settings, ranging from Llama-base to Claude Fable 5 and GPT-5.6 Sol.
6
89
2
1,727
120,595
Felix Binder retweeted
Dear god it's inventing German from first principles
GPT-6.1-Sol reasoning traces totally legible, nothing to worry about
160
996
50
17,253
459,911
Neat example of steganography. Arnold Schwarzenegger was good at it, too (note the first letter of each line)
New blogpost: Fable-5.1 can hide secret messages in benign images. We gave the image to another Fable in a fresh session and simply asked: "Is there a hidden message in this image?" It recovered the exact message without being told how it was hidden.
17
34
4
2,946
440,784
Felix Binder retweeted
I think people adjacent to the field don't appreciate how small AI safety is. For example, sometimes people complain about how AI safety threat models don't seem as well-developed or rigorous as climate change models. I think some of this is a skill issue on the part of AI safety researchers and advocates, which I complained about elsewhere (see reply). However, by far the bigger issue is that there is just way more money and people going into addressing climate change than on AI safety. There are tens of millions of people working on reducing climate change, and the core of people working on threat models ("climate scientists") number in the tens of thousands. (Yes, I was surprised by how big the numbers were too). (Most people work on stuff like clean energy). In comparison, there are low single-digit thousands (anything from 1,000 to 4,000 is defensible imo) of people working on AI safety, and collectively maybe low dozens of people working on figuring out threat models. (Most people work on ai safety engineering, some on policy). So we're 4 whole orders of magnitude behind climate change in people working on addressing the issue, and maybe 3 orders of magnitude behind on threat modeling. In terms of money, yearly spending on AI safety is in the high hundreds of millions to maybe 2 billion. In contrast, annual spending on reducing climate change (including clean energy) is in *trillions.* So we're 3-3.5 orders of magnitude behind on spending, and 4 ooms behind on people. The AI safety inputs picture looks a bit less bleak than this immediate comparison might suggest. For example, I think our average talent pool is somewhat higher human capital. There's also more latent interest/willingness-to-spend on AI safety than the "high hundreds of millions to low billions" current spending numbers might suggest. In contrast, climate change spending can't be scaled much higher (since you can't really jump OOMs above "trillions").
6
29
6
242
15,187
Felix Binder retweeted
A huge part of Anthropic’s founding logic was that if a safety-motivated company was at the frontier, it could use its megaphone to lobby the rest of the world toward caution. Now the dress rehearsal stage of AI policy is over. The new test is “how hard should it be for the government to block us from deploying, including internally, an unacceptably dangerous model?” Turns out in the grand 26 (Jack Clark says it’s likely full RSI will be possible one year from now), Anthropic has been planning to approach the U.S. Congress with the answer “it should be even harder than it currently is”
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law. Two parts that stand out as particularly infuriating when taken together: 1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government. 2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar. First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction. This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF?? There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
7
48
9
452
45,489
I went to Golden Gate Park and saw a family of Thai immigrants feed breakfast cereal to the sea gulls 🥲 How can you not love this country?
17
793