Mathias Gehrig (@MathiasGehrig) header
Mathias Gehrig (@MathiasGehrig) profile picture
Zurich, Switzerland
Joined May 2019
Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)Photo by Mathias Gehrig (@MathiasGehrig)
Mathias Gehrig retweeted
absolute fanfic. AI has real, immediate, tangible dangers that we need to focus on, for instance the impact on the job market and making sure random script kiddie can't use it for cyber offense. But this "oh once it's superintelligent it will take over and ensave us, maybe kill us all!" is pure sci-fi fanfic and distracts from the real (and difficult) issues, which linear probing won't solve. It's worth repeating the LLMs don't just "decide" to do something and then just "do it" in a void. They generate tokens based on our prompts, only as long as we call `model.forward()`, and on giant clusters with terabytes of weights, that's impossible to "quickly self-replicate". We also need to regularly remind that the OAI/HF incident was specifically a _guardrail-free_ cyber eval where the model is _instructed_ to hack, to find out how good it is at it. Some recent personal event made me realize there needs to be more push-back on this doomsday nonsense, and we need to re-focus the discussion on real dangers/issues. I won't argue much in the replies though, not interested in peer-reviewing Terminator fanfic.
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years). I’ll say a bit more about each of those claims. It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim… I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see nitter.cf/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo… If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…) Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.) I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon. Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything. I don’t know what you should do. I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon. (It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.) I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
115
187
40
1,700
134,985
Good. Now apply the same logic to Russia.
An absolute master class from US Secretary of State @SecRubio on the Islamic Republic Regime. Listen to every word!
37
Mathias Gehrig retweeted
Attention all companies: I never want to use your AI assistant on your website. I want to use *my* AI assistant on your website
234
479
134
8,277
253,075
Mathias Gehrig retweeted
AI may well be able to write massive software projects “from scratch”, but that almost certainly isn’t the fastest / highest performance / highest security / most token efficient way to do things. Good building blocks matter. In addition to RL’ing the agents on final task loss, we should try to get them to regularize the building blocks as well. What would the standard library look like that makes the agents most effective? It is likely at the limit that it turns out to be some alien organization of code, but given the existing human-bias in training, I bet that in the near term it would be lovely code that old programmers would appreciate. I suspect the same could be said for mathematics, but that is out of my lane.
365
454
81
5,689
745,814
Mathias Gehrig retweeted
Carl Sagan’s prediction about America, made 31 years ago.
92
1,682
144
6,087
417,331
Replying to @vtchakarova
And every european nation thinks they are the one that will cleverly play the large blocks and their stupid local neighbors. Basically either our politicians get their shit together and try to become one large block in their action or we are the next latin america (bunch of countries, some richish most poor, all under permanent leverage by US+CH)
4
2
56
2,952
Blessed by Claude. But for how long? 😅
76
Mathias Gehrig retweeted
Europe’s founders need a European standard to compete globally from day one. Europe’s leaders committed to delivering one. But their teams are negotiating away the very things that make it one. There is still time to change course if Europe's Heads of State step in now.
33
178
59
850
115,903
At this point the whole EU Inc negotiations are so lost in the details that everyone forgot the main goal: Creating a standard. Basically, allow founders to raise quickly globally and use stock options to hire competitively. Let’s say it quite clearly: 1) If EU Inc isn’t providing a standard it’s useless. We already got 30+ legal entities in Europe. Experienced founders use Delaware Inc already because of this. 2) If EU Inc doesn’t fix stock options Delaware Inc is still the better option. I understand the there are complex details and people are worried about loopholes. But you can’t lose the actual goal of it because of it. An EU Inc that doesn't provide a standard is pointless. Nobody will care about intricacies of compromises reached in political committees if the market can't use the outcome. If you are in contact with your political leaders (prime ministers, presidents, et al) please reach out to them. Or to founders/investors in your network who might have that access. 🔥🇪🇺✌️ nitter.cf/euinc_petition/status/…
Europe’s founders need a European standard to compete globally from day one. Europe’s leaders committed to delivering one. But their teams are negotiating away the very things that make it one. There is still time to change course if Europe's Heads of State step in now.
47
181
27
939
68,355
Mathias Gehrig retweeted
❗️Last ​​night, the operation of Starlink terminals in Ukraine was restricted. Connectivity began to drop when moving at speeds exceeding 160 km/h. As a result, a number of critical missions were disrupted. SpaceX likely took these measures at the request of the Trump administration. Also last night, when Kyiv was attacked by ballistic missiles, Ukraine did not receive intelligence regarding their launch— a failure that could have led to higher casualties. Ukraine did not attack Russian oil refineries, yet Trump has already begun imposing sanctions on my country. Meanwhile, sanctions against Russia have been eased.
1,124
6,231
758
23,153
1,332,376
I suspect many are rightfully pessimistic about the state of academia, not basic research. Starting a PhD now is betting on there being change to the very old and slow academic system. Risky bet if you ask me.
One of the worst forms of brainrot AI has cultivated is pessimism about basic research, ie the idea that important work can only happen inside a frontier/neo lab and only with 10k+ GPUs, so the rest should not even bother. What a bleak way to think about science. And it's false.
2
163
Catering to the wrong market? Backprop is painful for recurrent methods and requires workarounds for non-differential ops. Comparing on standard architectures is good but will likely not convince anyone
We've figured out how to *pretrain transformers* with zeroth-order optimization and no backprop. Many of the core assumptions in optimization research are completely wrong. (paper out soon)
91
This should be standard imo
At our new startup, there are ZERO non-competes and we give each employee the choice of either 1 or 3 months notice periods, upon their request. Competition is for winners. BTW, today is the last day to join the founding team.
1
84
Asymmetric fixed wing performing robotic take-off and landing live in front of hundreds of people. Kudos to the team behind that. Truly badass.
𝐀𝐬𝐜𝐞𝐧𝐝 𝟐𝟎𝟐𝟔: 𝐓𝐡𝐞 𝐅𝐥𝐲𝐢𝐧𝐠 𝐑𝐨𝐛𝐨𝐭 𝐈𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞 𝐄𝐫𝐚 𝐢𝐬 𝐡𝐞𝐫𝐞, 𝐚𝐧𝐝 𝐰𝐞’𝐫𝐞 𝐣𝐮𝐬𝐭 𝐠𝐞𝐭𝐭𝐢𝐧𝐠 𝐬𝐭𝐚𝐫𝐭𝐞𝐝 It is an honor and a joy to work with the critical industries our civilization runs on to build a new era of Flying Robot Infrastructure that helps keep us all safer and makes our world more efficient. Together with our customers we have made immense progress towards this future over the last year. You can see it in the metrics - 2M flights/year and exponentially growing - and you can see it in the thousands of individual stories where a drone helped save a life, protect an asset, or just save someone from tedious manual work. But we’re still just getting started. On stage at Skydio Ascend, we announced two new categories of products that will shape the next decade: • The software Commands: DFR Command, Asset Command, and Security Command. We’re now tapping the true potential of software-defined, autonomous, dock-based drones with industry-specific applications for Public Safety, Energy Infrastructure, and Physical Site Security. Each Command brings together specialized autonomy, customized interfaces and workflows, and deep integrations with complementary software and systems to orchestrate fleets of flying robots and deliver end-to-end solutions for each industry. This makes drones easier to adopt and integrate than ever before, and orders of magnitude more effective once you’re operating them. • F10 Lightrunner: This is the most advanced fixed-wing drone that has ever existed. It’s launched and caught with a robotic arm (RTOL); its gimbal is on the wingtip which means it never gets obstructed by the wing (SideEye); and it has the evolution of @SkydioHQ Autonomy for fixed-wing aircraft (Foresight). All of these together represent one of the great feats of engineering in the history of aircraft, but we didn’t do it just for fun. F10 will bring the benefits of Drone as First Responder and fully automated drone inspection of critical infrastructure to all the landmass in the free world with 100mph speed and up to 30-mile coverage radius. We build the coolest technology in the world, with the best team in the world, to serve the most important customers in the world. LFG.
3
164
Mathias Gehrig retweeted
In Dune, unlike in most scifi, the problem with AI wasn't machines rebelling against humans but rather the rise of an all-powerful class of AI technocrats. "Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them." This could be the real existential risk of AI.
23
42
7
470
25,717
So many coincidences recently
Every Twitter take this weekend: “The best and most appropriate approach to AI is - completely coincidentally - the one that will make me incredibly rich. How lucky you are that my incentives are perfectly aligned to your wellbeing. God bless capitalism. God bless America”
1
168
Mathias Gehrig retweeted
Why you can’t do RSI as fast as you like -
59
123
49
1,018
475,737
Bullish on fast fine-tuned policies as subagents of more general models. Bearish on LLMs writing tools to control robots.
Recently, there have been a lot of impressive demos of AI agents, like Claude, controlling robots. I wrote a short blog post with my thoughts on the advent of these "robot-use agents." web.mit.edu/phillipi/www/wri… I think it's an important change in the trajectory of robotics!
150
Mathias Gehrig retweeted
ngl, doing software development in the age of (good) LLMs feels far more like caring for a bonsai tree than doing engineering
43
127
17
3,159
74,022
LLMs so safe now they can't help making code safe. Make it make sense.
129