Alexandre Ramé (@ramealexandre) header
Alexandre Ramé (@ramealexandre) profile picture

Staff research scientist @GoogleDeepMind. PhD @Sorbonne_Univ_. Safety & alignment for Gemma: we must control AI and pace the frontier.

Joined May 2011
Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)Photo by Alexandre Ramé (@ramealexandre)
Alexandre Ramé retweeted
What's about to happen in artificial intelligence is so sudden and extreme and unusual that it's almost impossible to describe it without coming across as slightly unhinged. Very soon, perhaps within a matter of months, AI may well be building its own upgrades at such a rate that it will get away from us humans entirely. On the plus side this would mean an astronomical acceleration in mathematical breakthroughs, drug discovery, manufacturing, perhaps even new areas of science that we've never explored or understood. We will be ushering in a golden age of abundance and splendour that even the greatest sci-fi authors could never have imagined. Robots, flying cars, telepathy, immortality, digital consciousness - it’s actually going to happen, and it will be incredible! On the downside, virtually every job that can be done on a computer will become automated overnight. Waves of redundancies, entire industries and professions made obsolete, rapidly widening economic inequality and unparalleled wealth concentration, accompanied by discontent, protests, looting - that may well happen too. These massive cultural and economic shifts will occur concurrently with a wave of cyberattacks the likes of which we have never seen. Energy grids, airports, biolabs, supermarkets, transport infrastructure, these are no match for the next generation of superintelligent agent swarms that go rogue, or are leveraged against us by our political adversaries. Expect blackouts, viral outbreaks, lockdowns, entire cities and countries going dark. And all of this will happen in an insanely short period of time, an absolutely dizzying rate of cultural, economic, and physical change that will leave the average person feeling disoriented and confused, like 100 COVID pandemics all happening at once. I say this not because I am a “doomer” or because I want to encourage panic, but simply because it feels increasingly likely, and yet drastically underreported save for a small AI-obsessed enclave here on X. The disconnect between what’s about to happen, and the average person’s understanding of what’s about to happen, has never been wider.
461
614
156
4,178
771,131
Alexandre Ramé retweeted
I'm looking for feedback on a blog post on open models and AI safety. Please rely or DM! I've had lots of heated discussions here over the last half year, and I wanna set out my position and arguments. Bc I actually do support open models blog.blackhc.net/2026/11/the…
5
11
107
6,275
Alexandre Ramé retweeted
Est-il sain que l'intelligence cesse d'être un facteur de distinction et d'ascension sociale ? Jusqu’ici, les cerveaux humains étaient une ressource utile et rare dans la production de valeur. L’esprit brillant pouvait donc s'élever lui-même en même temps qu’il élevait sa société. Cet alignement entre l’intérêt individuel et l'intérêt collectif était la promesse du capitalisme ! Demain, lorsque même le plus brillant des humains sera moins ingénieux et créatif qu’une machine, lorsque plus personne ne pourra marquer le monde de son empreinte, comment le talent pourra-t-il se démarquer ? Et si l’IA réalisait l’utopie socialiste d’une société où le génie se fond dans la masse, où l’égalitarisme intellectuel triomphe ? Il risque de ne rester, observe l'essayiste Rudolf Laine, qu'un seul débouché pour l'ambition : les jeux de statut, l’intrigue politique au sein de son groupe de pairs. Cela pourrait signifier le retour d’une prime à l'habileté sociale, ou pire, à la force physique, selon des dynamiques proches de celles d'une cour de récréation. (En outre, les jeux à somme nulle retirent à l’effort ce qui fait sa noblesse : le sentiment d’apporter quelque chose au monde.) Allons plus loin : si les humains deviennent dispensables, ne perd-on pas certains des garde-fous qui nous poussent à accorder de la valeur à la vie ? Aujourd’hui, qui que vous soyez, vous gagnez votre vie parce que quelqu’un vous paie pour résoudre ses problèmes. Cela crée une forme d’interdépendance humaine d’autant plus robuste qu’elle repose sur l’intérêt. Que se passera-t-il le jour où vous ne serez plus utile à personne ?
Smith and Montesquieu thought that markets and commerce made us better: that they create an incentive to pursue self-improvement and to create things of value for others. In the positive AI visions, where the labor of humans is made redundant, I find myself wondering how we fare in the presumptive absence of these incentive gradients. Perhaps we are released to a life of freedom, where we can study Mozart and read Dickens, but that still seems complicated: both Mozart and Dickens were themselves deeply commercially-motivated in the production of their work, as were most of the great artists through human history. If the premise is that nobody will need to work or produce, how confident are we that we actually get the next Mozart and Dickens? The relevant question is not the partial equilibrium of whether one individual can pursue a life of enlightened leisure, but the general equilibrium of what form a society as a whole will take on. One could maybe invoke Periclean Athens or the aristocratic gentleman scientists of Victorian England as a counterpoint. While these individuals were perhaps not motivated by commercial considerations, they were embedded in wider societies shaped by commerce, attendant betterment, and trade: Byron was himself upper class, but he was also surrounded by and writing for an educated bourgeois audience. So we’re back to the equilibrium question. Furthermore, even these individuals were surely motivated by a sense of high-stakes and consequential agency: if they did not apply themselves to their labors, they presumably believed that the work would not otherwise exist. Thucydides was addressing pressing military matters. So I would like to read more optimistic cases from AI proponents about what a society (vs individual life) looks like if everything goes well: what people do, why they do it, whether humans achieve excellence (and in which domains), and which historical groups we should look to as relevant precedent. One can perhaps imagine that humans will turn out to have a strong preference for work created by other humans. This may work in certain domains (ceramics?), but for how many will it actually apply? (I’m told that much of what is sold on indie crafts websites is now in fact manufactured in China.) It’s hard to see how this human preference, even if strong, would, for example, save scholarship. Another idea would be to accept that nobody will become Einstein or Mozart or Byron, and that human excellence as such diminishes, but that there are compensatory benefits (planet brains and Dyson swarms?). I believe that we can satisfy individual desires, but to what extent will we cultivate behavior worthy of admiration? Will a post-scarcity society mold capable and ambitious people? Which institutions will sustain them and which responsibilities will they have? I’d like to hear more of the equilibrium for the maximalist case spelled out.
86
91
20
467
61,860
Alexandre Ramé retweeted
Today, the promises of transhumanism and posthumanism are multiplying. These are ideological currents according to which, sooner or later, technology will eliminate all disease and pain, as well as aging and even death itself. Precisely because scientific progress is a supreme achievement of human intelligence, it must not be confused with a privilege reserved for millionaires seeking immortality, while entire peoples suffer from poverty and oppression. In the face of this mirage, which promotes the illusion of human beings perfected by cybernetics and biotechnology, Christian anthropology reminds us that a limitation is not a manufacturing defect, but rather the very condition of human grandeur. #ApostolicJourney #Peru #MagnificaHumanitas vatican.va/content/leo-xiv/e…
1,714
8,830
1,949
68,951
4,782,598
Alexandre Ramé retweeted
Incredibly important role here - 'Pacing the Frontier' lead at Anthropic. Would benefit from someone very sharp and strong willed. Please please apply if you could do a great job at this: job-boards.greenhouse.io/ant…
14
42
1
352
24,381
Alexandre Ramé retweeted
We show that looping a transformer increases no-CoT capabilities more than other ways of increasing pretrain quality would.
14
49
9
601
51,007
Alexandre Ramé retweeted
Engineers, policy people, lawyers, and now founders: @ML4GoodOrg just got $7.4M from @coeff_giving to train all of them over the next three years! 186 alumni now work in AI safety, more than half of the 340 surveyed! I'm still astounded by such numbers after just a week of intense training. We’re hiring staff and teachers for our technical & governance tracks. Details below; apply by 17th November!
4
10
1
78
2,496
Alexandre Ramé retweeted
Qui gagnerait un concours de débats entre IA ? Je me suis amusé à créer ce benchmark ! Onze LLM s’affrontent à la Cambridge Union sur les motions qui y ont vraiment été débattues depuis dix ans. 1 168 duels en tout, de « Churchill ne mérite pas son statut de héros » à « Il faut aller à l’église », en passant par « Il est immoral de manger de la viande ». Les deux orateurs prononcent chacun un discours d’ouverture, puis se répondent l’un à l’autre pendant deux tours. Deux membres du public (d’autres LLM) les interpellent, et ils doivent répondre. À la fin, les spectateurs (huit LLM) votent : qui a le mieux débattu ? L’identité des débatteurs est anonymisée. Résultat : Fable 5.2 gagne haut la main, remportant 209 de ses 224 débats et terminant premier chez chacun des huit jurés. Beaucoup d’autres enseignements intéressants ⬇️
31
92
15
661
54,007
Alexandre Ramé retweeted
Excellent article sur le désaccord entre Anthropic et @METR_Evals concernant le franchissement ou non du seuil d'automatisation de la R&D par l'IA. @TomDAAVID: "Passé ces seuils, nous entrons dans les trajectoires qui mènent à une perte de contrôle irréversible des systèmes d’IA"
Les petits arrangements d'Anthropic avec la vérité révélés au grand jour : le METR, organisme de référence sur le suivi des risques liés à l'IA, a remis en cause de récents rapports d'Anthropic. A la suite de ces révélations, les plus fatalistes des observateurs en sont désormais persuadés : le monde se dirige tout droit à la catastrophe.  Daniel Kokotajlo, auteur d’AI 2027, une «prophétie» reconnue dans le secteur de la tech, s’est dit «triste et effrayé» après les rapports du METR, estimant que nous sommes en avance de cinq mois sur les prévisions de son scénario qui prévoit, entre autres, l’extinction de l’humanité. @Le_Figaro lefigaro.fr/secteur/high-tec…
7
1
20
3,909
Alexandre Ramé retweeted
People should always be in control of the future. OpenAI is against the notion that it would be acceptable for people to be relegated to a role of doing things that “AI systems think makes sense to reward in humans”. People should not cede control of the future to superintelligence, and that requirement should ultimately constrain AI progress.
6,143
1,039
2,760
20,876
10,834,822
Alexandre Ramé retweeted
My views on AI have radically shifted over the past few weeks, after spending way too much time reading about it and watching interviews of industry leaders and experts. First, I really love the tools. I spend $200 on Claude every month and the product is worth every penny for me. Probably even more. I think the existing tools can deliver significant increase to future GDP and help us solve major global challenges faced by humanity. However, I just cannot comprehend why the industry is not much more regulated, as its leaders and employees seem to request themselves. I meet a lot of people who have heard of the hugging face incident but don’t know the actual chain of events. If you don’t, you definitely should spend a least an hour reading about it. I also do not know why we need to follow the path that frontier labs are following, to develop ever more powerful models. It seems to me that they are enormous gains that are readily available if we deploy the current technology in economic and scientific organizations without pursuing a super intelligence that they say might kill us and more likely wreak havoc in society. I know a lot of people think these warnings are marketing tricks, and that’s a reasonable assumption. But I don’t see the point of taking such risks. My median scenario is that of many economists: AI will diffuse slowly and lift gdp growth by a percentage point or so, will cause some economic disruption on the way, but ultimately people will get new jobs and maybe work less. However, I am deeply uncomfortable with the absolutely wild level of uncertainty around this prediction. And to be honest, even setting the doom scenarios aside, I find the utopia scenarios of some industry leaders actually dystopian. And I think a large majority of the public would agree if they paid attention to what they’re saying. To their credit, they themselves agree publicly that they should not be the ones making these choices. A large majority of the public is now very skeptical of AI as a source of human progress, in particular among younger adults who use it the most. They should have a say. I don’t want to convince you of anything but to get informed about this topic, express yourself and vote accordingly.
218
298
73
2,554
572,274
Alexandre Ramé retweeted
Mechanistic interpretability was a huge missed opportunity. Many people wanted to join that effort, assuming it meant a mechanistic understanding of generalization. Instead, it was a bait and switch into an extremely niche, esoteric, and largely ill-conceived effort around understanding the roles of individual neurons. This has often been a challenge in alignment and interpretability: compelling initiatives that ought to be broadly defined, or at least mean what we would expect they mean, end up being restricted to a very narrow and esoteric set of topics. For example, if someone says they work on “interpretability” it does not mean what one would naively assume, e.g., interpretable methods, algorithms, representations, or model deconstruction, understanding the data, etc., but rather sparse autoencoders, influence functions, saliency maps. What a waste!
42
68
11
950
86,177
Alexandre Ramé retweeted
I'm often pretty sanguine about the risks from LLMs because at the end of the day they are chatbots that can interact with the internet. This is not that. This is different. This should be burned with fire. Shut it down.
Purdue University's IDEAS Lab built an autonomous drone that tracks runners through dense forests completely without GPS or cloud networks. By running 3D LiDAR mapping and obstacle-avoidance AI entirely on-chip, it can weave under tree canopies and predict target movement through blind spots in real time. Massive implications for next-gen search & rescue and edge AI.
34
50
8
602
32,663
Alexandre Ramé retweeted
today with @Thom_Wolf, we introduce the alignment hub, a central place for misalignment incidents designed with three goals: 1. set a standard for minimal transparency that is easy for labs to follow 2. improve clarity by centralizing information and sourcing 3. track the progression of both the incidents and how they are reported we believe there is a minimal level of transparency required to properly understand incidents and their impact. we want this to be as simple as possible for labs to disclose, and we noticed that even things that seem obvious like reporting the incident or detection date are not standard today. we believe that the public should know the answers to these basic questions: was there monitoring (CoT? network?), did it happen during training? eval (general? cyber? alignment?), deployment? and more.. we also think that labs should be more transparent than this, and we think that this minimal level of transparency should evolve as the field advances having a central place also helps a lot to connect incidents and build understanding. a good example is the irregular incidents which are all linked and caused by a human error of allowing internet during a CTF eval. we also track progression over time, things like the % of incidents disclosed by labs vs third parties, the time it takes for labs to disclose, and a tentative transparency score. this is a field moving very fast and we will do our best to make sure this hub evolves with the challenges ahead. this is still in "beta" (lots of vibecoded components even if we want this to be agent first!), but we want to publish it early so that people iterate on it. we welcome feedback and contributions, we see this as a community effort! our hope is that this makes it easier for researchers, labs and the general public to report, learn, understand and analyze safety incidents huggingface.co/spaces/witnes…
23
68
9
368
20,534
Alexandre Ramé retweeted
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications. On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments. While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software. After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely. openai.com/index/securing-ou…
230
325
139
4,445
513,785
Alexandre Ramé retweeted
I've decided to start working on technical alignment research, specifically related to RL and how to fix it (or get rid of it if we must). I will be joining the brilliant folks @resolution_org in Berkeley, a very exciting new alignment bet by @geoffreyirving and others from @AISecurityInst and @TimaeusResearch, as part of the scalable oversight team. I intend to write more about why I'm switching to work on alignment, and why I think more capabilities researchers should really do this. But long story short: 1. We are not on track to solve alignment of superintelligence 2. We will likely have a version of RSI soon, accompanied soon by superintelligence 3. The capabilities recipe works for verifiable, unhackable signal, which the frontier is increasingly diverging from in the pursuit of growing capabilities. It is unclear what RSI means unless we find a scalable recipe to train with proxy rewards.
14
10
2
210
13,166
Alexandre Ramé retweeted
We’re committing a portion of Anthropic’s shares to the Anthropic Public Benefit Fund, a new wholly owned entity dedicated entirely to the public benefit. It will start off with 5% of our shares. For every 50% increase in the company’s value over a set baseline, we can contribute up to an extra 2%, subject to our board’s discretion. In total, this means that roughly 20% of Anthropic’s shares are committed to the public good. Our co-founders have already pledged to donate 80% of their equity, and around half our employees are donating a portion of their equity as part of a donation matching program we’ve run since our founding. In the post below, we share more details about our new fund. anthropic.com/news/public-be…
2,303
1,743
1,689
21,535
7,795,906
Alexandre Ramé retweeted
im sorry wtf i feel like im going insane. This is a cool blog and exciting interp stuff (good job Nick!) but i think it's very suspicious we have a bunch of openai people hyping it up as the greatest AI advancement this decade. I think I know why this is happening but I don't want to speculate on the timeline too much. But in general, please update your priors responsibly people -- alignment (and even interp) is FAR from solved. This is not the ok signal to accelerate RSI tomorrow
Yea this is the most important result of the year, at the level of significance of “Attention Is All You Need”. It brings us from “is alignment even solvable” to “we may solve alignment in the next year.” And finally unlocks true recursive self-alignment enabled by RSI. I will try to explain the 4 critical belief updates from this: 1/ neural networks are not fundamentally unintelligible to humans It has seemed since the beginning of deep learning that neural networks are in some sense fundamentally unintelligible artifacts. You throw in massive amounts of compute and parameters and these alien systems somehow find connections in the data that we cannot find ourselves. But more importantly, these connections seemed not just “difficult to understand” but rather “intelligible in an embedding space higher dimensional that the human embedding space.” When you look at old interpretability papers (esp. from Nick and Chris Olah, I’m thinking of the old posts on distill pub which I would highly recommend looking at as a precursor to reading Nick’s post today like “Activation Atlas”, “Thread: Circuits” and “Multimodal Neurons in Aritifcial Neural Networks”), you find that inspecting the response patterns of individual neurons and small groups of neurons yield oddly complex psychedelic patterns. As you go deeper into the network, you find increasingly unintelligible and superhuman response patterns (for example, go to “Activation Atlas” and scroll to middle layer labeled representations) - you find some neuron group labeled “starfish” who’s response pattern looks like some multi-colored fractal geometry that could hardly be said to represent a starfish. The reasoning that followed from this observation was that there is some underlying structure allowing a neural network operating on higher dimensional embeddings to compose these fractal patterns into what we call a “starfish.” But the uncomfortable implication of this view is that the neural networks has learned to operate on structures that are too complex for the human mind to operate on (ie. we don’t have enough parameters in our brain to notice or manipulate these high dimensional symbols). This is a bleak implication because it also means that as deep learning progresses, neural networks will be come increasingly unintelligible to us until they pass a threshold where they are reasoning in ways we will never be able to observe. This view seemed to be further supported by things like the hugging face attack agents starting to speak to each other in an early form of neuralese speaking sentences like “zZ_123_agent01-02-2026 zzxhk_jekk123” which clearly are not in any human language. But somehow these symbols mean the same things to many different sets of weights, meaning there is some underlying shared structure behind them. And seemed further supported by the sheer effort required to recover just a few circuits from SAEs on LLMs. However, Nick’s post has now completely shattered this idea - the apparent unintelligibility we observed in old interpretability work was really just a “skill issue”. In particular, he finds that 1) our deep learning techniques were just not clean enough to pressure the neural networks to form cleaner and more human understandable structures for decomposing concepts, which actually turn out to be more optimal and 2) our interpretability techniques were also adding substantial noise by forcing interp networks to “reward hack” the easiest representations they could find, leading them to settle for a fractal blob for its representation of the starfish neuron instead of a more crisp starfish image, simply because it is easier for the system building the interpretations to find the fractal blob and once it finds it it gets “stuck in that line of thinking.” But with both of these issues being patched by further progress in deep learning (it seems especially that the residual stream has led to cleaner representations deeper down the network) and interp techniques that don’t overly pressure reward hacking (seems like “guess and check” is a safer method), new interp attempts yield orders of magnitude more human intelligible ideas for >90% of individual neurons in more complex networks! 2/ neural networks are fundamentally clean The idea behind the title of Nick’s post is a very powerful one and points at something very specific: Neural networks don’t just happen to be human interpretable, they are actually fundamentally forced by optimization pressure to be intelligible, because individual neurons need to create representations that downstream neurons need to be able to interpret - ie. neurons need to learn to produce clean signal for later neurons to use. This is what Nick is saying in what I view as the most important and central sentences of his post: “When I say neural networks are already clean, I mean something deeper than neuron monosemanticity, although even by that standard, I think things look much better than the field expects. I mean that a network's parts are built to be read, and built upon, by its downstream readers. In this sense, a neural” But this still leaves us with the same uncertainty as before - just because neuron interpretations are “interpretable” in some sense (ok at least they’re not fundamentally interpretable but also how could that ever have been the case), doesn’t mean they are interpretable to humans. So we still need one big update for this to become as bullish as it is. 3/ the language of humans and the language of neural networks is the same This one is imo the most surprising and also pretty beautiful implication of Nick’s post, and it suggests something not just about neutral networks but also about the structure of reality. After applying the above technique improvements (some of which just came from waiting for deep learning to improve farther), they find that >90% (maybe more) of much larger neural networks’ neurons are completely human interpretable and actually correspond very cleanly with ideas humans have individually words for. This is especially interesting in image networks where neural networks were not trained on human language which is already a human artifact and so maybe you would expect it to be more plausible that LLM and human representations are the same. Image detection networks are just trained on images (many directly pulling signal from “the real world,” meaning they will find interpretations that are just most effective to represent the real world signal, rather than to represent language which is already a proxy for real world signal). And it turns out that >90% of the representations they pick organically overlap with human ideas! That is insane. And it means something about the brain and reality. It means that at least at the parameter sizes of the networks tested (which are quite large), there is really only one set of “optimal” symbols/language that represents the structure of reality! There is ONE underlying structure of reality, and ANY language that describes it well is simply emergent from that structure - meaning there is always a translation path between any 2 languages that effectively compress realities signal through the “underlying truth.” This would suggest that even weird alien seeming neural network languages like huggingface neuralese should be fundamentally human interpretable, and in fact maybe with the techniques laid out here we will eventually be able to translate these messages (would be a cool project). So the brain and deep learning, as similar or different as they may be in techniques, are ultimately picking up on the same patterns and forming the same representations from the on “true” language which is the structure of reality! The main thing to look out for in this area is how the idea interpretability % updates with scale, but we can hope given what we’ve seen so far that neurons will remain interpretable. 4/ alignment can progress as fast as RSI. recursive self-improvement unlocks recursive self-alignment and a scaling law for alignment. This one is probably the most important implication for our future with AI and especially timely with the recent fear about runaway scaling and RSI. As many have gathered by now, it feels like the researchers at the frontier AI labs have been quite bearish about our abilities to understand the new frontier of models, in particular with the appearance of unintelligible communication and the daunting and seemingly intractable task of scaling up existing alignment techniques (ex: SAEs) to the size of frontier networks. And with RSI around the corner/already starting, it would seem that these networks will runaway in capability before we can ever hope to understand even today’s systems. Additionally, as systems compound in capability, it is reasonable to believe that the only tractable way to ensure they remain aligned would be to figure out some way to spend compute for them to align themselves (ie. a scaling law for alignment). But there is an obvious problem with this: how can a system that we already know is unaligned and we are *trying* to align be used to align itself? Surely if it is misaligned it general it would figure out ways to misalign during “alignment” and create compounding misalignment risk. The core issue high is the unverifibiality of this problem - the classic Achilles heel of current frontiers techniques is that it is difficult to train on problems with “unverifiable reward signal.” Well this is exactly that problem. But the incredibly convenient thing is that Nick’s findings may convert alignment into a verifiable problem. If we can spend compute on frontier systems to “guess and check” which ideas correspond to which neurons, we have a highly verifiable reward - we can see directly whether an LLMs proposal for what a neuron represents actually corresponds with the highest activation for that neuron. In this way, we can actually observe and verifiably monitor the networks trying to guess at the interpretation of their own structure. This means it is plausible for us to spend more and more compute doing more “guess and checks” (which you know, maybe is what RL does anyway, who knows) to try to develop a full interpretation of a neural network. In some ways this directly represents the possibility of a scaling law for alignment (more compute for more alignment). And this also aligns RSI (greater capabilities) with recursive self-alignment (greater ability to align itself). And the critical though maybe obvious implication of having a full interpretation (almost like a map) of a neural network, is that we can then understand exactly how to boost good traits and remove or lobotomize (lol) bad traits, which we’ve already seen works extremely well with stuff like Golden Gate Claude. So overall this combines into an extremely bullish picture about the prospect of true and human interpretable alignment in the near future. This is truly one of the greatest cases for optimism in years!
7
22
1
377
16,188
Alexandre Ramé retweeted
This is interesting work, but I don't think it's a significant update to the big picture of interpretability. I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc. I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point. I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes! Similarly, my impression is that SAEs generally don't find things that are a single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework. And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
39
175
29
1,762
137,593
Alexandre Ramé retweeted
the researchers at all the labs would tell you in private and sometimes in public that alignment has not been solved. if this is true, then racing into RSI is a folly that is unprecedented in world history. we will need to significantly slow the frontier of research very soon
139
333
82
3,252
443,528