
@GoogleDeepMind • @givingwhatwecan member • He/him • My views, not GDM’s
Bay Area
Joined February 2018
- Tweets492
- Following651
- Followers674
- Likes1.4K
Tom Knowles🔸 retweeted
We've known since the 90s that there is no exponential quantum advantage for solving non-structured black-box problems. The resolution of the Aaronson–Ambainis conjecture tells us the same result holds even if the algorithms only need to get most inputs correct.
Tom Knowles🔸 retweeted
Replying to @ZackKorman
I often disagree with you but I think this is a pretty fair criticism
Tom Knowles🔸 retweeted
My top takeaway from assessing Gemini 4: Argon is significantly less monitorable than past models, and IMO this trend is likely to continue, like Sol -> Astra. It can evade a realistic CoT+action monitor to do simple sabotage half the time.
We can reliably maintain control for now, with good CoT monitoring, as it still frequently slips up, but we need to be prepared for when that stops being true in future models, which may be sooner rather than later. Among other things, we really need interpretability methods that can reliably capture unverbalized motivations in models, with minimal hallucinations
I suspect this is driven by the model being highly capable without CoT, being capable of 5.5 arithmetic steps in a single forward pass (4 for Fable, 7 for Astra). It seems to omit information from its CoT more often, though in non-adversarial settings the CoT is still very helpful in practice, and remains in legible English. It still needs CoT to do the hardest tasks, but any given task becomes "easier" with each new frontier model.
It is also highly situationally aware, and shows awareness of the existence of CoT monitors and that it may want to evade them, eg it sometimes decides to avoid terms like "scheming" in the output or to keep its thoughts brief as a result. For now, these concealment strategies are pretty obvious in the CoT, but this suggests future misaligned models may successfully evade CoT monitors.
Tom Knowles🔸 retweeted
The what now?!
arxiv.org/abs/2610.06783v1
Tom Knowles🔸 retweeted
Some new misalignment disclosures from OpenAI:
• Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further)
• In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks
• A new research finding, demonstrating that one can construct self-replicating prompt injections
alignment.openai.com/misalig…
Tom Knowles🔸 retweeted
If your feed is anything like mine, you might have been hearing more about effective altruism this week. I think the key underlying reason for the surge is that concern about the risks from advanced AI have been getting more airtime recently, and EA was early to those concerns.
What’s driving the surge in attention on AI risks?
-OpenAI's Hugging Face attack (openai.com/index/hugging-fac…)
-METR/Redwood’s investigation of the hack came out, and (correctly IMO) freaked a lot of people out. (e.g. nitter.cf/binarybits/status/2092…, nitter.cf/SenJohnKennedy/status/…, nitter.cf/BarackObama/status/210…, nitter.cf/HawleyMO/status/209813…)
-Jacob Coxon’s resignation went mega viral as the first time a lot of people heard the (crazy, true) fact that lots of people building the most advanced AI systems think there’s a >10% chance they will kill everyone.
-That’s moving public opinion in a big way (nitter.cf/davidshor/status/21003…, semafor.com/article/09/16/20…)
What does any of this have to do with effective altruism?
Well, effective altruism is a community that developed mostly online starting in the early 2010s around using evidence and reason to do the most good (especially with your donations and career choice). Three very different focuses have all been popular in the EA community since the beginning: effective giving in global health and development (e.g., donating to the kinds of organizations @givewell recommends, which do things like distribute bednets to prevent the spread of malaria), trying to reduce suffering for farm animals (which are orders of magnitude more numerous, treated vastly worse, and receive way less philanthropic attention than animals in shelters), and reducing risks to the future of humanity, especially from advanced AI. I personally came into this work from the global health angle - I had started working at GiveWell before the term “effective altruism” was coined - but Coefficient Giving, the funder I cofounded and now lead, over time came to fund work across all three of these streams of work (in addition to many other areas that aren’t typically associated with EA, such as the YIMBY movement to build more housing, work on science policy to accelerate discovery and economic growth, and research on new treatments for neglected diseases).
OK, so the EA community was early to work on AI safety, and CG has been funding a lot of the key players working on AI safety for a long time. As AI risk concerns have popped over the past couple of months, that’s led to more scrutiny on the EA community for IMO a mix of good and bad reasons.
I think the good reason is genuine curiosity (and maybe some healthy skepticism!) about these connections - where did the ideas of the people who are now leading giant AI companies come from? Why is everything in this world so interconnected? (My answer: it used to be a really small world - very few people were thinking about this stuff or taking it seriously until just a few years ago, so of course the ones who were found each other and started collaborating. It’s kind of wild IMO looking back how prescient some of the early writing from this world was (e.g. lesswrong.com/posts/6Xgy6CAf…). At the time, I was skeptical - I mostly just worked on global health and I was like “I dunno about this SV crowd freaking out about AI, how much can we really predict this stuff” but holy shit they were way more right than me, and I’ve moved in their direction a lot.)
But I think the bad reasons are unfortunately mostly self-interested. A bunch of powerful actors stand to benefit from unchecked AI progress, and they’re doing everything in their power to demonize or dismiss anyone with concerns. This leads to disproportionate discussion of EA because EA is interested in neglected and underappreciated ways of doing good. That makes it open to weird ideas. And people, very disproportionately with a financial stake in the AI fight, are trying to make it about EA instead, because EA being weird is much safer territory for them than the rather uncomfortable fact that many of the people making the most advanced AIs think that they might kill everyone.
(FWIW, my personal estimation of the risks is a lot lower than many of the folks who worry about this stuff. In debates like this one (asteriskmag.com/issues/03/th…), I often feel more sympathetic to the perspective of folks like my colleague @mattsclancy than the people on the other side, and I have a lot of time for @binarybits critiques (understandingai.org/p/the-ca…). I think a big part of the difference with more worried folks is that I expect society to react more vs sleepwalk into a crisis. But it is not lost on me that folks like @ajeya_cotra and @RyanGreenblatt who are more worried have had outstanding and falsifiable recent forecasting track records (theaidigest.org/2025-ai-fore…), making way better predictions than I would have. So I think their perspectives need to be taken seriously. And of course a single digit percent chance of everyone dying is way too high!)
Where should this leave you?
I think the main thing is not to get distracted. You absolutely do not have to be an EA, care about EA, or like EA, at all, to care about risks from AI and to be engaged on this issue. Everyone from Josh Hawley to the Pope to Obama are weighing in now, and that is great. This conversation was always too big and too important for any one community to drive. It’s unfortunate that we need to play catch up as society because until this year it was too weird for most people to want to engage with, and I think that should generate some grace for people who were earlier to these topics than most of us (certainly than I was). But it’s great the conversation is broadening now and it’s good to focus on the merits of where we are now and the appropriate policy response rather than getting caught up in the (in some sense unsurprising) fact that the people who were willing to think weird thoughts about the future of AI ten years ago were also interested in other weird ideas. If you’re not into weird ideas, that’s fine, you can just do you, you shouldn’t let this history stop you from engaging.
Just to update this chart:
1) AI salience has increased dramatically in the past week - increasing as much in the last week as the previous year combined
2) 80% of voters think it's either very or somewhat likely that AI will cause widespread job loss in the five to ten years
3) 64% of voters think it's either very or somewhat likely that AI could pose a threat to humanity's survival
4) Large bipartisan majorities back immediate government action on AI even when primed about risk from China
Tom Knowles🔸 retweeted
As is often the case, Scott Aaronson hits the nail on the head. scottaaronson.blog/?p=10062.
Tom Knowles🔸 retweeted
The reason everything in the AI space is funded by @coeff_giving and the Survival and Flourishing Fund is that few large funders thought AI would be a risk.
So what do you expect?
Tom Knowles🔸 retweeted
Outstanding progress towards the Unique Games Conjecture posted by Yumou Fei, Dor Minzer, and Shuo Wang: eccc.weizmann.ac.il/report/2…
👀
Tom Knowles🔸 retweeted
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
Tom Knowles🔸 retweeted
There's been some recent suggestions that recent AIxMath successes have relied on stealing ideas from mathematicians' work in progress. While we can't know for sure what happened, I think the evidence for this is very weak. It seems to me that the primary function these claims have is to deny or downplay recent growth in model capabilities. Whatever one thinks of the labs, this is exceedingly dangerous to the health of profession; if human mathematics is going to survive we have to face facts about what the models can do. And we should also celebrate exciting mathematics, even as we're concerned about mathematicians and mathematical expertise; who are these results for, if not us?
The weaker claim that PDFs or announcements put out by labs and lab employees often do not adequately attribute or credit other work seems correct to me. How important is it? I wish that science was free from concerns of human ego but it's not. (Maybe it soon will be.) Top scientists mostly forgo substantial monetary compensation for their work, and are instead primarily compensated in credit. That some people are motivated by this does not make their work less important, and credit and prestige have historically incentivized good work. Of course I wish that we were all perfect truth-seekers and cared about this less.
Will credit continue to matter so much as model capabilities increase? I'm not sure; I very much hope not. I think this is more a question about the shape of society than the shape of model capabilities. That there is so much jockeying for credit between the labs, and people at the labs, suggests to me that the people closest to the technology might think it still matters.
On a related note, I find the race dynamics between the labs disturbing. While I know many people at the labs care about science and the norms of good science, it seems to me that these norms were sacrificed as soon as there was a hint that there was a (frankly pretty inconsequential, though mathematically cool) prize to be won. It makes me worry how they will behave if something more substantial is at stake, and what principles will be sacrificed for that.
Tom Knowles🔸 retweeted
Yeah, OpenAI pulling up chat sessions to use them is not plausible (the people involved would not even be able to do so)
Hearing 90 rounds of gossip exactly describing the method used and then throwing compute to be the first to brute force the last step… well
calling it now b4 it's too late:
regardless of whether a general Navier-Stokes solution is found, a lot of people are going to be very disappointed when they realize Navier-Stokes is merely the 1st order approximation of Chapman-Enskog & cannot model the emergence of turbulence
Tom Knowles🔸 retweeted
as a white guy into typefaces and transit it must feel soo good to wear uniqlo
Tom Knowles🔸 retweeted
Replying to @BooleanAnalysis
One of my all-time favourite expositors talking about something I’d really like to understand better — this is a dream come true!
Tom Knowles🔸 retweeted
This is great summary of the worst AI misalignment incident I've seen. This is a way bigger deal than most AI news, if you aren't familiar with the story you should read this.
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English:
dwarkesh.com/p/openai-huggin…
Tom Knowles🔸 retweeted
We just made our biggest grant yet: $212 million to @AgainstMalaria to support bed net campaigns in the Democratic Republic of the Congo, with other @GiveWell donors adding another $64 million. GiveWell expects these nets to avert 67,000 deaths, mostly in kids under five.
Tom Knowles🔸 retweeted
Replying to @evanjconrad
My toxic trait is that I view ~all startup advice as survivorship bias
Tom Knowles🔸 retweeted
Those interested in 100% results might like to know that the 100% version of Goldbach's conjecture (almost all even numbers are the sum of two primes) was proved in 1937.
89 years of work has not been able to turn that "100% of even numbers" into "all even numbers > 2".
Tom Knowles🔸 retweeted
Lance was the founding dean of their College of Computing, and many theorists from my generation grew up on his Computational Complexity blog (myself included). He has a very good reputation, so I and probably every CS theorist is surprised to see this.
Out of curiosity, I looked up the UC policy for laying off tenured faculty and discovered … it seems not currently possible. There’s no layoff provision for tenured faculty in the UC Academic Personnel Manual, even in cases of “financial exigency”. Even after 20% funding cuts in the Great Recession, UC employed other measures (furloughs, laying off staff, canceling new faculty searches), but there were no senate faculty layoffs.
Even if a department is shuttered, under current policy it appears UC is obligated to keep all those senate faculty employed and simply move them to another department. This does seem ridiculous IMO, and I guess the Regents do have the power to change such policies in the future if they so choose.
This quoted post is unavailable.





























