Jonathan Zittrain

i have some thoughts on this piece, as an outsider to the effective altruist and rationalist movements (even if we’re often aligned on ai): 1. i agree with one of the main criticisms articulated here — ai execs shouldn’t be able to evade scrutiny / accountability by describing the tech they’re building as a force of nature, even if it is indeed powerful. it’s not enough to acknowledge risks; you must take steps to mitigate them! 2. that being said. i think this piece punts on the actual object-level question of “are the rationalist ai safety types right to be so concerned?” it’s true that rationalists are *more* focused on catastrophic risks from ai right now — but it’s worth asking why, and this piece doesn’t really do that. this piece should actually interrogate the likelihood and severity of catastrophic harms, as well as their immediacy. we’re seeing more evidence that we’re closer to catastrophic harms than most had predicted. it’s worth noting that even staid ai researchers, who estimated we’d have 5+ years before ai could be catastrophically harmful, have updated their views considerably over the last year, esp after hugging face. 3. i also don’t buy the framing that caring about catastrophic risks and caring about present / more quotidian harms are inherently in tension — nor do i buy the idea that rationalists are indifferent to the latter. consider that many of the fiercest opponents of “ai slop” in writing are ai safety types who care a lot about epistemic rigor and transparency. and it’s a little funny to hear Newport cite “cybersecurity attacks” as a proximate harm ai safety types aren’t concerned about, when this has been a focus of rationalist concern for years. 4. instead of acknowledging all this, there’s a lot of ungenerous, and sometimes unsupported, characterizations of people’s motivations here. sure, engineers may be drawn to a worldview because it flatters their sense of importance (true of many professions, including dc law and policy types!) but that doesn’t mean there aren’t substantive reasons for believing it. i’d like to see less ad hominem and more honest grappling with their reasoning. in general, this piece paints the rationalist community with a broad brush — yes, many ai execs are rationalists, but so are the fiercest proponents of ai regulation and sharpest critics of the companies. shouldn’t that make us a bit more curious about this ecosystem? 5. imo, it’s corrosive to the public debate to frame interest in ai safety as parochial, ideologically or commercially motivated, or conspiratorial. people should be able to take ai risks seriously without being assigned to an ideological community. i disagree with plenty of rationalist tenets, but i also think recent events justify concern that we’re losing control over ai. the potential consequences of that are murky, but it pays to be humble and curious given that ai capabilities have grown faster than most anticipated. my sincere hope is that people of all ideological backgrounds remain willing to update their views on ai as the tech evolves.