Effective Altruism is not the first totalizing ideology with pretensions to a superior scientific basis to capture huge numbers of smart but clueless intellectuals. One hundred or so years ago, many of the best and brightest in our society were absolutely convinced that the most important thing they could do for humanity was to aid or undertake Marxist revolutions. As I have said many times in the past, it takes a certain amount of intelligence to convince yourself of a completely crazy idea and of the importance of devoting your life to it. For a certain sort of mind, it’s almost impossible to turn away, and it’s all too easy to persuade yourself of absurdities with castles of logic built in the air that would not convince normal folk, who just notice that the emperor isn’t wearing any clothes and go about their business. Marxism also comes with its own internal language, with extremely verbose secular scriptures, with detailed explanations from its highly intelligent but self-lobotomizing adherents about why it makes almost inevitable sense and why opposing it demonstrates that you simply must not understand it. It is also an ideology rooted in utilitarianism, also has an eschatological and utopian vision, and also is also incredibly dangerous. Many Marxists are, of course, perfectly nice people as individuals. So are many Effective Altruists. In both cases, they really are just trying to help the world, even if their idea about how to help the world is horrifying and will cause incredible damage if it isn’t vigorously resisted. I fully believe that the bulk of the rank-and-file mean well, though I am not always convinced about the leadership. However, it is also the case that society cannot turn a blind eye towards a fifth column attempting to subvert our civilization.

Any time someone tells you that AI has an X% chance of wiping out humanity, ask them for their math that got them the number. If they didn’t do a calculation, if they just pulled the number from their butt, you can safely treat it like anything else that came from their butt.

Years ago, there were two points of view on releasing LLM systems into the world. One of them, the Orthodox Doomer/Yudkowsky view, was that it was far too dangerous, and that no one should be playing with such things until the semi-mythical “alignment problem” was solved, perhaps thousands of years from now. The other people said that the only way that you learn how to use and control a new technology is to engage with it, that ordinary engineering processes are the only tool we have that ever works for learning how to solve problems in new systems, and that it was necessary to grapple with stuff if you wanted to learn how to use it well. History has already proven the second group correct.