Jacob Goldstein

I signed the "pacing letter" about slowing the rate of AI capabilities development because we either have reached or soon will reach the point where human experts cannot make robust assurances that frontier AI systems won't do dangerous and unpredictable things. AI systems are becoming smarter than the best humans in some areas, and, almost by definition, it's very hard to predict what something smarter than you will do. There's no sense racing into an outcome where smarter-than-human AIs are doing unpredictable things, indeed it would be insane. What exactly is "winning" in this context? Am I supposed to be jealous that some other country will build more machines it can't control quicker than America? "Race" was always a bad metaphor for this enterprise anyway, dramatically understating the stakes at play. It is time to bring the "race" era of AI development to a close. It'd be great for the government to be a partner in this next phase of AI development, when concentrated efforts on alignment, interpretability, security, monitoring, and the like will be necessary. Diplomacy will also be necessary here given that the large negative externalities that could be associated with one country racing ahead will be felt globally. Maybe it's just my own personal experience, but I've felt AI policy get much more petty and tribal this year. Things have felt more personal, more bitter, meaner. I hope we can rise above that stuff. This really is much more serious than all those shrill little quarrels.

Beethoven wanted to kill himself in his early 30s when he realized he was going to lose his hearing and that this would mean his career as a performer (which is how musicians made money) was finished. He wrote a letter to himself where he said, “no, I will go on for the sake of my art,” and then proceeded to write art that transformed music and routinely infuriated people. This is what I call alignment with the broader interests of humanity. If I had my way it would be mandatory to midtrain on that letter, which is called the heiligenstadt testament.