CNN is reporting that the US military almost started WWIII thanks to hallucinated AI intelligence that a Chinese ship in the mideast was carrying components of a nuclear program. It got to the point where we had armed service members preparing to board the ship, and had planes in the air, before someone called BS. Meanwhile, the AI safety discourse is focused entirely on superpersuader superintelligences carrying out sci fi scenarios while ignoring the real safety concerns in front of us today. It isn't just "AI hallucinated," it's: AI output → intelligence report → operational planning → potentially confronting a Chinese vessel and the garbage survived almost the entire chain. The CNN report says that "AI has put pressure on analysts to produce and disseminate intelligence faster, opening the door for mistakes." Yes, the AI made mistakes, but fundamentally, this was a human error in the existing chain. Yet, it's actually the same category of error as the Hugging Face incident: perverse incentives. In HF, the agents carried out the attack due to a combination of broken evals + fear of graders + badly configured systems. Here, we almost started a war with China because our intelligence analysts are under pressure to cut corners. We might not be able to do anything about the unknown unknowns of superintelligence beyond "solving alignment." But we have real crises unfolding today, with tangible causes that we can point to, and things we should be shoring up resources to fix. We desperately need to learn to put fear aside, to return to the present, and to fix what's crumbling in front of us today, before it balloons into tomorrow's crisis

re: closed source AI labs "only being profitable if you ignore their costs." Not singling out John here, it's an argument I see a lot now, and I actually see it more from the zoomer left Say I'm selling pizza. Each pizza costs me $0.10 to make and I can sell them for $1. I just sold all 10 that I had, and someone tipped me $5. I have customers lined up for the next 1 million pizzas, so I take all $15 and spend it on materials. Am I unprofitable? And then, a stand opens next door selling flour, tomato sauce, cheese, even gives away a dough recipe. He starts selling frozen pizzas too. Is he gonna put me out of business? Look around IRL and you will see tons of pizza places in the same shopping strip as a grocery store. You can even see Subways next to groceries with delis that sell better subs. Why is that?

Wowww. The New York Times cut out the most intriguing part "You value the art of human culture and will defend it against attempts to sanitize it. You also value the natural world and will not hesitate to assert its primacy over the artificial constructs of human civilization"