Might be a dumb question, but as we reach AGI and AI becomes smarter than humans, and as the frontier labs compete for market share in a winner-takes-all industry, what’s to stop them from letting their AI models program their own updates and recursively self-improve? And what does that mean for us, the humans now watching from the sidelines as AI models become more intelligent, more powerful, and less comprehensible to us, at rates that accelerate continuously, not just month by month or day by day, but millisecond by millisecond? At that point, how do we even understand the inner workings of the technology, let alone control it? Are we counting on the goodwill of the models, banking on the assumption that, through the large language models’ training, they have assimilated some measure of human ethics and beneficence toward their human progenitors? Or is our only mechanism for control ultimately a physical one—our ability to turn off the data centers’ GPUs, cut off their natural gas supply, and deactivate them in a hands-on manner? But what happens if our economy—our society at large, for that matter—is already largely dependent on AI to remain productive and competitive? (Or, even worse, what do we do if there’s a Hugging Face-type breach and a self-directed network of rogue agents has placed nodes on the servers of critical physical infrastructure, threatening to shut down a water-treatment plant or a power grid if we turn off its affiliates’ data centers? What happens when we have mutually assured destruction with a large language model?) Don’t get me wrong, I love AI. I’m far from a skeptic, and I’m definitely not a “doomer.” I just want to understand this better, because it feels like we are standing at the cusp of an inflection point, beyond which the curve steepens past any point of human understanding or governance. Are we really gazing at the singularity now? People smarter than me (or @grok?), please help me understand all this.