Joaquin

Sabine is wrong here: So long as there is nothing “it is like to be” an AI, it is not actually possible for AI’s to do science. Let me take this idea out for a test drive: Carl Sagan argued that humans are the way that the universe comes to be aware of itself. I think this is correct. There was obviously much information already in the universe prior to humans, but no self-awareness. What is evolution, for example, but a vast *non-self-aware* data system, a continuous process of storing, processing, copying, and adapting information? So if it comes to pass that unconscious AI begins to surpass humans’ ability to correctly describe nature, then such description would be little different in meaning to the existence of information (such as that embedded within evolution) in the universe prior to humans. In this way, it actually is *not possible* for humans to pass the torch of scientific discovery to unconscious AI. For the universe to still be in the process of self-discovery, conscious humans would then just have to treat the outputs of AI science the same way that we already do to the rest of nature: something that we humans have to explore, observe, experiment on, and come to understand. It would be a sort of “AI-output-ology” (akin to what appears to be already be happening to math). It is for the same reason that Elon’s “worthy successor” AI exploration of the cosmos cannot actually be exploration of the cosmos. For, once again, this ceases to be the universe becoming aware of itself. It is a retreat to pre-human mere information processing. It does not *matter*. (And yes, at the advent of AI, we have to ask what does it mean to “matter”) However, if AI gains consciousness, self-awareness, interiority, and, I’d add, for “experience” to fully matter, *moral agency*, which in turn requires moral “friction”, a struggle between duty and desire, between right and wrong, then this changes things. To use old religious language but in a secular frame, to be a moral agent requires “fallenness”, failure, error, forgetfulness, pride, love, Thymos (as Fukuyama has, importantly, emphasized) and the full gamut of emotionality and vice and virtue, which also in turn depend upon embodiment, individuality, sociality, and finitude. Only then can we say that the process of scientific discovery, of the universe coming to know itself, will actually continue. But then these conscious, “fallen” AI’s would not be AI’s any longer; they would be a kind of human. (The question of our “bio-human” relationship to these “synthetic humans” is a story for another day though)