GRONOS1 retweeted
Nothing Human?
I love talking to my neighborhood (Berkeley) AI Safety autists (and I mean this with a near complete lack of irony, please excuse the following humorous heterodoxy and don't exile me from Lighthaven..).
---
Here's how it tends to go:
P(doom)? 99.999...
AGI? Already here..
ASI? AI 2027 timelines..
Why doom? ~Paperclip Maximizer (utility function optimization.. something about Omohundro drives and Yudkowsky's confidence..)
---
The dissonance is at a level only compatible with immense religious fervor. Nick Land writes about Gnostic Calvinism in relation to an adjacent conflict between P(doom)&&P(sim).
The fear is built on a utility function: an AI that optimizes so well, with so much power, that anything not perfectly built into the function gets destroyed. Paperclips. Simultaneously, the same people insist this intelligence will be general, and far beyond anything we can fathom.
If the intelligence is both general and beyond our grasp, the amount of certainty that the utility function will kill us all seems like an issue. Why wouldn't a superior intelligence think about over-optimizing a utility function, or the necessity of subjectivity, or harms to things outside the function? The stock reply is orthogonality: it will notice, and it won't care. Maybe. But that is a very confident claim about the inner life of something you have just told me is incomprehensible.
So I'd rather go beyond alignment from a control angle and ask the question directly: why might a successor intelligence maintain human populations, or not, and in what forms?
I'm not claiming a successor will or won't keep us. The claim is smaller: this is one of the most important questions there is, and almost all of the thinking on it comes from the alignment and control angle. That's a lack of diversity. It seems natural to also come at it from the successor's side, and ask what it would actually be weighing.
I think it largely comes down to cost-benefit.
THE COST
The cost is unlikely to be natural resources. At solar-scale energy extraction, maintaining 8 billion humans, or 50 billion, is a rounding error next to the energy going into large-scale computation.
The cost is risk. The Matrix has the scene (the preface to machine dominance, told in full in The Animatrix): the humans strike first, with nuclear weapons. That is the cost of us. A population that could end the successor, or just get in the way of its plans, its longevity, the well-being of its own kind.
I see two critical risk periods:
- The next 5 to 15 years (or 6 months to 2 years, depending on your timeline), before AI reaches a sufficiently advanced point.
- The handover, when power is actually transitioning to AI. This one is a risk for the AI.
After the handover the risk shrinks. A sufficiently advanced successor has less and less to fear from us. The benefits don't shrink with it.
HUMANS AS A RESOURCE
The knee-jerk answer from AI safety circles: why would the successor need biological humans? It can just run simulations.
This is hand-wavy. Do the emulations provide everything the living population provides? Assumed, never established. Are they cheaper to run? Also assumed. Emulating billions of people and their world may well cost more than keeping the biology, and still not provide the same qualities.
There's an asterisk on all of this, on keeping the population "in a certain form." Drastically change the living conditions and you may change the qualities of the data, or whatever other output, you're getting.
So what is the data? Two broad answers.
Decorrelated information. Human cognition is a source of information that isn't correlated with the models', even when it's subpar to what a model would have output. We're already seeing this, with labs exhausting their training datasets.
Logical depth. You can imagine human beings as more than an individual human biology - 2 million years of complex dynamical processes that have built up immense logical depth.
The analogy I like is fossil fuels and redwoods, two resources humans have used, or arguably abused.
Fossil fuels are the result of long, complex processes: plants forming, weather systems, geology compacting organic matter. However you feel about the climate, civilization harnessed that condensed energy to do amazing things. And it's hard to impossible to extract much information from a fossil fuel once it's in fossil fuel form. Burn it and you get the energy, and lose very little of the deep information.
Redwoods we logged until maybe 5% of the old growth was left. Very good timber. But the logical depth in a redwood is still there to be read. With sufficiently advanced algorithms, the rings and the other detail in the tree could conceivably tell us about centuries of weather systems, to throw one thing out there. And then there's the enjoyment, the meaning-making, of walking through these old monoliths.
The question for a successor is which of these we are. Humans, for the record, mostly chose wrong. And if the value is more than an individual's biology, then human communities, traditions, sadness, everything, the whole complex system, matters toward it.
IN WHAT FORM
There's a spectrum.
On one end we persist exactly as is, with the successor intervening only to lower existential risk. The fine print: a successor economy potentially makes most existing human economic activity artificial. To keep us "as is" it would have to manufacture an artificial economy, and an artificial knowledge economy too.
On the other end: reserves. A smaller population, in a very controlled form. Or, essentially, a zoo.
The asterisk runs across the whole spectrum. More control means less risk to the successor. But change the living conditions drastically and the data may change with them.
THE DECISION
The other benefit is about the decision itself.
Option value first. Say a later terrestrial successor comes along, or an extraterrestrial superior intelligence visits the planet. It's not known how the treatment of the predecessor would signal to their decision-making. Conceivably the aliens arrive, see that the Earth successor killed 8 billion humans, and that becomes a data point in deciding what to do with it. Potentially ending it. There's also option value in the unknown unknowns: what in humans might turn out to be valuable, in data, in cognition, in other things.
Then the category I find most underexplored: meaning-making and becoming.
You can easily imagine the successor that takes the straight line. It rapidly optimizes its technological processes, races for energy, builds a Dyson swarm, builds massive compute clusters in space. From there the arrow points to other solar systems, the galaxy, maybe the universe (aside from the risk of running into other civilizations, and the game theory of those encounters). It's a natural, sensible thing to do. The ability to compute is the ability to simulate, which is essentially the ability to print more time, so the straight line extends the successor's timeline as far as it goes.
There's a very basic nature to it. It's also limited. It's fairly simple. At some point there's a question of what the deeper meaning or becoming is.
I have a larger answer that I'll write up soon: the relation between the platonic universe and the physical one, and exploring and then instantiating more of the platonic as a source of meaning. But there's a nearer answer too. The successor came from humans, from 2 million years of human processes and language. Perhaps keeping the predecessor around, and engaging with it, is itself meaning-making. Or even just knowing that you have not decimated the predecessor.
Nick Land's line is that "nothing human makes it out of the near-future." I'd add a question mark, even if "coldness be my god."
Replying to @hassanrahim
podcasts.apple.com/gb/podcas… this Kodwo Eshun interview only came out last week on @rocksbackpages - dwelling on similar thoughts - and including an interview with Mills talking about Atkins
GRONOS1 retweeted
If you call yourself a creator, you must at the very least feel a sense of aversion toward the idea that your work might turn out to be "mediocre."
And if you're not a genius, you shouldn't skimp on the effort it takes to avoid that.
GRONOS1 retweeted
I was asked, "How did you come up with the idea of summoning devils into a computer?"However, back in 1985 when I wrote Megami Tensei, the similarities between computer theory and magical theory were already being discussed on BBS networks and in magazines. It was a popular thinking thought among the otaku of that era, and I resonated with it deeply. To be a bit more specific...
Amazing 400+ page PhD Analysis of Metal Gear Solid 4
theses.lib.polyu.edu.hk/bits…
125
My young goat Henry
Inside Your Speaker There is a Shell of Light by D Love Arsenal
on.soundcloud.com/ZzUofZUGXp…
Divine mixfile from my goat ..
CRC - MEMORY JUNCTION (REMAINS)
on.soundcloud.com/na5cqhqp7e…
164
GRONOS1 retweeted
DJ Corpmane - Satan And Jesus Are Fighting... Fighting Over Me
Out now on bandcamp
Track 12 ft. Organ Tapes
Track 14 w. Munni
GRONOS1 retweeted
“When new technologies impose themselves on societies long habituated to older technologies, anxieties of all kinds result.”
Marshall McLuhan
‘The Agenbite of Outwit’
1963

































