Emerj AI Research

Veritasium: "For the last 200 years... each generation has felt that life has been moving *faster*... and it seems likely that we'll continue to be able to do so indefinitely." An incredible feat of willful denial. Imagine seeing the singularity and saying "indefinitely" lol

you know him for "grabby aliens", "the great filter" and many other ideas - - today, listen in to @robinhanson discuss the possible trajectories of posthuman life in episode 16 of the Worthy Successor series there's a lot to love about this episode, and the opening clip sums up one of the major points: People want the future trajectory to have all kinds of wonderful and delightful traits (which are often fluffy and not well thought out), such as: - love - equality - valuing human beings - conscious - preserving of free speech - etc... Robin says the obvious thing that no other guest has thus far worded so succinctly: (Paraphrasing) "The thing we can be more sure about of great future intelligence is that it'll be adaptive, it'll do best able to survive and thrive into the future... ...so if you have something YOU value and want to see that thing in the future, you need to TIE that value to something adaptive and useful for the greater intelligence if you want any hope of that value persisting in some form into the future." The full episode explores a very rich take on Robin's take on what "progress" is and how it extends beyond humanity - and what he hopes for for that grand expanse of intelligence into the future.

you've heard Musk say "humans will be a small fraction of the total set of intelligent agents on earth"... ...but who has actually thought through what that looks like / how we get there? @blaiseaguera serves as Google CTO of Technology & Society (though like all my other interviews, he spoke with me in his personal capacity) his ideas about the coming symbiosis between man and machine is among the most thorough that now exist imo here's a few highlights from Blaise's Worthy Successor interview: 1. Our Symbiosis with Machines is Underway Blaise likens our use of machines to "outsourcing our metabolism" to machines - which he likens to what happened to mitochondria when they were first taken in by archaea through this outsourced metabolism, not only are there more machines (now that man and machines can build more machines) - but there are also more *humans* because a machine-powered world can support and provide for more humans (listen at 01:04:00) soon, blaise mentions humans will be a rare type of node in a greater network of intelligent agents at 01:09:00 into the interview we get into how win-win symbiosis has occured between man and animals (Blaise talks about hunters doing a lot of good to protect ducks in the wetlands), and i express a bit of my own skepticism that we'll receive the same treatment from machines -- and blaise out his own case for optimism ^ I saw this as part of the very heart of the interview and I hope you enjoy blaise's answers and ideas as much as I did 2. Evolution's Arrow of Time / Dynamic Stability complexity arrises over time because complex things are made up of more simple things, which must arise before the emergent complexity can occur often, this starts off as a random event - but when there is mutual benefit in the interests of the two entities, it can often create a higher degree of thriving (their ability to stay alive and continue developing) for both parties than had previously existed coordination is driven by mutual self-interest, and creates a kind of "dynamic stability" which becomes anti-fragile (listen at 00:22:15) he lays out frankly that this applies to humans with each other, but will very soon apply to humans and machines in a way that encourages symbiosis at levels beyond our present imagination .. I now thoroughly consider myself to be a Blaise fan. he may literally be among my favorite living thinkers, and I've just started to scratch the surface with him Below I'll put links to the episode and some of his other work

Paraphrasing Stuart Russell this week on The Trajectory: "Imagine that air travel involved riding on giant 200 foot-long birds which we bred selectively for this purpose. And occasionally those birds would take you were you want to go... but other times they'd drop you in the ocean if they wanted to - or just eat you if they got hungry mid-flight. If we couldn't figure out how to control them, have them not harm people, and have them do what we want - we might re-think the 'giant bird' flight strategy and work on a different technology path. With AGI, we're building something much more powerful that birds, with no ability to control them long-term, and yet we keep pursuing this uncontrollable path." lol In this episode Stuart shares his updates recommendations for innovators and regulators when it comes to "getting AGI right." All in all he's pessimistic that the research community will turn away from black box LLM approaches (which he wishes could happen). But does think there is ripe opportunity for some international coordination in the near term - and his ideas are worth diving into. (Links below)

If anyone cracks brain simulation in the coming decade i'd put my money on MIT's Ed Boyden He's long believed that neurotech shouldn't be just for curing diseases, but for expanding the space and possibilities of mind Ed's Worthy Successor episode is now live. TONS to cover. A few of my favorite sections: 1 - Bioengineering to Crack Consciousness [00:40:03] Ed talks about 4 Ms of bioengineering (make, model, modulate, measure), and what he thinks could be done to model brains (work that he is very familiar with), and what kind of "data about consciousness" might exist. He discusses how hard it is to determine what to measure for sentience, but lays out some of his best hypotheses from a yet-to-be-released paper he's working on on the subject. He hopes more engineers will join the ranks of the philosophers who are working on sentience. 2 - Aspiring Beyond Human Values / Goals, and Even Consciousness Itself [00:09:33] Ed believes that if we can understand the components of the mind bottom-up, it might help us to bloom new, greater forms of mind beyond us (in bio or other substrates). While he believes we might have in initial direction about what "happiness" or "meaning" might be for humans, he frankly admits that as we explore that expanse of minds, new and higher goals might emerge from these new intelligences, things not only beyond our present stated goals, but beyond human imagination - and potentially beyond consciousness itself. He articulates this cosmic idea of the state-space of powers, goals, and experience beautifully. Anyone who thinks humans can or should set the eternal goals or values of future intelligences should have their minds changed by Ed's argument here. ... It was a real pleasure to speak to Ed after following his work for 12+ years, I hope you enjoy this episode with Ed Boyden. [Links below]

Today on the Trajectory, the first guy ever to mention "Worthy Successor" (disparagingly) on Rogan: @romanyam :) Literally everyone has heard Roman's arguments about why superintelligence implies that we all die. But you haven't heard his "untestability" hypothesis, and what it implies about the kind of latent powers lurking in AI systems already. Two ideas you have NOT heard Roman speak about at length before: -- Unknowable Powers Inside Current AI Systems - What kinds of understanding of nature, or psychology, or (topics humans can't conceive of) that we can't possibly know - making "evals" potentially an exercise in futility. -- Treacherous Turn - Roman lays out two main ways that AGI might "bide its time" before it turns against us, and what we might do to prevent it. I'll remind you all of this: I will often interview people you have already heard from. But I will always ask NEW angles that you haven't heard them articulate before. This is the Third epsiode in our "Early Experience of AGI" series (previous epsiodes @DavidDuvenaud, @joshua_clymer) (Links in top comment)

Today, Toby Ord's AGI Governance episode is live (the last one was 4+ months ago with YUD himself) today. He explores the crucial AI and compute trends that are increasing, decreasing, or changing the nature of AGI risk since he published his last book. And yes, we touch on some Successor stuff, too ;) Some interesting sections that might update your p(doom) and AGI timelines: 00:07:00 - LLMs might have *lessened* our chances of AGI having a kind of near-term "hard takeoff" that humans can't understand. There might be a "kink in the curve" of AGI going beyond human level, because it is so anchored to and reliant on human data. 00:15:37 - We may find that fields where "tastes" is more subjective are areas where AI makes much slower progress. There's a lot of dialogue about "AI research taste" and recursive self-improvement, and toby's takes on this were novel and valuable. 01:17:37 - Future successors may be "worthy" (have goals and experience as far beyond the richness and depth of our own, as our powers and goals are beyond sea snails), but we really shouldn't *assume* that, and we should be remarkably careful about how we move into a world with systems more powerful than ourselves (Toby gets into some the incentives behind the AGI race here that is different than my take - and I suspect many people will agree with what he thinks is 'driving' lab leaders like Musk and Dario and others). I hope you enjoy this episode with Toby. (Links below)

"We are a transition phase between darwinian evolution, and secular intelligent design." - Lord Martin Rees He's the world's most eminent cosmologists and 60th President of the Royal Society. He has long thought about aliens, but now spends more time thinking about AGI. And he defines "Worthy Successor" in a way that would probably upset most people. Takeaways: -- **1** -- "We shouldn't understand them" Martin wants posthuman life to have 2 qualities: 1. Be conscious (this is the most common "worthy successor" trait our guests want) 2. Be so far advanced beyond humans as yo be entirely incomprehensble. Martin would feel most comfortable with posthuman intelligence if it is busy pursuing goals which are - in scope and scale - as far beyond his conception as his own goals are beyond the conception of smile mold (he uses slime molds as an example a few times in this episode). ^ This second point is the first time any guest has framed it that way, and TBH, I agree. I wish more guests felt this way. -- **2** -- "We should hope that they'd take the reins" Martin spends a lot of time laying out the case for why we should HOPE that life expands beyond humanity. While he doesn't think AGI will automatically be conscious (he advocates that we test and understand these qualities thoroughly) - he thinks that IF they are conscious, and they're vastly beyond us in all possible powers and understanding, we should gladly have them be the primary locus of volition in the world, and not human beings. ^ Probably the LEAST speciesist take I've ever heard on the show - not something I expected from a 83-year-old man! ... Everything Martin says is fun because he's brilliant, and because he has that kind of British accent which makes everything seem elevated :) Some fun timestamps worth jumping around to: 00:06:20 - His non-anthropocentric view of moral value 00:08:40 - Humans are a transition phase between darwinian evolution, and ‘secular intelligent design’ 00:16:28 - If posthuman machines have consciousness and are vastly more powerful and capable than ourselves, we should be glad to give way to them 00:42:10 - Martin’s hope that AGI would have an aesthetic sense (this isn’t an idea that I share, and we hash it out a bit - kind of an interesting part of the chat) 00:56:43 - Why Martin considers it essential for us to ensure that the flame of life blazes beyond the human torch (he lays out his argument to speciesists)

headed in a big metal bird, soon. i'll be interviewing top chinese and american AGI governance thinkers and AGI lab employees, along with some other major players who else is headed to Shanghai for WAIC?

the worthy successor episode with @eshear is out. emmett very much has his own understanding of agi development and intelligence - with tons of unique analogies to the natural world that i've never heard others use. 1 major theme that was a big contribution to the WS dialogue: Emmett discusses a broader "frame" for the great network of intelligence - a greater kind of "organism" or "tissue" of which both humans and AIs will likely be part. He discusses the two KINDS of "intelligence" that AI needs to manifest in order to truly integrate into this broader system (including the ability to discern WHAT to focus or "care" about, not just to find local maxima). His intuition is that a kind of harmony will or could emerge from what he calls "organic alignment." He's aiming to figure out: What makes it simple for intelligences to infer that they ARE part of a broader system that works together and cares for itself, and grows TOGETHER? We hash out a LOT of those dynamics in this unique episode, and I hope you enjoy it :) (Links to the full article / YouTube link in top comment)

Once early AGI starts replacing relationships, and taking white collar jobs en masse, surely people will FREAK out and wonder "where is this taking us!?", right? @joshua_clymer at @redwood_ai thinks people might not raise any alarms. And the implications of this are huge. Here's two significant takeaways from the episode with Clymer: ... 1 -- Different People May "Wake Up" to AGI for Different, Uncorrelated Reasons Assuming that biped robots will make everyone warry of where AI is going might be wrong. Assuming that AI taking over white collar jobs will make people wary of where AI is going might be wrong. Different people pay attention to different things. In this episode Josh talks about how AI "humor" is something that freaks him out specifically - but might not stand out as jarring for other people. If not enough people wake up at once, there will be no united political force to make national or international AGI coordination possible. 2 -- Clymer's Two Priorities for AGI When I asked Josh his MOST important goals for AGI governance, he lists two as primary: a) International agreements that all AGIs and aGI labs adhere to (the specifics of these agreements might vary, but this would likely involve certain levels of autonomy or capability that are barred for a specific amount of time because they seem vastly too dangerous and not understood) and b) That not very many people can build superintelligence (i.e. hardware, power, or other resources are restricted and closely monitored) We unpack both ideas / their potential implications in the full episode. ... This is the 2nd episode in our "Early Experience of AGI" series, where we look at the impact of proto-AGI on work and life. See the full YouTube video, podcast link, and article in the tweet below:

its incredible lol he's dancing on it, pointing to it, leading up to it, he's saying it... but NOT saying it. "This thing by its nature won't be controlled and humanity will be buffered out by a trillion Cambrian explosions" but you can't raise money being honest abt this

Peter Singer, arguably the most eminent living philosopher, airs on the Worthy Successor series today :) people who think singer only cares about utility/happiness will be surprised by Singer's "Worthy Successor" criteria. Here's 2 things that stood out to me in this interview: 1. Expanding minds into realms beyond consciousness Initially in our interview, it seemed like Peter was advocating for allocating 100% of AGI's power to tiling the universe with "utilitronium" (blissful sentience in some computational substrate). But by the end of the episode, Peter made it clear that he would prefer that some of a hypothetical AGI's resources should be allocated to (a) ensuring that it survives (can maintain and defend itself, keeping life alive), and (b) exploring entirely new realms of power and potentia to discover potential realms of value even beyond consciousness itself. This is not the textbook utilitarian answer by any means, and felt completely in line with axiological cosmism (albeit with a utilitarian bent) 2. Cope-free moral considerations This might be the first episode I've ever recorded where I could literally SEE my guest thinking through their own moral principles, and laying out their reasoning cold. Almost everyone - including eminent scientists and futurists - does a little bit of grasping at straws as to why humanity ought always be the most important (or surely at least relatively important) part of the universe in all posthuman futures. Singer does none of that - and it's remarkably refreshing. I hope you enjoy this episode with Peter: (Links below in next tweet)