Researcher. AI, games, markets, open-endedness, evolution. Professor @nyuniversity @NYUGameLab Head of AI @the_nof1 Co-founded @modl_ai Rogueliker.
New York City
Joined January 2009
- Tweets23.4K
- Following1K
- Followers23.9K
- Likes67.9K
Pinned Tweet
Guess what has arrived in physical form? The second edition of the Artificial Intelligence and Games book by @yannakakis and me! 530 pages of everything you wanted to know about AI for games and games for AI. – at Malmo, Sweden
Once again, I sit down to do some useful work and end up writing what I actually think instead, then post it before I sober up and nerf the text
How can I not retweet this
…hey, investors, look here
We mapped funding raised vs founder citations across 32 neo-labs
Their founding teams have amassed 8.35M citations and have raised nearly $26B
Discovery Loop leads with 1.59M founder citations, followed by SSI (915K) and AMI Labs (752K)
The two are correlated, but a huge citation count certainly isn’t a prerequisite to be a neo-lab (see Standard Intelligence and Flapping Airplanes)
Seriously, if the tokenizer treats it as a single token, is it likely to in some sense be treated as more primitive or elemental further down the layers? If it's a single token, it's something that just exists in the world; if it's multiple tokens, you have to assemble it. Or?
Imagine training a model to trade like Warren Buffett and Charlie Munger. As long-horizon as it gets. Sparse rewards, but *very* sparse actions. Extremely imbalanced reward distribution. Really makes you think about how by "capability increase" we mean tasks that are easy for RL
Recordings of a seminar on the future of games where I spoke last week. Among other things, I compared Stockholm to San Francisco. Not in a good way. (I also offered thoughts on the future of games.)
Futures of Games, Julian Togelius, Media Evolution, Malmö (ten days ago)
youtube.com/watch?v=KISSb_TR…
Julian Togelius retweeted
Working with AI requires perpetually operating at the edge of sanity. You have to recalibrate your mental model of AI capabilities and your own job description every few months, without getting delusional about seemingly infinite productivity gains, or feeling threatened by losing what you thought was the uniquely human part of your job, or obsessing over the perfect setup, or losing your mind over the clumsiness and jaggedness of agents, or ...
Maybe AI literacy needs to be less about prompting and more about improving emotional regulation.
1. I obviously agree
2. Interesting timing
People should always be in control of the future.
OpenAI is against the notion that it would be acceptable for people to be relegated to a role of doing things that “AI systems think makes sense to reward in humans”.
People should not cede control of the future to superintelligence, and that requirement should ultimately constrain AI progress.
All of this makes sense to me. The point about RSI potentially (or even likely) being narrow is also in agreement with my complementary intelligence thesis, where what're seeing is an increase in the magnitude of a particular intelligence vector, which may not mean much when projected onto other intelligence vectors.
As AI labs race toward Recursive Self Improvement, there’s a possibility that what they are actually racing toward is Recursive Self Extinction — of their business model. Here’s my reasoning.
1. Progress on verifiable tasks continues. In particular, improving the cost efficiency of existing capabilities is a highly verifiable task. For the past few years, cost per equivalent capability has dropped by at least 10x per year. I think this progress will be sustained for a few more years, and even accelerate as labs use AI more and more heavily for AI research in the pursuit of RSI. There is of course a ceiling to efficiency improvements, but I don’t think we’re close to that ceiling.
2. Progress on most unverifiable tasks remains frustratingly hard. AI inherits the same limitations (ambiguous success criteria, slow and noisy feedback) that limit human performance at tasks ranging from writing to governance. The optimization paradigm generally doesn’t result in superhuman capabilities on such tasks, because whatever proxy you optimize for gets Goodharted. For example, I recently pointed out how over-optimization explains Claude’s overly dense and ornamental writing style that’s since been dialed back.
3. RSI ends up being narrow. A consequence of the above two points — even if some version of RSI is achieved, it results in an explosion in some dimensions (efficiency, verifiable tasks) while falling short of producing something that is superintelligent in any real sense or capable of automating a broad swath of jobs in the economy.
4. More than one lab stays at the frontier. This means the market will remain competitive, and developers can’t charge high margins to compensate for rapidly falling inference prices. Any compute scarcity that allows higher margins is temporary, in my view. I also think it’s likely that most of the innovations continue to make their way into open-weight models with some lag, so open-weight models are at cost parity for inference or only slightly more expensive. Note that efficiency tricks and verifiable capabilities tend to leak much faster than unverifiable ones. Distillation could be one reason, but simply knowing that something can be sped up 10x tends to have a focusing effect on research and innovation.
5. Jevons’ paradox is not a magic bullet. If the above points hold, there are very few applications that will consume so many tokens that they could generate the astronomical amounts of revenue implied by the labs’ valuations. We’re talking about a 10x growth in tokens year after year, just to keep revenue flat given falling prices.
Even today’s behemoth tasks like migration of million-line legacy codebases will likely become cheap within a few years. While new applications will open up, there won’t be orders of magnitude more new ones. As for unverifiable tasks, inference scaling rapidly hits diminishing returns. Ultimately you have to have a human in the loop who’s accountable for the result. Spending more on tokens doesn’t let customers escape that responsibility — and only makes the human review bottleneck worse.
6. What about scientific research? Science is an endless frontier, and more tokens could perpetually mean more discoveries and inventions, in theory. But the unfortunate thing about research is that researchers don’t get to capture the value it creates, which is why it requires government funding in the first place. Even if it’s true that AI could enable great scientific innovations, I don’t think AI companies will make much money off of that. I think they are currently investing in AI-for-science not as a future revenue stream but as a loss leader because they want to show the public that it could be beneficial to humanity.
7. The one escape hatch is moving “up the stack”, by which I mean quickly transitioning from selling tokens to an enterprise software like model that relies on high switching costs, deep integration into workflows, and control over systems of record. In this world, enterprises will purchase “digital workers” or pay per solution (e.g. resolved support ticket) rather than tokens, so it’s good or neutral for AI companies if inference costs fall. And open models don’t matter as much — Linux being free didn’t make most enterprises switch away from Windows, because they paid for polish, peace of mind, support, and familiarity, not the code itself. I think this is the labs’ best bet for becoming sustainable businesses (though the potential lock-in is concerning for their customers!). The problem is that the labs have many hurdles to clear to get there, most of all their mindset, and the faster they race toward RSI, the shorter the window they give themselves for this transition. The pure-play AI labs are the ones who are farthest from being able to make this transition, and AI companies with a long history in enterprise software are relatively well positioned.
In short, the race to RSI ends up being a race to eliminate the labs’ main avenue for capturing the value that they create — the fate of many past transformative technologies such as railroads, airlines, and fiber.
The labs’ bet is that RSI produces a durable, decisive lead that escapes commoditization. My view is that there are two different improvement loops that have been conflated, and they will come apart. The efficiency improvement loop will accelerate, but a broader improvement loop that might lead to AGI or superintelligence will require ever-increasing amounts of compute, talent, data, and real-world feedback that can only come from gradual deployment, and the money to fund all this will have to come from massive revenue growth. And the irony is that the first improvement loop is undermining exactly that revenue growth.
The premise in my chain of reasoning that I’m least confident in is #5 — that Jevons’ paradox will eventually stop being able to compensate for efficiency improvements. A few months ago I was much more of a believer in continued, nearly unbounded token volume growth, but the more I’ve thought about it the more skeptical I’ve become.
To better understand why the premises and outcome I've laid out are at least a possibility:
* My AI-as-normal-technology essay and follow-up writings with @sayashk on our newsletter normaltech.ai/
* My ICML talk laying out my skepticism of the claim that RSI will lead to superintelligence normaltech.ai/p/what-will-be…
* Ramez Naam’s review of the evidence questioning the prospects of an intelligence explosion (@ramez) noahpinion.blog/p/wheres-the…
* My piece with Akash Kapur (@akashkapur) on whether AI labs will move up the stack normaltech.ai/p/up-the-stack…
* Many pieces by Nathan Lambert (@natolambert) such as “I expect rapid progress but not towards general superintelligence.”
Slopaganda (n): Promoting the creation and dissemination of slop. Example: It looks like you are writing an email. Click here to write it with AI.
Julian Togelius retweeted
Okay:
- Attached bottlecap
- Schengen, expanded to Croatia, Romania and Bulgaria
- The euro joined by Croatia and Bulgaria
- Air passenger rights (older, 2004, newly updated 2025)
- Galileo satellite navigation
- EU-Canada trade deal (CETA)
- EU-Mercosur trade deal
- EU-Japan trade deal
- EU-Singapore trade deal
- EU-Vietnam trade deal
- EU-Indonesia trade deal
- End of roaming charges, extended to 2032
- Price cap on calls and texts to other EU countries
- USB-C common charger
- Geo-blocking ban for online shopping
- Streaming subscriptions that work abroad
- Payment Services Directive (PSD2), no card surcharges and open banking
- Instant payments regulation
- EU Digital Identity Wallet
- eCall automatic crash alert in new cars
- Automatic emergency braking in new cars
- EU-Alert emergency warnings on phones
- Single Digital Gateway for cross-border admin
- No more apostille stamps between member states
- WiFi4EU free public wifi
- DiscoverEU free Interrail passes for 18-year-olds
- Erasmus+ (older, 2014), budget nearly doubled
- European Universities alliances
- European Solidarity Corps
- EU Digital COVID Certificate
- Joint COVID vaccine procurement
- Health Emergency Preparedness and Response Authority (HERA)
- EU4Health
- Europe's Beating Cancer Plan
- European Health Data Space
- NextGenerationEU joint borrowing
- SURE short-time work loans
- 15% global minimum corporate tax
- European Market Infrastructure Regulation (EMIR 3), euro clearing back in the EU
- Anti-Money Laundering Authority (AMLA)
- European Green Bond Standard
- Permanent Structured Cooperation (PESCO) on defence
- European Defence Fund
- European Peace Facility arms for Ukraine
- EU military training mission for Ukrainian troops
- Act in Support of Ammunition Production (ASAP)
- Strategic Compass and the 5,000-strong rapid deployment capacity
- European Defence Industrial Strategy
- Security Action for Europe (SAFE), €150bn joint defence loans
- ReArm Europe / Readiness 2030
- First ever Commissioner for Defence and Space
- Military mobility funding
- IRIS² sovereign satellite constellation
- EU Space Programme and the EU Agency for the Space Programme
- rescEU shared firefighting fleet
- 5G security toolbox
- Foreign investment screening
- Foreign Subsidies Regulation
- International Procurement Instrument
- Anti-Coercion Instrument
- Dual-use export control regulation
- Temporary protection for Ukrainian refugees
- Candidate status and accession talks for Ukraine and Moldova
- Ukraine Facility
- €90bn joint-borrowing loan to Ukraine
- Sanctions on Russia (older, 2014), big packages from 2022
- Russian banks cut from SWIFT
- Russian oil embargo and oil price cap
- Russian gas import phase-out
- Western Balkans Growth Plan
- Global Gateway
- REPowerEU
- Gas storage regulation
- Joint gas buying through AggregateEU
- Coordinated gas demand cut
- Windfall levy on fossil energy profits
- Electricity market reform
- European Climate Law, net zero by 2050
- Carbon Border Adjustment Mechanism (CBAM)
- Emissions Trading System reform and the Social Climate Fund
- Renewable Energy Directive, 42.5% by 2030
- European Wind Power Package
- European Hydrogen Bank
- Methane regulation
- Nature Restoration Law
- Ambient air quality directive
- Drinking water directive
- Urban wastewater directive, polluter pays
- Critical Raw Materials Act (CRMA)
- Conflict minerals regulation
- European Chips Act
- Net-Zero Industry Act
- Batteries Regulation with battery passport
- European Battery Alliance
- Important Projects of Common European Interest (IPCEI) on batteries and hydrogen
- EuroHPC supercomputers, including JUPITER
- InvestAI and the AI gigafactories
- Horizon Europe and the European Innovation Council
- Ecodesign regulation, no destroying unsold clothes
- Right to repair
- Ban on products made with forced labour
- Ban on vague greenwashing claims
- General Product Safety Regulation
- Common Agricultural Policy (older, 1962), current version 2023 to 2027
- Unfair trading practices directive protecting farmers
- Geographical indications reform for food, wine and spirits
- EU-China agreement protecting European regional products
- Geographical indications for crafts, like Murano glass and Delft Blue
- Creative Europe (older, 2014), current programme 2021
- Culture Moves Europe grants for artists
- Music Moves Europe
- European Year of Cultural Heritage
- European Capitals of Culture (older, 1985)
- New European Bauhaus
- European Collaborative Cloud for Cultural Heritage
- Ban on importing looted cultural goods
- 30% European content on streaming platforms
- Culture Compass for Europe
- European Media Freedom Act
- Anti-SLAPP directive protecting journalists
- European Public Prosecutor's Office (EPPO)
- Rule of law conditionality on EU funds
- Annual Rule of Law Report
- Article 7 against Hungary
- Whistleblower protection directive
- European Labour Authority
- Work-life balance directive, paternity leave
- Adequate minimum wages directive
- Pay transparency directive
- Women on boards directive
- Violence against women directive
- European Child Guarantee
- European Accessibility Act
- European Disability Card
- Rail passenger rights (older, 2007), recast 2021
- Trans-European Transport Network (older), revised 2024
- Single Market Emergency Instrument
- EU Inc (proposed)
- Digital euro (in negotiation)
- Next EU budget at close to €2 trillion (proposed)
- European Competitiveness Fund (proposed)
- AgoraEU culture and media programme (proposed)
- Savings and Investments Union (proposed)
- European supervision of big stock exchanges and crypto firms (proposed)
- Scaleup Europe Fund (launching)
- Startup and Scaleup Strategy
- Digital Omnibus simplification (proposed)
- European Business Wallet (proposed)
- European Innovation Act (expected)
- Quantum Act (expected)
- Cloud and AI Development Act (expected)
- Chips Act 2.0 (expected)
- Biotech Act (proposed)
- Advanced Materials Act (expected)
- Industrial Accelerator Act (expected)
- Circular Economy Act (expected)
- Critical Medicines Act (proposed)
- EU Space Act (proposed)
- European Grids Package (proposed)
- 2040 climate target of 90% (in negotiation)
- European Defence Industry Programme (EDIP)
- Defence Readiness Omnibus (proposed)
- Military mobility package (proposed)
- European Drone Defence Initiative and Eastern Flank Watch
- High-speed rail linking European capitals (plan)
- Single cross-border train ticketing (expected)
- EU Customs Authority (in negotiation)
- European Affordable Housing Plan
- Union of Skills and cross-border recognition of qualifications
- EU-India trade deal (in negotiation)
- Montenegro and Albania accession
- Qualified majority voting in foreign policy (under discussion)
Svenskan skriver om rekryteringarna av bland annat mig till Sverige med Wallenbergpengar. Ser fram emot flytten!
Fulltext av artikeln för dem som inte har SvD-prenumeration:
När Wallenbergsfären erbjöd 80 miljoner kronor bytte AI-professorn Julian Togelius
New York mot Göteborg. Nu värvas flera toppforskare till Sverige – men ingen tjänst
utlyses. ”Det är inte helt oproblematiskt.”
Han och frun hade funderat på ett sabbatsår eller två i Sverige. Men efter ett telefonsamtal från Devdatt Dubhashi, chef för AI och datavetenskap på Chalmers, blev planen en annan.
– Jag tänkte, men fan, vi provar, säger Julian Togelius, professor i datavetenskap vid New York University och en av forskarna som Wallenbergsfären värvat till Sverige med mångmiljonpaket. Ändå går han ner i lön med upp till en fjärdedel när han byter New York mot Göteborg.
– Ingen matchar amerikanska löner. Å andra sidan betalar vi just nu 55 000 svenska kronor i månaden i dagisavgifter, säger Togelius, som är svensk medborgare och uppväxt här, men haft hela sin forskarkarriär utomlands.
Ytterligare tre professorer har värvats under året: biokemisten David Fenyö från New York University till Lunds universitet, materialkemisten Alexander Bismarck från Wiens universitet till Chalmers, och matematikern och sociologen Mason Porter från UCLA till Stockholms universitet.
”Det är mycket positivt och glädjande att vi tillsammans med universiteten kan erbjuda så attraktiva rekryteringspaket att världsledande forskare väljer att flytta sin forskning till Sverige”, säger Peter Wallenberg Jr, ordförande i Knut och Alice Wallenbergs stiftelse, KAW, i en skriftlig kommentar.
Måste flytta till Sverige
Alla paket är lika stora: 80 miljoner kronor varav 30 miljoner från universitetet och 50 miljoner från KAW. Pengarna ska gå till att bygga upp en forskargrupp. Ett krav är att forskaren ska flytta till Sverige permanent och tillbringa minst 80 procent av sin sammanlagda arbetstid vid ett svenskt universitet. Pengarna får användas under tio år.
– För pengarna får du ett antal doktorander och postdoktorer. Det blir inte jättemycket pengar egentligen, för oss är det en rimlig nivå, säger Anders Palmqvist, Chalmers vicerektor för forskning.
Julian Togelius, vars forskning handlar om AI och spel, ser framför sig att han ska anställa åtta–nio doktorander och två eller tre postdoktorer. Hittills har han fått 643 ansökningar till de tjänster han utlyst.
– En del går till att täcka mina lönekostnader, så det är kanske 50 eller 60 miljoner som går till doktorander och annat. Det är stort internationellt sett, men inte gigantiskt.
Den stora vinsten är för honom en helt annan.
– Jag har berättat för folk i New York att jag fått en tjänst där jag inte behöver oroa mig för finansiering de närmaste åtta–tio åren. Då säger de bara ”wow”.
Mer osäkert i USA
Tre av fyra rekryteringar sker från USA, där villkoren för forskare förändrats snabbt. Sedan 2025 har Trumpadministrationen frusit eller avslutat drygt 8 000 forskningsanslag, enligt den oberoende databasen Grant Witness. Julian Togelius har inte känt av något politiskt tryck, men mycket har blivit mer osäkert under de senaste åren.
– Det är kanske den absolut största fördelen med den här rekryteringen: stabila villkor. Att veta i förväg att vi kan betala lön till doktoranderna. Så är det inte i USA just nu.
Satsningen landar mitt i en forskningspolitisk tvist – ska universiteten satsa på enskilda stjärnor eller fler doktorander och lektorer? Anders Palmqvist jämför satsningen med hur mycket Chalmers satsar på nyrekrytering av biträdande universitetslektorer, med målet att ge yngre forskare möjlighet att utvecklas.
– Där lägger vi nästan 100 miljoner om året, av våra egna resurser. Så vi lägger mer på
rekrytering av unga än på topprekryteringar.
Ingen tjänst utlystes
Under hösten 2025 bjöd KAW in sina strategiska program att nominera forskare. Fjorton namn kom in – bland dem Togelius och Bismarck, som föreslogs av Chalmers – och stiftelsens vetenskapliga råd valde med stöd av internationella sakkunniga ut åtta som erbjöds paket. Fyra har tackat ja, två har tackat nej och med två pågår förhandlingar.
”Generellt är lön och pension en försvårande faktor när man rekryterar seniora forskare
internationellt”, skriver Wallenbergstiftelsernas kommunikationschef Carina Dahlberg i ett mejl.
Alla fyra rekryterade är män. Av de åtta som erbjöds paket är tre kvinnor – de två som tackade nej och en av de två som förhandlar. Stiftelsen uppger att 43 procent av de drygt 1 000 forskare den finansierar är kvinnor.
Tjänsterna har inte utlysts. Stiftelsen uppger att den inte ställt några krav på hur lärosätena tillsätter dem. Fackförbundet Sveriges universitetslärare och forskare välkomnar att toppforskare värvas till Sverige men är kritiskt till hur det gått till i det här fallet.
– Vi är för en öppen utlysning och skulle föredra det framför ett annat förfarande. Det här är inte helt oproblematiskt, säger förbundsdirektör Lars Geschwind.
Att lärosätena själva skjuter till 30 miljoner per rekrytering gör att det blir en avsevärd satsning, menar han.
– Då får den externa finansiären, i det här fallet stiftelsen, en väldigt stor betydelse för hur ett lärosäte utvecklas strategiskt. Och det finns alltid alternativkostnader – man kunde ha satsat det här på andra forskare eller andra nivåer, säger Geschwind.
Att hans egen tjänst inte var öppen att söka ser inte Julian Togelius som problematiskt.
– Jag hade en lysande tjänst och skulle aldrig ha sökt en öppen position. Öppnar man en
sådan här tjänst får man tusentals sökande med tveksamma meriter.
Flyttlasset över Atlanten går i slutet av året. Göteborg är nytt för hela familjen.
– Vår teori är att det är ett bättre ställe för barnen att växa upp. Funkar det inte så flyttar vi igen.
Wallenberg värvar: 80 miljoner för Sverigeflytt
svd.se/a/4OPwaq/wallenbergs-…
