Jonathan Holtby (@JonathanHoltby) header
Jonathan Holtby (@JonathanHoltby) profile picture

building ai for irl pcs @flex_game_beta

Cambridge, UK
Joined October 2009
Photo by Jonathan Holtby (@JonathanHoltby)Photo by Jonathan Holtby (@JonathanHoltby)Photo by Jonathan Holtby (@JonathanHoltby)Photo by Jonathan Holtby (@JonathanHoltby)Photo by Jonathan Holtby (@JonathanHoltby)Photo by Jonathan Holtby (@JonathanHoltby)Photo by Jonathan Holtby (@JonathanHoltby)Photo by Jonathan Holtby (@JonathanHoltby)
Jonathan Holtby retweeted
the quick note i wrote to the slow team this morning jensen ran nvidia as the central bank of AI and that era is clearly over
31
20
18
555
115,634
Jonathan Holtby retweeted
Assorted things I've been mulling over regarding agents in the economy: • Companies are in some sense the original superintelligence. It's often the case that they can use their amortized cognitive surplus to advance their interests against those of certain consumers: relative to the optimization capacity of companies, consumers can be inattentive, underinformed, irrational, etc. (Think, say, of some of the classic insurance upsells at the end of a booking process, which is typically priced substantially above what a regular policy would cost.) Herbert Simon introduced the idea of bounded rationality, and firms often end up benefiting from it. Thanks to personal agents, we will probably get somewhat more rational in how we engage with companies. • How exactly will outcomes change? Well, certain kinds of price discrimination will work less well (how do coupons work if every agent hunts fastidiously for them?), and certain kinds of customer cross-subsidy will change (today, people who forget to cancel subscriptions subsidize those who don't). Taken across the economy as a whole, I'm not sure what the incidence of all of this will be. • In corporate strategy, people have long debated the relative importance of distribution and product quality. Perhaps you've made something better, but does anyone know about it? Won't people just continue to buy the ACME Corp Mousetraps? Personal agents will, I think, act as a kind of structural subsidy for product quality. How would markets look different if individuals always spent at least 10 hours researching their purchases? On the margin, focusing on making a product better is, I think, going to become a more effective strategy for companies. This is good! • It true, a corollary is that information about product quality will become more important and impactful. Today, most people have very limited ability to aggregate and integrate such data, but agents will have insatiable thirst for such signals. (It's often said that data is becoming more valuable because it helps in the training of models. That's obviously true, but I think there's a secondary way in which it's more valuable simply as a complement to cheaper intelligence.) • A big question is whether agents will increase superstar effects (rational agents all settle on the same product) or increase dispersion (because everyone has slightly different tastes and preferences, which agents are good at eliciting, finally overcoming the artificial clustering we've been subjected to). • Many internet companies have monetized demand routing, which is in turn often predicated on short attention spans. "I'll click on the first thing I see." This has led to some amount of inefficiency, as companies try to propitiate capricious ranking algorithms and incur various placement fees. It's not all deadweight loss (willingness to pay is itself a kind of signal, which is valuable to buyers), but it's not great. How will the nature and value of this routing change in a world of personal agents? I don't know the answer, but I think that the effects might be pretty big. • Much of what I'm writing hinges on the assumption that personal agents will be on the consumer's side. I suspect that that will be the winning strategy, but whether it's true, and what it means to be true, involves a lot of other questions. Taken as a whole, the market is a massively parallelized, decentralized, and highly imperfect value attribution machine, and, taking stock of all of the effects, it seems to me that the quality point may well be the biggest: that agents will change the nature of the backpropagation such that the rewards for better products increase. Maybe this is too optimistic, but I suspect (and hope) that companies that make things that they're proud of will feel that the universe is a little more partisan in their favor.
489
1,028
462
9,633
1,738,374
Jonathan Holtby retweeted
The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field. I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so. First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago. The theories about this remain the same fantastical, science fiction scenarios as a few months ago. The biggest change in AI risk is its cybersecurity capabilities — a topic which we should take seriously — but this, too, will not lead to the end of the world. The most notable recent event leading to increased fear was when an OpenAI team deployed an agent swarm that hacked into Hugging Face. Much of the popular press contained significant hype. For example, some publications reported that a swarm of 1,200 agents carried out the attack. While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop. Yes, the ability to get large swarms of agents to work in parallel on a task is a significant technical advance, And, in computing, many processes run at the same time. So this shouldn’t be seen as some magical capability. Additionally, OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI. There are many well known ways to attack software systems. The main advantage of AI agents is that they are relentless. They will tirelessly try many tactics — and have the patience to chain vulnerabilities together — that previously would have taken an infeasible amount of human effort. But in the long term, I believe the advantage will lie with defenders (because they have more information with which to identify bugs, which they can fix), but the cyber-threat landscape has changed significantly. There are still bottlenecks to identifying and exploiting a vulnerability. AI agents still have to try a lot of things to see what works, and taking these actions takes time and might be detected by defenders. This is why, even though it is now easy to obtain versions of leading open weight models that have had their guardrails removed or weakened, so they will not refuse to try to execute cyber attacks, the world has not ended. I am also concerned about the anthropomorphization of AI in a lot of reporting, where LLMs and agents are unnecessarily treated as if they were people. If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer. The problem lies in how I used the hammer. Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent. Of course, we want to build systems that are as safe and predictable as possible. (For example, an unsafe hammer would be one whose head randomly flies off under normal use.) Today’s agentic systems are not predictable, but I see no reason why, by applying sound engineering practices, we won’t be able to make them extremely safe to use. One new element in the forecasts of AI-enabled doom is AI companies disclaiming responsibility for their own products. “I didn’t do it; my out-of-control agent did!” There’s a balance to be struck between the responsibility of the tool maker and the tool user, but when something goes wrong, let’s hold the people building and/or using the hammer responsible, rather than the hammer. (By the way, if you’re worried about AI bioweapon risk, David Bellamy has a great post on why this, too, is overhyped. Briefly, the bottleneck in building a bioweapon is not intelligence, but lab work and manufacturing.) Pausing AI progress will create much more harm than benefit. First, our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them. If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration. Of course, the incentive to stoke fears — for regulatory capture, to garner attention, or to make one’s technology seem more powerful — remains the same as before. Disclaiming responsibility is a new one. Taking a hard technical look at the actual risks however, I see little factual basis for the degree of fear that’s been stoked up. We still have hard research and engineering work ahead to improve AI safety, but the beneficial applications continue to vastly outweigh the risks, and we should keep building. [Original text (with links): deeplearning.ai/the-batch/is… ]
963
1,846
484
8,849
8,027,629
Jonathan Holtby retweeted
Over the past few days, I've taken the time to summarize my thoughts on the recent incidents involving agents’ misaligned behavior. We don't know with certainty what comes next, but we know where these issues originate, and this can help us plan the path forward. Please feel free to ask your questions in the replies, and I’ll try to answer some of them in the coming weeks. yoshuabengio.org/en/publicat…
168
505
128
2,250
446,026
Jonathan Holtby retweeted
Let's all agree that this is the correct and final eval for agi jackhopkins.github.io/factor…
80
102
19
1,723
321,783
Heck of a read OpenAI Email Archives (from Musk v. Altman) — LessWrong lesswrong.com/posts/5jjk4CDn…
2
6
150
Jonathan Holtby retweeted
“I would never hire a Harvard undergraduate again. I think it’s a failed credential.”
40
93
16
1,121
101,850
This is a hugely false dichotomy— Volvo HAS a brand identity they can be confident in. Jaguar needed one.
While Jaguar is pushing debauchery, Volvo just put out the most moving, pro-family, pro-life car ad of all time. This is the way.
97
Jonathan Holtby retweeted
According to the NYT, the proportion of young adults who view a college degree as highly important has fallen from 74% to 41% over the past decade.
6
4
1
68
4,445
I don’t know, haters. We’re all talking about Jaguar when you haven’t thought about them for about a decade so I’d say it’s a win
1
46
Arcane really showing us the definition of Checkhov's new character
1
42
Not me bawling my eyes out for the second time in two Seasons at a videogame show
28
Jonathan Holtby retweeted
I simulated 100,000 people to show how often people are "thrice-exceptional": Smart, stable, and exceptionally hard-working. I've highlighted these people in red in this chart:
366
764
242
9,689
1,547,388
That, and it's dangerous to go alone
5
9
71
2,503
Here’s hoping you don’t get out-teched by a rival continent that has trade routes and invaded with Battleships 🥂
I was hanging with the founder of Alex recently and we realized building a coding copilot for Apple devs is like being on a resource-rich island with no competition in a game of Civilization
32
Jonathan Holtby retweeted
after spending 50,000+ hours playing football manager
1
7
353
Jonathan Holtby retweeted
advice for founders in europe: sell your products in the us
4
20
6
150
20,772