Teo Ruiz

Loved this conversation between @dwarkesh_sp, @fchollet and @mikeknoop! youtu.be/UakqL6Pj9xo Highly recommend listening to the full episode, but here are a few interesting notes and thoughts: (1) It’s fascinating to see how much experts in AI/ML disagree not only about the future (e.g. how much room there is for further scaling of LLMs, Hinton/Altman/Sutskever vs LeCun, Gary Marcus etc) but also about the capabilities of LLMs *today*. Dwarkesh shared an anecdote of talking to experienced ML researchers (who didn’t know about ARC) and showing them an example ARC problem. They said, 'of course, an LLM would be able to solve something like this’. They took a screenshot of it and put it into ChatGPT – it didn’t get the pattern. (2) François argues that LLMs are not intelligent. No matter how intelligent e.g. ChatGPT appears to be – it’s all just pattern-matching and memorization. If you look at some of the remarkable achievements of ChatGPT (e.g. interpreting an essay that it has never seen before; learning a language based on a grammar book and a dictionary alone; …), it’s hard to imagine how it can do this without intelligence. However, the size of these models is so far beyond what we can grasp that we can’t trust our imagination here. The counter-argument is that humans (in general, not only VCs 🤣) rely on a lot of pattern-matching, too. Here’s where ARC comes in: These puzzles are designed to be easy for humans, but hard for AI (see arcprize.org). (3) Another argument against François’ position might be that it’s anthropocentric. Machine intelligence will look very different from human intelligence, so let’s not theorize too much and just look at the results. If a computer can do lots of tasks that previously required human intelligence … well, then I guess the computer is intelligent. Let’s give the machines the benefit of the doubt instead of continuously moving the goalpost. ;-) A corollary of the tight definition of intelligence is that if or when a machine can do the job of, say, a junior lawyer or junior developer, while not being intelligent based on François’ reasoning, that job apparently doesn’t require intelligence. (4) It may seem like this is all just semantics, but there are some real implications because the level (or absence) of intelligence of LLMs will tell us something about the speed of future progress in AI (obviously a hot topic in SaaS these days). It should also have an impact on how much research should go into other AI areas outside of LLMs. Quoting François: “OpenAI basically set back progress towards AGI by quite a few years, probably like 5-10 years. […] They also triggered this initial burst of hype around LLMs. Now LLMs have sucked the oxygen out of the room. Everyone is just doing LLMs. I see LLMs as more of an off-ramp on the path to AGI actually. All these new resources are actually going to LLMs instead of everything else they could be going to. If you look further into the past to like 2015 or 2016, there were like a thousand times fewer people doing AI back then. Yet the rate of progress was higher because people were exploring more directions.” (I don’t agree with this but it’s a refreshing perspective and great food for thought.)