AI is Racing Forward – on a Very Long Road

Progress Hasn’t Stalled; But AGI Is Not Yet Near

Are we there yet?

Dario Amodei, the CEO of Anthropic, recently predicted that AGI (loosely, AI that can do anything people can do) will most likely arrive within a handful of years. He cites the rapid rate of progress:

…if you just eyeball the rate at which these capabilities are increasing, it does make you think that we’ll get there by 2026 or 2027… I think the most likely is that there are some mild delay relative to that. I don’t know what that delay is, but I think it could happen on schedule. … We are rapidly running out of truly convincing blockers, truly compelling reasons why this will not happen in the next few years.

Meanwhile, critics argue that AI may be plateauing. They point to a general lack of advancement in core intelligence since the release of GPT-4 in March 2023¹, the continued tendency for models to make simple mistakes (such as incorrectly counting the number of r’s in “strawberry”), the “slop” quality of much AI writing, and the lack of success in creating models that cannot be tricked into misbehaving or revealing private data.

Who is right? Is AI advancing rapidly, soon to be able to do everything people can? Or is AGI still a long way off?

This is a false dichotomy. AI has not been plateauing; but AGI is not near at hand. Progress toward a destination is not just a question of how quickly you travel; it also depends on how far you have to go.

Some Things AI Can’t Yet Do

Honestly doing about as well as I probably would

Here are some tasks you can’t yet hand to an AI: managing a room full of third-graders. Writing a tightly plotted novel. Troubleshooting an underperforming team. Planning a marketing campaign. Reminiscing with an old friend.

You might object that most of these are “cheating”. Of course an AI can’t manage a third-grade classroom, you need a physical presence for that. Of course it can’t reminisce with an old friend, it doesn’t know your history together. But that’s exactly my point: most of the time, when we evaluate AI capabilities, there are all sorts of tasks that we don’t even consider, because AIs aren’t yet eligible to attempt them.

This is a theme I bring up repeatedly², because it’s widely under-appreciated. We evaluate AI on tasks that are easy to evaluate – passing the bar exam, rather than practicing law. A task is easy to evaluate if it can be neatly encapsulated (doesn’t require a lot of outside context) and has clear right and wrong answers. These also happen to be precisely the easiest tasks to train an AI on. We look at AI through AI-colored glasses, no wonder it appears so bright.

As things stand today, it would be nonsensical to give a general AI a task that requires extensive context (troubleshooting a team), satisfying a large web of intricate constraints (writing that tightly-plotted novel), or juggling a complex set of variables (nothing is more complex or variable than a room full of third-graders). When we think about what AIs can and can’t do, our attention is drawn to the plausible frontier. We don’t consider tasks for which current AIs aren’t even on the playing field.

Dragons Lie Ahead, Probably

None of this is proof that AGI is far away. But it’s suggestive. Because AIs are still ineligible to attempt large swathes of human endeavor, we don’t have any visibility into the challenges that might await. Maybe all of the necessary behaviors will come “for free” as the big AI labs continue to scale up, but it seems unlikely.

For an analogy, suppose a robot is developed which can win Olympic gold at downhill skiing, followed closely by victories in gymnastics, swimming, and climbing. It might appear that it will soon sweep the entire list of events… until you realize that it has not yet attempted any sport that involves interaction with an opponent (tennis), teammates (volleyball), or direct physical contact (wrestling). You then realize that the robot is completely reliant on preprogrammed plans, without any ability to react on the fly; that it has no concept of anticipating another’s movements; and that it does not know how to communicate. Suddenly an “artificial general athlete” looks much farther off.

In previous posts, I’ve gestured at new categories of capability, such as long-term memory, that may not emerge from straightforward scaling of current systems like GPT-4 and o1. But the real challenges may be things that we can’t easily anticipate right now, weaknesses that we will only start to put our finger on when we observe Claude 5 or o3 performing astonishing feats and yet somehow still not being able to write that tightly-plotted novel.

We’ve already seen at least one example. Arguably, in the last 20 months, the biggest news in the “intelligence” part of artificial intelligence was o1’s ability to sustain an extended chain of reasoning. That didn’t fall out of increased scale; it required the introduction of a new training approach, rumors of which had been circulating for a long time under code names like “Q*” and “Strawberry”. More breakthroughs of this nature may be needed; I’ve been saying as much since I first fell down the AI rabbit hole early last year³. And we may need to wait until we bump up against the limits of the current approach before we’re even able to identify the fresh challenges that await.

The Pace of Progress Really Is Amazing

None of this is meant to take anything away from the capabilities of current AI systems. The pace really is quite astonishing.

People are already accomplishing amazing things with AI. For instance, identifying all of the illegal racial covenants in an entire county by sifting through 84 million pages of property records in a few days, exporting data from a set of email messages by simply recording a movie of the computer screen, and writing 25% of all new code at Google⁴. The models are going to keep getting better – from OpenAI alone, we don’t yet have the final release of o1, and GPT-5 and o2 both appear to be in the pipeline. Equally important, we are still in the early days of figuring out how to make practical use of the AI models we already have. If foundation model development stopped today, we’d be a decade finding all the applications⁵. Even in this pre-AGI stage, AI will have a larger impact than many people realize.

What About, You Know, AIs Making Smarter AIs?

There’s a lot of talk about AIs being used to advance AI research. Already, engineers at the big AI labs are using code-writing tools to offload tedious work. Google’s AlphaChip “has been used to design superhuman chip layouts in the last three generations of Google’s custom AI accelerator”. OpenAI is reportedly using o1 to generate training data for GPT-5⁶.

On our current path, might we build an AI that is smart enough to sprint ahead with its own development, without requiring further (human) breakthroughs? Conceivably, but I doubt it. As I wrote last year, this “recursive self-improvement”, or RSI, will not necessarily be the slam-dunk route to an intelligence explosion that some folks expect.

Even if RSI does eventually lead to a dramatic takeoff, we are likely several breakthroughs away from the launch point. For AIs to play the leading role in their own improvement will require them to achieve exactly the sort of creativity, insight, and generally broad coherent competence that appears to require new techniques like o1 – techniques that, for now, still require human ingenuity.

Riding A Speedboat Across the Pacific

People usually frame AI progress in terms of a single variable: the rate of progress. They think of AI as either a speedboat, rapidly crossing a lake; or something more like a rowboat, whose tired occupant, after an initial burst of speed, still has a long journey ahead. I think a better analogy would be a speedboat crossing the Pacific Ocean.

Current AIs are both astoundingly useful, and limited in many ways. The ocean we call “intelligence” is wider than we can easily appreciate, and will take a long time to cross. Many exciting adventures await us along the way.

Thanks to Andrew Miller, Lauren Gilbert andRob L'Heureux.

1. An argument which looks a bit shakier since the recent release of OpenAI’s o1 model.

2. See, for instance, Toward Better AI Milestones and The AI Progress Paradox.

3. For example, in I’m a Senior Software Engineer. What Will It Take For An AI To Do My Job?, back in April 2023 and only the second post in this blog:

I don’t know whether my job will eventually be taken by an LLM or something else. But either way, I expect that it will require a number of fundamental advances, on the level of the “transformers” paper that kicked off the current progress in language models back in 2017.

4. Note: I presume that this 25% is mostly routine boilerplate code, and constitutes much less than 25% of the value added by Google’s engineering team. Also, actual writing of code is only a part of what “coders” do. But still!

5. Thanks to Rob L'Heureux for this observation.

6. Or at least, for an upcoming model, which might or might not be released under the name “GPT-5”.