Although the details are different and the time frame somehow even more aggressive, Aschenbrenner’s trend-line arguments and conclusions are quite similar to what Ray Kurzweil argued in The Singularity is Near (2005). I’d go so far as to characterize the arguments in Situational Awareness as neo-Kurzweilean with a geopolitical twist. It’s strange to me that Aschenbrenner doesn’t even seem aware of the Singularity / “Rapture of the Nerds” debates from the mid-2000s.
> When he says “we just need to teach the model a sort of System II outer loop”, there’s an awful lot hiding in that “just”. The entire argument is full of handwaves like this.
The key impression I got from all this is that Leopold has access to the private conversations in top-tech companies, the off-the-record discussions, and most importantly the mood. My conclusion: they are not discouraged in any way. We might be discouraged about the lack of agents right now and the huge challenge of System II, they are not.
And that can only be because they have two or three things working now, two or three things to try next to improve that, and two or three things to try after that. That's the way I get optimistic on a problem, when the solution tree is boundless. I think Leopold's optimism reflects the optimism of the top engineers at the top AI companies.
What I suspect is going on is that OpenAI and others are already sitting on huge discoveries that they would prefer never see the light of day. Secrecy and closed AI is quickly becoming the new norm, Leopold was just a bit early seeing (and pushing) the importance of that.
(This is of course very disturbing to me that so much important technology advance could be done in complete secret for the next decade or more, leaving us in the dark about the true capacities of AI models.)
On System II unhobbling was it just handwaving or did he, in fact, give away a bit of why this might not be such a large problem? Consider:
From the website:
"But unlocking test-time compute might merely be a matter of relatively small “unhobbling” algorithmic wins. Perhaps a small amount of RL helps a model learn to error correct (“hm, that doesn’t look right, let me double check that”), make plans, search over possible solutions, and so on. In a sense, the model already has most of the raw capabilities, it just needs to learn a few extra skills on top to put it all together. "
From the interview:
"In the short timelines AI world, it’s not that hard. The reason it might not be that hard is that there are only a few extra tokens to learn. You need to learn things like error correction tokens where you’re like “ah, I made a mistake, let me think about that again.” You need to learn planning tokens where it’s like “I’m going to start by making a plan. Here’s my plan of attack. I’m going to write a draft and now I’m going to critique my draft and think about it.” These aren’t things that models can do now, but the question is how hard it is."
From the interview:
"What humans do is a kind of in-context learning. You read a book, think about it, until eventually it clicks. Then you somehow distill that back into the weights. In some sense, that's what RL is trying to do. RL is super finicky, but when it works it's kind of magical."
I submit that this confidence comes from secret conversations and direct experience: the AI companies are doing RL on planning tokens.
> For my part, both paths look perilous, so I hope to hell that ASI is a lot farther away than 2030.
Agreed. While taking Leopold's predictions seriously could be dangerous, ignoring them could be dangerous too. We need more time!
For many decades, we've been hearing about impending breakthroughs in cancer treatments and anti-aging therapies. There has been incremental progress, but people still die of cancer and they still get old.
Tesla claimed they would create a breakthrough in electric vehicles, and they were right.
I will be very surprised if OpenAI or their peers are sitting on one or two tricks that will blow AGI wide open. I think the problem is just inherently more complex than that. We'll see! I'd hate to bet the fate of humanity on it, and yet we seem to be doing exactly that.
> And that can only be because they have two or three things working now, two or three things to try next to improve that, and two or three things to try after that. That's the way I get optimistic on a problem, when the solution tree is boundless.
The solution tree for aging is also boundless.
"there are only a few extra tokens to learn" -> this is sweeping so much under the rug. It would be like saying "to make it in the NBA, I just need to be good at passing, shooting, and guarding". Still, we'll learn a lot in the coming years!
In my grasp of the evidence, my main personal concern is confirmation bias: yes, I would like to see AGI cracked and therefore it's going to be easy to fool myself into thinking I'm seeing signs of that.
Leopold is young, never met a problem he couldn't solve I'm sure, so feels invincible. Not clear how long or deeply he's grappled with his own biases.
My theory that AI labs will be increasingly secretive, protective, holding each new discovery close to the chest is also consistent with DL hitting a scaling wall and the search tree of promising solutions yielding only withered leaves, true.
But where I think the evidence comes down on the side of AGI in our decade is that deep-neural-networks were solely unlocked ("bitter lesson") by growth in computation power (AlexNet in 2012). And since then, further compute advancements, massive investment, and more and more of the world's brightest minds have accelerated performance and scale of DL far beyond Moore's law (https://epochai.org/trends#compute-trends-section). This juggernaut I think is completely unprecedented in human history and it feels to me like it won't be slowing down.
I agree that the ongoing success of scaling is the strongest argument in favor of the idea that AGI may not be far off. Extraordinary claims require extraordinary evidence; near-term AGI is an extraordinary claim; the sustained returns to scale are our primary candidate for extraordinary evidence.
Steve- I never thought that situational awareness would be something we talk about in the context of technology, let alone life, so this is very informative. I appreciate you sharing. Hope you're well this week. Cheers, -Thalia
Although the details are different and the time frame somehow even more aggressive, Aschenbrenner’s trend-line arguments and conclusions are quite similar to what Ray Kurzweil argued in The Singularity is Near (2005). I’d go so far as to characterize the arguments in Situational Awareness as neo-Kurzweilean with a geopolitical twist. It’s strange to me that Aschenbrenner doesn’t even seem aware of the Singularity / “Rapture of the Nerds” debates from the mid-2000s.
See also https://www.dwarkeshpatel.com/p/leopold-aschenbrenner
> When he says “we just need to teach the model a sort of System II outer loop”, there’s an awful lot hiding in that “just”. The entire argument is full of handwaves like this.
The key impression I got from all this is that Leopold has access to the private conversations in top-tech companies, the off-the-record discussions, and most importantly the mood. My conclusion: they are not discouraged in any way. We might be discouraged about the lack of agents right now and the huge challenge of System II, they are not.
And that can only be because they have two or three things working now, two or three things to try next to improve that, and two or three things to try after that. That's the way I get optimistic on a problem, when the solution tree is boundless. I think Leopold's optimism reflects the optimism of the top engineers at the top AI companies.
What I suspect is going on is that OpenAI and others are already sitting on huge discoveries that they would prefer never see the light of day. Secrecy and closed AI is quickly becoming the new norm, Leopold was just a bit early seeing (and pushing) the importance of that.
(This is of course very disturbing to me that so much important technology advance could be done in complete secret for the next decade or more, leaving us in the dark about the true capacities of AI models.)
On System II unhobbling was it just handwaving or did he, in fact, give away a bit of why this might not be such a large problem? Consider:
From the website:
"But unlocking test-time compute might merely be a matter of relatively small “unhobbling” algorithmic wins. Perhaps a small amount of RL helps a model learn to error correct (“hm, that doesn’t look right, let me double check that”), make plans, search over possible solutions, and so on. In a sense, the model already has most of the raw capabilities, it just needs to learn a few extra skills on top to put it all together. "
From the interview:
"In the short timelines AI world, it’s not that hard. The reason it might not be that hard is that there are only a few extra tokens to learn. You need to learn things like error correction tokens where you’re like “ah, I made a mistake, let me think about that again.” You need to learn planning tokens where it’s like “I’m going to start by making a plan. Here’s my plan of attack. I’m going to write a draft and now I’m going to critique my draft and think about it.” These aren’t things that models can do now, but the question is how hard it is."
From the interview:
"What humans do is a kind of in-context learning. You read a book, think about it, until eventually it clicks. Then you somehow distill that back into the weights. In some sense, that's what RL is trying to do. RL is super finicky, but when it works it's kind of magical."
I submit that this confidence comes from secret conversations and direct experience: the AI companies are doing RL on planning tokens.
> For my part, both paths look perilous, so I hope to hell that ASI is a lot farther away than 2030.
Agreed. While taking Leopold's predictions seriously could be dangerous, ignoring them could be dangerous too. We need more time!
For many decades, we've been hearing about impending breakthroughs in cancer treatments and anti-aging therapies. There has been incremental progress, but people still die of cancer and they still get old.
Tesla claimed they would create a breakthrough in electric vehicles, and they were right.
I will be very surprised if OpenAI or their peers are sitting on one or two tricks that will blow AGI wide open. I think the problem is just inherently more complex than that. We'll see! I'd hate to bet the fate of humanity on it, and yet we seem to be doing exactly that.
> And that can only be because they have two or three things working now, two or three things to try next to improve that, and two or three things to try after that. That's the way I get optimistic on a problem, when the solution tree is boundless.
The solution tree for aging is also boundless.
"there are only a few extra tokens to learn" -> this is sweeping so much under the rug. It would be like saying "to make it in the NBA, I just need to be good at passing, shooting, and guarding". Still, we'll learn a lot in the coming years!
In my grasp of the evidence, my main personal concern is confirmation bias: yes, I would like to see AGI cracked and therefore it's going to be easy to fool myself into thinking I'm seeing signs of that.
Leopold is young, never met a problem he couldn't solve I'm sure, so feels invincible. Not clear how long or deeply he's grappled with his own biases.
My theory that AI labs will be increasingly secretive, protective, holding each new discovery close to the chest is also consistent with DL hitting a scaling wall and the search tree of promising solutions yielding only withered leaves, true.
But where I think the evidence comes down on the side of AGI in our decade is that deep-neural-networks were solely unlocked ("bitter lesson") by growth in computation power (AlexNet in 2012). And since then, further compute advancements, massive investment, and more and more of the world's brightest minds have accelerated performance and scale of DL far beyond Moore's law (https://epochai.org/trends#compute-trends-section). This juggernaut I think is completely unprecedented in human history and it feels to me like it won't be slowing down.
I agree that the ongoing success of scaling is the strongest argument in favor of the idea that AGI may not be far off. Extraordinary claims require extraordinary evidence; near-term AGI is an extraordinary claim; the sustained returns to scale are our primary candidate for extraordinary evidence.
https://amistrongeryet.substack.com/p/the-ai-progress-paradox was my attempt to reconcile the ongoing returns to scale with the idea that additional breakthroughs will be needed to achieve AGI.
Steve- I never thought that situational awareness would be something we talk about in the context of technology, let alone life, so this is very informative. I appreciate you sharing. Hope you're well this week. Cheers, -Thalia