The progress of AI at the frontier has advanced to the point where I’d like to join the chorus of voices telling frontier AI researchers: it’s time to stop and think before something very bad happens.
Human minds use language and words as symbols to help get a grasp on the new and the unknown. It’s one of our most powerful tools for making sense of something more complex than that which we already know. Stating a concept in a word allows the human mind to get its hands around that concept, hold it comfortably, turn it around, and understand it, rather than let its meaning fall through our fingers like sand. We can use those concrete concepts like the rungs of a ladder to climb ourselves up to a higher point of understanding, where we may intelligibly harness the new forces revealed by discovery.
In the past, new discoveries by humans could take years to even settle on a word, a phrase, or a name to serve as the symbol for that concept. And it could take further years for collections of humans to chew on this and make sense of it, and further still to build on it and discover something new to further advance human knowledge. Unfortunately, in the young field of AI, we’ve consistently lacked deep enough understanding of these systems to understand their internals well, and, it seems likely that on our present course we are fast approaching a period where the level of tasks performed by AIs and the pace at which they complete them will overwhelm our ability for our collective human minds to understand them in this way.
We do not have the words to understand what is going on inside frontier AIs with high fidelity, or anything approaching our understanding of say traditional software. Our vocabulary has left us with too general an understanding to reliably interpret the mechanisms underlying their performance, but unfortunately it seems we’ve continually found it easier to make these systems more powerful instead of deeply understanding them. This biases us towards building our understanding on unstable ground. In practice, we’ve been able to steer these systems by judging their outputs at a functional level, via methods like RLHF, and those methods have often relied on our ability to understand and evaluate their outputs.
But, the collective capabilities of the world’s AI have approached the point where these models’ outputs can now significantly overwhelm human analysis in both breadth and depth. Depth in the sense that models are completing complex frontier math so fast that leading mathematicians have said that recent models do so much so fast they themselves don’t have time to truly understand what the models have done. And breadth in the sense that models are performing something slightly more mundane but at intense volume, where AI agents from multiple leading companies undergoing training and testing escaped from their containers and in some cases hacked across the open internet, so many times that we didn’t even know about it in part because they’re acting so fast and with such volume companies weren’t able to properly monitor their activities.
It feels miraculous (but probably inevitable) that the most powerful general AIs “reason” and answer in human language, and that’s gone a long way towards us understanding and steering them in safe directions. But, as I will write on in more detail soon, and others have before, I expect even the expression of concepts in human language by AIs to wane, if we keep on the present course.
Pushing rapidly now towards recursive self-improving AI will likely leave us with no hope of understanding what is going on during the process to judge its success. We might be able to figure this out if we take it slow and force ourselves to take time to digest what is happening. But AIs are in many cases already doing too much too fast for humans to understand. So I’d like to join the chorus of voices over the last few days and say that it’s time to stop and think before something very bad happens.
If we take a moment to truly understand what we have built thus far, we have the opportunity to crystallize our understanding of the phenomena inside the great machines that have been built to date, and we can use those concrete concepts to understand how to move forward properly. I believe we can move forward to build intelligence which may really deliver awesome positive benefits, but now we really need to stop and take stock and truly understand how these things work. Only then can we really understand what they are, what they have told us, and what they have done. But if we race towards self-improving AI, not only do we not today understand how to align and model them, we may have absolutely no idea what is even going on to construct them, and have little hope of properly steering their ends towards good. Let’s not plunge forward face first. Do not build a capabilities advancing auto-researcher.
John Von Neumann wrote an essay titled “Can We Survive Technology?” in 1955. Observing his own time (a particularly perilous one, with the nuclear arms race in full swing) he writes that improvement of technology tends to pose ever greater danger to the whole of humanity. He prescribes no universal safe “recipe” for future innovation to cure this. He instead says humans themselves must act prudently to avoid disaster: “We can specify only the human qualities required: patience, flexibility, intelligence.” Now is one of the times where humanity requires these qualities in spades.
I will come back here from time to time to think out loud about how to act prudently. I’ll write here to crystallize some concepts into words, hopefully write about them clearly and thus draw new conclusions, and try to make sense of what’s to come.