Everything AI did this year was crazy and none of it was weird

Since Dave and I attended CHAI's annual meeting at Asilomar back in June, I’ve had a split feeling about recursive self-improvement (RSI). We heard about AI improving itself being the goal of the labs while, at the same time, also that AI improving itself as a red line that no one should do. Since then we've had agent swarms, math proofs, and models that run their own experiments. And now METR says Anthropic has doubled its research speed. It's left me with a question I really don’t love. Is this it? Are we about to be subsumed?

But I think the RSI story smuggles in the idea that enough search eventually gets you out of what can be searched. Push the frontier hard enough and it relinquishes itself. I don’t believe that—maybe I’m wrong—but I’ve spent a few weeks trying to work out why I don’t, and that’s the point of this essay.

Then Dwarkesh Patel put out a video called Recursive self-improvement has started. His view now is that we’re a couple of years, possibly months, from AI that’s as good as the best humans at AI research, and after that it’s a coin toss between a software-only singularity and what he calls a spark: three to six years of progress compressed into one.

At the core of his argument is one idea that clashes with my view: that AI research is special. In math the prize is new theory. In AI research, he says, nobody needs a new theory. You take pieces that already exist, try them in a new combination, train a small model, and see if the score gets better. And he expects labs to be able to do that thousands of times a day. So the one thing AI hasn’t shown it can do, ask a new question, is the one thing AI research doesn’t need.

Part of it, I’ll admit, is that I don’t like the feeling. I think human ingenuity is part of what being human is, not a nice-to-have. Watching the math community grieve this year is hard. I’m not ready to conclude that’s what's coming for everyone else. But I think even though we’ve seen crazy stuff, we haven’t seen weird stuff. Or weird enough.

Here’s what I mean by weird.

Adam Mastroianni put a slide up at our Summit. If humans are to remain relevant, we must become stupider, stranger, weirder, riskier, less legible, more open-minded, more idiosyncratic, more human. His point was that we’ve always been more limited by insight than intelligence, and AI is going to speed up everything on the intelligence side and do roughly nothing for insight. I absolutely love this distinction and I believe it still holds when we look at what AI can and cannot do.

So here’s my position and I agree with Adam. We need insight more than we need intelligence, and insight is primarily human. Not only human—I’m not making that claim—but primarily, for two reasons. One is that insight is inexpensive. Einstein didn’t need more data or more compute. He just needed to drop one assumption. The other is that insight is about meaning. Something has to bother you first. Einstein’s 1905 paper opens with a magnet and a coil: physics gave two different explanations for the same current, depending on which one you said was moving, and he couldn’t let that go. The problem mattered to him, so he noticed it, which sent him looking for the assumption underneath. A search doesn’t get bothered. It finds what passes the check it was given and hands the result to someone who decides whether it matters.

What I’m watching for

Of everyone talking about super-human AI capabilities, Demis Hassabis is still the one I watch most. He's remarkably consistent in how he talks about superintelligence and he recently offered up a new test. If you give an AI only what physicists knew in 1905, does it come up with relativity?

Every physicist alive then had the same data and the same theory as Einstein, but something about his brain was different and he stepped out of it all. Nothing in the data said time runs differently for different observers, so nobody was even considering such a weird angle. He dropped that assumption (although he probably never thought of it as anything so ordinary as an assumption) and the idea appeared. That’s insight, not intelligence. Margaret Boden called it transformational creativity: you change the space of possible ideas instead of looking deeper inside it.

Stuart Kauffman has the widest frame on this which he calls the adjacent possible. Whatever exists now, there are things sitting right next to it that could exist next. They are one step away, yet you can’t skip ahead to two steps. And once you take the step, what counts as next door has changed. The first eye made a world where camouflage was valuable. Nobody could have anticipated it, and Kauffman’s claim is that nobody can, ever. The space of what’s possible can’t be written down before you move, because moving is how it comes about in the first place.

So the question for AI is whether it ever actually steps into the adjacent possible the way a biological intelligence does, so that what’s reachable afterwards is different. I don’t think it has. Yet. But this is the year to ask, because swarms and RSI are the first things that look, from the outside, like they might.

What just happened at Anthropic