When AI Builds Itself, What's Left for Us?

Anthropic warned last month that AI may soon be able to design its own successors. I'm grateful someone is saying it plainly—and a little unsettled by how alone they are in saying it.

Somewhere in Anthropic’s new essay on recursive self-improvement, past the charts and the benchmarks, there’s a line from one of the company’s own engineers that I haven’t been able to shake. He describes the strange ache of a day when everything works—when the code runs, the agents do their jobs, and he’s left quietly wondering whether any of it still needs him.

I read that and thought: there it is. Not the cinematic version of AI risk, but the small, human one. The version most of us will actually meet first.

The argument itself is straightforward, and Anthropic makes it without melodrama. For most of AI’s history, humans drove every step of building it. That’s changing fast. More than 80% of the code Anthropic now merges is written by its own model; the typical engineer ships several times the work they did two years ago. Follow that line far enough, the company says, and you reach a system capable of designing and training its own successor—improvement that compounds without us in the loop. They’re careful to say we aren’t there yet, and that it isn’t inevitable. They’re just as careful to say it could arrive before our institutions are ready.

What strikes me isn’t the claim. It’s who’s making it.

This is a company that just filed confidentially for one of the largest IPOs in history, at a valuation approaching a trillion dollars—a company with every commercial incentive to talk about upside and only upside. Instead, it published a sober document explaining why its own product might be the most consequential and hardest-to-control technology in the room. The point, Jack Clark told Axios, was simply to “give people a sense of what’s coming.” That’s leadership—the unglamorous kind, where you raise a hard subject before anyone forces you to.

And it throws into relief how quiet everyone else is.

This same week, the official response came into focus, and I’m honestly not sure it meets the moment. The president signed an executive order asking frontier labs to voluntarily hand the government a look at their most powerful models for up to thirty days before release, so agencies can screen them for security risks. It’s a reasonable first instinct. But I keep turning over the same question: what does a thirty-day review accomplish against a system whose defining trait is that it improves itself faster than we can study it? I want it to be a first step. I worry it’s being treated as a finish line.

Then there’s the other headline. The White House confirmed it’s in talks to take a financial stake in OpenAI—the company would donate equity to seed a “Public Wealth Fund,” and, as the president framed it, “the American public essentially becomes a partner.” A senator has gone further, proposing a 50% government stake in the leading labs.

I want to be fair to this, because the impulse behind it is a good one. If AI is going to generate enormous wealth while displacing the work that wealth used to flow through, then giving ordinary households a direct claim on the upside is a serious answer to a serious problem. I don’t want to wave that away.

But I can’t ignore the tension underneath it either. A government that becomes a shareholder in the companies reshaping the workforce, the classroom, and the professions now holds a financial interest in that reshaping succeeding. The body meant to referee the game becomes a player with a position in it. And in that arrangement, who is left to speak for the people on the other side of the disruption—the ones whose jobs and schools are the thing being reorganized?

Here’s the detail that stayed with me. According to the reporting that broke the story, Anthropic isn’t part of those equity conversations. The lab raising the alarm is, so far, also the one not lining up for a government ownership stake. Make of that what you will. I found it clarifying.

So what does this actually mean for us—for the engineer on his too-smooth afternoon, and for the rest of us watching from outside the lab?

Here Anthropic offers something I found genuinely steadying, almost in passing. Even if model development goes fully automatic, they note, most of life won’t. There are things more intelligence simply cannot rush. A drug still has to be lived with for years before we truly know what it does. An election still arrives when the calendar says it does. A stranger still doesn’t become an old friend by the end of a weekend.

I spend my days thinking about how AI fits into healthcare, and that rings true in the most concrete way. I can imagine a model designing a better clinical trial overnight. I cannot imagine it compressing the years it takes to learn, in a real body over real time, that a therapy is safe. The lab may run at the speed of compute. Care still runs at the speed of trust.

That gap—between a recursive intelligence improving itself upstream and the slow, stubbornly human world downstream—is where I think the next few years actually get decided. Not in whether the curve bends, but in what we choose to do while it hasn’t yet.

I don’t think the answer is to pretend we can stop any of this. I agree with the realists: we are going to get there. What I want—what I think this week should make all of us want—is more of exactly what Anthropic just modeled. Tell people the truth early. Build the verification tools that would let a coordinated slowdown actually mean something. Bring the public into the deliberation now, while “recursive self-improvement” is still a phrase we’re explaining ahead of time, rather than one we’re apologizing for after.

For the moment, one company is doing the hard, honest talking. I’m grateful for it.

I’d just feel a great deal better if it had some company.