Anthropic warns that it’s very close to RSI capability, aka self-building AI

Most of the attention on Anthropic's amended S-1 went to the money: revenue is climbing fast, and so are costs. The more consequential lines are in the risk factors. The company now says it has already used AI to build models whose jumps in capability it didn't recognize until afterward. It also says its internal estimates suggest it may soon hit its own Responsible Scaling Policy threshold for progress dramatically sped up by AI.

That threshold is the company's working version of recursive self-improvement, where AI gets good enough at building AI that the pace of progress takes off. Anthropic says no current model has crossed it. It also says it doesn't expect to safely let Claude design and build later frontier generations on its own without meaningful human oversight, and that it's weighing policy changes to keep humans in control.

When an idea that many people dismissed as a buzzword earlier this year shows up as a disclosed risk in an IPO filing, the conversation has changed. This is no longer a thought experiment. It's a statement a company has to stand behind to investors.

I want to take this seriously and still keep both feet on the ground. Seriously, because of where it's written. An S-1 isn't a blog post or a conference keynote. Lawyers read every word, and an admission that the company has missed its own acceleration before, and may again, is a strange thing to include if the only goal were marketing. It reads like a company trying to get on the record before something happens.

The skeptics have a point, too, and TechCrunch makes it: an S-1 is both disclosure and sales pitch. Saying your models are close to building themselves is also a way of telling investors you're near the frontier. Both readings can be true. Our read is that the disclosure is sincere and it is also useful to Anthropic. Neither of those cancels the other.

The line that worries me most isn't the threshold. It's the admission that leaps in capability were noticed only after they happened. Every safety framework the labs have published assumes you can measure a capability before you deploy it or build on it. If the measuring tools lag the thing being measured, the framework becomes a record of what already happened instead of a gate. That's the real risk here if it scales: not a sudden robot takeover, but a steady loss of the ability to say what the latest model can actually do.

There is a strong case for the useful version of this. AI that speeds up research, catches bugs, and helps engineers move faster is the reason a lot of people are building it in the first place. We don't think the answer is to stop labs from using their own tools. The answer is checks that keep up with the speed: independent evaluations, honest reporting when a threshold is close, and a human sign-off that means something rather than just a step in a checklist.

Anthropic says it's considering changes to its own policy to keep humans in charge. That's welcome, and it's also voluntary. A company policy can be revised by the same company that wrote it, under the same commercial pressures this filing describes. If the leading labs really believe they're near this line, this is the moment for outside oversight with real teeth, before the first model is officially designated as past the threshold.

Anthropic's Responsible Scaling Policy sets capability thresholds that trigger stronger safeguards. One of those covers dramatically accelerated progress attributable to AI. The new language came in an amended S-1 filed Monday as the company moves toward a public listing. Claude's coding, math, and cybersecurity abilities have improved sharply this year, which is part of why the claim is getting attention.