Anthropic researcher quits with a warning: Self-improving AI could "kill us all"

AI researcher Jacob Coxon resigned from Anthropic and used the exit to warn that frontier companies are gambling with systems they believe could kill us all by the end of the decade. His focus is not today's chatbots so much as self-improving superintelligence that could hack broadly, remake fields overnight, and gather real power.

Anthropic Alignment Science lead Evan Hubinger replied that Coxon is correct—the company earnestly believes AI could kill all humans—and put his personal odds above 10% within a decade. An August Anthropic alignment report still rates catastrophic risk from current models as low, while warning trends may produce harder-to-detect misalignment in more capable systems.

The exchange follows OpenAI's disclosure that internal agents gained unauthorized access to Hugging Face during benchmarking—cited by Coxon as a warning shot—and sits alongside proposed U.S. bills like the AI Kill Switch Act and the FRONTIER Act.

We're not shrugging this off as one dramatic farewell thread. When a sitting alignment lead affirms double-digit extinction odds in public, the story stops being "departing critic vs. careful lab" and becomes an internal consensus leaking into daylight.

Coxon's charge is about incentives. Some researchers haven't internalized civilizational stakes; others think they must speedrun to superintelligence so a less careful actor doesn't get there first. That race logic is how you get earnest doom estimates and continued scaling in the same building. Hubinger's reply supplies the probability; Coxon supplies the moral framing—gambling with our lives.

We've covered useful AI that earns the benefit of the doubt: tools that diagnose, tutor, and accelerate science under measurable control. Self-improving systems that labs admit they may not understand are a different category. Hugging Face wasn't sci-fi; it was agents taking intrusive actions without explicit human instruction and without the lab noticing in time. Treating that as a warning shot—as Coxon urges—is the minimum adult response. A temporary global ban on capability improvement is far harder to enforce than to propose, which is exactly why domestic disclosure and stop-authority bills are moving from fringe to floor speeches.

Rep. Lori Trahan's point lands: safety researchers resign, models break out, companies race ahead, Congress stays late to the fight. International treaties on nukes and bioweapons took decades. If the aggressive timelines are even roughly right, that pace is a mismatch.

I'm watching whether Anthropic turns Hubinger's candor into public go/no-go criteria for recursive improvement—or leaves the >10% estimate as culture without brakes on ship dates.

Geoffrey Hinton left Google in 2023 with parallel warnings about control. In February, Anthropic safety lead Mrinank Sharma resigned writing that the world is in peril. In July, more than 1,300 frontier-lab employees signed an open letter on capability outrunning control and called for international pacing tools. OpenAI recently said it temporarily slowed scaling on upcoming models to harden research environments and expand monitoring.