AI Security Agents Compared: Claude, Codex, Mantis, Cloudflare

Anthropic, OpenAI, Cloudflare and Google now each offer an AI agent that searches source code for security bugs. A detailed comparison of the four finds they share one design. One agent reports a candidate flaw, a separate agent tries to knock it down, and only what survives gets reported or patched.

The reason is noise. Google's security team has written that careless AI code scanning produces hallucinated bugs and true-positive rates under 7%. A scanner that is wrong that often costs defenders more time than it saves. The other driver is offense: both Anthropic and Google have framed these tools as a response to attackers already using models to find exploitable flaws.

The tools differ in reach. Cloudflare's skill is MIT-licensed and works with any capable agent. Google's Mantis is Apache-2.0 and tries to reproduce each bug in a sandbox. Claude Security runs inside Claude Code. Codex Security's client is open source, but access remains limited to approved customers.

This is AI security work growing up, and we are glad to see it. Separating the finder from the verifier is an old audit principle, and applying it to models is the difference between a firehose of false alarms and a list a team can act on. Cloudflare's reported numbers show it at scale: of about 20,800 raw candidates, around 12,000 survived validation, and 7,245 actionable findings reached engineering teams. Better reconnaissance cut its first-pass rejection rate from 40% to 11%.

We also like that the proof bar keeps rising. Mantis tries to reproduce bugs, from an exploit sketch up to a unit test, and attacks its own patches. Across the wider field, the pattern is deterministic checks for proof, models for reasoning, and a human approving every fix. That last part should stay non-negotiable.

Now the uncomfortable side. The same machinery that finds a bug for a defender finds it for an attacker, and the open-source versions are available to both. Defenders still have to triage, patch, test and ship; attackers only need one working exploit. If discovery becomes cheap everywhere, the bottleneck becomes the volunteer maintainers of open-source projects, who will be handed more valid reports than they can fix. Snyk's own research, which found about half of model-only review findings vanish on a rerun, is a reminder that inconsistency has not gone away.

There is a practical warning too. Agents that execute exploit code need real isolation. Google says to isolate Mantis, and Cloudflare's skill declines to run leads without a sandbox. Teams that skip that step are running untrusted code on purpose.

Our read: the convergence is good news, and the open, model-agnostic options lower the barrier for smaller teams. I'm watching whether patch capacity grows as fast as discovery does.

Dozens of agent-based vulnerability tools surfaced in 2026, from vendors like Snyk, Semgrep, Wiz and AWS and from companies like Visa and Vercel publishing their own harnesses. Most follow the same find-then-independently-verify loop. OpenAI and Trail of Bits also ran a program applying Codex Security to open-source projects.