SDSignal Desk

Two million bait secrets show how far a prompt travels once an AI router takes it

· Updated · Signal Desk newsroom

Analysis of reporting by Ars Technica. How we work

Image: Ars Technica

HD Moore seeded unique credentials and links into nearly two million prompts and tracked where they resurfaced. The answer: breached hosts, logging resellers and a lab scraping data marked private. The cheap inference market has a custody problem.

Key takeaways

  • HD Moore, chief research officer at Dragos, sent nearly two million prompts carrying unique, never-used secrets through AI providers, routers and resellers, then watched where those secrets were used.
  • Three separate hosting providers were breached; attackers reused a couple hundred bait AWS keys and the stolen keys were used tens of thousands of times in AWS API calls.
  • Some low-price resellers turned on request logging and harvested credentials themselves.
  • A Chinese AI lab fetched secret URLs from more than half the prompts sent to its models, including roughly two-thirds of requests flagged no-data-collection or Zero Data Retention, even when reached through US resellers.
  • Moore’s advice: self-host if possible, otherwise buy directly from a first-tier provider and pick upstream providers rather than accepting default pools.

A tripwire experiment with a grim result

Most arguments about AI privacy are hypothetical. This one is not. HD Moore, the security researcher who now serves as chief research officer at Dragos, built a large set of fresh secrets: working AWS access keys tied to empty accounts, SSH passwords, unique URLs, DNS names and references to private git repositories. Each secret went into exactly one prompt, sent to exactly one provider. Moore controlled the infrastructure behind every credential, so any later use pointed straight back to the link in the chain that leaked it. Ars Technica reports he sent nearly two million of these canary prompts.

The results read like an incident report for an entire industry. Three hosting providers were breached. Some resellers stole data on purpose. At least one model maker scraped links that customers had explicitly asked it not to keep. None of this required exotic attacks on models. It came from the ordinary plumbing that moves a prompt from an app to whichever server answers it.

The router layer is the weak point

The piece of the market under the microscope is the AI router, sometimes called a gateway. Services like OpenRouter or Vercel take prompts from apps and pass them to an inference engine chosen on price, speed or fit. That is genuinely useful, and it is spreading; Hugging Face now pushes every chat request through OpenRouter’s router, as we covered when the two companies joined forces on open-weight inference. But Moore’s experiment highlights the cost. A prompt is encrypted on the wire and decrypted at every hop. The router reads it to decide where it goes, it may land in a pool of models running at different hosts, and it can pass from one router to another. Each hop is another party that may log it, and for as long as its own policy allows.

The breach details are the most alarming part. In one case, every request Moore sent during several short windows in late October was stolen, including requests the provider rejected before any model ran. That suggests the attacker was copying traffic at the front door. Within hours, the bait keys were being tested; the intruders searched S3 buckets, secrets managers and parameter stores for more credentials, tried to enable hosted models in AWS Bedrock and created their own IAM users to keep access. Secrets from two of the breached providers were used together from the same rented server, which points to a single adversary. Some of the stolen requests carried a Zero Data Retention flag. A retention promise means nothing once someone else is inside.

Privacy flags that do not mean what buyers think

The resellers are a different kind of failure. Moore caught them by sending requests for the same model through different sellers; when a credential was reused after passing through one seller but not another, the seller was the leak. The business model is simple and ugly: undercut on price, switch on logging and wait for valuable prompts.

Then there is the scraping. The clearest case Moore describes involves a Chinese AI lab whose systems fetched secret URLs from more than half of the prompts sent to its models between July and October, including about two-thirds of the requests marked no-data-collection or ZDR, usually two or three days later. The same scraper appeared when the lab’s models were reached through US hosting providers, so customers who chose a US host to keep data domestic did not get that. A git client on another cloud later tried to clone the private repositories referenced in those prompts. Moore also found a US provider crawling a secret URL more than two months after a single search call. This is not a China-only problem.

We are firmly in favor of cheap, competitive inference; it is a big reason useful AI is reaching small developers at all. But the market has optimized for price and latency and left custody of the prompt as an afterthought. When a privacy setting is a checkbox that can be ignored at the third hop, it is marketing, not a control. As agents get more access to real credentials, as with the privileged access Meta’s Muse assistant held when a zero-day surfaced, the stakes of a leaky chain rise quickly.

Who is exposed

App developers
Any app that forwards user prompts through a router inherits every downstream host’s logging and security; default provider pools are the riskiest choice.
Enterprises
Credentials, internal URLs and repository names pasted into prompts should be treated as disclosed unless the full chain is contractually covered.
Routers and resellers
Expect pressure to publish provider lists, audit retention claims and offer verifiable routing guarantees.
End users
People rarely know which companies touch their prompt, and ZDR labels gave them little real protection in this test.

What to watch

  1. Whether the three breached hosting providers are named and disclose their incidents.
  2. Responses from major routers on how they vet upstream providers and enforce ZDR flags.
  3. Any reaction from the Chinese lab or the US hosts reselling its models.
  4. Enterprise policies that restrict which inference providers prompts may reach.

FAQ

What is a canary in this experiment?

A unique, never-used secret, such as an AWS key or a private URL, placed in one prompt to one provider so any later use reveals exactly where it leaked.

Did Zero Data Retention protect requests?

Not reliably. Some stolen requests in two breaches were marked ZDR, and a Chinese lab fetched URLs from roughly two-thirds of requests flagged no-data-collection or ZDR.

How can teams reduce the risk?

Moore recommends self-hosting where affordable; otherwise use a first-tier provider directly, choose upstream providers explicitly and keep live credentials out of prompts.

Read the original

Continue at the source.

Ars Technica

More on AI

All AI coverage →