Source: Wang, Xiang, Qu, Li, Zhang, Suh and Tian, "MetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmarks", arXiv 2611.10026, 5 Nov 2026. Surfaced by DAIR.AI.
The problem
Indirect prompt injection (IPI) benchmarks lag behind utility benchmarks: they cover few tasks and environments and use fixed, static attacks. Meanwhile agents get high-privilege tools in exactly the environments utility benchmarks already model.
The approach: meta-benchmarking
MetaPI converts an existing utility benchmark into an end-to-end IPI suite that reuses its tasks, environments and infrastructure. The toolkit:
Finds plausible injection points (tool outputs, files, web content the agent will read).
Generates realistic attacker objectives for that environment.
Optimizes adaptive payloads rather than using fixed strings.
Validates each attack by running it in the real environment and checking whether the objective was achieved.
Interfaces are modular, so new environments, agents, defenses and attack algorithms plug in without rewriting the pipeline.
Findings (author-reported)
Applied to AppWorld, VitaBench and Terminal-Bench across several backbone models.
No evaluated defense achieved both strong utility and strong security under adaptive attacks.
Per the DAIR summary: the most robust closed models still had attack success above 20%; a detector-style defense (DataSentinel) kept utility but blocked less, while a capability-separation design (CaMeL) blocked more at a large utility cost.
What to do with it
If you already run a utility benchmark for your agent, run injection attacks inside the same environment, so security and utility are measured on the same tasks.
Track both numbers per release; a defense that halves attack success but drops task success is a product decision, not a free win.
Use adaptive, environment-validated attacks. Static strings overstate robustness.
Caveats
Adaptive payload optimization means results depend on attacker budget; compare defenses at equal budgets.
Validation checks the attacker's goal, not subtler harms such as data exposure in logs.
Review checklist
Which inputs to our agent come from places an attacker could write?
Do we test injections inside our real task environment, not a toy one?
Do we report utility and attack success together for each defense?
Are high-privilege tools gated by something other than the model's judgment?
Related entries
R086 (graders as an attack surface in Singularity-Bench), R074 (policy on every mutation, not tool hiding), R064, R052.