R070 · Agent principles and patterns › Evaluation data for agents

SynthCorp: benchmarks for data agents generated from a simulated company

Read the original

evalsagent-reliability

Sources: Zhang, Garg, Lai, Zhen, Sharma, Pan, Lin and Halevy, SynthCorp: A Platform for Creating Benchmarks for Data Agents, arXiv 2611.14765, 10 Nov 2026 (v1.0; the authors say the benchmarks will be released soon). Surfaced by DAIR.AI on LinkedIn, 2026-11-13, which attributes it to Google.

The problem

Benchmarks for LLM data agents (agents that answer questions over a company's databases, documents and catalog) saturate quickly, and a fixed dataset can't be aimed at one skill. Hand-labelling new tasks over real enterprise data is slow and leaks private data.

How SynthCorp works

It generates the company's data by simulating the business that would have produced it.

  1. Event types. Given a scenario and the complexity dimensions to test, it synthesizes reusable business event types (an order, a refund, a supplier switch) and compiles each into a program over fixed simulation primitives.

  2. A virtual calendar. Event instances are scheduled over time, mixing routine events with disruptions, so tables, documents and catalog entries stay consistent with each other.

  3. Execution with a log. Running the calendar materializes the data estate (relational databases, a document store and a data catalog) and records every operation in a simulation log.

  4. Tasks with gold programs. Analytical questions are generated, and their gold answers are derived as executable programs from the log.

Because the log records what caused each row, a candidate query can be audited against causal provenance, so gold answers are checked without manual annotation.

First release

Why it matters

Caveats

Connections

See R070 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.