Sources: Zhang, Garg, Lai, Zhen, Sharma, Pan, Lin and Halevy, SynthCorp: A Platform for Creating Benchmarks for Data Agents, arXiv 2611.14765, 10 Nov 2026 (v1.0; the authors say the benchmarks will be released soon). Surfaced by DAIR.AI on LinkedIn, 2026-11-13, which attributes it to Google.
The problem
Benchmarks for LLM data agents (agents that answer questions over a company's databases, documents and catalog) saturate quickly, and a fixed dataset can't be aimed at one skill. Hand-labelling new tasks over real enterprise data is slow and leaks private data.
How SynthCorp works
It generates the company's data by simulating the business that would have produced it.
Event types. Given a scenario and the complexity dimensions to test, it synthesizes reusable business event types (an order, a refund, a supplier switch) and compiles each into a program over fixed simulation primitives.
A virtual calendar. Event instances are scheduled over time, mixing routine events with disruptions, so tables, documents and catalog entries stay consistent with each other.
Execution with a log. Running the calendar materializes the data estate (relational databases, a document store and a data catalog) and records every operation in a simulation log.
Tasks with gold programs. Analytical questions are generated, and their gold answers are derived as executable programs from the log.
Because the log records what caused each row, a candidate query can be audited against causal provenance, so gold answers are checked without manual annotation.
First release
Seven data estates with up to 1,000 operational tables, and 256 tasks across five domains.
The estates pass deterministic checks (foreign-key violations, undeclared writes, recurrence violations, execution failures) with a near-zero error rate.
Three models evaluated across four data-context settings: the benchmark separates both data contexts and model tiers.
Multi-event composite tasks, ones that need several business events chained together, still expose reasoning failures in state-of-the-art agents.
Why it matters
Generated benchmarks don't saturate the same way. When a model starts to ace a release, you generate harder estates on the dimension you care about.
The log is the oracle. Building test data from a simulation that records causes gives you ground truth for free. The same trick works for any agent eval where you can simulate the world that made the data.
Context matters as much as the model. That the benchmark separates data-context settings is a reminder to evaluate what the agent is shown (catalog, schema docs, examples), not just which model it runs.
Caveats
The benchmarks aren't public yet, so the results can't be reproduced.
Simulated businesses are cleaner than real ones. Real estates have undocumented columns, conflicting sources and history nobody logged, so scores here are likely an upper bound.
Only three models were evaluated.