R029 · Agent principles and patterns › Evaluation data for agents

Synthetic enterprise tenants for evaluating Copilot

Read the original

evalsbenchmarksdata-privacyagent-reliabilityhuman-oversight

TL;DR

Microsoft Research describes its Industry Evaluation Data Generation Platform (blog, 30 Oct 2026, Chatterjee, Sahijwani, Chen, Hwang, Jacobs and others; surfaced by Microsoft Research's LinkedIn post). To evaluate an assistant that works over a company's mail, meetings and documents without touching customer data, it generates whole synthetic organizations ("tenants") with users, histories and workflows, then generates grounded evaluation queries against them.

How it works

1. Synthetic environments. Each tenant is an isolated organization with reporting lines, teams, project timelines, emails, meetings and documents. Public signals calibrate who exists and what they do; generation adds time so that state changes across weeks.

2. Query generation, six stages. A library of generic workplace tasks is filtered to an industry, adapted with domain language, turned into queries grounded in the tenant's actual data, rewritten to sound human, then validated for accuracy and uniqueness.

3. Quality framework. Synthetic data is scored on structural fidelity (hierarchies, team sizes, cross-team links), contextual relevance (do the workflows and queries match real usage), behavioral realism (human-like communication) and content safety (no harmful content, no PII, no impersonation of real people).

4. Automation and reproducibility. The same configuration regenerates an identical dataset, so evaluations compare cleanly across releases and teams share one format.

Privacy posture. No customer data at any stage, public sources reviewed by legal, lineage tracked through the pipeline. That is what lets teams reuse the datasets without per-use privacy review.

Tradeoffs and failure modes

Implications

For anyone evaluating an agent that works over private organizational data, a generated tenant is a credible substitute for production data, and reproducibility from config is the feature that makes it a regression suite rather than a demo. The missing step is calibration against real outcomes.

Sources

Connections

See R029 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.