TL;DR
Microsoft Research describes its Industry Evaluation Data Generation Platform (blog, 30 Oct 2026, Chatterjee, Sahijwani, Chen, Hwang, Jacobs and others; surfaced by Microsoft Research's LinkedIn post). To evaluate an assistant that works over a company's mail, meetings and documents without touching customer data, it generates whole synthetic organizations ("tenants") with users, histories and workflows, then generates grounded evaluation queries against them.
Three tenants so far: pharmaceutical, capital markets and an IT/software company, each with hundreds of users and full communication histories.
Grounded in public data: O*NET for roles and activity mix, job postings for titles and hierarchies, SEC filings for scale and business units.
Tenants evolve over time (meetings, threads over weeks, org changes), because many real queries are time-relative ("what did we decide last Thursday?").
Reported outcomes are about the data and the process: a 39% gain on an internal composite quality metric, some tenants above 0.9 of 1.0, dataset creation from days to automated runs, an estimated $100K saved per dataset, and adoption as the first standardized Copilot evaluation baseline.
How it works
1. Synthetic environments. Each tenant is an isolated organization with reporting lines, teams, project timelines, emails, meetings and documents. Public signals calibrate who exists and what they do; generation adds time so that state changes across weeks.
2. Query generation, six stages. A library of generic workplace tasks is filtered to an industry, adapted with domain language, turned into queries grounded in the tenant's actual data, rewritten to sound human, then validated for accuracy and uniqueness.
3. Quality framework. Synthetic data is scored on structural fidelity (hierarchies, team sizes, cross-team links), contextual relevance (do the workflows and queries match real usage), behavioral realism (human-like communication) and content safety (no harmful content, no PII, no impersonation of real people).
4. Automation and reproducibility. The same configuration regenerates an identical dataset, so evaluations compare cleanly across releases and teams share one format.
Privacy posture. No customer data at any stage, public sources reviewed by legal, lineage tracked through the pipeline. That is what lets teams reuse the datasets without per-use privacy review.
Tradeoffs and failure modes
No link to real-world quality is shown. The post reports how realistic the synthetic data scores on Microsoft's own metric, not whether Copilot scores on these tenants predict how it does for real customers. That correlation is the claim the whole method rests on.
Self-graded realism. The composite quality metric is internal and not described in detail; "0.9 out of 1.0" has no external reference point.
Model-generated world, model-tested model. Tenants and queries are generated with AI and then used to test AI. Shared blind spots (what generators find typical, how they phrase things) can make the test easier than real use.
Clean data by construction. Real tenants are messy: duplicate files, stale permissions, half-finished threads, contradictory documents. Unless the generator plants that mess deliberately, retrieval and grounding failures that dominate production will be under-sampled.
Three industries. Coverage is narrow, and the most valuable failure modes may be in verticals not yet modeled.
Single-turn focus so far. The authors list multi-turn agentic tasks and actions, not just answers, as open problems.
Not public. No dataset, generator or metric definitions are released, so the approach can be copied but not reproduced.
Implications
For anyone evaluating an agent that works over private organizational data, a generated tenant is a credible substitute for production data, and reproducibility from config is the feature that makes it a regression suite rather than a demo. The missing step is calibration against real outcomes.
Ground generated orgs in public role, hierarchy and company-size data rather than inventing them
Make time a first-class dimension: histories, decisions and changes that relative-date queries can test
Deliberately inject real-world mess (stale, duplicated, conflicting, permission-restricted content)
Validate every generated query against the tenant's data before it enters the suite
Regenerate from a pinned config so results compare across releases
Check, at least once, that scores on synthetic tenants track a small set of consented real-world outcomes
Related entries
R016 Human verification of AI output: ground truth as a versioned, audited object, the governance this kind of dataset needs.
R024 InfraRSI: same principle of locking the measurement before testing the system; here the locked artifact is the generated tenant.
R027 The Tool-call index: another case where aggregate measurement surfaces failures that individual runs hide.
R005 Second Brain OS: evals as the layer most agent stacks underbuild.
Sources
From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence, Microsoft Research blog, 2026-10-30
Microsoft Research LinkedIn post, 2026-11-10