TL;DR
Ethan Mollick argues that "humans must check AI output for factual errors" is an outdated default, citing two results: agents beat 2025 human searchers on BrowseComp (about 91% vs 25.3%), and on DeepFact, PhD specialists working alone got only 60.8% of planted test claims right. Both numbers are real, but neither paper tests everyday work, and DeepFact's best result (90.9%) is humans auditing agent evidence, which is human verification in a different shape, not its end.
The defensible takeaway for engineers: stop designing humans as line-by-line error catchers on long AI output, because they are bad at it under load. Design them as auditors of disputes and evidence that agents surface, with versioned, traceable labels. Mollick's "good riddance" overstates what the data shows; the DeepFact paper's own framing (expert as auditor, not labeler) is the stronger and more useful claim.
The post and its chart
Mollick's post (7 Nov 2026, about 420 reactions) makes two moves. First, recent models beat human experts at finding facts and checking claims, so worry about factual errors is miscalibrated. Second, humans still need to judge viewpoints and approaches, but "humans as AI error-checkers" is a role we should be glad to lose.
His chart, "AI agents vs. humans on factual tasks", puts two benchmarks side by side and warns in its footer that the scores describe separate tasks and should not be pooled:
| Panel | Comparison | Human | Agent |
|---|---|---|---|
| Finding facts on the web (BrowseComp) | Matched the reference answer | 25.3% (2025 human searchers, ~2 h per task) | 90.4% to 92.2% (GPT-5.6 Sol, GPT-5.6 Sol Ultra, GPT-6 Astra, Claude Opus 5; lab-reported, 2026) |
| Checking research claims (DeepFact, ACL 2026) | Correct verdict on 143 planted claims | 60.8% (PhD specialists alone); 90.9% (humans auditing agent evidence) | 83.2% (GPT-4.1 agent), about 89% (GPT-5 agent), 93.0% (agents auditing agents, best pairing) |
The chart also draws the DeepFact trivial baseline: always answering "unsupported" scores 83.9%, because 120 of the 143 test claims were deliberately made false. The 2026 BrowseComp model scores are as charted from OpenAI's announcement pages; I did not check them independently.
The comments push back from law, finance and medicine: courts are sanctioning firms for filed hallucinations, a retired bank director still finds errors, and a government legal adviser using Opus 5.5 daily says quality drops as reports get longer and that accountability for the final product does not delegate. Two comment threads are vendor pitches.
BrowseComp: built to be hard for people
BrowseComp (Wei et al., OpenAI, April 2025) is 1,266 questions whose answers are short and easy to check but very hard to find: trainers wrote each one by starting from a fact and stacking constraints until neither they nor the models of the day could find it quickly. The 25.3% human score is 317 of 1,255 answers matching the reference. Most of the gap is surrender, not wrong answers: trainers gave up on 888 of 1,255 questions (70.8%) after two hours, and when they did answer, they matched the reference 86.4% of the time (317 of 367).
So the panel shows that agents with search and test-time compute out-persist people on needle-in-haystack retrieval. It says little about verification: BrowseComp asks the agent to find a fact, not to catch an error in a draft, and the asymmetry it was built on (hard to find, easy to verify) is exactly the case where a human check is cheap.
DeepFact: experts are poor labelers and good auditors
DeepFact (Huang, Ribeiro, Hardalov, Dhingra, Dreyer, Saligrama; Duke, Amazon AGI, BU; ACL 2026) asks how to benchmark fact-checkers for deep research reports, where one report holds hundreds of claims and one claim can take hours to settle.
Micro-golds. PhD students each picked research questions in their own field, got AI-written reports, and labelled about 40 high-risk sentences per report. Hidden among them (25% of items) were known-answer claims: real sentences with an injected error (a misattributed citation, a causal inversion, an over-generalization) at a 4:1 ratio to verified-true ones. Unassisted, the experts got 60.8% of these right, despite high self-reported confidence and pay tied to passing the hidden tests.
Audit-then-Score (AtS). Labels become versioned and revisable. A challenger verifier that disagrees with the benchmark must submit evidence; an auditor (an expert or an agent) accepts the revision only if the new rationale beats the old one; then the challenger is scored on the updated benchmark. Over three audit rounds with stronger challengers, expert accuracy rose to 90.9%. The three AtS rounds took 34.5% of the 400-plus expert-hours; the first independent labelling pass took the rest.
The experts only beat "always say unsupported" from Round 2 on, so the 60.8% is worse than a constant answer on this deliberately error-heavy set.
How humans and agents interacted. In the human-agent decision flows (Table 1), humans adopted a correct agent verdict in 22.4% of cases and resisted a wrong one in 13.3%; they were misled by a wrong agent in 2.8%, stayed wrong despite a right agent in 5.6%, and both were wrong in 11.2%. Agent auditors were non-regressive in every pairing, but auditing only consolidates the evidence on the table; improving the benchmark needs a stronger verifier to find new evidence.
The verifier. DeepFact-Eval reads the whole report, plans broad search queries, summarizes retrieved documents, then asks depth questions per document before deciding. On DeepFact-Bench (944 claims, 20 reports, six fields) it scores 83.4% accuracy with GPT-4.1 and 87.2% with GPT-5, against 52.5% to 58.5% for snippet-based checkers (SAFE, VeriScore, FactCheck-GPT, FIRE) and about 69% for GPT-Researcher and SmolAgents. It costs about $1.16 per claim; grouping 10 related claims cuts that to $0.21 at 76.3%. On SciFact and Factcheck-Bench, re-auditing its disagreements lifts its estimated accuracy from 84.6% to 94.7% and 82.0% to 93.0%: many "errors" were bad gold labels.
Where the data supports Mollick, and where it is stretched
Supported.
Unassisted human review of long AI output is a weak control. DeepFact's experts, in their own fields and paid to pass hidden tests, missed about two in five planted errors; the pilot notes attention decaying across long sessions and second reviewers deferring to the first. "A human looked at it" is not evidence of correctness.
Agent verifiers now beat snippet-matching fact-checkers by 25 to 30 points and catch much of what people miss, at roughly $0.20 to $1.20 per claim.
Static human gold labels are themselves noisy, so "the model disagreed with the human" increasingly means the human was wrong.
Stretched.
The best number is a human-in-the-loop number. 90.9% is experts auditing agent rationales; agents auditing agents reach 93.0% in the chart's best pairing. The paper's thesis is to change the human's role from labeler to auditor, not to remove it.
The test is error-heavy by design. 84% of micro-golds are false, so a constant "unsupported" scores 83.9%; the GPT-4.1 verifier (83.2% on the chart) is no better than that constant on this set. The chart says so; the post text does not. Base rates in everyday work are very different, and precision on true claims matters more there.
Finding is not checking. BrowseComp measures persistence at search on questions built to be easy to verify. It supports "use agents to look things up", not "skip review of what agents write".
"Everyday work" is not tested. Neither paper measures the errors that matter in practice: subtle omissions, wrong framing, stale sources, or errors that are correlated across models. The legal commenters' point stands: where an error has a high cost and someone is accountable, review is a liability control, not only an accuracy control.
Agents-checking-agents has a monoculture risk. DeepFact itself schedules an expert re-audit after 5% of labels change, because repeated agent revisions can drift toward the challenger's own reasoning and retrieval habits.
What it means for engineering
Treat human review of agent output as a system you design and measure, with the human placed where DeepFact shows people are strong: adjudicating a specific disagreement with evidence in front of them.
Make agents produce checkable rationales. Every factual claim carries its evidence and a verdict, so a reviewer judges an argument instead of re-deriving a fact.
Route by disagreement. Run an independent verifier (a different model or retrieval path); send only conflicts and high-risk claims to a human. DeepFact's ablation shows auditing 75% of conflicts gets within 1.4 points of auditing all of them by round 3.
Seed canaries. Plant known-bad items in review queues, as DeepFact's micro-golds do, to measure reviewer accuracy rather than assume it. The same trick measures your automated verifier.
Version the ground truth. Golden sets and eval labels should be revisable with a changelog and rationale, and scores tied to a version, or a better model will look worse against stale labels.
Guard the agent-only loop. When agents review agents, diversify models and retrieval, and schedule a human re-audit when cumulative changes pass a threshold.
Review checklist
For this workflow, what is the error cost, and who is accountable for the output?
Does the agent emit claim-level evidence a reviewer can check in minutes?
Is there a second, independent verifier, and does disagreement route to a person?
Do we measure reviewer accuracy with planted items, not self-reported confidence?
Are eval labels versioned, and can a model's challenge with evidence revise them?
Is our test set's base rate of errors close to production, so accuracy numbers mean something?
Is there a drift check when agents audit agents?
Related entries
R012, AI crash projects for public good: Mollick's earlier post. Its AMIE study only worked with a physician watching every chat; DeepFact suggests that watcher is best used as an auditor of flagged disputes.
R006, harness terms and R005, five-layer map: the evals and gate layers this changes.
R014, MIT Intro to AI: teaches "verify AI output" as a user habit, the default Mollick is arguing against.
Sources
Ethan Mollick, LinkedIn post, 7 Nov 2026, and its chart.
Huang et al., DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality, arXiv 2603.05912v2, ACL 2026.
Wei et al., BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents, OpenAI, April 2025 (human baseline from its Table 2).