AI crash projects for public good: Mollick's post and the evidence behind it

TL;DR

Ethan Mollick argues that today's models are already good enough for national-scale "crash projects" in tutoring, healthcare second opinions, financial literacy and government services, and that labs and policymakers are under-investing in them. He backs this with three 2026 papers; the lead one is a randomized trial showing that plain ChatGPT access raised learning by 0.28 SD, with 0.21 SD still there a week later.

The common thread across all three papers is that the value depends on how the AI is used, not only on the model. Students who let AI write for them kept nothing. Users who wrote weaker prompts got worse financial plans. The medical agent was safe only under full-time physician supervision. For anyone building agents, the lesson is that the product is the interaction design and the oversight around the model, and that is the part these crash projects would actually have to build.

The post

Ethan Mollick's LinkedIn post (Wharton, about 11 hours old when read on 2026-11-06; 238 reactions, 56 comments) has no links. It attaches screenshots of the first pages of three papers, one per proposed area:

The fourth area, making government services easy to access, has no paper attached. His argument is that these are old, hard problems where earlier technology capped the gains. They need careful design and studies before rollout, and they may fail, but they need no model beyond what exists today.

The comments mostly argue about where to start. Scam avoidance gets the most support, because its success metric is cheap: did the person send the money or not. One builder's comment generalizes well: in their education tool, the model may phrase the answer but may not decide it, and if the model call fails, the user still gets the plain deterministic result.

Main paper: AI access and learning (Contractor and Reyes)

Off-the-shelf ChatGPT raised how much students learned in a fixed 35 minutes, and about half the gain was still there a week later. That gain came from students who used it as a tutor. Students who had it write for them showed no lasting gain.

Design. The study ran at Middlebury in spring 2025 with 211 undergraduates (205 attended both sessions). Each student had 35 minutes to learn an unfamiliar topic (blockchain, carbon capture or CRISPR) and write a roughly 500-word essay. Labs were randomized to AI-allowed, with a logged-in GPT-4o account, or AI-forbidden. All tests were taken without AI, right after the session and again a week later, along with a second essay written with no resources. The design fixes time on task, so it measures learning per unit of time, not time saved. Compliance was checked by proctors, ChatGPT logs, self-reports and an AI-text detector (Pangram). The trial was preregistered (AEARCTR-0015806).

Results (intent-to-treat, in control-group SD units):

• Scale. 0.28 SD is nearly three times the median education RCT (0.10 SD in Kraft 2020). It is close to structured human tutoring (0.29 SD) and above the 0.19 SD mean of 14 other AI-and-learning experiments.

• Mechanism. Total time didn't change, but students shifted it from writing (53% of time in the control group, 5.3 points less with AI) to reading and searching (44%, 4.4 points more). Enjoyment rose 11%. Essays did not become more alike.

• Use type matters most. About 70% of the AI-allowed group used it. The authors coded each conversation log: 61% of users used AI as a tutor (augmentation), 27% mixed tutoring and writing, and 9% had it write for them (automation).

The automation group's essay advantage fell by 1.64 SD once AI was taken away. They turned in the best work and kept the least. Students also misjudge the effect: control students predicted AI would raise their score by 25.5 points, 6.5 times the measured effect, while students who had used it guessed close to the real number.

Caveats a reviewer should raise.

• Time on task was held fixed. The authors say this does not show that wider AI use raises learning, because in real coursework students use AI to spend less time.

• AI access also raised rule-breaking on the first test. Violations explain 17 to 22% of the immediate gain. Excluding self-reported violators shrinks it to 0.15 SD.

• The use-type split is descriptive, not randomized, and the automation group is small (about 7 students, my estimate from the reported shares). The authors lean on six other experiments that randomize use mode, and four of the six agree.

• The setting is one selective college (mean SAT 1386) with 5- and 10-question tests. The week-later effect is only marginally significant.

• Read note: the arXiv HTML rendering has broken number macros in the prose. I took figures from the paper's macro table and from the passages that rendered correctly.

Supporting papers

Healthcare: AMIE in a real urgent-care clinic (Brodeur et al., BIDMC and Google)

In a real clinic, a Gemini 2.5-based diagnostic chat agent took patient histories about as well as the clinic's physicians, but only with a physician watching every session live. It was a single-arm study: 100 adults chatted with AMIE up to five days before their urgent-care visit, and 98 also saw their PCP.

• Safety. There were zero safety stops. Supervisors stepped in three times: to rule out an emergency, to clarify when to seek emergency care, and to fix a date AMIE got wrong.

• Accuracy. Checked against chart review 8 weeks later, the final diagnosis was in AMIE's list 90% of the time, in its top 3 75% of the time, and its first guess 56% of the time. Blinded raters scored AMIE's diagnoses and plans as similar to the PCPs' on quality, appropriateness and safety. PCPs won on practicality (p = 0.003) and cost (p = 0.004), likely because AMIE had no chart access, no physical exam and listed more possibilities.

• Users. Patients' attitudes toward AI improved after the chat (p < 0.001). Among PCPs who read the transcript, 75% found it helpful, and they likened it to a third-year medical student's write-up. Visits shifted from gathering data to checking it.

• Limits. Patients knew they were watched, which likely means a Hawthorne effect and no adversarial prompting. Pregnancy, mental-health and emergency cases were excluded. Patients needed a laptop. It wasn't connected to the EHR, so only 73% of PCPs saw the transcript in time. Confidence in the model's reasoning traces didn't predict accuracy. Google funded and co-ran the study, and the code is closed.

Financial literacy: AI financial advice over a lifetime (Choukhmane et al., MIT Sloan, NBER w35574)

Following ChatGPT 5.2 or Gemini 3 Flash advice would move people toward textbook life-cycle behavior, but the quality of the advice tracks the quality of the question. The authors collected three prompts each from 1,000 survey respondents (their situation, how much to save, how to invest). They fed those prompts to the models for simulated people from age 22 to 90, with income shocks, taxes and market returns, and turned the answers into decisions.

• What works. The advice smooths spending, builds cash buffers, diversifies into stock funds and cuts equity after age 45. It brings up liquidity in 83% of answers when only 6% of users asked about it.

• What doesn't. It adjusts poorly after income shocks, lets the portfolio drift instead of rebalancing, and draws down too little in retirement. A richer system prompt fixed saving advice but not rebalancing.

• The prompt gap. Prompts from users with low financial literacy produced about $50K (4.1%) less wealth at 60. Prompts from users with no prior AI experience produced about $100K (5.7%) less. Prompts written by women produced about $60K (4.1%) less. Two-thirds of the gender gap comes from how the prompt is worded. One-third comes from the model reacting to the stated gender alone.

• Limits. These are simulated outcomes, not observed behavior. The figures come from the MIT Sloan summary; I didn't open the full PDF.

What it means for builders

Mollick is right that the model isn't the bottleneck. All three papers show the bottleneck is the system around the model: how users are steered, what the agent can see, and who checks it. A crash project is mostly harness work.

• Design for augmentation, and measure it after the tool is gone. Assisted output is a poor proxy for learning. The automation users turned in the best essays and kept nothing. This applies to engineers too: the paper cites Shen and Tamkin (2026), where developers who used AI to learn an unfamiliar library scored worse on a later unassisted test. Coding agents that write the code for juniors carry the same risk.

• Don't make outcomes depend on how well users prompt. Weaker prompts led to worse financial plans. A product should elicit the missing facts itself (income, buffers, horizon) instead of answering whatever was asked. Give the model the method, not just the question, and test whether its advice changes with gender or wording.

• Oversight costs money. AMIE's zero safety stops came with a physician watching every session live. A national rollout needs a cheaper gate: risk-tiered triage, human-on-the-loop for low-risk cases, and evals in place of a live observer. That is R005's gate and evals layers and R006's guardrails.

• The model phrases, deterministic code decides. The commenter's pattern (rules compute the answer, the model explains it, and failure falls back to the plain result) suits finance and benefits navigation.

• Pick problems with a cheap success metric. Scam avoidance (did the money move) is easier to evaluate than learning or health outcomes, and evals come first.

Review checklist for a public-good AI program

• Is the outcome measured without the AI present (retention, unaided performance), not just with it?

• Does the design steer users toward explanation and away from delegation (hints before answers, withholding the answer)?

• Does it work for users who prompt badly, and is advice quality checked across literacy, gender and language?

• What is the oversight model at scale, what does it cost per session, and which cases skip it?

• Where does deterministic logic decide, and what does the user see when the model call fails?

• Is there a randomized evaluation before rollout, with a business-as-usual control?

Related entries

• R005 (gate and evals layers) and R006 (guardrails, human-in-the-loop): AMIE's live physician is the gate, and the open problem is replacing it with evals.

• R004: prompting with Goal + Context. The financial paper measures what happens when users don't supply the context.

Sources

• Ethan Mollick's LinkedIn post, read on 2026-11-06

• Contractor and Reyes (2026), arXiv 2607.08849, full HTML text of sections 1 to 7; also IZA DP 18792

• Brodeur et al. (2026), arXiv 2603.08448v2: abstract, results, discussion

• MIT Sloan press summary, 2026-05-21, for NBER w35574 (paper PDF not opened)