Sources: Pushpdeep Mishra's LinkedIn post resharing InfoQ's post; InfoQ article, 2026-11-10 by Wejdan Bagais, Pushpdeep Mishra and a third author (InfoQ Certified Architect Program capstone; reviewer Hien Luu).
In one paragraph
The authors tested the common assumption that stacking checks around an LLM makes it more accurate. On an adverse-drug-event extraction task (the CADEC corpus: 1,250 forum posts, each reaction a span coded in SNOMED CT), they ran six layers on the same records with each one priced: a deterministic vocabulary check, self-correction on provable failures, three-sample voting, a second-model judge, refusal that holds doubtful records for a person, and the person. The layers did not add up. They sorted answers by how far to trust them and corrected almost none. Voting cost about 420K tokens a run and flipped as many right answers to wrong as the reverse. Self-correction fired two or three times a run. The judge separated right from wrong well once it was shown the same candidate menu the extractor saw, but no later layer read its verdict. The real ceiling was detection: no layer can recover a mention the extractor never quoted.
The system under test
The model reads, the vocabulary knows. The extractor (gpt-oss:20b, zero-shot) quotes each reaction as a span. A retriever with no LLM in it embeds the quote (granite-embedding:30m, 384 dimensions, local) over 227,554 SNOMED description rows and shows the model the 20 nearest concepts. The model picks one; it is never asked for a code.
Why that shape. Measured on 40 documents, three cold runs each. Recalling codes from memory was 8 to 12 times worse than naming the concept and looking it up. It returned null on up to 38% of records, and 13% to 18% of the codes it did give exist in no SNOMED release.
Baseline. Zero-shot end-to-end F1 was 0.47 to 0.50 (lenient) and 0.39 to 0.43 (span-exact), against 0.72 for a supervised system fine-tuned on 875 CADEC files. The held-out split of 60 documents was run once.
Measure the noise floor first
Three identical runs at temperature zero differed by four F1 points. Of 233 grouped mentions, only 70% matched on both span and code across all three runs. Four other model families (llama3.1:8b, mistral:7b-instruct, qwen3:8b, granite4:micro-h) were byte-identical across repeats; the mixture-of-experts extractor was the only one that varied, and only inside a full run. From then on, every change was reported as a range across three runs.
What each layer bought
Vocabulary check (0 tokens). It changes no answer. It gives each one a verdict: REJECT (the code doesn't exist, or the span isn't in the post), ACCEPT (the span matches one of the concept's names), or BAND (no evidence either way). Code can prove an answer wrong but never right.
Self-correction (about 1.5K tokens). It fires only on REJECT, which the model rarely triggers, so it was effectively unmeasured.
Voting (411K to 432K tokens). Net -1 to +1 codes per run, destroying 2 to 5 right answers. The voter is the answerer, so a vote carries no information the answer lacked.
Second-model judge (84K to 88K tokens). A 3.2B judge shown only the span and a code barely told right from wrong. Shown the menu, it separated them 3.4 to 4.2 times, and it could say the right answer wasn't on the menu. Nothing downstream used its verdict.
Refusal (0 tokens). On ACCEPT, 21% to 23% of records pass, at 0.74 to 0.82 accuracy; the rest go to a person. A confidence threshold was dropped because the extractor reported confidence 1.0 on two-thirds of answers, right or wrong.
Removing the three paid layers changed what refusal passed by one record in one run and none in the other two.
Why the layers don't stack
Withholding is the only lever. Any verdict raises the accuracy of what passes by holding answers back, so accuracy flatters every withholding layer. The authors report yield (right answers over all records) beside accuracy because yield can't be gamed that way.
Verdicts nest. Nearly everything the free check endorses, the judge endorses too, so a second verdict mostly confirms the first. Where all verdicts agree, answers are almost all right; where only a vote agrees, almost all wrong.
Detection is the ceiling. Once a mention is found, the right concept reaches the menu 93% of the time and gets picked 81% of the time. But nearly half the annotated mentions were never quoted as annotated. The causes were boundary conventions, single words left untouched and idioms read literally, not missing medical knowledge. Domain models (BioMistral-7B, SapBERT as encoder) didn't help. A worked example in the corpus's conventions and a rule for denied reactions did.
Defects lived between layers. Voting overwrote codes without re-running the vocabulary check, so records shipped marked verified for codes they no longer held. F1 could not see it: fixing the bug left the held-out F1 at 0.204.
Does it generalize?
The authors re-ran on seven more corpora and five model families. "Layers sort, they don't fix" held. The free check held for clinical vocabularies (the endorsed lane was right 76% to 100% of the time) but reversed for gazetteers, where a place name can match and still be the wrong place (7% to 26%). It never fired on financial tag names. On the financial corpus the model read well but coded badly, mostly by habitually picking the menu's first line. Seventeen matrix cells ran with the wrong prompt or model, caught only because each run saved its received configuration.
Tradeoffs and caveats
One task family (entity extraction plus normalization), small open-weight models, and a small judge. Absolute numbers would move with frontier models. CADEC is public and likely in training data, though the between-layer comparisons should hold.
The gold labels aren't the only right answer: an equally valid general concept can score as wrong.
The person layer wasn't measured. Neither was a second-reader layer, the one design that could lift the detection ceiling.
It is a capstone article, not a peer-reviewed paper, though the authors publish code, a run ledger and decision records.
Review checklist for a reliability stack
Does any layer change answers, or does it only withhold them? Report yield beside accuracy.
Is the identifier produced by a lookup rather than generated?
Is confidence anchored in something that doesn't resample (a vocabulary, a schema, a compiler)? Has it been tested on the answer key and on your own vocabulary, where it may reverse?
Have you measured run-to-run variance (at least three runs) before claiming any gain?
Does something downstream read every verdict a layer writes?
If a later layer rewrites an answer, does it re-run the earlier checks on that answer?
Where does the pipeline lose: finding, retrieving or picking? Fix that stage before adding checks above it.
Are split votes and "not on this menu" verdicts used offline as a to-do list, rather than paid for online?
Related
R020 (the model proposes, the harness enforces): the same principle, with code making the guarantees and the model only proposing.
R032 (Strands Decider): picking from offered options instead of generating, which this study measures directly.
R016 (do humans still need to check AI's facts?): where human verification belongs.
R015 (workflows over agents): deterministic steps around a narrow model call.