A curated path for engineers moving into AI systems: LLM fundamentals, prompting, context engineering, harness, evals, observability. The list is sound and current, and three of its sources (Anthropic's harness-design and prompt-versioning pieces, OpenAI's harness engineering) carry most of the new ideas; its gap is the loop that ties evals and traces back into the harness.
The post
Shivani Virdi (NeoSage, ex-Microsoft, author of R002) posted a text-only list of 19 sources in six stages, each with a one-line thesis. About 1,170 reactions, 52 comments and 134 reposts. Twelve of the 19 are Anthropic pages.
| Stage | Thesis in the post | Sources |
|---|---|---|
| 1. LLM fundamentals | Hallucination, cost and context limits all follow from how models are built | Karpathy, Zero to Hero; Stanford CS336 |
| 2. Prompting in production | A prompt is code: it ships, breaks things and needs tests | Anthropic prompting best practices; Anthropic cookbook: prompt versioning and rollback; Microsoft Learn: advanced prompting for production agents |
| 3. Context engineering | Tools, history, memory and retrieved documents compete for one window | Anthropic: effective context engineering; Manus: lessons from building Manus; Anthropic: writing effective tools for agents; Anthropic memory tool docs; Anthropic: Contextual Retrieval |
| 4. Harness | The model picks the next step; the harness runs it, feeds back the result, enforces limits | Agent SDK: how the agent loop works; Anthropic: effective harnesses for long-running agents; Anthropic: harness design for long-running apps; OpenAI: harness engineering |
| 5. Evals | A change can fix one case and break another; evals tell you before users do | Anthropic: demystifying evals for AI agents; DeepEval: AI agent evaluation |
| 6. Observability | If you can't see each step, you can't debug the agent | Opik tracing concepts; OpenAI Agents SDK tracing; MLflow tracing |
Two comments add the substance the post lacks. One proposes a seventh stage, failure analysis: read traces, classify misses, and turn them into evals. Another ties every production failure to the prompt version that caused it and reruns those cases before each new version ships, so the eval set becomes a release history.
What the key sources say
Four of the sources are where the list's ideas are sharpest.
Harness design for long-running apps (Anthropic, Prithvi Rajasekaran, 2026-03-24). A planner, generator and evaluator, borrowed from GANs. Two failure modes drive the design: models lose coherence (some show "context anxiety" and wrap up early) as the window fills, and agents grade their own work leniently. The fix for the second is a separate evaluator, tuned to be skeptical, that drives the running app through Playwright and fails a sprint when any criterion falls below a hard threshold. Generator and evaluator negotiate a per-sprint "contract" of testable behaviours before code is written, and talk only through files.
Cost: a retro game maker took 20 minutes and $9 solo against 6 hours and $200 with the full harness; only the harness version actually played.
Out of the box, Claude was a poor QA agent: it found real bugs, then talked itself into approving. Tuning meant reading the evaluator's logs and patching its prompt where it diverged from the author's judgment.
On Opus 4.6 the author removed sprints and per-sprint grading one component at a time. A DAW build ran 3 h 50 min for $124.70, and a single end-of-run QA pass still caught stubbed core features.
The general rule: every harness component encodes an assumption about what the model cannot do, so re-test each one when a new model lands. The evaluator is worth its cost only where the task is beyond what the model does reliably solo.
Harness engineering (OpenAI, Ryan Lopopolo, 2026-02-11). Five months, roughly a million lines and about 1,500 merged PRs with no hand-written code; three engineers averaged 3.5 PRs each per day, and throughput rose as the team grew to seven. The engineer's job became asking "what capability is missing, and how do we make it legible and enforceable for the agent?"
AGENTS.md is a 100-line table of contents; a structured docs/ tree (design docs, exec plans with decision logs, quality grades) is the system of record, checked by linters and a recurring doc-gardening agent.
Architecture is enforced mechanically: fixed layers per domain with allowed dependency directions, custom lints whose error messages carry remediation instructions for the agent.
Each worktree boots its own app plus an ephemeral logs, metrics and traces stack the agent queries with LogQL and PromQL, so "no span over two seconds" becomes a checkable goal.
Merge gates are minimal because corrections are cheap and waiting is expensive. Drift is handled like garbage collection: background agents scan for deviations from written "golden principles" and open small refactor PRs, replacing a weekly 20% clean-up day.
Prompt versioning and rollback (Anthropic cookbook, 2026-04-07). Managed Agents stores the system prompt server-side; every agents.update mints an immutable version and sessions pin one. The worked example scores a ticket-routing agent on 20 labelled tickets, ships a v2 routing rule, sees billing drop from 4/5 to 2/5, and rolls back by pinning v1 with no deploy. The notebook says plainly that agents.update has no approval step and any workspace key can call it, so the control is to have production pin an explicit version and put that pin under review.
Demystifying evals for AI agents (Anthropic, 2026-01-09). The vocabulary the list needs: task, trial, grader, transcript, outcome (the environment's end state, not what the agent says), evaluation harness versus agent harness. Code, model and human graders each have a place; capability evals should start at a low pass rate, regression evals near 100%, and capability tasks graduate into the regression suite.
The remaining sources (Karpathy, CS336, the context-engineering, tool-writing and Contextual Retrieval posts, Manus, the Agent SDK loop and memory-tool docs, and the three tracing docs) are well-known references.
Assessment
A good list for its stated reader, an engineer starting out, and unusually current: half of it is from 2026. Its weakness is shape, not selection.
What it gets right. It treats a prompt as a versioned, tested artifact and puts evals and tracing on the path rather than as extras. The harness stage pairs a lab view (Anthropic's generator-evaluator) with a practitioner view (OpenAI's legibility and mechanical enforcement), which is the right tension to study.
What it misses.
The loop. The six stages read as a line, but in production they are a cycle: traces feed failure analysis, failures become eval cases, and evals gate the next prompt, context or harness change. The top comment's "failure analysis" stage and the prompt-version release history are that loop; the post names neither.
Harness subtraction. Rajasekaran's most transferable lesson, re-testing each component when a model ships and removing what is no longer load-bearing, has no stage of its own. Without evals it cannot be done safely.
Change control for prompts. The versioning cookbook itself warns that server-side updates bypass code review. The list frames prompt versioning as a convenience, not as a governance problem.
Security and permissions. Nothing on sandboxing, tool permissions or prompt injection, which R006's commenters and R021 treat as core harness concerns.
Vendor skew. Twelve of 19 are Anthropic pages; there is little on open-weight serving, cost, or non-Claude agent stacks.
The order question. A commenter praised fundamentals-first. For an engineer who already ships software, the faster route is probably evals and tracing first on a small real agent, then the rest as each failure demands it. That is how both the OpenAI and Anthropic harness posts describe their own learning.
Review checklist for a team using the list
Every production prompt has a version id, and the version production pins is changed only through review.
Each failure from production is filed as an eval case tagged with the prompt, model and harness version that produced it.
Capability and regression suites are separate; regression stays near 100% and capability tasks graduate into it.
Graders check outcomes (environment state), not the agent's own claim of success.
Work and judgment are done by different agents or contexts; the judge's leniency is measured against human calls.
Traces cover every tool call and are queryable by the agent itself, not only by humans.
Project rules live in a short map plus checked docs, and the checks run in CI.
On each model upgrade, harness components are removed one at a time and re-evaluated.
Tool permissions, sandboxing and injection handling are reviewed with the harness, not after it.
Related entries
R005 Second Brain OS: the five-layer map: almost the same stack (context, loop, gate, harness, evals) drawn as layers to diagnose, where this list draws it as stages to learn.
R006 6 must-know terms for AI agent harnesses: its commenters asked for exactly the observability and trajectory evals this list adds.
R002 The 5 layers of agent memory: same author; the memory-tool and context sources here are the reading behind it.
R020 The model proposes, the harness enforces and R017 Elephant-Goldfish: the same separation of maker and judge as Anthropic's generator-evaluator harness.
R016 Do humans still need to check AI's facts?: calibrating a lenient judge against humans, the step Rajasekaran describes doing by hand.
R004 The best Claude setup is no setup: prune harness config after model upgrades; Anthropic's harness post shows how to do it with an eval instead of a wipe.
R021 DeepSeek's Super-agent: the permission and gate layer the list leaves out.
Sources
The LinkedIn post (Shivani Virdi, about a week before 2026-11-10)
Harness design for long-running application development (Anthropic, 2026-03-24)
Harness engineering: leveraging Codex in an agent-first world (OpenAI, 2026-02-11)
Prompt versioning and rollback (Anthropic cookbook, 2026-04-07)
Demystifying evals for AI agents (Anthropic, 2026-01-09)