R022 · AI education › Learning paths for AI engineers

Shivani Virdi's six-stage AI engineering reading list

Read the original

learning-resourcespromptingcontext-engineeringagent-harnessevalsobservability

A curated path for engineers moving into AI systems: LLM fundamentals, prompting, context engineering, harness, evals, observability. The list is sound and current, and three of its sources (Anthropic's harness-design and prompt-versioning pieces, OpenAI's harness engineering) carry most of the new ideas; its gap is the loop that ties evals and traces back into the harness.

The post

Shivani Virdi (NeoSage, ex-Microsoft, author of R002) posted a text-only list of 19 sources in six stages, each with a one-line thesis. About 1,170 reactions, 52 comments and 134 reposts. Twelve of the 19 are Anthropic pages.

StageThesis in the postSources
1. LLM fundamentalsHallucination, cost and context limits all follow from how models are builtKarpathy, Zero to Hero; Stanford CS336
2. Prompting in productionA prompt is code: it ships, breaks things and needs testsAnthropic prompting best practices; Anthropic cookbook: prompt versioning and rollback; Microsoft Learn: advanced prompting for production agents
3. Context engineeringTools, history, memory and retrieved documents compete for one windowAnthropic: effective context engineering; Manus: lessons from building Manus; Anthropic: writing effective tools for agents; Anthropic memory tool docs; Anthropic: Contextual Retrieval
4. HarnessThe model picks the next step; the harness runs it, feeds back the result, enforces limitsAgent SDK: how the agent loop works; Anthropic: effective harnesses for long-running agents; Anthropic: harness design for long-running apps; OpenAI: harness engineering
5. EvalsA change can fix one case and break another; evals tell you before users doAnthropic: demystifying evals for AI agents; DeepEval: AI agent evaluation
6. ObservabilityIf you can't see each step, you can't debug the agentOpik tracing concepts; OpenAI Agents SDK tracing; MLflow tracing

Two comments add the substance the post lacks. One proposes a seventh stage, failure analysis: read traces, classify misses, and turn them into evals. Another ties every production failure to the prompt version that caused it and reruns those cases before each new version ships, so the eval set becomes a release history.

What the key sources say

Four of the sources are where the list's ideas are sharpest.

Harness design for long-running apps (Anthropic, Prithvi Rajasekaran, 2026-03-24). A planner, generator and evaluator, borrowed from GANs. Two failure modes drive the design: models lose coherence (some show "context anxiety" and wrap up early) as the window fills, and agents grade their own work leniently. The fix for the second is a separate evaluator, tuned to be skeptical, that drives the running app through Playwright and fails a sprint when any criterion falls below a hard threshold. Generator and evaluator negotiate a per-sprint "contract" of testable behaviours before code is written, and talk only through files.

Harness engineering (OpenAI, Ryan Lopopolo, 2026-02-11). Five months, roughly a million lines and about 1,500 merged PRs with no hand-written code; three engineers averaged 3.5 PRs each per day, and throughput rose as the team grew to seven. The engineer's job became asking "what capability is missing, and how do we make it legible and enforceable for the agent?"

Prompt versioning and rollback (Anthropic cookbook, 2026-04-07). Managed Agents stores the system prompt server-side; every agents.update mints an immutable version and sessions pin one. The worked example scores a ticket-routing agent on 20 labelled tickets, ships a v2 routing rule, sees billing drop from 4/5 to 2/5, and rolls back by pinning v1 with no deploy. The notebook says plainly that agents.update has no approval step and any workspace key can call it, so the control is to have production pin an explicit version and put that pin under review.

Demystifying evals for AI agents (Anthropic, 2026-01-09). The vocabulary the list needs: task, trial, grader, transcript, outcome (the environment's end state, not what the agent says), evaluation harness versus agent harness. Code, model and human graders each have a place; capability evals should start at a low pass rate, regression evals near 100%, and capability tasks graduate into the regression suite.

The remaining sources (Karpathy, CS336, the context-engineering, tool-writing and Contextual Retrieval posts, Manus, the Agent SDK loop and memory-tool docs, and the three tracing docs) are well-known references.

Assessment

A good list for its stated reader, an engineer starting out, and unusually current: half of it is from 2026. Its weakness is shape, not selection.

What it gets right. It treats a prompt as a versioned, tested artifact and puts evals and tracing on the path rather than as extras. The harness stage pairs a lab view (Anthropic's generator-evaluator) with a practitioner view (OpenAI's legibility and mechanical enforcement), which is the right tension to study.

What it misses.

  1. The loop. The six stages read as a line, but in production they are a cycle: traces feed failure analysis, failures become eval cases, and evals gate the next prompt, context or harness change. The top comment's "failure analysis" stage and the prompt-version release history are that loop; the post names neither.

  2. Harness subtraction. Rajasekaran's most transferable lesson, re-testing each component when a model ships and removing what is no longer load-bearing, has no stage of its own. Without evals it cannot be done safely.

  3. Change control for prompts. The versioning cookbook itself warns that server-side updates bypass code review. The list frames prompt versioning as a convenience, not as a governance problem.

  4. Security and permissions. Nothing on sandboxing, tool permissions or prompt injection, which R006's commenters and R021 treat as core harness concerns.

  5. Vendor skew. Twelve of 19 are Anthropic pages; there is little on open-weight serving, cost, or non-Claude agent stacks.

The order question. A commenter praised fundamentals-first. For an engineer who already ships software, the faster route is probably evals and tracing first on a small real agent, then the rest as each failure demands it. That is how both the OpenAI and Anthropic harness posts describe their own learning.

Review checklist for a team using the list

Sources

Connections

See R022 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.