R110 · Agent architecture › Mental models of the agent system

harness-profiler: a profiler for Claude Code skill and agent factories

Read the original

observabilityagent-harnessagent-skillsinference-costprompt-cachingharness-config

Aleksandr Furmanov's open-source Claude Code plugin records every skill, subagent and tool call through hooks and draws a session as a UML-style sequence diagram, one bar per context, with duration and new, cached and output tokens per context. Its point: once a harness grows to dozens of skills and agents, a run can report PASS while doing redundant work, and only a call tree with costs shows it.

How it works

Capture is a set of async Claude Code hooks; analysis is an offline tree builder over the trace. Everything is one dependency-free Node file (hp.js) behind a POSIX launcher.

Rejected designs worth noting: Perfetto and Gantt or flame timelines (readers mistook overlapping calls for nesting), per-OS binaries (unreadable hooks), fetching the tool on first run (breaks behind cloud network policy), and a Stop hook pushing traces to a side branch.

What it found: a "final" validation that redid everything

The blog post's real case cost up to a minute and 48k new tokens per change, and both runs reported PASS. The task was a small Ruby SDK change (reject a blank API key). The workflow validates twice: after the code change, then again after the agent trims its planning doc to a decision record. The second run is meant to be cheap: the validator consults the trace and reuses any check that passed after the last change to its files, so only the doc checks should rerun.

RunDurationNew tokensCached tokensOutput tokens
First validation69 s66.7k426.4k2.9k
Final validation60 s48.0k447.0k3.2k

The final run reran the Rails review (14 s, 41.4k new tokens), tests and lint, though only Markdown had changed. The author spotted it visually: two validation bars of near-equal length with identical children. Two root causes: the code was left uncommitted, so the reuse rule saw every Ruby file as changed, and the reuse script crashed under an old system Ruby. The fix was to commit before final validation and run the script under the project's Ruby.

The transferable lesson is the last step: once you know what a healthy trace looks like, encode it as a check-trace skill (ordered skill calls, which calls must run in fresh contexts, no duplicate skill runs per story) and run it after sessions. That turns the profiler from a debugging aid into a behavioural regression test for the harness.

Limits and tradeoffs

The profiler sees only what hooks and transcripts expose, so it shows structure and spend, not content or causality.

Versus built-in tracing. A commenter points to Claude Code's OpenTelemetry tracing beta (CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 plus an OTEL_TRACES_EXPORTER), where subagent spans nest under the parent's tool span. If that holds, OTel is the better fit for fleet-wide dashboards in an existing backend, and harness-profiler's additions are skill boundaries, inline-versus-fork distinction, per-context token kinds, and a zero-infrastructure local page. The two are complementary rather than competing.

On the billing claim. The post says Anthropic moved Claude Enterprise to token billing in April and GitHub moved Copilot in June; it says "reportedly" for the first. The argument does not need it: per-context spend matters for latency and rate limits even on flat plans.

Review checklist

  1. Record a baseline trace of a known-good run for each main workflow before changing skills; diff later traces against it.

  2. Decide which skills must run in a fresh context (validators, reviewers) and assert that in a check-trace skill; inline-versus-fork is both a correctness and a cost choice.

  3. Watch the new column, not totals: copies of the parent inflate cached cheaply, while repeated fresh contexts drive real spend.

  4. Look for repeated children under sibling bars (the double-validation pattern) and for any reuse or caching rule that depends on git state.

  5. Run doctor in CI or session start so a missing Node or stale committed hook doesn't silently record nothing.

  6. Treat traces as sensitive: they hold tool arguments and result first lines. Keep them excluded from git and think before publishing one.

  7. If you already ship OpenTelemetry, decide which questions each tool answers rather than running both by default.

Sources

Connections

See R110 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.