R064 · Agent principles and patterns › Evaluation data for agents

SkillEvaluator: measure whether an agent skill helps, then delete the ones that don't

Read the original

evalsagent-skillsharness-configbenchmarks

Sources: André Lindenberg's LinkedIn post (2026-11-12); NVIDIA/SkillEvaluator README (Apache-2.0); Kevin, Raghavan, Puget et al., Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills, arXiv 2608.20614 (20 Aug 2026, 15 pages); NVIDIA Developer blog.

The claim

Most teams that adopt agent skills (folders of instructions and scripts in the Agent Skills format) accumulate them without ever measuring whether they help. SkillEvaluator treats a skill as an executable artifact: run the same task, with the same model, sandbox and grader, once with the skill and once without, and report the difference. Across a large catalog, that tells you which skills to keep, fix or delete. The post's figures: about 800 GitHub stars, Apache-2.0.

How it works: three independent tiers

TierQuestionWhat runsNeeds
1. ValidationIs the skill safe and well-formed?Structural quality score (0 to 100, pass at 70), SkillSpector skill-security scan, Gitleaks, optional Semgrep, optional LLM rubricScanners; no model for the static score
2. DeduplicationDoes guidance repeat or overlap other skills?Embeddings plus chat; --catalog compares across the whole skill catalogEmbedding and chat endpoints
3. Live evaluationDoes the skill help the agent?Paired agent runs with and without the skill in a sandbox (Harbor, Docker by default), graded on six runtime metricsA supported agent (Claude Code, Codex and others), provider access, a sandbox

validate runs all three. Tier 1 always gates, Tier 2 gates unless disabled, Tier 3 gates only when asked (--block-on-agent-eval). If a skill has no eval cases, an "autopilot" writes one starter case; create-eval-dataset --full generates more, but negatives only exist if someone writes them in EVAL.md. A scanner that cannot run reports INCOMPLETE and exits nonzero rather than passing silently. CI templates for GitHub Actions and GitLab check changed skills on each PR and can post reports as PR comments.

The evidence (paper)

The paper calls the method ACES and its headline number Skill Lift: the added value of one skill for a fixed task, harness, workspace and scorer. Trajectories are normalized to a shared format (ATIF) before grading, so the same scorer works across harnesses.

What is transferable

  1. Measure a skill against its absence, not against a rubric. The with/without pair under a fixed model, sandbox and grader is the unit of evidence. Everything else (style, structure, security) is a precondition.

  2. Deduplicate at the catalog level. Overlapping skills compete for the same trigger and dilute routing; a catalog-wide similarity pass is cheap compared with live runs.

  3. Gate in CI on what changed. Run static and dedup checks on every PR that touches a skill; run live evaluation on a schedule or before release, because it costs model time and sandbox time.

  4. Re-run after model upgrades. Lift is defined for a fixed model and harness, so a skill that helped last quarter's model can be dead weight on this one. This gives the "prune old config" advice an evidence base instead of a wipe.

  5. Fail closed on missing evidence. INCOMPLETE instead of a green check when a scanner or eval did not run is a pattern worth copying in any agent-config pipeline.

Caveats

Bottom line

A practical, open implementation of the obvious-but-rarely-done experiment: does this skill make the agent better at this task? The three-tier split (cheap static checks always, catalog dedup often, paired live runs when it matters) is the right cost shape. Treat the paper's averages as "skills usually help a bit", and use the tool to find the quarter that don't.

Connections

See R064 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.