Sources: André Lindenberg's LinkedIn post (2026-11-12); NVIDIA/SkillEvaluator README (Apache-2.0); Kevin, Raghavan, Puget et al., Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills, arXiv 2608.20614 (20 Aug 2026, 15 pages); NVIDIA Developer blog.
The claim
Most teams that adopt agent skills (folders of instructions and scripts in the Agent Skills format) accumulate them without ever measuring whether they help. SkillEvaluator treats a skill as an executable artifact: run the same task, with the same model, sandbox and grader, once with the skill and once without, and report the difference. Across a large catalog, that tells you which skills to keep, fix or delete. The post's figures: about 800 GitHub stars, Apache-2.0.
How it works: three independent tiers
| Tier | Question | What runs | Needs |
|---|---|---|---|
| 1. Validation | Is the skill safe and well-formed? | Structural quality score (0 to 100, pass at 70), SkillSpector skill-security scan, Gitleaks, optional Semgrep, optional LLM rubric | Scanners; no model for the static score |
| 2. Deduplication | Does guidance repeat or overlap other skills? | Embeddings plus chat; --catalog compares across the whole skill catalog | Embedding and chat endpoints |
| 3. Live evaluation | Does the skill help the agent? | Paired agent runs with and without the skill in a sandbox (Harbor, Docker by default), graded on six runtime metrics | A supported agent (Claude Code, Codex and others), provider access, a sandbox |
validate runs all three. Tier 1 always gates, Tier 2 gates unless disabled, Tier 3 gates only when asked (--block-on-agent-eval). If a skill has no eval cases, an "autopilot" writes one starter case; create-eval-dataset --full generates more, but negatives only exist if someone writes them in EVAL.md. A scanner that cannot run reports INCOMPLETE and exits nonzero rather than passing silently. CI templates for GitHub Actions and GitLab check changed skills on each PR and can post reports as PR comments.
The evidence (paper)
The paper calls the method ACES and its headline number Skill Lift: the added value of one skill for a fixed task, harness, workspace and scorer. Trajectories are normalized to a shared format (ATIF) before grading, so the same scorer works across harnesses.
Static checks and live results measure different things. On 145 real skills, structural scores and LLM-judge scores barely agree (Spearman ρ = 0.14). A clean, well-written skill is not evidence that it helps.
Most skills help, on average, but not all. Across 947 scored paired cases from 58 of 64 production skills and four harnesses, mean composite lift is 0.213 (95% CI 0.197 to 0.230); outcome-only lift (accuracy and goal accuracy) is 0.180. Lift is positive in 72.8% of paired cases, so roughly a quarter of cases show no gain or a loss.
The gains are mostly in process. The largest improvements are in skill execution, behavior checks and efficiency: whether the agent found the skill, routed to it, followed the workflow and used tools well. Document scans cannot see any of that.
What is transferable
Measure a skill against its absence, not against a rubric. The with/without pair under a fixed model, sandbox and grader is the unit of evidence. Everything else (style, structure, security) is a precondition.
Deduplicate at the catalog level. Overlapping skills compete for the same trigger and dilute routing; a catalog-wide similarity pass is cheap compared with live runs.
Gate in CI on what changed. Run static and dedup checks on every PR that touches a skill; run live evaluation on a schedule or before release, because it costs model time and sandbox time.
Re-run after model upgrades. Lift is defined for a fixed model and harness, so a skill that helped last quarter's model can be dead weight on this one. This gives the "prune old config" advice an evidence base instead of a wipe.
Fail closed on missing evidence.
INCOMPLETEinstead of a green check when a scanner or eval did not run is a pattern worth copying in any agent-config pipeline.
Caveats
Vendor tool, vendor paper. The paper's authors build the tool; the production skills are largely NVIDIA-internal. No independent replication yet.
Lift depends on the cases. One autopilot starter case per skill is thin coverage, and generated cases share the generator's blind spots. Without authored negative cases (tasks where the skill should not fire), lift overstates value and never catches harmful triggering.
Composite lift mixes outcome and process metrics. The outcome-only number (0.180) is the one to quote when the question is "does the task get done"; process gains can reflect the grader rewarding the skill's own workflow.
Cost and variance. Paired live runs per skill per harness add up across hundreds of skills, and the abstract reports means and CIs over cases, not run-to-run noise for a single skill. A small lift on one skill needs repeated trials before you delete or keep it.
Free tier is NVIDIA's. The easiest model-backed path uses NVIDIA Build keys; other providers are configurable.
Bottom line
A practical, open implementation of the obvious-but-rarely-done experiment: does this skill make the agent better at this task? The three-tier split (cheap static checks always, catalog dedup often, paired live runs when it matters) is the right cost shape. Treat the paper's averages as "skills usually help a bit", and use the tool to find the quarter that don't.