Testing Agent Skills Before They Reach Production: A Validation Framework

A skill file is untested code until proven otherwise

A skill file looks like documentation -- markdown, prose, maybe a YAML header -- and that appearance is exactly why it so often ships without the scrutiny a function with the same blast radius would get. But a skill is executable in every sense that matters: it's read by a model and directly shapes what that model does on every request it matches. An untested skill is untested code that happens to be written in English.

This is deliberately scoped narrower than general agent evaluation. Evaluating whether an agent's output is good (see this site's coverage of agent evaluation at scale) is a downstream question about a whole system's behavior. This article is about the upstream question: before a specific skill file is trusted at all, does it actually do what its description claims, does it avoid firing when it shouldn't, and if it claims to self-check its own output, does that self-check actually catch a bad answer?

Golden-example sets: the skill's own unit tests

The direct analogue of a unit test suite for a skill is a set of golden examples: realistic inputs paired with the expected shape of output, not necessarily the exact text (LLM output isn't that deterministic) but the properties a correct response must have. For a skill like this site's code-review skill, a golden example is a diff with a known, deliberately-planted bug, and the assertion isn't "the model says X" but "the model's findings include a finding anchored at the planted bug's line, with a correct failure scenario."

Build the set from two sources deliberately. Positive cases -- inputs the skill should fire on and handle well, covering the range of what "review this diff" can actually mean (a one-line fix, a multi-file refactor, a diff with no real bugs at all, where the correct output is "nothing found"). Negative cases -- inputs that are adjacent to the skill's domain but should NOT trigger it, or should trigger a different, more specific skill instead. A skill validation suite that only tests positive cases will pass while the skill quietly fires on everything.

- id: assignment-in-condition
  input: "diff:\n+ if (user.plan = 'pro') { return 0.2 }"
  expect:
    finding_present: true
    anchors_line: 2
    fix_mentions: "=="
- id: clean-diff-no-issues
  input: "diff:\n+ function add(a, b) { return a + b }"
  expect:
    finding_present: false
    explicit_no_issues_statement: true   # not silence -- a stated verdict
- id: not_a_review_request
  input: "explain what this function does"
  expect:
    skill_should_not_match: true

Regression testing on every edit

A skill file is edited far more casually than a function usually is -- it's prose, so the temptation to tweak a sentence "real quick" without re-running anything is strong, and that's exactly the moment regressions sneak in. Wording changes that seem cosmetic can change model behavior in ways a code diff never would: reordering two instructions, softening an imperative into a suggestion, or trimming an example can measurably shift what the model does on the same input.

The fix is procedural, not clever: the golden-example set from the previous section is not a one-time acceptance test, it's a regression suite that reruns on every edit to the skill file, the same discipline as running a test suite before merging a code change. Track pass rate over time per skill; a skill whose golden-set pass rate drops after an edit that was supposed to be unrelated is telling you the edit had a side effect the author didn't intend -- exactly the signal a code diff's test suite exists to surface.

Because model outputs aren't bit-for-bit deterministic, "pass" has to mean the output satisfies the expected properties (finding present at the right anchor, correct verdict on a clean case) rather than exact string match, and a flaky-looking regression is worth investigating the same way a flaky test in any other system is worth investigating rather than dismissing as noise.

Testing for the false-positive problem specifically

The single most damaging failure mode for a skill isn't that it misses something -- it's that it reports something that isn't real, because a handful of false positives destroys a reader's trust in every subsequent true positive from the same skill faster than an equivalent number of false negatives does. Testing for this needs its own dedicated pass, separate from ordinary correctness testing.

Concretely: take a set of inputs engineered to look superficially like the skill's trigger pattern without actually containing the defect (code that looks unusual but is correct, a diff that touches a security-sounding filename but changes nothing security-relevant). A skill passes this pass only if it correctly reports "no issue" on every one of them. A skill that has never been tested against deliberately-suspicious-looking-but-actually-fine inputs has an unmeasured false-positive rate, which in practice means an unknown one.

This is also where you test whether a skill's own anti-false-positive design actually works. This site's code-review skill explicitly instructs a verify pass -- "for each candidate finding, state the concrete input that triggers it, or drop it." Testing that instruction means feeding it a plausible-sounding but ultimately baseless finding and confirming the verify step actually drops it rather than rubber-stamping the first draft. An instruction to self-verify is itself a claim that needs a test, not a guarantee that needs none.

What a validation harness actually looks like

In practice this doesn't need bespoke tooling: a small script that loads a skill's markdown body, constructs a prompt per golden example (system context plus the skill body plus the example input), calls the model, and checks the response against each example's expected properties is enough to start. The properties worth asserting on are structural, not exact-text: did a finding appear, is it anchored where expected, does a "no issues" case actually produce an explicit no-issues statement rather than just silence that could equally mean the model didn't understand the task.

Run the harness in CI on any change to a skill file, the same trigger as a code test suite, and block the merge on regression the same way. The cost is a handful of model calls per skill per change -- cheap relative to a skill silently regressing in production and eroding trust in every skill on the same site or team before anyone notices which one caused it.