fix(scripts): reconcile the evals gate with the warrant policy's explicit skips

[kyle-sexton]

Context

Split out of #3127 item 4. The documentation half landed in #3127's PR (docs/MIGRATION-PLAYBOOK.md § "Evals — warrant policy and consumer-verify recipe"); the script-side decision is deliberately left here because scripts/ was outside that PR's fence.

The disagreement

docs/MIGRATION-PLAYBOOK.md:303-310 states evals are warranted, not mandatory, and names two explicit skip classes: a pure-reference skill (playbooks:fable-5, tdd, …) and a skill in a hook plugin.

The gate disagrees, unconditionally:

• scripts/check-changed-skills.sh:69-72 adds --require-evals for every skill whose SKILL.md is new or modified.
• plugins/skill-quality/scripts/check-skill.sh:690-696 (Check 14) then hard-FAILs on a missing evals/evals.json — for any skill shape. Its own header (:15-17) says so explicitly.
• There is no allowlist file, frontmatter opt-out, plugin-type detection, or hook-plugin awareness in either script. Neither script has any notion of "hook plugin".
• One CI caller: .github/workflows/ci.yml:1328, which does not pass the flag — the wrapper adds it internally, so it cannot be overridden.

Consequence: a "skip" verdict in the playbook survives only until someone touches that SKILL.md. In #3115, 9 of 13 new eval sets exist for exactly this reason — the gate won by construction, not by argument.

#3127's doc half narrowed the hook-plugin skip (its own parenthetical rationale, "no model-invoked skill", is false for a hook plugin that ships a setup skill) and recorded the gate as operative. The pure-reference skip is still a live contradiction.

Two exits, pick one deliberately

1. The gate learns a recorded skip. The repo's established idiom is an exemptions file carrying the verdict — scripts/skill-count-claim-exemptions.txt is the model. A skip becomes a reviewed, diffable line rather than an implicit absence. Recommended: it preserves the warrant policy's reasoning while keeping CI fail-closed for anything unrecorded.
2. The policy drops "skip". Evals become mandatory for every skill; docs/MIGRATION-PLAYBOOK.md's warrant rule is rewritten as an authoring-quality guide rather than a ship/skip test.

Whichever wins, the losing surface changes in the same PR so the two stop disagreeing.

Related

• fix(docs): setup-contract drift — unstamped --config claim in three docs, no SSOT for the contract, stale PR-section config #3127 — parent; carries the documentation half.
• fix(plugins): correct the unstamped --config reconfiguration claim (#3111) #3115 — where the gate's cost was paid (9 hook-plugin setup-skill eval sets).

[kyle-sexton]

[kyle-sexton]

This was generated by AI during triage.

Triage result

Verified. Both cited enforcement points still exist on main unchanged: scripts/check-changed-skills.sh still passes --require-evals unconditionally for every skill whose SKILL.md is new or modified, and plugins/skill-quality/scripts/check-skill.sh Check 14 still hard-FAILs on a missing evals/evals.json for any skill shape, with no allowlist, frontmatter opt-out, or plugin-type awareness in either. docs/MIGRATION-PLAYBOOK.md § "Evals — warrant policy and consumer-verify recipe" still states the disagreement is live and explicitly names this issue as the place it gets settled: "the standing resolution ... is tracked in issue #3135 and is not settled here." No exemptions-file mechanism for evals currently exists (the repo's only precedent of the kind, scripts/skill-count-claim-exemptions.txt, covers a different claim). Not a duplicate, not already implemented, and not closeable alongside #3127 — the script-side decision was deliberately left out of that PR's fence.

Cluster check. Cross-referenced the other #3127 residuals (#3136 org-agnosticism enforcement, #3137 check-only carve-out validation, #3138 follow-up chores) — none share this decision. No decision-carrier clustering applies.

Routing: human-gated. This is a repo-wide policy call — whether evals stay warranted-not-mandatory (with an explicit, gate-honored skip class) or become mandatory for every skill in the fleet — recorded on a repo-owned policy surface (the playbook itself) as still open, not merely unimplemented. The losing surface (playbook wording or gate script logic) changes in the same PR, per the issue's own framing. That is cross-repo/product-policy territory, not a bounded implementation detail, so it is routed to a human rather than defaulted.

Agent Brief (for the human decision, then delegable follow-through)

Type: Task
Summary: Decide whether the evals-warrant policy's explicit "skip" classes survive contact with scripts/check-changed-skills.sh / plugins/skill-quality/scripts/check-skill.sh's unconditional --require-evals gate, then reconcile whichever surface loses in the same change.

Current behavior:
docs/MIGRATION-PLAYBOOK.md documents evals as warranted-not-mandatory, with an explicit skip for pure-reference skills and a narrowed skip for hook-plugin skills. The CI gate ignores both skips: any skill whose SKILL.md changes must ship evals/evals.json or CI fails, regardless of shape. The playbook already documents this exact contradiction and cites this issue as unresolved.

Desired behavior — pick one exit, then implement it fully:

• Exit A (recommended in the issue body — human veto welcome). The gate learns a recorded-skip mechanism: an exemptions surface (mirroring the existing scripts/skill-count-claim-exemptions.txt idiom) that a skill can be listed in to record a reviewed, diffable "evals not warranted" verdict. Both gate scripts consult it before failing. The playbook's warrant-rule section is updated to point at the exemptions surface as the mechanism that makes a recorded skip durable across future touches of that SKILL.md, rather than describing a skip that CI silently overrides.
• Exit B. The warrant policy drops "skip" outright. Evals become mandatory for every skill shape. docs/MIGRATION-PLAYBOOK.md's warrant-rule section is rewritten so it no longer promises a skip the gate never honored — reframed as an authoring-quality guide, not a ship/skip test. Every skill currently relying on an unrecorded skip verdict (pure-reference skills, hook-plugin skills without a setup skill) needs an evals set or an explicit accepted-debt trail.

Whichever exit wins, the losing surface (gate script logic under Exit A rejection, or playbook wording under Exit B rejection) is corrected in the same change — the two must stop disagreeing.

Key interfaces:

• The CI gate's --require-evals enforcement path (currently unconditional) — needs a recorded-skip lookup under Exit A, or stays unconditional under Exit B.
• The warrant-rule section of the migration playbook — states the policy either way; must match whatever the gate actually enforces once this lands.
• An exemptions-surface format, if Exit A is chosen — same shape/idiom as the existing skill-count-claim exemptions file (reviewed, diffable, one line per recorded verdict).

Acceptance criteria:

• The playbook's warrant-rule section and the CI gate's actual enforcement agree — no remaining sentence describing a skip the gate does not honor, and no remaining unconditional FAIL for a case the policy still calls a skip.
• Every currently-skip-classified skill (pure-reference, hook-plugin without a judgment-bearing skill) either gets a recorded skip verdict (Exit A) or ships evals/evals.json (Exit B).
• CI's changed-skill gate passes on a repo-representative sample covering both a recorded-skip case (if Exit A) and a mandatory case.
• This issue and the playbook's "not settled here" sentence are updated to reflect the closed decision.

Out of scope:

• Re-litigating the hook-plugin setup-skill narrowing already resolved in fix(docs): setup-contract drift — unstamped --config claim in three docs, no SSOT for the contract, stale PR-section config #3127's doc half.
• Writing or improving the content of any individual skill's evals — this is about gate/policy agreement, not eval quality.
• Any other fix(docs): setup-contract drift — unstamped --config claim in three docs, no SSOT for the contract, stale PR-section config #3127 residual (fix(scripts): consolidate org-agnosticism enforcement behind its stated rule #3136, fix(scripts): validate the check-only carve-out's surface, not the keyword #3137, chore: follow-ups surfaced by the setup-contract campaign (toolchain pin, CLAUDE_PLUGIN_ROOT liveness, Windows worktree test failures) #3138) — independent decisions, not clustered with this one.