quality-net: skill changes ship with zero regression detection — 120 evals.json unexecuted, skill-quality:check uninvoked (staged eval/gate/pass^k umbrella)

[kyle-sexton]

Category: quality infrastructure (umbrella) · Source: control-tower gap-hunt 2026-07-19 (skill-catalog audit + external research), operator-directed proactive sweep

Problem — the repo's single biggest quality gap

This repo edits its own skills daily via the autonomous pipeline, yet has NO skill-regression safety net at any layer:

• 120 of 146 skills ship evals/evals.json; nothing executes them. CI's plugin-gate runs run-plugin-tests.sh (*.test.sh) + validate-plugins.sh (manifest schema); validate-plugin-contracts.mjs has no eval handling — evals.json is not even schema-checked.
• skill-quality:check (17-check static gate: frontmatter, trigger-keyword preservation vs HEAD, line caps, broken refs, evals presence) is invoked by nothing — not CI, not claude-review, not the work lane. Its validate-evals action is also uncalled.
• The work lane's high-blast-radius rule makes the orchestrator manually eyeball skill diffs — hand-reimplementing what the static gate does deterministically.

Every merged skill change is an untested behavior change.

Staged path (each stage independently valuable)

1. Deterministic CI lane (cheap, now): run skill-quality:check static gate + validate-evals on any PR touching plugins/*/skills/** — replaces the work lane's manual diff-read gate.
2. Model-graded eval execution of CHANGED skills in the work-lane post-green pass: single judge call per dimension returning 0.0-1.0 + pass/fail (Anthropic guidance: per-dimension isolated judges reduce hallucination; 0.0-1.0+pass/fail "most consistent and aligned with human judgements"). Start with 20-50 tasks harvested from real pipeline failures. Source: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
3. Frozen regression suite: tasks the pipeline previously passed, run on every skill/prompt change, near-100% pass expected; any dip = silent drift caught. This is the net the rule→skill absorption lifecycle (source-control(babysit-prs): absorb v4 loop-prompt operating rules into the skill #477-feat(claude-ops or session-flow): loop-prompt authoring skill — generate and maintain standing loop-lane prompts #480) currently lacks.
4. pass^k consistency metric as the feat(source-control): babysit autopilot merge tier — bot-account review + criteria-gated merge #476 autopilot unlock criterion: gate autonomous merges on probability ALL k trials succeed (pass^k), not pass@k — quantitative unlock instead of judgment call.
5. Clean-state eval isolation: each trial from fresh state; Anthropic caught agents reading prior-trial git history and inflating pass rates. Matters as soon as stages 2-4 exist.

Relations

#476 (stage 4 is its unlock criterion), #477-#480 (absorption needs the regression net), #513 (mini-SDLC verify lane consumes these gates), #445 (mechanical-conformance CI backlog — this extends it to behavior).

[kyle-sexton]

This was generated by AI during triage. · Category: quality infrastructure (umbrella) · Priority: priority: high (the repo's single biggest quality gap — every merged skill change is an untested behavior change). Routed needs-human + wayfind: design. Decompose candidate.

Why needs-human

A 5-stage quality-net umbrella spanning cheap-CI-wiring through pass^k autopilot-unlock design. The staging, the eval-harness design, the pass^k-as-#476-unlock metric, and clean-state isolation are genuine design + product-sequencing decisions the maintainer must own — not a defaultable fork.

Peelable now (does not need the umbrella decision)

Stage 1 is ready-shaped and shouldn't wait: a deterministic CI lane running the existing skill-quality:check static gate + its validate-evals action on any PR touching plugins/*/skills/** (silent-skip-gate shape). This replaces the work lane's manual high-blast-radius diff-read with a deterministic gate and wires up two skills that are currently invoked by nothing. If the maintainer wants, split Stage 1 into its own status: ready issue immediately; Stages 2-5 stay here for design.

The design decisions (Stages 2-5)

• Stage 2 — model-graded eval execution of CHANGED skills in the work-lane post-green pass (per-dimension isolated judges, 0.0-1.0 + pass/fail per Anthropic guidance). Seed 20-50 tasks from real pipeline failures.
• Stage 3 — frozen regression suite (previously-passed tasks, near-100% expected; any dip = silent drift). This is the net the source-control(babysit-prs): absorb v4 loop-prompt operating rules into the skill #477-feat(claude-ops or session-flow): loop-prompt authoring skill — generate and maintain standing loop-lane prompts #480 rule→skill absorption lifecycle currently lacks.
• Stage 4 — pass^k consistency metric as the feat(source-control): babysit autopilot merge tier — bot-account review + criteria-gated merge #476 autopilot unlock criterion (probability ALL k trials pass — quantitative unlock, not judgment). Directly gates feat(source-control): babysit autopilot merge tier — bot-account review + criteria-gated merge #476.
• Stage 5 — clean-state eval isolation (fresh state per trial; prevents prior-trial git-history inflation).

Recommended path

Peel Stage 1 to ready now; run /work-items:decompose on Stages 2-5 into per-stage issues; sequence Stage 4 with #476. Relations: #476 (unlock), #477-#480 (absorption needs this net), #513 (mini-SDLC verify lane consumes these gates), #445 (mechanical-conformance CI — this extends it to behavior).

[kyle-sexton]

This was generated by AI (needs-human decision-brief pass).

Decision brief (2-minute read)

Problem. This repo rewrites its own skills daily through the autonomous pipeline, yet no layer executes any behavioral regression check — so every merged skill change is an untested behavior change. The triage note already peeled Stage 1 (deterministic CI lane) as ready-shaped; what remains for you is the sequencing/design decision on the model-graded stages (2-5) and, specifically, whether the cheap deterministic lane ships now on its own.

Today (verified).

• 120 of 146 skills ship evals/evals.json (find plugins -path '*/skills/*/evals/evals.json' = 120; SKILL.md count = 146). Nothing executes them.
• evals.json is not even schema-checked: scripts/validate-plugin-contracts.mjs has zero eval handling (a schema exists at plugins/skill-quality/reference/evals.schema.json, unused by CI).
• skill-quality:check exists (plugins/skill-quality/skills/check/SKILL.md, engine plugins/skill-quality/scripts/check-skill.sh — its Check 14 only WARNs on missing evals) and is invoked by no CI lane. ci.yml jobs are: hygiene, zizmor, hook-utils-sync, standards-contract-sync, cross-plugin-source-drift, silent-skip-gate, plugin-gate, miro-plugin, runner-policy, ci-status. None run or schema-check evals; the only eval strings in ci.yml are grep-exclusion paths. Confirmed: no eval-execution lane exists.

Options.

• A — RECOMMENDED: ship Stage 1 now as its own lane; keep Stages 2-5 here for staged design. Wire a deterministic CI lane running the existing skill-quality:check static gate + validate-evals on any PR touching plugins/*/skills/** (silent-skip-gate shape, mirroring the existing silent-skip-gate job). This replaces the work lane's manual high-blast-radius diff-read with a deterministic gate and activates two skills currently invoked by nothing — zero model cost, uses only what already exists. Stages 2 (model-graded judges on changed skills), 3 (frozen regression suite), 4 (pass^k as the feat(source-control): babysit autopilot merge tier — bot-account review + criteria-gated merge #476 unlock metric), 5 (clean-state isolation) stay as a decompose target. Tradeoff: Stage 1 alone catches static/structural regressions, not behavioral drift — but it is the cheapest real net and unblocks the others independently. Reversibility: high — a single removable lane; per-stage issues are independently droppable.
• B — design the whole eval harness (Stages 1-5) before shipping anything. One coherent net, no half-measures. Tradeoff: leaves the biggest quality gap open for the full design cycle while the pipeline keeps merging untested skill changes; Stage 1's value is deferred for no reason. Reversibility: high (nothing shipped yet) but the exposure window is the cost. Not recommended — Stage 1 has no dependency on the rest.
• C — model-graded only, skip the static lane. Go straight to per-dimension judge calls on changed skills. Tradeoff: highest signal but per-run model cost and hallucination-management overhead from day one, and it re-implements deterministic checks the static gate already does for free. Reversibility: medium. Not recommended as the first move.

Cross-links.

• feat(source-control): babysit autopilot merge tier — bot-account review + criteria-gated merge #476 (autopilot merge tier — APPROVED, staged: source-control:babysit-prs: gh-bot.sh bot token generation fails, worker writes fall back to personal identity #450 → babysit-prs: merge-step no-monitor STOP, bare-wrapper pipe rule, snapshot exit-code taxonomy, cross-PR dependency channel #373/fix(source-control): review-trigger head-staleness hardening (BLOCKED-behind guard, stale-head reactions) #327/fix(source-control): require per-thread pins for autonomous bulk babysit thread resolves #326 → implement): Stage 4's pass^k is the quantitative autopilot-unlock criterion feat(source-control): babysit autopilot merge tier — bot-account review + criteria-gated merge #476 consumes. This is the load-bearing link, not a footnote.
• CI-gate backlog: automatable conformance checks surfaced by the #313 fleet audit #445 (CI-gate backlog): sibling gate, but mechanical-conformance only (CHANGELOG/version/frontmatter). quality-net: skill changes ship with zero regression detection — 120 evals.json unexecuted, skill-quality:check uninvoked (staged eval/gate/pass^k umbrella) #530 extends gating from conformance to behavior — complementary, not overlapping.
• work-items(work): per-issue execution should compose installed skills as a role-separated mini-SDLC (implementer ≠ reviewer ≠ verifier, ephemeral fresh-context agents) #513 (mini-SDLC verify lane) consumes these gates; source-control(babysit-prs): absorb v4 loop-prompt operating rules into the skill #477-feat(claude-ops or session-flow): loop-prompt authoring skill — generate and maintain standing loop-lane prompts #480 (rule→skill absorption) is the lifecycle whose regression net Stage 3 supplies.
• source-control: no read/consume-only posture — "safe" babysit tier and pull-request monitor write to GitHub autonomously #440 (outward_write_mode: report-only, default report-only): keep every eval/gate lane a pure measurement surface (no outward writes) so it stays report-only-compatible.
• Sibling umbrellas — discriminator: quality-net: skill changes ship with zero regression detection — 120 evals.json unexecuted, skill-quality:check uninvoked (staged eval/gate/pass^k umbrella) #530 = signal absent (evals/gates never wired or invoked, no verdict produced); ci: portability lint lane — enforce declared agnosticism mechanically (19-issue coupling class + top review-churn source) #531 = coupling/agnosticism axis; convention: false-green checks — surfaces report success while the capability is dead or findings invisible (liveness-assertion umbrella) #532 = signal false (check runs green and lies). Cross-link, do not merge.

Decision asked: Ship Stage 1's deterministic skill-quality:check CI lane now and decompose Stages 2-5 separately (Option A) — accept?

[kyle-sexton]

Operator decision (2026-07-19, decision-session batch 3; relayed).

ACCEPTED — the decision brief's RECOMMENDED option verbatim.

Re-intake: removing needs-human, adding priority: needs-triage for routing.

[kyle-sexton]

This was generated by AI during triage.

Routing status: ready per ratified decision. Operator ACCEPTED Option A verbatim: ship Stage 1 now as its own CI lane — wire skill-quality:check's static gate + validate-evals on any PR touching plugins/*/skills/** (silent-skip-gate shape, mirroring the existing silent-skip-gate job), replacing the work lane's manual high-blast-radius diff-read. Stages 2-5 (model-graded judges, frozen regression suite, pass^k autopilot-unlock metric for #476, clean-state isolation) stay a separate decompose target — not scoped into this ready item; recommend /work-items:decompose on this issue once Stage 1 lands, to split 2-5 into independently-sequenced follow-ups.

Priority: scaling-chain member (rule 8b names #530 directly) — kept priority: high. wayfind: design removed (sequencing decision settled).