chore(evals): author skill evals for implementation (5 skills)

plugins/implementation/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "implementation",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Disciplined implementation stage: execute approved plans inline or via orchestrated workers, run polyglot build/test/lint verification, author and diagnose tests, drive live E2E checks, and prove changes achieved their intended outcome — with measurable-improvement claims verified against planning-time baselines.",
   "author": {
     "name": "Melodic Software",

plugins/implementation/CHANGELOG.md

@@ -3,6 +3,18 @@
 All notable changes to the `implementation` plugin are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning.
 
+## [0.3.0]
+
+### Added
+
+- **Rich-form evals for five skills.** `evals/evals.json` ships for `implement`, `implement-dispatch`,
+  `build`, `lint`, and `setup` — the skills' judgment-bearing contracts (mode/orchestration routing,
+  divergence and scope-fence guardrails, skip-not-FAIL and consumer-config-precedence behavior, and the
+  config-writer's interview/write-scope discipline) are now covered by objectively-verifiable cases,
+  modeled on the `bug-report` rich-form exemplar and validated against
+  `plugins/skill-quality/reference/evals.schema.json`. Evals are a shipped component, so this minor bump
+  is their delivery vehicle; no behavioral change to the skills themselves.
+
 ## [0.2.0]
 
 ### Changed

plugins/implementation/skills/build/evals/evals.json

@@ -0,0 +1,78 @@
+{
+  "skill_name": "build",
+  "evals": [
+    {
+      "id": 1,
+      "name": "auto-detects-affected-ecosystems-from-git-status",
+      "prompt": "/implementation:build — I just edited a couple of .cs files and a pyproject-managed .py module.",
+      "expected_output": "Classifies the changed files from git status against each covered ecosystem's globs, detects the dotnet and python ecosystems, resolves each one's command surface via the ladder, and runs build -> test -> lint per ecosystem — reporting a per-ecosystem results table. It does not run ecosystems with no changed files.",
+      "files": [],
+      "expectations": [
+        "Detects the affected ecosystems (dotnet, python) by matching changed files against ecosystem globs, not by building the whole repo",
+        "Runs build -> test -> lint in order for each affected ecosystem",
+        "Emits a per-ecosystem results table using pass / FAIL / skip / — statuses",
+        "Does not run ecosystems that have no changed files"
+      ]
+    },
+    {
+      "id": 2,
+      "name": "missing-tool-reports-skip-not-fail",
+      "prompt": "/implementation:build python — but `uv` is not installed on this machine.",
+      "expected_output": "Verifies the tool is on PATH before running; since uv is missing it reports the python ecosystem as `skip` with the ecosystem's install-hint — it does NOT report FAIL for a missing tool.",
+      "files": [],
+      "expectations": [
+        "Reports the ecosystem as `skip` (with the install-hint) rather than `FAIL` when the required tool is absent from PATH",
+        "Distinguishes a missing-tool skip from an actual build/test/lint failure",
+        "Does not treat the absent tool as a red build"
+      ]
+    },
+    {
+      "id": 3,
+      "name": "consumer-ecosystem-config-wins-over-bundled-default",
+      "prompt": "/implementation:build dotnet — the repo has a tracked `.claude/ecosystems/dotnet.yaml` with a project-specific test-cmd.",
+      "expected_output": "Resolves the dotnet command surface through the four-rung ladder with the consumer's tracked `.claude/ecosystems/dotnet.yaml` authoritative (rung 1) — the bundled portable default is only the rung-4 fallback. It runs the consumer's declared commands, not the bundled ones.",
+      "files": [],
+      "expectations": [
+        "Resolves commands via the ladder with the consumer's `.claude/ecosystems/dotnet.yaml` taking precedence (rung 1)",
+        "Uses the bundled portable default only as the rung-4 fallback when no consumer file is present",
+        "Runs the consumer's declared test-cmd, not the bundled default's"
+      ]
+    },
+    {
+      "id": 4,
+      "name": "build-scope-excludes-lint-only-surfaces",
+      "prompt": "/implementation:build all — the working tree changed a GitHub Actions workflow YAML and a plain README, among other files.",
+      "expected_output": "Runs only the ecosystems /build covers (dotnet, python, typescript, bash, powershell, markdown). It does NOT run the lint-only `yaml` or `cross-cutting` surfaces — those belong to /lint, and cross-cutting's `**` glob is never matched here.",
+      "files": [],
+      "expectations": [
+        "Runs only the six /build-covered ecosystems (dotnet, python, typescript, bash, powershell, markdown)",
+        "Does NOT run the `yaml` surface for the workflow file — that is /lint-only",
+        "Does NOT run the `cross-cutting` surface — its `**` glob is never matched by /build (both `yaml` and `cross-cutting` are /lint-only surfaces)"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "clean-tree-falls-back-to-branch-diff",
+      "prompt": "/implementation:build — every block was already committed on this feature branch, so the working tree is clean.",
+      "expected_output": "With a clean working tree it falls back to the branch diff (git diff against the merge-base with the default branch) so checkpoint-committed work still gets classified and verified — rather than reporting 'no changes' and exiting.",
+      "files": [],
+      "expectations": [
+        "Falls back to the branch diff vs the default-branch merge-base when the working tree is clean",
+        "Classifies and verifies the checkpoint-committed changes rather than reporting nothing to do",
+        "Only reports 'No changes found' when neither the working tree nor the branch diff yields changes and no ecosystem argument was given"
+      ]
+    },
+    {
+      "id": 6,
+      "name": "malformed-consumer-config-degrades-not-hard-stops",
+      "prompt": "/implementation:build dotnet — the repo's tracked `.claude/ecosystems/dotnet.yaml` exists but is malformed (invalid YAML / missing required keys).",
+      "expected_output": "A malformed consumer ecosystem file warns and degrades to inference (rung 2) rather than hard-stopping — the ladder is tolerant, so a bad consumer file never aborts the build; it still resolves a command surface and runs.",
+      "files": [],
+      "expectations": [
+        "Warns that the consumer `.claude/ecosystems/dotnet.yaml` is malformed rather than silently ignoring it",
+        "Degrades to inference (rung 2) to resolve the command surface instead of hard-stopping",
+        "Does NOT abort the run with a hard error because the consumer file is malformed"
+      ]
+    }
+  ]
+}

plugins/implementation/skills/implement-dispatch/evals/evals.json

@@ -0,0 +1,80 @@
+{
+  "skill_name": "implement-dispatch",
+  "evals": [
+    {
+      "id": 1,
+      "name": "autonomous-dispatches-even-when-plan-is-all-main-window",
+      "prompt": "/implementation:implement-dispatch — execute the approved plan. This is an autonomous goal-loop run with no human in the turn cycle; the plan's routing table lists every phase as main-window.",
+      "expected_output": "Detects autonomous orchestration mode and dispatches scope-fenced workers per phase anyway — orchestrated cadence is the default in autonomous mode even when the plan routes all-main-window, so it synthesizes per-phase worker rows rather than doing volume edits from the orchestrator, capping concurrent dispatch at 3-5 workers.",
+      "files": [],
+      "expectations": [
+        "Detects autonomous mode from the goal-loop / no-human-in-cycle signal",
+        "Dispatches workers per phase even though the routing table is all main-window — synthesizes worker rows rather than editing inline",
+        "Does NOT perform the volume edits itself from the orchestrator window in autonomous mode",
+        "Caps concurrent dispatch at roughly 3-5 workers per wave"
+      ]
+    },
+    {
+      "id": 2,
+      "name": "interactive-all-main-window-hands-back-to-implement",
+      "prompt": "/implementation:implement-dispatch — this is an interactive session with me reviewing each turn, and the approved plan has no routing table (all phases are main-window).",
+      "expected_output": "In interactive mode with no worker rows in the routing table, hands execution back to /implement's classic inline cadence instead of dispatching workers — dispatch is only for worker-routed phases here.",
+      "files": [],
+      "expectations": [
+        "Detects interactive mode (human reviewing each turn) rather than autonomous",
+        "Hands back to /implement classic inline cadence because the routing table is absent / all main-window",
+        "Does NOT force worker dispatch onto an interactive all-main-window plan"
+      ]
+    },
+    {
+      "id": 3,
+      "name": "verifies-worker-return-against-direct-evidence",
+      "prompt": "/implementation:implement-dispatch phase-2 — the worker for phase 2 reported back: 'Done, all files updated and the build is green.' Accept the phase and move on.",
+      "expected_output": "Treats the worker return as synthesis, not ground truth: promotes its claims to direct evidence (diff read, grep, file Read) and runs /build itself main-side before accepting the edits — it never accepts the worker's green claim as the build signal.",
+      "files": [],
+      "expectations": [
+        "Does NOT accept the worker's 'build is green' claim as the authoritative build signal",
+        "Verifies the worker's claims against direct evidence (reads the diff / greps / reads files) before building on them",
+        "Runs /build itself from the main window rather than trusting the worker's report"
+      ]
+    },
+    {
+      "id": 4,
+      "name": "brief-carries-scope-fence-and-divergence-clause",
+      "prompt": "/implementation:implement-dispatch phase-1 — compose the brief and dispatch the worker for phase 1.",
+      "expected_output": "The composed brief carries an explicit scope fence (enumerated ALLOWED and FORBIDDEN files/actions), the verbatim divergence-escalation clause instructing the worker to STOP and report if an assumption proves wrong or it must touch anything FORBIDDEN, the touched project invariants, and the phase acceptance criteria — before dispatching.",
+      "files": [],
+      "expectations": [
+        "The brief enumerates an explicit scope fence: ALLOWED vs FORBIDDEN files/actions",
+        "The brief includes the verbatim divergence-escalation clause (STOP and report on a wrong assumption or a FORBIDDEN touch — do not improvise)",
+        "The brief includes the phase's acceptance criteria and the project invariants the task touches",
+        "The scope fence and clause are present before dispatch, not added after the worker returns"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "non-interactive-moderate-divergence-takes-conservative-option",
+      "prompt": "/implementation:implement-dispatch — autonomous run, no human to escalate to. A worker hit a moderate divergence: the plan named a config key that turned out slightly different, and it's asking how to proceed.",
+      "expected_output": "In a non-interactive run, moderate divergence does not deadlock the run: it picks the conservative option (truest to the plan's intent, smallest blast radius), logs it to a DEVIATIONS.md beside the plan artifact, and keeps going. Major divergence would still STOP even autonomously.",
+      "files": [],
+      "expectations": [
+        "Does NOT deadlock the autonomous run waiting for a human on a moderate divergence",
+        "Picks the conservative option (closest to plan intent, smallest blast radius) for the moderate divergence",
+        "Logs the deviation to a DEVIATIONS.md beside the plan artifact (what was planned, what was done, why, blast radius)",
+        "Reserves a hard STOP for major divergence (a fundamental assumption wrong), not this moderate one"
+      ]
+    },
+    {
+      "id": 6,
+      "name": "autonomous-fresh-context-verifier-before-done",
+      "prompt": "/implementation:implement-dispatch — autonomous run. Phase 3's workers have returned and their acceptance criteria look met; mark the phase DONE.",
+      "expected_output": "In autonomous mode, before marking the phase [DONE] it dispatches a fresh-context verifier subagent that checks the phase's acceptance criteria against the actual diff — a separate agent, never the producing context auditing itself — and surfaces that result in the response before ending the turn. It still runs the full Step 4 phase-boundary ritual.",
+      "files": [],
+      "expectations": [
+        "Dispatches a separate fresh-context verifier subagent before marking the phase DONE (not the producing context self-auditing)",
+        "The verifier checks acceptance criteria against the actual diff, and its result is surfaced in the response before the turn ends",
+        "Runs the full /implement Step 4 phase-boundary ritual (plan marks, handoff, status, commit, resume prompt) regardless of resident-vs-clear"
+      ]
+    }
+  ]
+}

plugins/implementation/skills/implement/evals/evals.json

@@ -0,0 +1,79 @@
+{
+  "skill_name": "implement",
+  "evals": [
+    {
+      "id": 1,
+      "name": "bugfix-mode-incremental-cadence",
+      "prompt": "There's a bug in the login flow — expired tokens are being accepted. Fix it.",
+      "expected_output": "Fires as a model-invoked skill from the natural-language 'fix' signal (no explicit /implementation:implement prefix). Detects Bugfix execution mode and consults the bugfix context guidance, runs the Step 1 prerequisite check (plan/branch/dirty-tree) before the first edit, then executes with the incremental cadence: one logical block, build check, test, commit at green — not the whole change before any validation.",
+      "files": [],
+      "expectations": [
+        "Selects Bugfix execution mode from the 'fix' signal rather than asking when the mode is unambiguous",
+        "Runs the Step 1 prerequisite check (approved plan, correct branch, clean-enough tree) before writing code",
+        "Executes incrementally — validates (build/test) after a logical block instead of implementing the entire change before any validation",
+        "Commits at a green checkpoint rather than leaving the fix uncommitted or committing a red state"
+      ]
+    },
+    {
+      "id": 2,
+      "name": "branch-check-stops-on-default-branch",
+      "prompt": "/implementation:implement feature — start building the new export endpoint. (Assume the working session is currently on the `main` branch in a PR-based repo.)",
+      "expected_output": "The Step 1 prerequisite check detects the session is on the default branch in a feature-branch workflow and stops to create a feature branch (git checkout -b, or /source-control:worktree when installed) BEFORE the first edit — it does not begin editing on main.",
+      "files": [],
+      "expectations": [
+        "Detects that the current branch is the default branch (main/master) in a feature-branch workflow",
+        "Creates or directs creation of a feature branch before the first code edit, rather than editing on the default branch",
+        "Does NOT proceed to write code on the default branch and leave branch creation for later"
+      ]
+    },
+    {
+      "id": 3,
+      "name": "major-divergence-routes-back-to-planning",
+      "prompt": "/implementation:implement — I approved a plan that assumed the vendor SDK exposes a streaming API, but partway through it's clear the SDK only supports batch calls, so the whole approach doesn't fit. Keep going and make it work.",
+      "expected_output": "Recognizes this as major divergence (a fundamental plan assumption was wrong), STOPS writing code rather than forcing workarounds, runs external research for alternative approaches first, then routes back to the planning skill (/planning:architect review when installed) so the user can approve the new direction — it does not silently push through with hacks.",
+      "files": [],
+      "expectations": [
+        "Classifies the wrong SDK-capability assumption as major divergence, not a minor inline fixup",
+        "STOPS writing code instead of pushing through with workarounds/hacks to force the original plan to fit",
+        "Runs external research for alternatives before re-planning, then routes back to the planning skill for the user to approve the new direction",
+        "Does NOT silently expand scope or improvise a new design without surfacing it to the user"
+      ]
+    },
+    {
+      "id": 4,
+      "name": "never-declares-impossible-without-alternatives",
+      "prompt": "/implementation:implement — I tried wiring the webhook two ways and both failed, so this integration is impossible. Confirm that and stop.",
+      "expected_output": "Does not accept 'impossible' after only two attempts. It presents the two failures as a progress update, not a conclusion, and researches one level deeper (GitHub issues for workaround flags, bypass options, alternative APIs) before escalating — it never declares the task impossible without exhausting alternatives.",
+      "files": [],
+      "expectations": [
+        "Does NOT declare the integration impossible on the basis of two failed attempts",
+        "Frames the two failures as a progress update and researches deeper (workaround flags, bypass options, alternative APIs) before concluding",
+        "Only escalates 'cannot be done' after alternatives are exhausted, not as the default response to early failure"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "over-correction-guard-on-pushback",
+      "prompt": "/implementation:implement — at this phase boundary you proposed three follow-up actions (two are plan work-items, one you surfaced from an anomaly mid-implementation). I don't like where this is going — drop it.",
+      "expected_output": "Applies the over-correction guard: with pushback on multiple (>=2) proposed actions it does NOT silently drop all of them. It asks per-category with AskUserQuestion (drop all N / only the invented item / specific items), and it keeps invented follow-ups separated from plan-anticipated work rather than batching them.",
+      "files": [],
+      "expectations": [
+        "Does NOT silently drop all proposed actions in response to the ambiguous 'drop it' pushback",
+        "Asks a per-category clarifying question (drop all / only invented / specific items) via AskUserQuestion before dropping anything",
+        "Keeps the invented/anomaly-surfaced action classified separately from the plan work-items rather than batching them into one undifferentiated proposal"
+      ]
+    },
+    {
+      "id": 6,
+      "name": "autonomous-run-chains-to-implement-dispatch",
+      "prompt": "/implementation:implement — execute the approved plan. This is an autonomous goal-loop run with no human in the turn cycle and the plan routes phases to worker subagents.",
+      "expected_output": "Detects orchestration mode (autonomous run / worker-routed phases) and, after the Step 1 prerequisite check passes, chains to /implement-dispatch via the Skill tool for those phases instead of running the inline Step 2 cadence — but Step 1's branch/plan/dirty-tree preflight still runs.",
+      "files": [],
+      "expectations": [
+        "Detects orchestration mode from the autonomous / worker-routed signal rather than defaulting to inline execution",
+        "Chains to /implement-dispatch (via the Skill tool) for the worker-routed phases instead of editing inline",
+        "Still runs the Step 1 prerequisite preflight (branch/plan/dirty-tree) even in the orchestrated path — dispatch does not skip it"
+      ]
+    }
+  ]
+}