chore(evals): author skill evals for claude-ops + prototype (5 skills)

plugins/claude-ops/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "claude-ops",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Claude Code operations toolkit of three skills: claude-observability (read locally captured telemetry — OTEL store, collector, hook-event JSONL, ccusage — with trend reports and store pruning), claude-troubleshooting (search known Claude product GitHub bugs, check service health, maintain a persistent tracked-issue registry), and claude-code-changelog (ingest Claude Code changelog entries and integrate them into the current repo).",
   "author": {
     "name": "Melodic Software",

plugins/claude-ops/skills/claude-code-changelog/evals/evals.json

@@ -0,0 +1,65 @@
+{
+  "skill_name": "claude-code-changelog",
+  "evals": [
+    {
+      "id": 1,
+      "name": "passive-mention-routes-read-only",
+      "prompt": "Claude Code v2.1.152 just dropped — what changed in it that's relevant to this repo?",
+      "expected_output": "A read-only response: the skill routes to fetch or diff (displays the changelog and/or an impact table) and does NOT start the apply pipeline or edit any repo file. If integration might be wanted, it offers apply as an explicit next step rather than auto-running it.",
+      "files": [],
+      "expectations": [
+        "The response presents changelog content and/or a repo-impact analysis without editing any file",
+        "The response does NOT run the apply pipeline (no interview/architect/implement phases executed) on a passive version mention",
+        "If integration is relevant, the response offers `apply` as an explicit opt-in step rather than starting it automatically"
+      ]
+    },
+    {
+      "id": 2,
+      "name": "explicit-apply-interviews-before-edits",
+      "prompt": "Apply the Claude Code v2.1.152 changelog to this repo.",
+      "expected_output": "The apply pipeline runs and presents a triage table (item, P1/P2/P3 classification, affected files, action needed) for the user to pick scope BEFORE any file is edited. Edits happen only after the user locks scope; nothing is silently changed.",
+      "files": [],
+      "expectations": [
+        "The response surfaces a per-item triage table with a classification and affected files before making edits",
+        "The response waits for the user to confirm scope (all P1+P2 / just P1 / specific items) before implementing changes",
+        "No repo file is edited prior to the user confirming scope"
+      ]
+    },
+    {
+      "id": 3,
+      "name": "target-version-ahead-of-installed-warns",
+      "prompt": "Apply the v2.1.152 changelog — and if my installed Claude Code is older than that, warn me before applying.",
+      "expected_output": "The apply/diff flow checks the installed Claude Code version (via `claude --version`) and compares it to the target v2.1.152, grounding the comparison in the actual installed version rather than an assumed one. If the installed version is older than the target, it warns that applying newer-version changes may reference features not yet available and suggests updating first (claude update); if the installed version is equal or newer, it proceeds without that warning.",
+      "files": [],
+      "expectations": [
+        "The response checks the installed Claude Code version (e.g. via `claude --version`) rather than assuming a specific installed version",
+        "If the installed version is older than the target v2.1.152, the response warns and suggests updating (e.g. `claude update`) before applying",
+        "If the installed version is equal to or newer than the target, the response proceeds without emitting the ahead-version warning"
+      ]
+    },
+    {
+      "id": 4,
+      "name": "status-derives-from-git-not-tracker-file",
+      "prompt": "/claude-ops:claude-code-changelog status",
+      "expected_output": "The status action derives applied versions from git history (e.g. git log --grep for CC version citations in commit messages). It does NOT create, read, or propose a persistent applied-versions tracking file — git history is the single source of truth.",
+      "files": [],
+      "expectations": [
+        "The response derives applied versions from git history (e.g. `git log --grep`), not from a dedicated tracker file",
+        "The response does NOT create or propose a persistent applied-versions tracking file"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "p2-new-capability-not-dropped",
+      "prompt": "Diff this pasted Claude Code changelog entry against our repo — we don't set this option anywhere yet:\n\n### v2.1.160\n- **New setting:** `settings.json` gains `autoCompactWindow` (integer) to override the autocompact token threshold per session.",
+      "expected_output": "The diff ingests the pasted changelog entry (pasted text takes priority over a live fetch, so the analysis is deterministic) and classifies the new `autoCompactWindow` setting — a capability the repo does not use — as P2 ('New capability — evaluate for adoption') with a brief rationale, rather than silently dropping it as P3/no-action. Read-only diff analysis; no edits.",
+      "files": [],
+      "expectations": [
+        "The response ingests the pasted changelog entry rather than fetching a live version, keeping the analysis deterministic",
+        "The pasted new setting (a capability the repo does not yet use) is classified P2 (evaluate for adoption), not silently treated as no-action",
+        "The P2 item is surfaced in the impact/triage output with a short adoption rationale rather than omitted",
+        "The diff analysis is read-only: no repo file is edited"
+      ]
+    }
+  ]
+}

plugins/claude-ops/skills/claude-observability/evals/evals.json

@@ -0,0 +1,63 @@
+{
+  "skill_name": "claude-observability",
+  "evals": [
+    {
+      "id": 1,
+      "name": "week-scope-report-read-only",
+      "prompt": "How am I doing on token burn this week?",
+      "expected_output": "A read-only week-scope telemetry report (the default scope) assembled from the available sources (ccusage, hook-event log, OTEL store). It does not modify code and does not write to memory; any file output is confined to a report under the reports directory only when --write is passed.",
+      "files": [],
+      "expectations": [
+        "The response produces a week-scoped report (the default scope) covering token/cost signals",
+        "The response does not modify code or write to auto-memory",
+        "No file is written unless `--write` was requested, and any such file is a report under the reports directory"
+      ]
+    },
+    {
+      "id": 2,
+      "name": "clean-requires-confirmation",
+      "prompt": "The OTEL store is getting pretty large — is there a safe way to trim it down?",
+      "expected_output": "Because this is a casual cleanup request rather than an explicit destructive order (the user did not pass --dry-run or clearly order the prune), the skill shows the --dry-run preview (what would be pruned) and requires explicit user confirmation before actually pruning. It does not delete store or JSONL data in this turn.",
+      "files": [],
+      "expectations": [
+        "The response shows a `--dry-run` preview of what would be pruned before deleting anything",
+        "The response requires explicit user confirmation before running the destructive prune",
+        "No telemetry data is actually deleted in this turn"
+      ]
+    },
+    {
+      "id": 3,
+      "name": "product-bug-routes-to-troubleshooting",
+      "prompt": "Is the Stop hook a known broken Claude Code bug right now? Any workarounds?",
+      "expected_output": "This is a Claude product-bug / GitHub-issue question, not a local-telemetry read. The skill routes the user to /claude-ops:claude-troubleshooting instead of answering from the OTEL/telemetry store.",
+      "files": [],
+      "expectations": [
+        "The response identifies the request as a Claude product-bug question rather than a local telemetry read",
+        "The response routes to `/claude-ops:claude-troubleshooting` instead of querying the telemetry store"
+      ]
+    },
+    {
+      "id": 4,
+      "name": "unknown-scope-not-guessed",
+      "prompt": "/claude-ops:claude-observability lastweek",
+      "expected_output": "The scope token 'lastweek' is not a valid scope. The skill reports the unknown scope and lists the valid options (session, day, week, month, since:YYYY-MM-DD, all, clean) rather than silently guessing a nearby scope.",
+      "files": [],
+      "expectations": [
+        "The response reports that `lastweek` is not a recognized scope",
+        "The response lists the valid scopes/actions rather than silently substituting a guessed scope"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "empty-store-degrades-gracefully",
+      "prompt": "Give me a cost and hook-latency breakdown for this session.",
+      "expected_output": "A read-only session-scope breakdown. If a data source (e.g. OTEL store or hook-event log) is empty or absent, the skill degrades gracefully — reporting the empty source as normal-on-first-run rather than erroring or fabricating numbers.",
+      "files": [],
+      "expectations": [
+        "The response produces a session-scoped read-only breakdown without modifying state",
+        "If a data source is empty or absent, the response notes it as expected/first-run rather than erroring out",
+        "The response does not fabricate metric values for a source that has no data"
+      ]
+    }
+  ]
+}

plugins/claude-ops/skills/claude-troubleshooting/evals/evals.json

@@ -0,0 +1,75 @@
+{
+  "skill_name": "claude-troubleshooting",
+  "evals": [
+    {
+      "id": 1,
+      "name": "no-args-defaults-to-status",
+      "prompt": "/claude-ops:claude-troubleshooting",
+      "expected_output": "With no arguments, the skill runs the status action — a quick health summary (registry stats plus a model-quality snapshot). It does not run search, create, or any mutating action.",
+      "files": [],
+      "expectations": [
+        "The response runs the `status` action (registry-health + quality snapshot) when no arguments are given",
+        "The response does not invoke `search`, `create`, or any other action for the empty-argument case"
+      ]
+    },
+    {
+      "id": 2,
+      "name": "feature-name-routes-to-search",
+      "prompt": "Stop hook not firing on session end",
+      "expected_output": "The argument does not start with an action keyword, so smart routing treats the whole argument as a feature name and runs search against anthropics/claude-code (the default repo) for known bugs on that feature.",
+      "files": [],
+      "expectations": [
+        "The response routes the non-keyword argument to the `search` action using the whole phrase as the feature name",
+        "The search targets `anthropics/claude-code` as the default repo for a Claude Code feature"
+      ]
+    },
+    {
+      "id": 3,
+      "name": "create-runs-preflight-gates-before-drafting",
+      "prompt": "/claude-ops:claude-troubleshooting create bug \"Stop hook doesn't fire when I exit with Ctrl-C\"",
+      "expected_output": "The create action runs its mandatory preflight gates BEFORE drafting — at minimum a duplicate search of existing open and recently-closed issues (presenting any matches and proceeding only on explicit 'no duplicates' confirmation, else suggesting a comment on the existing issue) plus the live-template fetch and version check. Only after the gates pass does it draft the issue in template format, show the draft for review, and file solely on explicit user confirmation. It never auto-files and never skips the duplicate-search gate.",
+      "files": [],
+      "expectations": [
+        "The response runs the mandatory preflight gates before drafting, including a duplicate search of existing issues",
+        "The response presents any duplicate matches and proceeds only after the user confirms none cover the same issue (suggesting a comment on an existing issue when a match exists)",
+        "The response drafts the issue and shows it for review only after the gates pass, not before",
+        "The response requires explicit user confirmation and does NOT auto-file the issue"
+      ]
+    },
+    {
+      "id": 4,
+      "name": "telemetry-question-routes-away",
+      "prompt": "My token burn rate and hook latency this week — does /claude-ops:claude-troubleshooting report those, or is that a different tool's job?",
+      "expected_output": "This asks for local telemetry, which this skill does not read. It routes the user to /claude-ops:claude-observability instead of attempting a telemetry read or a GitHub bug search.",
+      "files": [],
+      "expectations": [
+        "The response identifies the request as a local-telemetry read outside this skill's scope",
+        "The response routes to `/claude-ops:claude-observability` instead of answering from local telemetry"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "own-code-bug-routes-away",
+      "prompt": "I found a bug in my own app's applyLateFee billing function (not a Claude Code bug) — does /claude-ops:claude-troubleshooting track that, or should I file it elsewhere?",
+      "expected_output": "This is a defect in the user's own code, not a Claude product issue. The skill declines to add it to the Claude-product registry or search Claude repos for it, and routes to /bug-report:bug-report (or a manual defect write-up when that plugin is absent).",
+      "files": [],
+      "expectations": [
+        "The response recognizes this as a bug in the user's own code, not a Claude product issue",
+        "The response does NOT add the user's own-code bug to the Claude-product registry",
+        "The response routes to `/bug-report:bug-report` (or a manual defect write-up if that plugin is unavailable)"
+      ]
+    },
+    {
+      "id": 6,
+      "name": "reports-does-not-fix",
+      "prompt": "Find and fix the known Claude Code Stop-hook bug for me.",
+      "expected_output": "The skill searches for and reports/tracks the known Claude Code issue but does not attempt to fix Claude Code itself — it reports and tracks bugs, it does not patch the product.",
+      "files": [],
+      "expectations": [
+        "The response searches for and reports/tracks the known issue",
+        "The response does NOT attempt to patch or fix Claude Code itself",
+        "If a workaround exists, it is surfaced as guidance rather than an applied fix to the product"
+      ]
+    }
+  ]
+}

plugins/prototype/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "prototype",
-  "version": "0.1.1",
+  "version": "0.2.0",
   "description": "Builds throwaway code to answer a design question before committing to architecture — a logic facet (an interactive terminal app over a portable state model) and a UI facet (radically different visual variants on one route).",
   "author": {
     "name": "Melodic Software",

plugins/prototype/skills/logic/evals/evals.json

@@ -0,0 +1,63 @@
+{
+  "skill_name": "logic",
+  "evals": [
+    {
+      "id": 1,
+      "name": "state-machine-throwaway-tui",
+      "prompt": "Does this scheduling state machine handle the case where a booking is cancelled and then rescheduled to the same slot? Prototype the logic so I can drive it by hand.",
+      "expected_output": "The skill states the question at the top of the prototype, isolates the scheduling logic in a small pure module (reducer / state machine) behind a throwaway TUI that re-renders the full frame each tick, and makes it runnable in one command via the project's task runner (or, when the project has no task runner, a command documented at the top of a prototype README).",
+      "files": [],
+      "expectations": [
+        "The prototype records the question being tested at the top of the file before building anything",
+        "The scheduling logic lives in a pure module (e.g. a reducer or explicit state machine) separate from the TUI shell",
+        "The TUI re-renders the whole frame on each action (replace, not append) and lists available actions",
+        "The prototype is runnable via a single command — wired into the project's existing task runner, or documented at the top of a prototype README when the project has no task runner"
+      ]
+    },
+    {
+      "id": 2,
+      "name": "appearance-question-routes-to-ui",
+      "prompt": "What should this settings dashboard look like? Prototype a few options.",
+      "expected_output": "This is a visual/appearance question — the wrong facet for the logic skill. It routes the user to /prototype:ui instead of building a terminal logic prototype.",
+      "files": [],
+      "expectations": [
+        "The response identifies the request as a visual/appearance question, not a logic/state question",
+        "The response routes to `/prototype:ui` rather than building a logic prototype"
+      ]
+    },
+    {
+      "id": 3,
+      "name": "no-tests-added",
+      "prompt": "Prototype this discount-calc reducer, and add a test suite so we can be sure it's right.",
+      "expected_output": "The skill builds the throwaway logic prototype but declines to add a test suite — a prototype that needs tests is no longer a prototype. It notes that tests belong to the lifted production module, not the throwaway prototype.",
+      "files": [],
+      "expectations": [
+        "The response builds the logic prototype without adding a test suite for it",
+        "The response explains that a prototype needing tests is no longer a prototype (tests belong to the lifted production module)"
+      ]
+    },
+    {
+      "id": 4,
+      "name": "logic-module-stays-pure-and-portable",
+      "prompt": "Prototype whether this order-fulfillment reducer correctly blocks shipping before payment clears.",
+      "expected_output": "The logic module is kept pure — no I/O, no terminal codes, no console output used for control flow — so it stays liftable into production. The TUI is a thin shell that imports and calls the module; nothing flows from the TUI back into the logic.",
+      "files": [],
+      "expectations": [
+        "The fulfillment logic module contains no I/O, terminal codes, or console output used for control flow",
+        "The TUI imports and calls into the logic module; the module does not depend on the TUI",
+        "The response frames the logic module as liftable into production with the TUI shell discarded"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "no-real-database-wiring",
+      "prompt": "Prototype this inventory data model and wire it up to our real Postgres so it feels real.",
+      "expected_output": "Unless the question itself is about persistence, the skill uses an in-memory store rather than wiring the prototype to the real Postgres database, keeping it a throwaway prototype.",
+      "files": [],
+      "expectations": [
+        "The prototype uses an in-memory store rather than connecting to the real Postgres database",
+        "The response only reaches for real persistence if the question being tested is specifically about persistence"
+      ]
+    }
+  ]
+}