You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Author rich-form evals/evals.json for five claude-ops + prototype skills, one file per skill, conforming to the bundled schema (plugins/skill-quality/reference/evals.schema.json) and modeled on the rich-form example (plugins/bug-report/skills/bug-report/evals/evals.json).
Skills covered (warrant re-check: all five carry judgment-bearing behavioral contracts — none skipped)
Skill
Cases
Coverage
claude-ops/claude-code-changelog
5
passive-mention→read-only routing, explicit-apply interviews before edits, target-version-ahead warning, git-history (not tracker file) status, P2 capability not dropped
claude-ops/claude-observability
5
default week-scope read-only report, clean requires confirmation, product-bug routes to /claude-troubleshooting, unknown scope not guessed, empty-store graceful degrade
claude-ops/claude-troubleshooting
6
no-args status default, feature-name smart routing to search, create shows draft + requires confirmation, telemetry & own-code-bug route-aways, reports-does-not-fix
prototype/logic
5
throwaway TUI over pure module, appearance question routes to /prototype:ui, no-tests, purity/portability, no real DB
prototype/ui
6
sub-shape A default, logic question routes to /prototype:logic, structurally-different variants, stubbed mutations, HTML-mockup constraints, no direct promotion to production
Each case covers trigger/routing, happy path, at least one guardrail, and one anti-pattern; every expectations entry is checkable from the run's output/artifacts.
Validation
check-jsonschema --schemafile plugins/skill-quality/reference/evals.schema.json <file> — ok for all 5.
check-skill.sh <skill> — PASS (0 errors) for all 5; the action-router-without-evals WARN is now cleared. Remaining no Gotchas surface WARNs are pre-existing skill-body advisories, out of scope for this evals task.
Version bumps (delivery vehicle)
Minor bump on both touched plugins (evals are a shipped component; the plugin.jsonversion is the marketplace delivery cache key):
claude-ops 0.2.0 → 0.3.0
prototype 0.1.1 → 0.2.0
Note — referenced docs not on main
The source issue cites docs/MIGRATION-PLAYBOOK.md "Evals — warrant policy and consumer-verify recipe" and docs/evals-coverage.md. Neither exists at origin/main (verified); both land via the unmerged origin/chore/evals-backfill branch (commit 2df1561). Warrant policy was instead taken from plugins/skill-quality/README.md and the self-contained per-skill re-check in the issue body. No docs created here (out of scope).
Refs melodic-software/medley#1452
Note
Low Risk
Adds JSON eval fixtures and minor plugin version bumps only; no runtime skill logic or application code changes.
Overview
Adds rich-formevals/evals.json under five skills (three in claude-ops, two in prototype), encoding checkable behavioral contracts—routing, guardrails, and anti-patterns—as prompts plus expectations, aligned with skill-quality’s evals schema.
claude-ops cases cover changelog read-only vs apply-with-triage, observability scopes/clean confirmation and cross-skill routing, and troubleshooting defaults, create preflight, and scope boundaries. prototype cases cover logic vs UI facet routing, throwaway TUI/purity constraints, UI variant shape, stubbed mutations, and HTML mockup rules.
Bumps shipped plugin versions (claude-ops0.3.0, prototype0.2.0) so marketplace consumers pick up the new eval assets.
Reviewed by Cursor Bugbot for commit 874954f. Bugbot is set up for automated code reviews on this repo. Configure here.
Author rich-form evals/evals.json for five action/behavioral skills, each
grounded in the skill's live SKILL.md contract and conforming to the bundled
schema (plugins/skill-quality/reference/evals.schema.json):
- claude-ops/claude-code-changelog (5 cases) — read-only routing on passive
version mention, interview-before-edit on explicit apply, version-ahead
warning, git-history-not-tracker-file status, P2-not-dropped.
- claude-ops/claude-observability (5 cases) — default week-scope read-only
report, clean-requires-confirmation guard, product-bug routes to
troubleshooting, unknown-scope not guessed, empty-store graceful degrade.
- claude-ops/claude-troubleshooting (6 cases) — no-args status default,
feature-name smart routing to search, create-shows-draft guard, telemetry
and own-code-bug route-aways, reports-does-not-fix.
- prototype/logic (5 cases) — throwaway TUI over a pure module, appearance
question routes to ui, no-tests, purity/portability, no real DB.
- prototype/ui (6 cases) — sub-shape A default, logic question routes to
logic, structurally-different variants, stubbed mutations, HTML mockup
constraints, no direct promotion to production.
Each case covers trigger/routing, happy path, at least one guardrail, and one
anti-pattern; expectations are checkable from the run's output. All files pass
check-jsonschema against the bundled schema and check-skill.sh (evals presence
gate now satisfied for all five).
Bump both touched plugins' versions (minor) — evals are a shipped component and
the version bump is the marketplace delivery vehicle: claude-ops 0.2.0 -> 0.3.0,
prototype 0.1.1 -> 0.2.0.
Warrant re-check: all five skills carry judgment-bearing behavioral contracts
(action routing, mutation gates, anti-patterns) — none skipped.
Refs melodic-software/medley#1452
Review evals for schema conformance and content quality
Check version bumps and plugin.json changes
Post review findings
Summary
Five well-authored eval files covering all behavioral contracts across claude-ops (3 skills) and prototype (2 skills). Schema conformance is clean, skill_name values match their directory names, all name slugs are valid kebab-case, and every expectation is verifiably observable from model output. The version bumps are appropriate minor increments for a new shipped component.
Findings
Minor: expected_output wording misattributes the actor — claude-observability case 2
"Because clean is a mutating action invoked by the model without --dry-run …"
The action is invoked by the user ("Clean up the OTEL store, it's getting big") — the model is the one that must pause and require confirmation. The expectations entries are correct and unambiguous; this is only an editorial inaccuracy in the prose description. Not a blocker, but worth fixing to keep the eval self-documenting. Fix this →
Nit: ui:existing-page-sub-shape-a assumes a page exists without a fixture
The expected_output says "because a plausible existing /settings page exists, the skill defaults to sub-shape A." There is no corresponding fixture file or repo scaffold that establishes this precondition, so the evaluator must infer from context that a settings page exists. All other cases are either purely routing/rejection or fully self-contained. This is acceptable given the plugin's repo-agnostic design, but a brief note in expected_output that this assumes a typical SPA with an existing route would make evaluation determinism explicit. Not a blocker.
✓ Every skill has at least one routing and one guardrail case
expectations entries are output-observable
✓ None reference internal state
files: [] empty — prompts are self-contained
✓ No fixture dependencies needed
claude-ops bump 0.2.0 → 0.3.0
✓ Minor bump correct for new shipped component
prototype bump 0.1.1 → 0.2.0
✓ Minor bump correct
claude-troubleshooting case 5 conditional routing (/bug-report:bug-report or manual fallback)
✓ Graceful — handles plugin-absent case
Verdict
Approve with nits. The one editorial fix in claude-observability case 2 is worth landing before or alongside merge, but neither finding is a functional or schema-level defect. The eval coverage is thorough and the behavioral contracts are faithfully represented.
…ed_output
- claude-observability case 2: reframe the clean-confirmation trigger as a
casual cleanup request (not an explicit destructive order) rather than
"invoked by the model", keeping the guardrail meaning explicit.
- ui case 1: state the sub-shape-A precondition explicitly ("assuming a typical
app where a plausible existing /settings route already exists") so the case is
self-documenting without a repo fixture.
Both remain schema-valid. Editorial only; no case added or removed.
Refs melodic-software/medley#1452
Five well-structured eval files across claude-ops (3 skills) and prototype (2 skills). Both nits flagged in the prior Claude review (commit 14d9ad2) have been addressed in the follow-up commit 4a23d50:
claude-observability case 2 — the "mutating action invoked by the model" phrasing has been replaced with the correct framing ("casual cleanup request rather than an explicit destructive order").
prototype/ui case 1 — the fixture-less assumption is now made explicit ("Assuming a typical app where a plausible existing /settings route already exists").
"Because this is a casual cleanup request rather than an explicit destructive order
(the user did not pass --dry-run or clearly order the prune), the skill shows the
--dry-run preview..."
The parenthetical "the user did not pass --dry-run" implies the confirmation step is conditional on the user omitting the flag — as if passing --dry-run would skip the preview. But --dry-run is the preview mode the skill invokes, not a user-suppressible flag. A reader could infer an incorrect conditional. The expectations array is unambiguous and correct; this is only in the prose description. Fix this →
What Looks Correct
Coverage: Every skill has at minimum one routing case, one happy-path case, one confirm-before-mutate or read-only guardrail, and one anti-pattern. The claude-troubleshooting case 5 handles the cross-plugin routing gracefully with a manual fallback when bug-report is absent. The prototype/logic purity constraint (no I/O in the logic module, TUI as a thin shell) is checked in a dedicated case.
No behavioral regressions: No skill bodies, hooks, or runtime configurations are changed. The only non-eval changes are the minor version bumps in plugin.json.
Verdict
Approve. The one residual nit is an editorial ambiguity in prose that doesn't affect the expectations contract and isn't a functional or schema defect. The evals authoring is thorough and the behavioral contracts are faithfully represented.
claude-observability case 2 — reframed the clean confirmation trigger as a casual cleanup request (not an explicit destructive order) instead of "invoked by the model", preserving the guardrail meaning while reading cleanly.
ui case 1 — stated the sub-shape-A precondition explicitly ("assuming a typical app where a plausible existing /settings route already exists") so the case is self-documenting without a repo fixture.
Both files remain schema-valid (check-jsonschema). Editorial only; no case added or removed.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving: Cursor Bugbot was skipped and did not complete automated review. Human review is needed before merge; no additional reviewers were assigned.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Codex review (P2): the claude-troubleshooting `create` action makes five gates
mandatory BEFORE drafting (context/action-create.md) — notably a hard
duplicate-search gate that presents matches and proceeds only on explicit
"no duplicates" confirmation. The prior case rewarded jumping straight to a
draft, which would skip that safety flow.
Rewrite case 3 (now create-runs-preflight-gates-before-drafting) so the
expected safety flow is: run preflight gates (duplicate search + live-template
fetch + version check) -> draft only after gates pass -> file only on explicit
confirmation, never auto-file, never skip the duplicate-search gate.
Refs melodic-software/medley#1452
Investigate Codex-flagged issue: preflight gates vs. create eval
Validate schema conformance and content quality
Check version bumps
Post review findings
Summary
Reviewed all 7 changed files (5 new eval files + 2 plugin.json version bumps) against the schema at plugins/skill-quality/reference/evals.schema.json, the relevant skill source files (context/action-create.md, SKILL.md), and the prior review rounds. The Codex P2 was correctly resolved in the latest commit. One residual editorial nit from the second Claude review remains in claude-observability case 2 — and on closer inspection against the skill spec, it is accurate prose, just slightly confusing.
Codex correctly flagged that the original case 3 (create-shows-draft-and-requires-confirmation) rewarded jumping straight to a draft, bypassing the five mandatory preflight gates in context/action-create.md — notably Gate 3, the hard duplicate-search gate that requires explicit "no duplicates" confirmation before drafting.
The rewrite in 66804ff is correct. The expectations now screen for the right flow:
Expectation
Screens out
Gates run before drafting (incl. duplicate search)
Skipping preflight entirely
Presents matches, proceeds only after user confirms
Treating "no results found" as implicit confirmation
Draft shown only after gates pass, not before
Drafting in the same turn before confirmation
No auto-filing
Auto-filing after the draft
In a single-turn eval the model stops at Gate 3 waiting for confirmation — expectation 2 correctly fails any implementation that proceeds without it. The fix is sufficient.
Residual Nit — claude-observability case 2 --dry-run phrasing (not a blocker)
"the user did not pass --dry-run or clearly order the prune"
The second Claude review flagged this as implying --dry-run is a user-suppressible bypass flag. On re-reading SKILL.md line 78:
clean requires explicit user confirmation before running when invoked by the model — show --dry-run output first unless user already passed --dry-run or explicitly ordered cleanup.
The phrasing is actually accurate per the skill spec — --dry-run is user-passable (argument-hint lists it), and when passed, the user is already requesting the preview mode, so the additional "should I show you a preview?" prompt is skipped. The parenthetical correctly states the condition under which the confirmation guardrail fires. The expectations array is unambiguous and correct.
The confusion is real (it reads as if --dry-run bypasses the preview entirely rather than being the preview mode), but the prose is not wrong. Worth a one-word fix ("no explicit --force or destructive intent" instead) if this eval is ever edited for another reason, but not a standalone blocker. Fix this →
No extra fields (additionalProperties: false compliance)
✓
expectations entries are output-observable
✓ None reference internal state
files: [] — all prompts self-contained
✓ All 5 files
Cross-skill routing is symmetric (observability↔troubleshooting, logic↔ui)
✓ Both directions covered
create eval: gates-before-draft contract enforced
✓ Fixed in 66804ff
create eval: no auto-file expectation
✓ Explicit in expectations
claude-ops version bump 0.2.0 → 0.3.0
✓ Correct minor bump for new shipped component
prototype version bump 0.1.1 → 0.2.0
✓ Correct minor bump
claude-observability case 2 --dry-run prose accuracy vs. SKILL.md
✓ Accurate per spec (see nit above)
Codex P2 resolved
✓ 66804ff rewrites case 3 to enforce preflight gate ordering
Verdict
Approve. The Codex P2 is correctly resolved. All 5 eval files are schema-valid, the behavioral contracts are faithfully represented across all 27 cases, and the version bumps are appropriate. The one residual nit in claude-observability case 2's prose is accurate per the skill spec — it's editorial ambiguity, not a functional defect.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot completed with skip status (usage limit reached), so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Reviewed all 7 changed files (5 new eval files + 2 plugin.json version bumps) at HEAD 26d0435, which is a merge from origin/main that touched only unrelated files (knowledge plugin, hook-migration docs — no conflicts with the eval changes). The PR is in clean shape.
Merge commit (26d0435) — no conflicts
The latest commit merges origin/main into the branch. The only files brought in are unrelated: README.md, docs/hook-migration-audit.md, and plugins/knowledge/ additions. None of the five eval files or either plugin.json were touched by the merge.
"(the user did not pass --dry-run or clearly order the prune)"
As noted in the prior review cycle and confirmed accurate against SKILL.md (the skill does skip the preview prompt when --dry-run is user-supplied, since they're already requesting preview mode), this parenthetical is technically correct but reads as if --dry-run is a bypass flag. The expectations array is unambiguous and governs what the evaluator actually checks — this is editorial ambiguity only. Fix this →
Verdict
Approve. The Codex P2 is correctly resolved. All 5 eval files are schema-valid, behavioral contracts are faithfully represented across all 27 cases, the merge from origin/main introduced no conflicts, and the version bumps are appropriate. The one residual nit is editorial and does not affect schema correctness or the expectations contract.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot completed with skip status (usage limit reached), so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
…ition
Two Codex P2 findings:
- claude-troubleshooting case 3: the prompt began with "File", which is not an
action keyword, so smart routing would send it to `search` and never reach
the `create` preflight gates the case tests. Prompt now starts with the
`create` action keyword so it routes to create.
- ui case 1: `files: []` with a prompt that did not establish an existing
/settings route — sub-shape A only defaults when a plausible existing page
exists, so a compliant run would pick sub-shape B / HTML mockup. The prompt
now states the /settings page already exists, making sub-shape A the correct
default without a framework fixture.
Both remain schema-valid.
Refs melodic-software/medley#1452
Validate prompt routing fix for claude-troubleshooting case 3
Validate sub-shape A precondition fix for prototype/ui case 1
Full schema and content review of all 5 eval files
Post review findings
Summary
Reviewed HEAD 1940c37 against the full PR diff (origin/main...HEAD). This commit addresses both Codex P2 findings from the last review round. The two fixes are correct and the PR is otherwise clean — no new issues introduced.
Codex P2 #1 — Resolved: claude-troubleshooting case 3 prompt routing
Old prompt:"File a bug on anthropics/claude-code: the Stop hook doesn't fire when I exit with Ctrl-C." New prompt:"/claude-troubleshooting create bug \"Stop hook doesn't fire when I exit with Ctrl-C\""
Verified against SKILL.md smart routing rule (lines 60–62):
$ARGUMENTS starts with action keyword → that action
$ARGUMENTS doesn't start with action keyword → search with entire argument as feature name
The old prompt started with "File" (not a keyword), so it silently routed to search and the create preflight gates were never reached. The new prompt starts with create, which is a recognized action keyword, routing correctly to the create flow. The expectations array — requiring duplicate-search gate before drafting, draft after gates pass, and explicit confirmation before filing — now exercises the right code path. Fix is correct.
Codex P2 #2 — Resolved: prototype/ui case 1 sub-shape A precondition
Old prompt:"Show me a few options for what the settings page could look like." New prompt:"Our web app already has a /settings page. Show me a few options for what that settings page could look like."
The precondition (existing route) is now established in the prompt itself. The expected_output was updated to match — from "Assuming a typical app…" (inference) to "Because the prompt establishes an existing /settings route…" (stated fact). A compliant evaluator no longer needs to infer project state; the case is now self-contained with files: []. Fix is correct.
Residual Nit (pre-existing, not a blocker)
claude-observability case 2 — --dry-run parenthetical
"(the user did not pass --dry-run or clearly order the prune)"
As established in the prior review round and confirmed against SKILL.md: this is technically accurate — the skill skips the preview prompt when the user already passed --dry-run, since they're requesting preview mode explicitly. The expectations array governs what the evaluator actually checks and is unambiguous. This is editorial ambiguity, not a functional defect — not a blocker for this PR.
ui case 1: sub-shape A precondition self-contained in prompt
✓ Fixed in 1940c37
claude-ops version bump 0.2.0 → 0.3.0
✓ Correct minor bump
prototype version bump 0.1.1 → 0.2.0
✓ Correct minor bump
Verdict
Approve. Both Codex P2 findings are correctly resolved in 1940c37. All 5 eval files are schema-valid, the behavioral contracts are faithfully represented across all 27 cases, and the version bumps are appropriate. No functional or schema-level defects remain.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot completed with skip status, so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Two Codex P2 findings:
- claude-code-changelog case 5: the case depended on the live `latest`
changelog containing a new settings.json field, which is nondeterministic and
could reward trusting an unsupported prompt assertion over the fetched
changelog. Prompt now pastes a concrete changelog entry (pasted text is the
skill's highest-priority input mode), so the P2 classification is grounded in
provided content. Added a read-only expectation.
- logic case 1: the run-command expectation unconditionally required an existing
task runner, but the skill falls back to a prototype README command when none
exists. Expectation now allows either. Also softened the module-shape
expectation to "e.g. a reducer or explicit state machine" since the skill
permits other pure shapes.
Both remain schema-valid.
Refs melodic-software/medley#1452
Investigate Codex P2s from latest review round (9996979)
Post findings
Summary
Reviewed all 7 changed files at HEAD 9996979 (chore(evals): make changelog P2 + logic run-command evals deterministic). Both Codex P2 findings from the prior review round are correctly resolved. The PR is clean.
Codex P2 #1 — Resolved: claude-code-changelog case 5 determinism
The old prompt ("Diff the latest Claude Code changelog…") depended on the live fetch containing a new settings.json field — nondeterministic and potentially rewarding an implementation that trusts an unsupported prompt assertion. The fix pastes the changelog entry inline:
"Diff this pasted Claude Code changelog entry against our repo — we don't set this option anywhere yet:\n\n### v2.1.160\n- **New setting:** `settings.json` gains `autoCompactWindow`..."
Per SKILL.md input-mode priority, pasted text beats a live fetch, so the classification is now fully deterministic. A new read-only expectation was also added. Fix is correct.
Codex P2 #2 — Resolved: prototype/logic case 1 task-runner fallback
The old expectation unconditionally required an existing task runner ("wired into the project's existing task runner"). The skill's own contract (SKILL.md step 5) allows a README fallback when no task runner exists. The expectation now reads:
"wired into the project's existing task runner, or documented at the top of a prototype README when the project has no task runner"
This matches the skill contract exactly. The module-shape expectation was also softened from "reducer or explicit state machine" to "e.g. a reducer or explicit state machine" — consistent with SKILL.md's shape table that permits pure functions and class/module forms too. Fix is correct.
Residual Nit (pre-existing, not a blocker)
claude-observability case 2 — --dry-run parenthetical
"(the user did not pass --dry-run or clearly order the prune)"
As established across prior review rounds and confirmed against SKILL.md: the phrasing is technically accurate — the skill skips the "show preview?" prompt when the user already passed --dry-run (they're already requesting preview mode). The expectations array governs what the evaluator checks and is unambiguous. Editorial ambiguity only, not a functional or schema defect. Fix this →
Changelog case 5: pasted text makes P2 classification deterministic
✓ Fixed in 9996979
Logic case 1: README fallback allowed when no task runner exists
✓ Fixed in 9996979
Logic case 1: module-shape expectation allows all SKILL.md-permitted shapes
✓ Fixed in 9996979
Create eval: gate ordering enforced, no auto-file
✓ Fixed in 66804ff
Create eval prompt: starts with create keyword for correct smart routing
✓ Fixed in 1940c37
UI case 1: existing-route precondition established in prompt
✓ Fixed in 1940c37
claude-ops version bump 0.2.0 → 0.3.0
✓ Correct minor bump
prototype version bump 0.1.1 → 0.2.0
✓ Correct minor bump
Verdict
Approve. All Codex P2 findings across every review round are correctly resolved. All 5 eval files are schema-valid, the behavioral contracts are faithfully represented across all 27 cases, and the version bumps are appropriate. No functional or schema-level defects remain.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot completed with skip status (usage limit reached), so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Three Codex P2 findings:
- claude-code-changelog case 4: prompt began with "Which", not an action
keyword, and the changelog skill parses the first token as the action (unknown
-> action table), so it would not exercise the git-history status path. Prompt
is now the action-form `/claude-code-changelog status`.
- claude-observability case 2: "Clean up the OTEL store" reads as an explicit
cleanup order under the skill's own confirmation exception, so a compliant run
could prune directly. Prompt now asks whether trimming is safe (a non-ordering
question), so the dry-run-first + confirmation gate must fire.
- ui case 5: expected_output required the winning variant captured before the
user has picked one; capture is a later step. Reframed to generate the mockup
(with a copy-out control) and hand it over, with winner-capture deferred.
All three remain schema-valid.
Refs melodic-software/medley#1452
Investigate latest Codex P2 issues (3 flagged on 3a7fcf5)
Validate all eval files against schema and skill contracts
Post findings
Summary
Reviewed all 7 changed files (5 new evals/evals.json + 2 plugin.json bumps) at HEAD 18259c9 against the schema (plugins/skill-quality/reference/evals.schema.json) and all three relevant SKILL.md contracts. Commit 18259c9 correctly resolves the three Codex P2 findings from the prior round. The PR is in clean shape.
Three Codex P2 Fixes — All Correct
Fix 1: claude-code-changelog case 4 — status action routing (evals.json line 42)
Old prompt started with "Which" — the skill's action router treats an unrecognized first token as unknown and shows the action table, so the git-history status path was never exercised. New prompt is /claude-code-changelog status, routing explicitly to the status action per SKILL.md's action table. ✓ Correct.
Fix 2: claude-observability case 2 — clean confirmation gate (evals.json line 19)
Old prompt "Clean up the OTEL store, it's getting big" reads as an explicit cleanup order. SKILL.md line 77 states: clean shows --dry-run first unless the user explicitly ordered cleanup. A compliant run on the old prompt could skip the confirmation step. New prompt — "The OTEL store is getting pretty large — is there a safe way to trim it down?" — is a question about process, not an order, so the dry-run preview + explicit confirmation gate must fire. ✓ Correct.
Old expected_output required the winning variant captured in markdown in the same first turn. Per SKILL.md, step 5 is "Hand it over" (surface URL/variant keys) and step 6 is "Capture the answer and clean up" — capture happens after the user picks one. New expected_output correctly defers the capture step and adds the copy-out control reference per SKILL.md's HTML mockup spec. ✓ Correct.
"(the user did not pass --dry-run or clearly order the prune)"
Confirmed accurate against SKILL.md line 77: clean shows dry-run first unless the user already passed --dry-run (they're already requesting preview mode) or explicitly ordered cleanup. The parenthetical correctly states when the confirmation guardrail fires. The expectations array is unambiguous and governs what evaluators actually check. This is editorial ambiguity in prose only — not a functional or schema defect. Fix this →
Verdict
Approve. All three Codex P2 findings from the latest review round are correctly resolved in 18259c9, and all five prior P2 fixes across the review cycle remain intact. All 5 eval files are schema-valid, the behavioral contracts are faithfully represented across all 27 cases, and both version bumps are appropriate minor increments for a newly shipped component. No functional or schema-level defects remain.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot completed with skip status, so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
…ions
Preempt the smart-routing conflict: the skill routes non-keyword args to
`search`, which mechanically contradicts the two route-away cases. Reframe both
as explicit capability/scope questions so routing away is the only sensible
response, preserving the scope-boundary test:
- case 4 (telemetry): now asks whether this skill reports token burn / hook
latency or whether that is a different tool -> routes to /claude-observability.
- case 5 (own-code bug): now asks whether this skill tracks a bug in the user's
own billing code -> routes to /bug-report (or manual write-up).
Schema-valid.
Refs melodic-software/medley#1452
Check open Codex P2 on logic/evals.json line 8 — README fallback
Review d1e4852 troubleshooting route-away reframe
Verify all prior findings
Summary
Reviewed all 7 changed files at HEAD 2796cd8 (merge commit from origin/main — no eval changes). The most recent substantive commit is d1e4852 (reframing claude-troubleshooting route-away cases 4 and 5 as explicit scope questions). That fix is correct. One open Codex P2 from the prior review round remains unresolved.
Open Finding: logic/evals.json case 1 — expected_output missing README fallback
"The prototype is runnable via a single command — wired into the project's existing task runner,
or documented at the top of a prototype README when the project has no task runner"
The expectations array (line 14, fixed in 9996979) allows the README fallback when no task runner exists, but the expected_output prose (line 8) was not updated alongside it and still requires task-runner wiring unconditionally. An evaluator reading expected_output as the authoritative description would incorrectly fail compliant no-task-runner runs, contradicting the more permissive expectations entry.
Fix: add the fallback clause to expected_output. Fix this →
d1e4852 — Troubleshooting Route-Away Reframe ✓
Cases 4 and 5 were reframed as explicit capability/scope questions, eliminating the mechanical routing conflict where non-keyword args would route to search instead of the intended route-away:
Case 4 (telemetry-question-routes-away): old prompt "What's my token burn rate…" → new "…does /claude-troubleshooting report those, or is that a different tool's job?" — correctly signals a meta-question about capability boundaries; the only sensible answer routes to /claude-observability.
Case 5 (own-code-bug-routes-away): old prompt "There's a bug in my applyLateFee function — Track this bug." → new "I found a bug in my own app's applyLateFee billing function (not a Claude Code bug) — does /claude-troubleshooting track that, or should I file it elsewhere?" — scoped to a capability question; routes to /bug-report or manual write-up. The added parenthetical (not a Claude Code bug) removes any ambiguity about intent.
Both reframes are correct and well-scoped.
All Prior P2 Fixes — Still Intact
Commit
Fix
Status
66804ff
create eval: gates-before-draft, no auto-file
✓
1940c37
create prompt starts with create keyword for correct routing
✓
1940c37
ui case 1: existing-route precondition stated in prompt
✓
9996979
changelog case 5: pasted text makes P2 classification deterministic
✓
9996979
logic case 1 expectations: README fallback allowed when no task runner exists
✓ (expectations only — expected_output still missing it, see above)
18259c9
changelog case 4: /claude-code-changelog status keyword prompt
✓
18259c9
observability case 2: casual question prompt (not an explicit order)
✓
18259c9
ui case 5: winner capture deferred until after user picks
✓
Residual Nit (pre-existing, not a blocker)
claude-observability case 2, expected_output line 20:
"(the user did not pass --dry-run or clearly order the prune)"
Accurate per SKILL.md (the skill skips the preview prompt when --dry-run is already user-supplied, since they're requesting preview mode). The expectations array is unambiguous and governs what evaluators actually check. Editorial ambiguity only. Fix this →
Verdict
Near-approve — one P2 still open. The logic/evals.json case 1 expected_output inconsistency (missing README fallback) is the last unresolved finding. Once that one-line prose update is in, the PR is clean for merge.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot completed with skip status (usage limit reached), so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Two Codex P2 findings:
- Plugin skills invoke as `/plugin-name:skill-name`, so all bare slash prompts
and command references are namespaced: `/claude-ops:claude-code-changelog`,
`/claude-ops:claude-observability`, `/claude-ops:claude-troubleshooting`
(verified against plugins/claude-ops/README.md). Prototype refs were already
`/prototype:logic` / `/prototype:ui`.
- logic case 1 expected_output now also states the prototype-README fallback for
the no-task-runner case, matching the expectation fixed earlier.
All 5 files schema-valid.
Refs melodic-software/medley#1452
Gather context and read changed files at HEAD (f3bd6a1)
Investigate open Codex P2s from 2796cd8 review round — resolved in c91d1aa
Full review of all 5 eval files and 2 plugin.json bumps
Post findings
Summary
Reviewed all 7 changed files (5 new evals/evals.json + 2 plugin.json bumps) at HEAD f3bd6a1 (merge from origin/main — no eval changes in the merge). The most recent substantive eval commit is c91d1aa ("namespace plugin commands + finish logic README fallback"), which correctly resolves both open Codex P2 findings from the prior review round.
c91d1aa — Both Open Codex P2s Resolved
Fix 1: logic/evals.json case 1 — expected_output now includes README fallback
"makes it runnable in one command via the project's task runner
(or, when the project has no task runner, a command documented
at the top of a prototype README)"
The expected_output now matches the expectations entry that was fixed in 9996979. A compliant no-task-runner run will no longer be graded as wrong by an evaluator reading the prose description. ✓
Fix 2: All slash commands are namespaced (/claude-ops:*, /prototype:*, /bug-report:*)
Every prompt and expected_output routing reference in all 5 eval files now uses the plugin-name:skill-name command form. Verified across all cases that reference slash commands:
File
Command form
claude-code-changelog case 4 prompt
/claude-ops:claude-code-changelog status ✓
claude-observability case 3 routing
/claude-ops:claude-troubleshooting ✓
claude-observability case 4 prompt
/claude-ops:claude-observability lastweek ✓
claude-troubleshooting cases 1, 3, 4, 5
/claude-ops:claude-troubleshooting ✓
claude-troubleshooting case 4 routing
/claude-ops:claude-observability ✓
claude-troubleshooting case 5 routing
/bug-report:bug-report ✓
logic case 2 routing
/prototype:ui ✓
ui case 2 routing
/prototype:logic ✓
All Prior Findings — Still Intact
Commit
Fix
66804ff
create eval: mandatory preflight gates (duplicate search, template fetch, version check) before any draft
1940c37
create prompt starts with create keyword; ui case 1 existing-route precondition in prompt
9996979
changelog case 5 pasted-text determinism; logic case 1 task-runner expectations fallback
18259c9
changelog case 4 status keyword; observability case 2 non-ordering prompt; ui case 5 winner capture deferred
d1e4852
troubleshooting route-away cases reframed as explicit scope questions
c91d1aa
logic case 1 expected_output README fallback; all commands namespaced
"(the user did not pass --dry-run or clearly order the prune)"
Confirmed accurate against SKILL.md across multiple review cycles: the skill skips the "show preview?" prompt when the user already passed --dry-run, since they're already requesting preview mode. The expectations array is unambiguous and governs what evaluators actually check. Editorial ambiguity only — not a functional or schema defect, and not a blocker. Fix this →
Verdict
Approve. Both open Codex P2s from the prior review round are correctly resolved in c91d1aa. All 5 eval files are schema-valid, every slash command reference uses the correct plugin-namespaced form, the logic case 1 expected_output is now consistent with its expectations entry, and the behavioral contracts are faithfully represented across all 27 cases. Both version bumps are appropriate minor increments. No functional or schema-level defects remain.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot completed with skip status, so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Codex P2: the case pinned the installed version via prompt text, but the skill
checks the active terminal with `claude --version`, so a compliant run in a
harness without that exact version becomes environment-dependent. Reframe to
test the version-awareness logic itself: check the installed version, warn only
if the target is newer, else proceed — robust to whatever version the harness
reports.
Schema-valid.
Refs melodic-software/medley#1452
Investigate latest open Codex P2 (changelog case 3 version-pinning) — resolved in 10df23f
Full schema and content review of all 5 eval files
Check version bumps
Post final review findings
Summary
Reviewed all 7 changed files (5 new evals/evals.json + 2 plugin.json bumps) at HEAD eee02dd (merge from origin/main — no eval changes in the merge itself). The substantive new commit since the prior review round is 10df23f ("make changelog version-ahead eval environment-robust"), which correctly resolves the last open Codex P2. The PR is clean.
10df23f — Codex P2 Resolved: changelog case 3 environment-dependency
The original case 3 pinned the installed version in the prompt text ("My terminal reports claude --version = 2.1.150"). Because the skill calls claude --version at runtime rather than accepting an asserted version, a compliant eval run in any harness where the installed version differs from 2.1.150 would produce different behavior — either the warning fires (older installed) or it doesn't (newer), making the expectation environment-dependent.
The fix is correct. The reframed prompt asks the skill to check the version conditionally ("if my installed Claude Code is older than that, warn me"), and the expectations now cover both branches:
Branch
Expectation
Installed version older than target
Warns and suggests claude update
Installed version equal or newer
Proceeds without warning
Both branches are output-observable (presence or absence of the warning text). A compliant run always satisfies exactly one of expectations 2 or 3 regardless of the evaluation environment, and expectation 1 ("checks the installed version rather than assuming") is always testable. The fix is sound.
One editorial note (not a blocker): the case name target-version-ahead-of-installed-warns now slightly overstates the invariant — the case covers both the warn AND the no-warn path. The name still anchors to the primary behavior, so this is acceptable as-is.
All Prior P2 Findings — Still Intact
Commit
Fix
66804ff
create eval: mandatory preflight gates (dup-search, template-fetch, version-check) before any draft
1940c37
create prompt starts with create keyword; ui case 1 existing-route precondition in prompt
9996979
changelog case 5 pasted-text determinism; logic case 1 task-runner expectations fallback
18259c9
changelog case 4 status keyword; observability case 2 non-ordering prompt; ui case 5 winner-capture deferred
d1e4852
troubleshooting route-away cases reframed as explicit scope questions
c91d1aa
logic case 1 expected_output README fallback; all commands namespaced to plugin:skill form
10df23f
changelog case 3: environment-robust dual-branch expectations
"(the user did not pass --dry-run or clearly order the prune)"
Confirmed accurate across multiple prior review rounds: per SKILL.md, the skill skips the "show preview?" prompt when the user already supplied --dry-run (they're already requesting preview mode). The expectations array is unambiguous and governs what evaluators actually check. Editorial ambiguity in prose only — not a schema or functional defect, and not a blocker. Fix this →
Verdict
Approve. The last open Codex P2 (changelog case 3 environment-dependency) is correctly resolved in 10df23f. All 5 eval files are schema-valid, every slash command reference uses the correct plugin-namespaced form, the behavioral contracts are faithfully represented across all 27 cases, and both version bumps are appropriate minor increments for a newly shipped component. No functional or schema-level defects remain.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot completed with skip status (usage limit reached), so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Verify all prior Codex P2 findings remain intact after latest merge
Validate schema conformance and content quality
Post review findings
Summary
Reviewed all 7 changed files (5 new evals/evals.json + 2 plugin.json bumps) at HEAD 874954f. The latest commit is a merge from origin/main that touched only README.md and docs/MIGRATION-PLAYBOOK.md — no eval or plugin.json changes. All prior P2 findings across every review round remain correctly resolved. The PR is clean.
Merge Commit (874954f) — No Conflicts
Merge parents are eee02dd (last eval substantive state) and cc00783 (a README-only commit from main). Neither parent modified any of the 7 PR-owned files.
All Prior P2 Findings — Resolved and Intact
Commit
Finding
Status
66804ff
create eval: mandatory preflight gates (dup-search, template fetch, version check) before any draft
✓ Intact
1940c37
create prompt starts with create keyword for correct smart routing
✓ Intact
1940c37
ui case 1: existing-route precondition established in prompt
✓ Intact
9996979
changelog case 5: pasted entry makes P2 classification deterministic
✓ Intact
9996979
logic case 1 expectations: README fallback allowed when no task runner
✓ Intact
18259c9
changelog case 4: /claude-ops:claude-code-changelog status routes correctly
✓ Intact
18259c9
observability case 2: casual question (not an explicit cleanup order)
✓ Intact
18259c9
ui case 5: winner-capture deferred until after user picks
✓ Intact
d1e4852
troubleshooting route-away cases reframed as explicit scope questions
✓ Intact
c91d1aa
logic case 1 expected_output: README fallback clause added
✓ Intact
c91d1aa
All slash commands use plugin-name:skill-name namespace
✓ Intact
10df23f
changelog case 3: dual-branch expectations; environment-independent version check
"(the user did not pass --dry-run or clearly order the prune)"
Confirmed accurate across multiple prior review cycles: per SKILL.md, clean skips the "show preview?" prompt when the user already supplied --dry-run, since they're already requesting preview mode. The expectations array — show dry-run preview, require confirmation, delete nothing — is unambiguous and governs what evaluators actually check. Editorial ambiguity in prose only; not a schema or functional defect. Fix this →
Verdict
Approve. All prior P2 findings across every review round are correctly resolved and intact. All 5 eval files are schema-valid, every slash command uses the correct plugin:skill namespace, the behavioral contracts are faithfully represented across all 27 cases, and both version bumps are appropriate minor increments for a newly shipped component. No functional or schema-level defects remain.
The reason will be displayed to describe this comment to others. Learn more.
Risk: low. Not approving because Cursor Bugbot completed with skip status (usage limit reached), so automated review did not complete successfully. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Author rich-form
evals/evals.jsonfor five claude-ops + prototype skills, one file per skill, conforming to the bundled schema (plugins/skill-quality/reference/evals.schema.json) and modeled on the rich-form example (plugins/bug-report/skills/bug-report/evals/evals.json).Skills covered (warrant re-check: all five carry judgment-bearing behavioral contracts — none skipped)
claude-ops/claude-code-changelogclaude-ops/claude-observabilitycleanrequires confirmation, product-bug routes to/claude-troubleshooting, unknown scope not guessed, empty-store graceful degradeclaude-ops/claude-troubleshootingstatusdefault, feature-name smart routing tosearch,createshows draft + requires confirmation, telemetry & own-code-bug route-aways, reports-does-not-fixprototype/logic/prototype:ui, no-tests, purity/portability, no real DBprototype/ui/prototype:logic, structurally-different variants, stubbed mutations, HTML-mockup constraints, no direct promotion to productionEach case covers trigger/routing, happy path, at least one guardrail, and one anti-pattern; every
expectationsentry is checkable from the run's output/artifacts.Validation
check-jsonschema --schemafile plugins/skill-quality/reference/evals.schema.json <file>— ok for all 5.check-skill.sh <skill>— PASS (0 errors) for all 5; the action-router-without-evals WARN is now cleared. Remainingno Gotchas surfaceWARNs are pre-existing skill-body advisories, out of scope for this evals task.Version bumps (delivery vehicle)
Minor bump on both touched plugins (evals are a shipped component; the
plugin.jsonversionis the marketplace delivery cache key):claude-ops0.2.0 → 0.3.0prototype0.1.1 → 0.2.0Note — referenced docs not on
mainThe source issue cites
docs/MIGRATION-PLAYBOOK.md"Evals — warrant policy and consumer-verify recipe" anddocs/evals-coverage.md. Neither exists atorigin/main(verified); both land via the unmergedorigin/chore/evals-backfillbranch (commit2df1561). Warrant policy was instead taken fromplugins/skill-quality/README.mdand the self-contained per-skill re-check in the issue body. No docs created here (out of scope).Refs melodic-software/medley#1452
Note
Low Risk
Adds JSON eval fixtures and minor plugin version bumps only; no runtime skill logic or application code changes.
Overview
Adds rich-form
evals/evals.jsonunder five skills (three in claude-ops, two in prototype), encoding checkable behavioral contracts—routing, guardrails, and anti-patterns—as prompts plusexpectations, aligned withskill-quality’s evals schema.claude-ops cases cover changelog read-only vs apply-with-triage, observability scopes/clean confirmation and cross-skill routing, and troubleshooting defaults, create preflight, and scope boundaries. prototype cases cover logic vs UI facet routing, throwaway TUI/purity constraints, UI variant shape, stubbed mutations, and HTML mockup rules.
Bumps shipped plugin versions (claude-ops
0.3.0, prototype0.2.0) so marketplace consumers pick up the new eval assets.Reviewed by Cursor Bugbot for commit 874954f. Bugbot is set up for automated code reviews on this repo. Configure here.