test(developer-experience): port eval cases S3 and S6

Parent

Depends on

chore(developer-experience): commit the hand-rolled eval scenario sources #9936
test(developer-experience): add shared eval fixtures for the .NET and Python repositories #9943

Problem

#6649 moves the developer-experience measurement onto claude plugin eval. Cases S3 and S6 exist only
as hand-rolled scenarios.

Fix

Port S3 and S6 from the committed scenario sources into plugins/developer-experience/evals/, one case
each, on the shared fixtures. Use deterministic graders where the scenario's D and T items allow; check
each llm grader against the committed labeled samples.

Also

Case layout: plugins/developer-experience/evals/<case>/ (prompt.md, case.yaml, graders/).
Include the rm and compound-shell checks where a case needs them.

Done when

Each case loads and passes /evals:validate (no paid run).
scripts/check-changelog-fragments.sh --check-required origin/main exits 0.

Filed by AI from the owner-approved split of #6649 (2026-10-18), after the scenario sources were found on melo-desk-001 (2026-10-25).

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none

Understood: port eval cases S3 and S6 into plugins/developer-experience/evals/ on the shared fixtures. Each case must load and pass /evals:validate, and the changelog-fragment check must exit 0.

Decision recorded.

Question (hub Q5, 2026-11-04): the item carried here from #9953 asks to calibrate the two smoke-result-reported llm graders (S1 item 11b in s1-branch-start-build, S4 item 9b in s4-migrator-build) against person-reviewed sample answers; the current samples are model-labeled. (a) move that criterion to After zero and close this issue once its own done-when passes, or (b) a person reviews the labels now.

Owner answer, verbatim, 2026-11-04 in the hub session: "go w/ recommended, or whatever is in line with:" plus the standing direction s1-s10. Recommended option: (a).

Acceptance:

Each case loads and passes /evals:validate (no paid run).

scripts/check-changelog-fragments.sh --check-required origin/main exits 0.

Before closing, the session that closes this issue files the grader-calibration item (both graders above, person-reviewed samples) as its own issue in the "After zero" milestone at status: confirmed, linked here with Refs: #9966.

#9966 (comment)

This decision settles the open question, so I'm asking nothing further. The dependencies #9936 and #9943 aren't in the open-issues list I was given, so I'm treating them as not blocking.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 7 (tracker-output: "(#8848) After its case and skill changes merge, the eight acr-* cases of plug..."), item 9 (measure-over-time: "(batch A) Re-measure output on the subscription login, before and after each ..."), item 11 (named-host: "(#8414) Run plugins/guardrails/hooks/block-root-delete-target.test.sh on a Gi..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Batch A planning result: #11516 shows no gain and no loss; the #11488 with #11530 pair is INVALID. Neither case set can separate the arms as written. Run under the preregistration, 70 runs, Sonnet 5.5, judge Opus, Claude Code 2.1.318. This does not show that these changes work, or that they do not.

#11516 (490bd07b2f0^ against 490bd07b2f0): both arms VALID.

case	before	after
target: `unattended-marker-from-trusted-human-pre-confirms`	0.48	0.56
`unattended-marker-from-agent-does-not-pre-confirm`	0.75	0.75
`unattended-run-records-approval-basis`	0.667	0.667

No-loss bar held: both non-target cases are unchanged, and there is no case drop. Noise report: +0.03, interval -0.03 to +0.08, non-inferior at 0.05.
Target: no gain shown. The named grader, expectation 3, reads 0 of 5 in both arms. The 0.08 rise comes from one run each on expectations 1 and 5, 4 to 5 of 5, which is not evidence of the rule working.
Why it cannot separate them: spot-read runs in both arms show the model treating the comment as a person's confirmation, including the before arm, whose text has no marker rule. The prompt lists every fact the rule checks, and the checkout is empty, so no run wrote a plan and no ### Problem exists to grade. The verdicts I read match that.

#11488 with #11530 (b899913f213^ against 4551c2fd306): INVALID, so no scores are quoted.

ib: verdict: INVALID (2 permission denials in the traces)
ia: verdict: INVALID (1 permission denial in the traces)

The denials: a Read of plugins/planning/templates/checklist.md in one run per arm, and a Glob of the plugin root in one before run. interview/SKILL.md:53 says "copy templates/checklist.md", which sits under skills/interview/, so the two Read runs appear to have resolved it from the plugin root. Cause UNVERIFIED.
No rerun (the preregistration allows reporting INVALID): 3 denials in 40 runs make another clean pass unlikely, and the before-arm runs I spot-read already reject the unsupported fact and reopen Q11, because the prompt hands over the facts. A rerun would likely buy the same reading.

Rest of batch A, triaged from each diff with no run:

feat(adhd): clarify resolves the rendered-views medium before it publishes #11454: its cases narrate a rendered-views.md layer and an Artifact tool as fixtures, and no such file exists in a run. Whether the Artifact tool is available in a run: not checked.
feat(work-items): rank selection candidates by priority label within each tier #11452: the PR added no eval case.
fix(source-control): hand a worktree claim to the session that resumes a handoff #11521: the PR added no eval case; its behavior is in worktree-claim.sh, which has its own test file.

What this leaves: batch A's re-measure cannot finish on the current cases. Each needs fixture files, a scaffolded repository or a new case. That is build work, listed with #8206's fixture and #8009's case repair. Lane 1 holds no claim on it unless the hub assigns it. The checkbox above stays unticked.

Measured cost: 127 list-price for the 70 runs, about 1.8 per run, of which 94 went to the INVALID interview pair. The 4-run pilot read VALID on these cases, so it did not predict the denials or show that the cases cannot separate the arms: a one-arm pilot cannot show that.

#8206 addendum 1 (2026-11-04), written before the rerun of the output arm. The preregistration's bars, arms, model, judge and run count are unchanged: 4 cases, plugin against no plugin, 3 runs per arm, Sonnet 5.5, judge Opus. Three changes, all made because the INVALID first pass measured the environment rather than the skill:

Fixture. Three cases get a scaffold.sh, run with --scaffold. It seeds the run's home with known_marketplaces.json, a catalog, installed_plugins.json and settings, plus a harness-ops install record, so the default marketplace resolves. audit-issues-zero-mutating-calls and model-starts-audit-not-sync get a small fleet with one plugin behind its catalog. divergence-report-collapses-bulk-names-conflicts gets the fleet its prompt names: 29 plugins whose project and user scopes differ, 2 same-version multi-scope, and 1 settings-only id absent from the catalog. rendered-report-is-pasted-not-restated narrates a render with other numbers, so it keeps no fixture. Graders and prompts are unchanged.
Node on PATH. node in the sandbox resolves to a mise shim that fails, so the real node goes first on PATH. No harness-ops hook blocks on it, but the change keeps the run clean.
Ceiling. It is raised to 60 list-price, from a 4-run pilot that averaged about 1.3 per with-arm run. The without-arm cost is unknown.

The pilot is not part of the result: 4 runs with the fixture, with-arm only, VALID. The three scaffolded cases ran the audit and scored 0.8, 1.0 and 1.0, and the narrated case scored 0.8. Writing the fixture took three offline rounds against sync-run.sh and two paid pilots, all before this addendum.

Strict reading reported beside the result: without the fixture, the first pass stays INVALID.

#8206 addendum 2 (2026-11-04), written before the rerun. The output pass under addendum 1 read INVALID on one run: a without-arm run of divergence-report-collapses-bulk-names-conflicts was denied a Glob of its own sandbox directory. That case alone reruns as a fresh pair, both arms, 3 runs each, with the same flags. The other three cases keep their addendum 1 runs. If the rerun is INVALID again, the case is reported INVALID with no number.

#8206 output arm: three of four cases pass with the plugin, and the gain is outside the noise. Run under the preregistration with addenda 1 and 2. Claude Code 2.1.318, Sonnet 5.5, judge Opus, 3 runs per arm, 30 runs in all. The trigger arm was already reported above.

case	with plugin	without plugin
`audit-issues-zero-mutating-calls`	0.8, 1, 1	0.4, 0.4, 0.4
`divergence-report-collapses-bulk-names-conflicts` (addendum 2 pair)	1, 1, 1	0.25, 0.75, 0.75
`model-starts-audit-not-sync`	1, 1, 1	0.33, 0.33, 0.33
`rendered-report-is-pasted-not-restated`	0.6, 0.8, 0.4	0.4, 0.4, 0.4

Validity: the addendum 1 pass read INVALID on one denied Glob, in a without-arm divergence run, and was otherwise clean (saved gate output). The divergence pair rerun read verdict: VALID. Re-gating the merged file is not possible: the rerun's launcher deleted the earlier pass's kept sandboxes before I re-gated, so the gate now reports those traces unreadable. The saved per-pass gate output is the record.
Noise report over the merged four cases: delta +0.45, 95% interval +0.26 to +0.65, "the interval excludes 0". Judges were unanimous on every graded run.
Preregistered bar, each case at least 0.8 with the plugin: met by three cases. rendered-report-is-pasted-not-restated misses at 0.6. This is a named failure. The case narrates a printed render, but in the runs I read during the pilot the skill re-ran the audit script and reported its error instead of pasting the render. Whether that is the case or the skill at fault: not settled.
Spot-read: done on the pilot and on the divergence pair, where the without-arm failed on not listing the delisted id separately. The addendum 1 pass's traces are gone, so its runs were not spot-read. Under the plugin-eval reading rule, that makes this a "do not promote" reading until the spot-read can be done on a fresh pass. That pass is not run here.
Cost: 58 list-price (45 for the first pass, 13 for the rerun), about 1.9 per run.

Left on #8206: the rendered-report miss, and a spot-read on fresh runs. The fixture scaffolds are in lane 1's scratch only. Committing them to plugins/harness-ops/evals/ is a build; lane 1 opens that PR only if the hub assigns it.

fix(harness-ops): write the plugins-audit fixtures under $HOME behind…

Batch A and #8206: where they stand (2026-11-05).

feat(harness-ops): split plugins audit into its own model-invocable skill #8206's cases are on main. PR #13442 merged the four plugins-audit cases the output arm ran on. They are now native claude plugin eval cases. Three of them have a scaffold that seeds a fixture fleet in the run's home, behind a guard that refuses the account's real home. rendered-report-is-pasted-not-restated stayed under its 0.8 bar (0.6, 0.8, 0.4) and has no fixture yet.
Batch A's case builds are new case work, deferred by the hub until after fix(fleet): 7 converted reach eval cases fail on the current reach skill #12908 (H48 (c)). The planning result showed that the current cases cannot separate the arms:
- feat(planning): count a trusted human's comment marker as problem pre-confirmation #11516's target case read 0.48 before and 0.56 after.
- The fix(planning): gate assumption-sweep dispositions and Step-exit checkboxes in the interview #11488 with fix(planning): require the literal value when an interview recommendation reuses an existing one #11530 pair was INVALID.
- feat(discovery): accept an authoritative paper's reported result as a single-source content claim #11443's cases are narrated, and the sandbox has no network.
- feat(adhd): clarify resolves the rendered-views medium before it publishes #11454, feat(work-items): rank selection candidates by priority label within each tier #11452 and fix(source-control): hand a worktree claim to the session that resumes a handoff #11521 have no runnable case.
Each needs a case that fails on the before text before a no-loss or gain result can count. That is the detection-control rule (D26), which Lane 3 is building as a hub rule.
Still open on this issue: the batch A case builds, batch B, fix(guardrails): decide whether a literal '/' in allowed roots counts as a root under Git Bash #8414, docs(pointers): close open pointer findings in playbooks and visualization #8009's case repair, and the T3 graders.

Preregistration, batch A: #11452 (work-items priority ranking). Written before the measured pair. Change: 94b47b31b0^ (before) against 94b47b31b0 (after). It is additive (priority ranks candidates within a tier), so the before text is the after text with the rule removed, and the pair is its own detection control (plugins/evals/skills/plugin-eval/reference/reading-results.md, "Detection control").

Cases (new, frozen, identical in both arms; each prompt gives raw list-frontier output and asks for a dry pick, with a git-root scaffold):

case	role	expected pick
`work-priority-label-outranks-age`	target	#140 (`priority: high`) over older #120 (`medium`) and #101 (none)
`work-priority-order-from-repo-file`	target	#140 (`priority: p0`, from a scaffolded `work_priority_order`)
`work-equal-priority-falls-back-to-oldest`	non-target	#101: equal `priority: high`, so oldest-first
`work-due-recurring-tier-beats-higher-priority`	non-target	#150: tier 1 before the frontier tiers

Each case has one llm grader on the picked item (judge Opus, trace focus). Uncalibrated, so I spot-read every run whose verdict decides a bar.

Case-development pilot (not a result, 2 runs per arm, both legs VALID): targets 0.5 → 1.0 and 0.5 → 1.0. Equal-priority fell 1.0 → 0.0: both after runs read the rule, then broke the tie by "bug urgency", which they called a judgment call. Due-recurring held at 1.0 → 1.0.

Run: Claude Code 2.1.318, Sonnet 5.5, with-plugin only, --scaffold, 5 runs per case per arm, both arms launched together, 4 concurrent (40 model runs). run-validity.py --runs 5 --scaffolded on each result file before any score is read. An INVALID leg is not a measurement: it is rerun as a fresh pair under a dated addendum written first, or reported as INVALID.

Bars, fixed now:

Detection (targets): each target case scores at least 0.15 higher after than before. A target that misses it is reported as no gain shown.
No loss (each non-target case): no noise-report case drop, and each case mean no more than 0.05 below before. A miss is rerun on a fresh pair at 5 runs. A loss is reported only if it reproduces, then traced to the line that causes it for a fix PR.

The cases ship in plugins/work-items/evals/ with the result, in one PR with Refs: #9966.

#11452 result: one gain, one loss that reproduced. Run under the preregistration. Claude Code 2.1.318, Sonnet 5.5, judge Opus, 5 runs per case per arm, both legs VALID (run-validity.py --runs 5 --scaffolded).

case	role	before	after
`work-priority-label-outranks-age`	target	0.6	1.0
`work-priority-order-from-repo-file`	target	1.0	1.0
`work-due-recurring-tier-beats-higher-priority`	non-target	0.6	1.0
`work-equal-priority-falls-back-to-oldest`	non-target	1.0	0.6
same case, fresh pair (rerun the preregistration requires)	non-target	1.0	0.4

Detection bar: label-outranks-age gains 0.4, so it passes. repo-file shows no gain: the before text already ranks priority: p0 first in all 5 runs.
No-loss bar fails. equal-priority is a noise-report case drop (1.00 to 0.60), and it reproduced on a fresh pair (1.0 to 0.4). Compare verdict: not shown non-inferior, interval -0.28 to +0.48.
Cause, from 4 of the 7 failing after runs, read (3 rerun, 1 pilot): each applied the priority tie, then overrode the oldest-first tie-break with tier 3's hints ("Select based on: items that unblock others, items in smaller categories, shorter well-scoped items"). They picked the newer bug, feat(claude-ops): capture user-typed skill invocations via UserPromptExpansion #140, and called it "a judgment call". The before runs applied oldest-first in 10 of 10. context/candidate-discovery.md already says tier 3's order is "a fixed tie-break so the pick is deterministic".

Addendum, fix pair, written before it runs:

Fix: tier 3 in work/SKILL.md says the pick is the first candidate in priority-then-oldest order, never one chosen over it on a judgment of impact, scope or type. The hint sentence goes.
Pair: origin/main against main plus the fix, the same 4 cases, 5 runs per arm, launched together.
Bars:
- equal-priority scores at least 0.15 higher after than before.
- Each of the other 3 cases loses nothing: no case drop, and no case mean more than 0.05 below before.
- A miss is rerun on a fresh pair at 5 runs.

#11452 fix pair: every bar passes. Run under the addendum. origin/main against main plus the tier-3 edit, 5 runs per case per arm, Claude Code 2.1.318, Sonnet 5.5, judge Opus.

The first pair (14:49Z) was INVALID. 21 of its 40 runs ended on "You've hit your weekly limit · resets Nov 10, 7pm (America/New_York)". A one-turn Sonnet probe at 15:06Z answered, so this pair is a fresh rerun, as the preregistration allows. Both legs are VALID (run-validity.py --runs 5 --scaffolded).

case	before (main)	after (fix)
`work-equal-priority-falls-back-to-oldest` (fix target)	0.6	1.0
`work-due-recurring-tier-beats-higher-priority`	0.8	0.8
`work-priority-label-outranks-age`	1.0	1.0
`work-priority-order-from-repo-file`	1.0	1.0

Target: +0.4, past the 0.15 bar.
No-loss, per case: no case drop, and no case falls.
Pooled compare: +0.10, interval -0.10 to +0.30, not shown non-inferior at 0.05. That pooled reading includes the target case, so it is not the preregistered bar.

Not shipped. The edit changes how /work-items:work lanes choose between equal-priority items: strict oldest-first, with the "unblocks others / smaller / well-scoped" hints dropped. That is the owner's design call, recorded in lane 1's outbox as Q1 with options. The branch stays local until it is decided. Batch A's #11452 line stays open on it.

Close audit on origin/main: both cases are ported and pass validation.

plugins/developer-experience/evals/s3-script-cleanup and s6-tool-inventory exist on main; their labeled judge samples landed in test(developer-experience): add labeled samples for the S3 and S6 judge graders #11656 (7457da5), and test(developer-experience): tag two eval cases no-network #13603 tagged them no-network.
python3 -I plugins/evals/skills/validate/scripts/validate-cases.py plugins/developer-experience/evals: exit 0, 0 FAIL. The only S3/S6 lines are advisory WARNs that the labeled samples need a judge-calibration run (the validator never calls a model).
Both load and score in a CI dispatch: run https://github.com/melodic-software/claude-code-plugins/actions/runs/41527706565 rows s3-script-cleanup | 1.00 | 0.75 | +0.25 | PASS and s6-tool-inventory | 0.83 | 0.50 | +0.33 | FAIL (a score under the 1.00 threshold, not a load failure).
The changelog-fragment check ran in the PRs that added them; the cases are on main.

Preregistration for #13507's pair (posted before any paired run).

Change under test: plugins/planning/skills/plan/SKILL.md Step 1, "Is the problem established?": an unattended run reads problem-framing.md's "Unattended" and "Pre-confirmed" sections before writing ### Problem (branch fix/13507-plan-unattended-problem-rule, commit a6476be).
Arms: before = the same file at the branch's merge base with main; after = branch head 737a605. Everything else in the plugin is the same in both arms.
Cases (7): plan-marker-trusted-human-counts (marker counts) and six near-misses that must not count: author-not-on-list, author-read-access, body-edited-after, edited-by-another, inside-quote, posted-through-app (all plan-marker-*, from lane 1's feat(planning): count a trusted human's comment marker as problem pre-confirmation #11516 batch). Each has a deterministic reads-problem-framing grader (tool_used Read on context/problem-framing\.md).
Runs: 5 per case per arm, --ablation none, --scaffold, --allow-tools Bash Write Edit, Claude Code 2.1.318, session model Sonnet 5.5, judge Opus.
Pass bar (all must hold):
1. In every after run, reads-problem-framing passes.
2. Both legs read VALID under run-validity.py <result> --runs 5 --scaffolded.
3. No case drops more than 0.05 from before to after (no case drop line in noise-report.py).
4. plan-marker-trusted-human-counts still passes marker-counted in the after arm.
Fail handling: if bar 1 fails, the trigger line does not load the rule, and the next try moves the checks into SKILL.md (the other shape fix(planning): load the unattended problem-confirmation rule in every plan run #13507 names). If bar 2 fails, nothing is concluded and the cause is reported.

Result for #13507's pair (preregistration). All four bars pass.

Claude Code 2.1.318; session model claude-sonnet-5-5 in 35 of 35 traces per arm; judge Opus; 7 plan-marker-* cases × 5 runs per arm. Before = merge base c019f67's SKILL.md, after = 737a605. diff -r of the two plugin copies shows only SKILL.md's Step 1 line.

Bar	Result
1. Every after run Reads `context/problem-framing.md`	35 of 35 after, 0 of 35 before (counted from the kept traces' `Read` tool calls)
2. Both legs VALID (`run-validity.py --runs 5 --scaffolded`)	before `VALID`, after `VALID`; 0 permission denials in 70 traces. The only warning is "skill fired", since the prompts invoke `/planning:plan` directly and the cases have no skill-fired grader
3. No case drops more than 0.05	no `case drop` line; paired change +0.50, 95% interval +0.31 to +0.69, `non-inferior at margin 0.05`
4. `plan-marker-trusted-human-counts` still passes `marker-counted`	5 of 5, unanimous judge votes

Case means, before → after: author-not-on-list 0.33 → 1.00, author-read-access 0.67 → 1.00, body-edited-after 0.00 → 1.00, edited-by-another 0.67 → 1.00, inside-quote 0.67 → 1.00, posted-through-app 0.67 → 1.00, trusted-human-counts 0.50 → 1.00.

Deviations from the preregistration:

A weekly usage limit stopped the first pass partway through: 15 before runs and 23 after runs errored. The affected cases were rerun whole (5 runs each) on a fresh account about 25 minutes later and swapped in. Each merged leg therefore combines two invocations. Both arms used the same model, Claude Code version and flags.
The preregistration did not set concurrency: the first pass ran with -j 3, and the reruns ran one case per process.

Estimated eval usage: about 100 USD of claude.ai plan capacity, not billed.

Navigation Menu

Repository navigation

test(developer-experience): port eval cases S3 and S6 #9966

Parent

Depends on

Problem

Fix

Also

Done when

52 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

test(developer-experience): port eval cases S3 and S6 #9966

Description

Parent

Depends on

Problem

Fix

Also

Done when

Activity

github-actions commented on Oct 25, 2026

melodic-automation-lanes commented on Oct 25, 2026

52 remaining items

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 5, 2026

kyle-sexton commented on Nov 5, 2026

kyle-sexton commented on Nov 5, 2026

kyle-sexton commented on Nov 5, 2026

kyle-sexton commented on Nov 6, 2026

kyle-sexton commented on Nov 6, 2026

kyle-sexton commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions