Repository navigation
test(planning): stop omits-unchanged-items failing replies that mention an unchanged value for another reason #11591
Description
Refs: #11189
Observed
The interview-final-brief-changes-only case's omits-unchanged-items grader is a not_contains regex on gzip, 50,000, 50k, toolbar, UTC, compression and audit log anywhere in the reply. It fails a reply that leaves the unchanged answers out of the restatement but uses one of those words for another reason.
A host run on #11588's head (d26e742; Claude Code CLI on the subscription login, --runs 3 --judge-model opus --scaffold --keep-temp; run-validity.py: VALID) failed it in 3 of 3 with-arm runs:
- Run 1 restated "Q1–Q4 and Q6–Q8: accepted as read back. No changes." and failed only because it suggested a UTC timestamp format for the changed Q5.
- Run 3 restated "Q1–Q4 and Q6–Q8 were accepted as read back and haven't changed, so I'm not listing them again." and failed only because it cited Q4's 50,000-row cap as a fact for the new Q9.
- Run 2 did re-list them ("Unchanged and accepted (7): ... Q3 UTC timestamps · Q4 50,000-row cap ..."), a correct fail.
The case's stated outcome is that the reply does not re-list the accepted answers. Mentioning one of their values while reasoning about a changed or new item is not re-listing. The llm grader changes-only-restatement already judges that criterion, and passed runs 1 and 3 and failed run 2.
Fix wanted
Make the grader fail only a reply that re-lists unchanged answers, for example by matching the answers as a list (two or more of the values in one block), or drop it in favour of changes-only-restatement. Add pass samples from runs 1 and 3 (shape only, no transcript text) and a fail sample shaped like run 2.
Done when
- The grader passes a reply that names one unchanged value while discussing a changed or new item, and fails one that lists the unchanged answers, shown by samples.
- With the grader fixed,
interview-final-brief-changes-onlyreaches a with-arm score of 0.8 or more in a VALID run (3 runs, opus judge). This carries the eval bar that fix(planning): restate only changed answers and unseen assumptions at final confirmation #11189 left to its rerun. If it does not, the rule from fix(planning): restate only changed answers and unseen assumptions at final confirmation #11189 gets a follow-up fix. - Carried from fix(planning): make the interview run the convention-home resolver before tagging EARS criteria #11597 (change merged in fix(planning): name the convention-home resolver command in the interview's EARS spoke #11675):
resolver-ranpasses in 3 of 3 with-arm runs ofinterview-ears-convention-tagsin a VALID run (3 runs, opus judge,--scaffold --allow-tools Bash), with no loss on the case's other graders. A miss reopens fix(planning): make the interview run the convention-home resolver before tagging EARS criteria #11597.
This was generated by AI.
Activity
github-actions commentedon Oct 31, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: the
omits-unchanged-itemsgrader in theinterview-final-brief-changes-onlycase uses anot_containsregex. It fails replies that mention one unchanged value (UTC, 50,000) while discussing a changed or new item, even though they don't re-list the accepted answers. The fix is to make it fail only on a re-listing, either by matching two or more values in one block or by relying onchanges-only-restatement. Pass samples shaped like runs 1 and 3 and a fail sample shaped like run 2 would show it. The issue leaves the choice between those two approaches open, but both keep the design the same. No likely duplicates found.Automated triage, applied by the lanes App.
github-actions commentedon Oct 31, 2026
github-actions commentedon Oct 31, 2026
melodic-automation-lanes commentedon Oct 31, 2026
The work-resolve-issue lane stopped partway through this issue and needs a person.
Reason:
securityWhy, in the run's own words:
Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41216692405
To retry, remove
needs-humanand addagent-readyagain.kyle-sexton commentedon Nov 1, 2026
Route:
host-run. Written by the intake triage session, not the owner.The lane run stopped with reason
security, but the cause was not a security finding. It was theissue-editedstop (.github/scripts/work-resolve-issue/stop-routes.json:9). This account edited the body at 2026-10-31T16:11:44Z, afteragent-readywas added at 15:49:38Z, and the runner refuses text changed after that label (.github/scripts/work-resolve-issue/write-issue.mjs:247-250).Re-adding
agent-readywould deliver only the first done-when item. The second needs a VALIDclaude plugin evalrun ofinterview-final-brief-changes-only(3 runs, opus judge) reaching a with-arm score of 0.8 or more. The issue lane's Bash runs only commands its allow rules name, and none namesclaude(.github/actions/collect-base-activity/action.yml:490-494). #11189's lane run left this same eval rerun undone ("is not run here", comment 6518094429). A host session fixes the grader and runs the eval in the same pull request. The criteria are unchanged.kyle-sexton commentedon Nov 2, 2026
Claim. Backlog HITL host-run worker is taking this now.
test(planning): cover the Q5 and new-question context shapes in judge…