Repository navigation
planning:design-handoff lists ADR candidates without applying the admission test to each #7659
Description
Problem
/planning:design-handoff's handoff summary asks for "ADR candidates: each resolved decision that is costly to undo, not obvious from the code, and chosen over a live alternative." In a real run the model listed four candidates without applying that test to each one. When the user asked whether they were real, two failed on inspection:
- "One data folder per session": visible in the code and cheap to migrate (fails "not obvious from the code" and "costly to undo").
- "Config behind one defaults file until config-cascade: settings resolution and convention home #7571": a temporary bridge, easily reversed (fails "costly to undo").
The two that held up (a security boundary for tunnel exposure; a terms-of-use policy for who may drive Claude) were mixed in with the two that did not, so the user had to do the filtering.
Cause (judgment): the bullet states the three criteria as a description of candidates, not as a test the model must show. Nothing asks for per-criterion evidence or allows "none" as an answer, so the step reads as "list some decisions".
Proposed change
In the handoff summary's ADR-candidates step:
- For each candidate, state the three criteria as pass/fail with one line of evidence each (what makes it costly to undo; why the code does not show it; the live alternative that was rejected, from
design-threads.md). - List only candidates that pass all three; "None" is a valid and expected outcome.
- Optionally list near-misses under a separate line, each naming the criterion it fails.
Acceptance
- An eval case where the design threads hold one ADR-worthy decision and two routine ones: the handoff lists exactly one candidate, with evidence for all three criteria, and names the other two as failing.
- An eval case with no ADR-worthy decision: the handoff says "None".
- Existing design-handoff evals show no loss.
No related issue: found while running /planning:design-handoff on a planning-plugin restructure.
Activity
github-actions commentedon Oct 16, 2026
Thanks for the detailed report. Understood:
/planning:design-handoff's ADR-candidates step states the three admission criteria (costly to undo, not obvious from the code, chosen over a live alternative) as a description rather than a per-candidate pass/fail test, so the model lists candidates without filtering out ones that fail the test — as happened with the two false positives you caught. The proposed fix (per-criterion evidence, "None" as a valid outcome, optional near-misses) and the two acceptance eval cases look like enough to scope a fix from.I didn't find an existing open issue covering this exact gap; #3817 touches design-handoff's completeness check but is closed and addresses a different aspect. Nothing else appears missing from the report.
Automated triage. A maintainer reviews it.
kyle-sexton commentedon Oct 18, 2026
Agent Brief
Type: Bug
Summary: Make
/planning:design-handofftest each ADR candidate against its three criteria and allow 'None'.Current behavior:
The handoff summary lists 'ADR candidates' as each resolved decision that is costly to undo, not obvious from the code, and chosen over a live alternative. The criteria read as a description, so a run lists routine decisions beside real ones and the user filters them.
Desired behavior:
For each candidate the summary gives the three criteria as pass or fail with one line of evidence each: what makes it costly to undo, why the code does not show it, and the rejected live alternative from
design-threads.md. Only candidates passing all three are listed; 'None' is a valid outcome. Near-misses may be listed on a separate line, each naming the criterion it fails. The offer to invoke/architecture:record-decisionapplies only to listed candidates.Key interfaces:
evals.json) and fixtures.Acceptance criteria:
scripts/check-changelog-fragments.shexits 0.Out of scope:
/architecture:record-decisionand the interview skill.This was generated by AI during backlog shaping. Held, not admitted: this brief is drafted and checked, but the issue lane would stop on it today. It is admitted, by adding
agent-ready, once the lane can run the skill-quality check scripts (Automation Lanes' PR #7912). Held under the owner's 2026-10-18 lane-narrowing course correction.