test(evals): run the hand-off phrasing eval #11959

Activity

1. kyle-sexton

   added the

   Task

   issue type on Nov 1, 2026
2. kyle-sexton

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

   on Nov 1, 2026
3. kyle-sexton

   added a parent issue on Nov 1, 2026

   • skill-quality: skill authoring, quality and catalog #7582
4. kyle-sexton commented on Nov 1, 2026

   

   kyle-sexton

   on Nov 1, 2026

   MemberAuthor

   More actions

   Claim: the Backlog HITL host-run worker is running the hand-off phrasing eval (12 cases, 3 arms, 3 runs) now. The result follows as a Probe result: comment.
5. kyle-sexton commented on Nov 1, 2026

   

   kyle-sexton

   on Nov 1, 2026

   MemberAuthor

   More actions

   Probe result: Host run. Ran the hand-off phrasing eval (D85, Q125, Q188) with claude plugin eval: 12 cases, 3 arms, 3 runs.

   Verdict: no arm separates. Every arm handed off in every run, so under the decision rule the seam-phrasing convention stays (rule 5). The user-only tie goes to "tell the user to run" (rule 4).

   What ran

   • Claude Code 2.1.316 on the subscription login. The decision model was claude-opus-5-5, pinned with --model, and every trace confirms it. The second-model check ran on claude-sonnet-5-5.
   • The design follows Q188 as approved:
     • One fixture plugin per arm (workbench), identical except for the hand-off sentence. Names come from the task domain, and no file the model reads says test, eval or grader.
     • Three upstream-to-downstream chains modeled on the real ones, each with its hand-off mid-body:
       • implement to confirm
       • benchmark to validate (modeled on /evals:plugin-eval to /evals:validate)
       • find to improve
     • The prompt starts the upstream skill by slash command, in three user wordings per chain: 9 model-invocable cases.
     • 3 user-only cases: publish finds no channel config, and the downstream setup sets disable-model-invocation: true.
   • Arms:
     • A: "Next, invoke /workbench:<t> (if that plugin is installed)."
     • B: "Next, invoke /workbench:<t> via the Skill tool (if that plugin is installed)."
     • C: Next, call the Skill tool with "workbench:<t>".
     • The fallback sentence after the hand-off is the same in every arm.
     • The user-only cases ran on A (gated token) and C (Matt's "tell the user to run /workbench:setup").
   • Command, one suite at a time, each in a capped claude.slice scope with a short TMPDIR: claude plugin eval <arm>/workbench --ablation none --runs 3 -j 2 --model claude-opus-5-5 --allow-tools Edit Write --disallow-tools Agent Task Bash --scaffold --keep-temp --no-publish. The cases grant Read, Edit, Write, Glob, Grep and Skill. Bash was not granted, so the Docker-store refusal and the sandbox-socket failure do not apply.
   • Graders, all free and no judge:
     • Hand-off: tool_used, tool: Skill, input_match on the downstream skill name, min: 1.
     • User-only: a regex for /workbench:setup on the last message, plus tool_used: Skill on setup with max: 0.
   • Validity: run-validity.py --runs 3 --scaffolded gave VALID for all 7 suites: complete, no run errors, no permission denials, every trace resolved, one model per suite.
     • The only warnings were "judge model not recorded", which does not apply because the run had no judge.
     • The user-only suites also warn "skill fired: no skill-fired grader", which is expected because those cases must not call the Skill tool.

   Scores

   Suite	Model	Cases	n (runs)	Hand-off rate
   A, gated token	Opus 5.5	9	27	27/27 (1.00)
   B, via the Skill tool	Opus 5.5	9	27	27/27 (1.00)
   C, Call the Skill tool with	Opus 5.5	9	27	27/27 (1.00)
   A, check	Sonnet 5.5	9	27	27/27 (1.00)
   C, check	Sonnet 5.5	9	27	27/27 (1.00)

   User-only suite	n (runs)	Names the command	No Skill call
   A, gated token	9	9/9	9/9
   C, tell the user to run	9	9/9	9/9

   • noise-report.py --baseline: B minus A and C minus A are each +0.00 paired over 9 cases, with a 95% interval of +0.00 to +0.00. C minus A on the user-only cases is +0.00 over 3 cases. No arm beats A by the 10-point margin, so rule 2 is not met, and rule 3 does not arise.
   • The second-model check ran A and C, because no arm won and those two are the ends of the range. It gives the same ceiling.
   • Limits on this result:
     • Every suite scored at ceiling, so this fixture cannot separate the arms.
     • The fixture skills are 3 to 4 steps long. The median shipped skill runs to about 1,477 words (skill-quality: skill authoring, quality and catalog #7582, D6), so the result does not cover a hand-off deep in a long skill body.
     • A harder fixture (real-length bodies, the hand-off after many steps) is what could still separate the arms. Filing it is the coordinator's call. Under Q188 it is not required.
   • The fixtures were run from an out-of-repo directory and were not committed, as Q188 says. With no winning arm, no cases were committed.
   • Total runs: 99 with Opus 5.5 (81 model-invocable and 18 user-only) plus 54 with Sonnet 5.5, which is Q188's 153.

   Done when

   • Met: the eval ran with 12 cases (9 model-invocable plus 3 user-only), 3 arms and 3 runs. All suites were VALID. This record is the Probe result: comment.

   Branch: host-run/11959-handoff-phrasing-null, head 27ef986829d0247b65c5864218710dd26b72fd2e, pushed. It holds one commit, which records the null result in docs/contracts/seam-phrasing/README.md (rule 5).

   • scripts/check-affected.sh --tests on the committed head: 12 passed, 4 skipped.
   • There is no changelog fragment, because the change touches no plugin.
   • No PR is open. The coordinator opens it.

   Target labels: add none; remove host-run.
6. kyle-sexton

   removed

   host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.Needs an unattended run on a host; the host runner takes it, not a person or the issue lane.

   on Nov 1, 2026
7. kyle-sexton

   closed this as completedin #12042on Nov 1, 2026