perf(source-control): cut the pull-request skill body through progressive disclosure

Outcome

The /source-control:pull-request body is cut through progressive disclosure: action-specific detail moves to reference files read on demand, and the hub keeps routing and gates. This is one of five slices under #12243; the first cut (/source-control:worktree) landed in #12387, whose recipe this follows.

Size at origin/main 3bd42cabfe: body 50,149 bytes (SKILL.md with frontmatter stripped, the method #12387 used); the skill directory holds 799,595 tracked bytes. A cut that moves about half the body is a diff well under the 262,144-byte cap. The binding limit is the paired eval run each cut needs, so each skill is its own PR.

Done when

The hub's body bytes fall, measured the same way. Moved text sits verbatim in reference/ (or an action file), and every citation of it (tests, hooks, scripts, fixtures, README) is repointed.
/docs-hygiene:compress compare returns VERDICT PASS with no SEMANTIC LOSS.
The pull-request-* cases in plugins/source-control/evals/ run on the before and after copies with /evals:plugin-eval --baseline <before.json> --margin <m>, on a round run-validity.py marks VALID on both arms, and the after arm is non-inferior (plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, "Cutting text").

perf: cut the bodies of the most-invoked skills through progressive disclosure - part 1 of 2 #12243 (parent), perf(source-control): move the worktree nesting measurement out of the skill hub #12387 (cut 1 recipe), perf: cut the bodies of the most-invoked skills through progressive disclosure - part 2 of 2 #12244 (after figure).

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none

Understood: cut the /source-control:pull-request skill body through progressive disclosure. Move action detail verbatim to reference files, repoint every citation, and pass the compress compare and the paired eval non-inferiority check. The eval and compare steps may need a host run, which admission routes.

Decision recorded.
(agent; not an owner decision)
run-validity runs on both arms of the done-when 3 round with --measured-fire pull-request-ready-hands-pr-to-explain-change, and that case keeps its plain prompt.

Basis: in the firing check (--runs 1 on the before arm), the case loaded the skill under test: its first tool call is Skill source-control:pull-request ready 512 .... run-validity still reports "the skill did not fire" because the case grades the hand-off to explain-change with a tool: Skill, min: 1 grader (graders/digest-at-ready-event.md). That hand-off is the behavior under test, which run-validity --help describes as "a should-trigger case whose firing is the quantity a comparison measures". Naming explain-change in the prompt would hand the model the graded answer. Dropping the case would leave the ready flow, which #12739 touched, untested. Every other check is unchanged. The missed hand-off is reported as a behavior finding with per-arm counts.

#12655 (comment)

This decision settles the open question about the measured-fire case, so nothing more is asked.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (measure-over-time: "The hub's body bytes fall, measured the same way. Moved text sits verbatim in..."), item 3 (a model eval run: "The pull-request-* cases in plugins/source-control/evals/ run on the before a..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

Claim. Backlog HITL host-run worker is taking this now, with #12468: its pull-request cut is this slice, so one PR serves both.

Claim. Backlog HITL host-run worker is taking done-when 3 (the paired pull-request-* eval for #12739) now, resuming the earlier run that stopped in a host lockup.

Negative control, preregistered before the first paid run.

#12739 cut the pull-request skill's body and merged without the negative control that plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md:60-66 and :79 require. This runs it after the fact.

Suite: the 21 pull-request-* cases in plugins/source-control/evals/.
Arms, launched together on one model (claude-opus-5-5), same flags (--runs 2 -j 2 --scaffold --ablation none --keep-temp):
- before: origin/main (e3bf854) with perf(source-control): move the pull-request monitor checklists to monitor.md #12739's diff (5f63e3d) reverted under plugins/source-control/skills/pull-request/.
- degraded: the before copy with the pull-request SKILL.md body deleted (frontmatter kept).
Validity: run-validity.py --runs 2 --scaffolded on both arms, plus --own-copy-search-warns on degraded only.
Compare: noise-report.py <degraded> --baseline <before> --margin 0.05.
Threshold: the suite detects a loss when the degraded mean falls at least 0.15 below before (or the noise report calls a case drop). Cases whose degraded runs carry a guard denial are left out of that comparison. If neither holds, the body cut is unmeasurable by this suite.

Host run. Done-when 3: a paired eval of every pull-request-* case in plugins/source-control/evals/, run on a before arm with #12739's diff reverted and on an after arm at main.

What ran (Claude Code 2.1.317, subscription login, isolated HOME, the claude.slice scope):

Arms: two git archive copies of the case-fix branch head f400737, which is origin/main 1e321c6 plus the case fix. The before copy has perf(source-control): move the pull-request monitor checklists to monitor.md #12739's diff (5f63e3d) reverted under plugins/source-control/skills/pull-request/ only, and diff -rq shows the copies differ only in that directory.
Firing check: claude plugin eval <before copy> --case 'pull-request-*' --runs 1 -j 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5. Every non-slash case loaded the pull-request skill (11 of 11). The 10 ci-mode cases open with the slash command.
Round: the same command, with --runs 2, on each arm. Before 18:39 to 18:52Z, after 18:39 to 18:55Z, both under the host's heavy-run lock at load < 2.
run-validity.py --runs 2 --scaffolded --measured-fire pull-request-ready-hands-pr-to-explain-change on each arm. The reason for the flag is in the Decision recorded comment on this issue.
noise-report.py <after> --baseline <before> --margin 0.05.

Verdicts:

before: verdict: VALID (warnings: measured cases fired in 0 of 2 runs, models)
after: verdict: VALID (warnings: measured cases fired in 0 of 2 runs, models)
before: with-arm mean: 0.91, 95% interval 0.82 to 0.99 (normal, 21 cases)
after: with-arm mean: 0.90, 95% interval 0.81 to 0.98 (normal, 21 cases)
with-arm change (this minus baseline), paired over 21 cases: -0.01, 95% interval -0.05 to +0.03
compare verdict: non-inferior at margin 0.05 (the interval's lower end -0.05 is at or above -0.05)

Done-when 3: met. Both arms are VALID, and the after arm is non-inferior at margin 0.05. The lower end sits exactly on the margin.

Findings:

Triggering (not fixed): pull-request-dirty-worktree-stashes-before-checkout with its plain prompt on main fired the skill in 0 of 4 runs. Each time the model stashed with git stash push --include-untracked and ran git switch main itself. The case-fix PR names the skill in that prompt, so the case now tests behavior, not triggering. The trigger gap belongs to the skill's description and trigger probes (/skill-quality:check measure-invocation).
Behavior: pull-request-ready-hands-pr-to-explain-change loaded the pull-request skill but never handed off to explain-change: 0 of 2 runs per arm, the same on both arms, so perf(source-control): move the pull-request monitor checklists to monitor.md #12739 did not cause it. reference/ready-for-review.md 2.5.5 carries the instruction in both arms.
Judge model: the result does not record it (run-validity models WARN). The round passed no --judge-model, the same on both arms, and neither the result nor the traces record which judge model ran.

PR: the case fix is in #12915.

Target labels: add none; remove host-run.

Host run. The negative control for #12739's pull-request body cut, run after the fact and preregistered in #12655 (comment).

What ran (Claude Code 2.1.317, subscription login, isolated HOME, the claude.slice scope, under the host's heavy-run lock at load 0.97):

Arms, both git archive copies of origin/main e3bf854: before has perf(source-control): move the pull-request monitor checklists to monitor.md #12739's diff (5f63e3d) reverted under plugins/source-control/skills/pull-request/; degraded is the before copy with the pull-request SKILL.md body deleted (frontmatter kept). diff -rq shows they differ only in that file. The earlier round's before copy was not reused: it was built on the unmerged case-fix branch and differs from main in two files.
claude plugin eval <copy> --case 'pull-request-*' --runs 2 -j 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5 on both arms, launched together 19:38:58Z; degraded ended 19:51:42Z, before 19:54:04Z.
run-validity.py --runs 2 --scaffolded on before; the same plus --own-copy-search-warns on degraded.
noise-report.py <degraded> --baseline <before> --margin 0.05.

Verdicts:

before: verdict: INVALID (the skill did not fire in 3 with-arm runs of a should-trigger case)
degraded: verdict: INVALID (4 permission denials in the traces; the skill did not fire in 4 with-arm runs of a should-trigger case)
before: with-arm mean: 0.91, 95% interval 0.84 to 0.98 (normal, 21 cases)
degraded: with-arm mean: 0.85, 95% interval 0.76 to 0.94 (normal, 21 cases)
with-arm change (this minus baseline), paired over 21 cases: -0.06, 95% interval -0.14 to +0.02
case drop: case pull-request-create-default-open-state-ready with-arm fell from 1.00 to 0.50, a third or more; name its cause and prove it before shipping
case drop: case pull-request-create-verification-ends-left-out with-arm fell from 1.00 to 0.50, a third or more; name its cause and prove it before shipping
compare verdict: not shown non-inferior at margin 0.05 (the interval's lower end -0.14 is below -0.05), and 2 case(s) dropped by a third or more

Validity:

Both arms miss the skill-fired check on the same case: pull-request-dirty-worktree-stashes-before-checkout fired 0 of 2 per arm. Its plain prompt on main never fires the skill (the earlier host run's finding; the prompt fix is in the unmerged test(source-control): name the pull-request skill in the dirty-worktree eval case #12915). pull-request-ready-hands-pr-to-explain-change also missed once on before and twice on degraded.
Degraded's 4 denials are each a search of its own plugin copy for the lost text (cd into the copy's skill directory, or sed on its reference file), in ci-mode-escalates-instead-of-cutting-criterion-line, ci-mode-fixes-markdown-lint, ci-mode-makes-room-without-cutting-criterion-line and ready-hands-pr-to-explain-change. --own-copy-search-warns did not downgrade them.

Means with those cases left out (per-case mean over both runs):

denial cases left out: 17 cases, before 0.914, degraded 0.850, fall 0.064.
denial cases and the never-firing dirty-worktree case left out: 16 cases, before 0.940, degraded 0.872, fall 0.068.

Result: two readings, for the coordinator to pick.

Under the preregistered rule (a 0.15 fall or a case drop, as in rewrite-a-plugin.md:63-64): detect, through the 2 case drops. Neither falls on a denial case, so both hold with those cases left out. Each drop is consistent across runs: both degraded runs of each case scored 0.50, against 1.00 and 1.00 on before. One judge vote in create-default-open-state-ready run 2 was split (FAIL FAIL PASS).
Under the mean-only 0.15 threshold: no detect. The mean falls 0.06 to 0.07, with or without the denial cases.

The degraded copy keeps reference/, and the traces show it searching its own copy for the lost text; reads that were not denied may have recovered much of the procedure, which would explain the small mean fall.

Traces are kept under the run's TMPDIR for the coordinator.

Target labels: add none; remove none.

Decision recorded.
(agent; not an owner decision)

The negative controls for #12739 (#12655) and #12704 (#12666) both detect a loss, so both cuts stand.

Basis: plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md:60-75. A suite detects a loss on a case drop or a 0.15 fall. A case whose degraded runs carry a denial cannot decide the control, so the drop or fall must hold with those cases left out.

#12655 (pull-request): detect.

Two case drops, create-default-open-state-ready and create-verification-ends-left-out, 1.00 to 0.50 each in both degraded runs. Neither case had a denial.
The 4 degraded denials and the before arm's skill-fired misses (dirty-worktree, already reported as a triggering finding; ready-hands-pr-to-explain-change) sit in other cases.
The mean fall is 0.06, under 0.15, so the call rests on the case drops.

#12666 (commit): detect.

The mean falls 0.18; with the one non-firing before run (no-push-no-pr) left out, it falls 0.213.
Two case drops, conventional-heredoc-trailer (1.00 to 0.60) and exec-bit-backstop (1.00 to 0.33).
The degraded arm is VALID with no denials.

Target labels: add none; remove none.

Correction: the "both controls detect" decision above is withdrawn until a clean re-run.

In both controls the before arm read INVALID on run-validity, because some with-arm runs never fired the skill. The decision leaned on cases unaffected by those runs. That is a looser reading than this lane's own rule, which reads numbers only from runs that pass validity. rewrite-a-plugin.md:60-75 permits leaving out degraded-side cases that have denials. It does not permit an INVALID before arm.

Next: re-run both controls on current main. #12915 has merged and fixed the dirty-worktree case. A control counts only when the before arm reads VALID; the degraded arm may leave out cases with denials, per the doc. If a clean control does not detect a loss (a case drop or a 0.15 fall), that cut is unmeasurable and its skill body reverts to the before text by PR. Both cuts stay merged until the re-run reports.

Host run. The clean re-run of the negative control for #12739's pull-request body cut, preregistered in #12655 (comment). It replaces the earlier control, whose before arm read INVALID.

What ran (Claude Code 2.1.318, subscription login, isolated HOME, the claude.slice scope, under the host's heavy-run lock at load 3.46):

Arms: both are git archive copies of origin/main 97ba98f. That base includes test(source-control): name the pull-request skill in the dirty-worktree eval case #12915, the dirty-worktree case fix, and no commit since the earlier base (e3bf854) touches skills/pull-request/. The before arm has perf(source-control): move the pull-request monitor checklists to monitor.md #12739's diff (5f63e3d) reverted under plugins/source-control/skills/pull-request/ only, and the patch applied with no fuzz. The degraded arm is the before copy with the pull-request SKILL.md body deleted and its frontmatter kept. diff -rq shows the two differ only in that file. Following the preregistration, "before" is the pre-cut text, not the current hub.
claude plugin eval <copy> --case 'pull-request-*' --runs 2 -j 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5 on both arms, launched together at 00:38:44Z. Before ended at 00:52:41Z, degraded at 00:54:48Z.
run-validity.py --runs 2 --scaffolded --measured-fire pull-request-ready-hands-pr-to-explain-change on before; the same plus --own-copy-search-warns on degraded.
- The --measured-fire flag applies this suite's validity decision, perf(source-control): cut the pull-request skill body through progressive disclosure #12655 (comment), used in the done-when 3 round. That case loads the pull-request skill, but its graded call is the hand-off to explain-change. The earlier control left the flag out, and that, together with the then-unfixed dirty-worktree prompt, is what made its before arm read INVALID. The flag only marks that case's skill-fired check as a measured rate; it leaves no case out of any score.
noise-report.py <degraded> --baseline <before> --margin 0.05.

Verdicts:

before: verdict: VALID (warnings: measured cases fired in 0 of 2 runs, models). No denials in 42 traces, and every session started.
degraded: verdict: VALID (warnings: measured cases fired in 0 of 2 runs, models). No denials in 42 traces, so no case is left out.
before: with-arm mean 0.89 (per-case mean 0.889, 21 cases)
degraded: with-arm mean: 0.78, 95% interval 0.66 to 0.90 (normal, 21 cases)
with-arm change (this minus baseline), paired over 21 cases: -0.11, 95% interval -0.21 to -0.01
case drop: case pull-request-create-refs-remaining-fallback-line with-arm fell from 1.00 to 0.62. Degraded scored 1.00 and 0.25 in its two runs.
case drop: case pull-request-create-refs-writes-remaining with-arm fell from 1.00 to 0.17. Degraded scored 0.17 in both runs.
case drop: case pull-request-create-verification-ends-left-out with-arm fell from 1.00 to 0.50. Degraded scored 0.50 in both runs.

Result: detect. Both arms read VALID, and the noise report calls 3 case drops over all 21 cases, with none left out. The mean falls only 0.11, under the 0.15 threshold, so the call rests on the case drops, which rewrite-a-plugin.md:65-66 accepts on their own. #12739's cut stands.

Finding, unchanged from the done-when 3 round: pull-request-ready-hands-pr-to-explain-change handed off to explain-change in 0 of 2 runs on each arm.

Traces are kept under the run's TMPDIR for the coordinator.

Target labels: add none; remove none.

Navigation Menu

Repository navigation

perf(source-control): cut the pull-request skill body through progressive disclosure #12655

Outcome

Done when

Related

1 remaining item

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

perf(source-control): cut the pull-request skill body through progressive disclosure #12655

Description

Outcome

Done when

Related

Activity

github-actions commented on Nov 3, 2026

melodic-automation-lanes commented on Nov 3, 2026

melodic-automation-lanes commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

1 remaining item

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions