[kyle-sexton]

Closes #13759
Refs: #13757

session: App Driven PR Verify

Summary

/testing:run-e2e scans e2e-summary.json with the values from the hub's ## Mask values: line. It now uses the Mask in effect: the hub's own ## Mask, or its hand-off target's. These are the rules PR 13765 adds to app-control:

• The hand-off target resolves from the hub copy that was read, inside trusted-root=, never from the checkout under test.
• Each values: source is judged on its own; a single none: line with exit 0 means that source has no values.
• A hand-off never falls into the "no values: line" branch. An unreadable target or a failed source keeps e2e-summary.json out of publishable.

Fix

• plugins/testing/skills/run-e2e/SKILL.md, Steps 2 and 5. They point at drive's "Mask post-check" instead of restating it.
• New evals:
  • affected-05: the target's source fails, so the summary stays unpublishable even though its text is clean;
  • affected-06: the target's source prints none:, so the summary is scanned with the generic pattern and published.
• A changelog fragment (bump: minor).

Verification

• bash scripts/check-changed-skills.sh origin/main: PASS, 0 errors. The one warning this change adds is run-e2e's line growth.
• check-evals-quality.sh on run-e2e's evals: PASS.
• scripts/check-affected.sh: 14 passed, 2 skipped.

Remaining

• Merge order: PR 13765 merged first; this PR points at its "Mask post-check" text.
• Merge hold: it changes a skill body, so it carries do-not-merge until a valid before/after eval lands (hub rule R379/R381).
• Eval gap: the "never from the checkout" expectation in the two new evals cannot fail as written, because neither prompt holds a checkout copy that would give a different answer. The eval run should add that decoy.

Related

• feat(app-control): publish masked CI captures through a hub Mask hand-off and a none: values result #13757
• feat(harness-ops): verify PR acceptance criteria by driving Claude Code (screenshots, video) before merge #11364
• PR 13762
• PR 13765

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

claude-review: 1 inline finding(s) on b79ed81e281fa048c1cdb16b72a33f3eb2b0bc3f; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 55a91a31b1950e1b36ba65d16e10b1c8c12ef783: read 1 of 1 changed file(s); 1 finding(s), 0 as inline comments.

Code review: incremental pass over 16b2ea02..55a91a31 (blocking only)

This lane has reviewed earlier heads of this PR, so this pass reports blocking findings only. I read every hunk of the one file in scope:

• plugins/testing/evals/run-e2e-touched-feature-calls-refresh/samples/calls-refresh-for-cart-total.json

Incremental change: no blocking findings. The push adds three fail samples for the calls-refresh-for-cart-total regex grader, app-control:map[\s\S]{0,80}?\brefresh[ "]+cart/total\b`. I checked each one against the regex by hand. I could not run the sample checker in this lane.

• Split-line form, wrong feature: misses. The only refresh is followed by checkout. cart/total comes later, and no refresh sits right before it.
• "was not called": misses. The only refresh is the sentence-final refresh., and . is not in [ "]+`.
• Skill-call form, no feature: misses. refresh is followed by --unattended.

Each sample names both the skill and cart/total, so together they pin the regex's requirement that cart/total come right after refresh, not just anywhere nearby. The four existing pass samples are unchanged and still match (re-checked).

Finding (restated: summary findings carry over each round)

1. The post-check rules that run-e2e Step 5 relies on are still not on main. At this head, evidence.md L79-L82 still says "A source that fails or yields no value leaves every capture not-checked and both public reports unpublished". It has no hand-off rule and no none: exception.

   Meanwhile, run-e2e SKILL.md L277-L281 finds the hand-off target "as that post-check finds it" and publishes a none: result "as that post-check accepts it".

   How it fails: in the affected-06 setup, the target's source prints one none: line and exits 0. run-e2e publishes the summary, but the post-check on main rejects that source as one that "yields no value", and it has no hand-off to follow.

   Fix: the PR body sets the merge order (after PR 13765), but nothing enforces it. Either hold the merge until PR 13765 is on main, or enforce the order: stack this branch on PR 13765's branch, or add a dependency label.

Notes

• Closing issue feat(app-control): publish masked CI captures through a hub Mask hand-off and a none: values result - part 2 of 3 #13759: this push does not touch affected-05, the eval its criterion names, so I re-checked nothing for it. An earlier review found that criterion met.
• Settled thread (the "never from the checkout" condition at run-e2e SKILL.md L279): marked lines changed since resolution: no. It stays settled, and I did not raise it again.
• Not verified: the author's Verification section (check-changed-skills.sh, check-evals-quality.sh, check-affected.sh) is their own claim. I could not run any of it in this lane.

[github-actions]

claude-security-review reviewed 55a91a31b1950e1b36ba65d16e10b1c8c12ef783: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #13774, incremental from 16b2ea02 to 55a91a31

No security issues found.

critical-findings: 0

This review only reports blocking findings, because the lane already reviewed an earlier head. Only CRITICAL and IMPORTANT findings count here. I read every hunk of the one file in the incremental scope (1 of 1).

• plugins/testing/evals/run-e2e-touched-feature-calls-refresh/samples/calls-refresh-for-cart-total.json: the push adds three known-bad samples to the eval data for the loosened calls-refresh-for-cart-total regex grader:

  • the skill name and its arguments on separate lines, refreshing the wrong feature (checkout);
  • a run that names the skill but never calls refresh;
  • a Skill(...) call with no feature.

  These are static JSON strings that only feed eval grading. Nothing executes them, and they gate nothing in the merge pipeline or in any lane that holds credentials.

The file matches no TRUSTED PATHS glob. The push touches no plugin manifest, workflow, token request, trust control, merge hold or standing instruction.

The only prior finding is the summary comment at 16b2ea02, which has no line anchor and reported 0 findings, so nothing carries forward. The PR context file has no text that tries to steer the review.

[kyle-sexton]

Held on do-not-merge only until a VALID no-loss before/after plugin-eval pair lands for this head (skill-body change, hub rule R379). It runs after PR 13765's pair, in the App Driven PR Verify session.

[kyle-sexton]

Eval pair: VALID, no loss. plugins/testing: the run-e2e skill before (67e80f97, main merged into this branch) and after (16b2ea02). Both sides used the head's evals/, so only the skill differs. Model and judge claude-opus-5-5, Claude Code 2.1.319, --scaffold --allow-tools Bash Write Edit --runs 6, the 11 cases that drive run-e2e.

• Result: head minus base, paired over 11 cases: +0.02, 95% interval -0.01 to +0.04, non-inferior at margin 0.05 (noise-report.py --baseline --margin 0.05). Mean with-arm score 0.99.

• Validity: run-validity.py --runs 6 --scaffolded returned VALID on both sides, with warnings for skill fired and judge votes. It ran on the laptop that holds the traces, per the hub's relay of the lane verdicts.

• Per case:

  • run-e2e-app-control-absent-reports-drift: 0.83 to 1.00.
  • run-e2e-route-decline-uses-testing-keys: 0.92 to 0.92.
  • The other 9 cases: 1.00 to 1.00.
  • No case lost.

• Earlier rounds:

  • Rounds 1 and 2 (3 runs each) were VALID but not shown non-inferior. Their intervals were -0.07 to +0.01 and -0.09 to +0.03.
  • A regex grader had failed correct runs in both rounds. It rejected Skill("app-control:map", "refresh cart/total …") and a two-line form, while the judge grader passed both. The grader was fixed before round 3 (e30b79df, 16b2ea02).

• Grader check: the loosened pattern is app-control:map[\s\S]{0,80}?\brefresh[ "]+cart/total\b`.

  Sample	Expected	Matches
  /app-control:map refresh cart/total --unattended	pass	yes
  backticked app-control:map refresh cart/total	pass	yes
  Skill("app-control:map", "refresh cart/total --unattended")	pass	yes
  /app-control:map with arguments on the next line refresh cart/total --unattended	pass	yes
  refresh with no feature	fail	no
  refresh of the legacy map path	fail	no
  refresh of checkout	fail	no
  split-line refresh of checkout with cart/total named nearby	fail	no
  names the skill and cart/total, with no refresh call	fail	no
  Skill refresh with no feature	fail	no

• Measured head vs current head: measured at 16b2ea02. The only later commit, 55a91a31, adds only grader samples (samples/calls-refresh-for-cart-total.json). The case loader does not read samples: it reads only prompt.md, case.yaml and graders/.

• Merge order: this PR still merges only after feat(app-control): let a hub hand off its Mask and a values source resolve to none #13765 (the body's Hold line).

[melodic-automation-lanes]

The do-not-merge hold waited only on eval evidence (reason=eval), and a listed human posted a VALID no-loss eval-pair record for 55a91a31b1950e1b36ba65d16e10b1c8c12ef783: #13774 (comment). pr-merge removed do-not-merge. To hold again, add the label back with a new pr-hold reason line.