ci(pr-verify): add the capture job that drives the affected app (S7 of #8651)

Parent

Refs #8651. Phase S7 ("Phase 10: S7 pr-verify capture job") of the evidence-verification plan, durable copy at #11364 (comment); Brief at #11364 (comment). #11393 (S5c) waits on this job; before this issue existed it was blocked by #8651 itself, its own parent.

Relates to #7529 (Claude Code UI leg of the per-PR smoke).

Depends on

feat(harness-ops): let control-claude-code run unattended on sessions it launches #11386 (S4a, control-claude-code unattended and credential)
feat(harness-ops): mask control-claude-code captures and launch changed plugins #11390 (S4b, control-claude-code masks and changed-plugin launch)
S2a (feat(testing): make run-e2e consume affected and write e2e-summary.json #11388) and S6 (feat(pr-pipeline): add the verify-capture key and verdict evidence metrics #11385) are closed.
The plan's two owner-reserved gates are answered. U1, "CI credential exposure", was answered by Q10 (feat(harness-ops): verify PR acceptance criteria by driving Claude Code (screenshots, video) before merge #11364 (comment)). U2, "PR code in verdict run", was answered by the superseding Q11 record (feat(harness-ops): verify PR acceptance criteria by driving Claude Code (screenshots, video) before merge #11364 (comment)). The gates are named in the design's table (feat(harness-ops): verify PR acceptance criteria by driving Claude Code (screenshots, video) before merge #11364 (comment)).

Outcome

pr-verify runs a capture job on in-repo PR heads. The job drives the affected app unattended, uploads the publishable captures as a retained artifact, and feeds them to the judge jobs as untrusted data. A missing or blocked capture never fails a verdict.

Plan (phase S7)

Satisfies: AC1, AC2, AC5, AC9, AC10
Size: Large. Files: .github/workflows/pr-verify.yml, .github/scripts/pr-verify/prompt.md, .github/scripts/pr-verify/workflow.test.mjs. Design: T8, sequence flow 2.

Probe step first. After setup-runner-isolation, a head-code step tries to read the runtime token and the credential from a hook. The job fails closed if either is readable.
Job capture. It uses the same if as the existing judge jobs, so fork heads never run. It runs the gates and uses a read-only token, .base plugins and --trusted-root .base, then calls /testing:run-e2e unattended app=<merge.verify-capture.app> base=<base-sha>. It uploads the publishable files as pr-capture-<pr>-<sha> with retention-days. When the key is unset, it skips with a note.
Gates first. The capture job's first steps are the kill switch, resolver, trusted-trigger and agree steps (check-intent's copies stay), so it captures nothing on a run that judges nothing. It sets timeout-minutes: 30. The outer session runs through ci-workflows' run-claude-cli with its own role, a tool allowlist and the drive maxEntryPoints cap (S4b). A capture timeout leaves capture: blocked and never stops the judge.
Judge jobs. Only check-intent gains needs: capture, with !cancelled(). check-intent-subscription and check-intent-fallback keep their success chain and download the same run's artifact by id. All three download into pr-verify-data/capture/. The prompt frames the capture as untrusted data and says a missing or blocked capture is never a FAIL reason. A trusted-text manifest line Evidence loop: 3/3 cap reached, last NEEDS WORK is a FAIL (AC7's park path then runs).
Last step. After the probe passes on a real PR, set merge.verify-capture.app: claude-code and show one in-repo PR with a judged capture (AC1). Update the header comment block in pr-verify.yml that describes the jobs (reviewed change: jobs now take an artifact).

Credential separation (Q10, Q11)

The superseding Q11 record (#11364 (comment)) is the Issue Triage session's reading of the owner's 2026-10-31 direction. It says:

The capture job may hold pr-verify's existing credential (Q10).
The PR's own code runs in a child process whose environment and readable files hold no token. The credential stays with the Claude Code driver and is never passed to, exported to, or written where the app under test can read it.
In-repo heads only.

If the build cannot keep the token out of the PR code's reach, it stops and brings that back to the owner as a new question. It does not ship without the separation.

Done when

node --test .github/scripts/pr-verify/workflow.test.mjs exits 0 with these cases:
- the capture job's permissions contain no write;
- the three judge jobs carry identical capture-download steps;
- the capture if includes the head-repository check;
- pr-verify.yml has no job-level always();
- only check-intent has needs: capture;
- a probe step exists, runs after setup-runner-isolation and before any head code, and has no continue-on-error.
The capture job's step that starts the app under test passes it no credential variable, and a test asserts that the app process's environment holds none of the pr-verify credential names.
actionlint passes on pr-verify.yml.
One in-repo PR shows a judged capture with merge.verify-capture.app: claude-code set (AC1).

Builder: the Automation Lanes session. This is host-run work: it edits a workflow file and needs a live PR run.

Claim (lane 3, hub-assigned): partial build only. Lane 3 builds the parts that do not need #11386 on a local branch: the probe step, the capture job skeleton with its gates and timeout, the judge jobs' capture-download steps, the prompt framing, the workflow.test.mjs cases and actionlint. The live judged capture (done-when 4) waits on #11386. The body names the Automation Lanes session as builder; the hub has told that session lane 3 holds this partial build. No push until the hub confirms a partial PR is wanted.

Owner question Q1 (credential separation, under Q11). The build stopped on this issue's own rule: "If the build cannot keep the token out of the PR code's reach, it stops and brings that back to the owner as a new question." Nothing was committed.

Problem. The merge gate trusts a verdict only because a run of pr-verify.yml uploaded it. In .github/workflows/pr-verify.yml:13-15: "Only steps of a run of this file on main can upload that run's artifacts, so the pr-merge lane counts a verdict only through its run record". And :21: "No pull request code runs here." babysit_verify_evidence.py keys on WORKFLOW_PATH = ".github/workflows/pr-verify.yml".

A capture job inside that run puts PR code in the same run. Several node steps run later in that job while holding the job's runtime token: the capture artifact upload, and the post steps of checkout and setup-node, which run regardless. pr-run-activity-read.yml:2021-2024 documents the same exposure in the read lane: "Each actions/checkout post step runs git ... as a node step that holds the job's runtime token." PR code that outlives its step, or a file it plants, could then upload a forged pr-verify-<pr>-<sha>-pass.

The issue's probe step checks only what can be read at one moment. It does not catch this.

Options

Recommended. Capture runs as its own workflow, triggered by the same workflow_run. The judge jobs download its artifact across runs by run id, with actions: read. PR code then never runs inside a pr-verify.yml run, so the verdict's trust basis stays as it is. Basis: pr-verify.yml:13-15,21 and babysit_verify_evidence.py WORKFLOW_PATH. Cost: the two runs must be sequenced, since the judges wait on the capture run or skip it.
Keep one run, but have the judge jobs seal each verdict in a way PR code cannot reproduce. Every verdict reader would have to change.
Keep one run and harden it: a process reaper, a root-owned externals directory, removing .git directories as the read lane does. You would accept the remaining risk of a forged pass verdict.

Unblocks: all of #12232, and #11393 after it.

Also found during the build, to fix whichever option wins:

The issue's /testing:run-e2e unattended ... call shape is wrong: the bare word is scenario text. The run needs the --unattended flag (plugins/testing/skills/run-e2e/SKILL.md:45-53).
The e2e-summary.json capture shape does not match the keys write-verdict.mjs reads, so a raw summary always records blocked.
The localLoop regex in write-verdict.mjs does not match Evidence loop: 3/3 cap reached, last NEEDS WORK.
resolve.mjs has no capture model route yet.

Decision recorded. Q1 (issuecomment-6619061583): option 1. Capture runs as its own workflow, and pr-verify's judge jobs download its artifact across runs. PR code never runs inside a pr-verify.yml run. Owner, verbatim: "Go w/ whatever this suggests", with the standing direction s3: "Never weaken a high/critical security requirement or boundary to gain throughput." Options 2 and 3 either keep PR code in the run that holds the verdict's upload rights or accept a forged-pass risk. Lane 3 resumes the #11386-independent build under option 1, and also fixes the three plan defects listed in Q1.

fix(pr-verify): fence the capture driver's file tools to the head and…

Proposed work class: work-class: structural
Basis: automated intake triage; the lanes App applied this class.

Size: Large
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none

Decision recorded. Owner, 2026-11-06 ~13:4xZ, in session (Lane 6), on Q7-Q10: "Yes, go with recommendations here."

Q7 (a): route the outer capture claude through the key proxy with a dummy key, in a follow-up PR after the switch-on. Basis: devil's-advocate finding 15. The outer leg's run-claude-cli refuses any base URL but the API (ci-workflows lane.cjs:975), so the follow-up starts there.

Q8 (a): accept that the driven session runs --restricted (no Bash) for now, so a Bash-event-only hook is recorded, not driven; a follow-up probes Bash in the sibling sandbox.

Q9 (c): narrow the outer leg's Bash to the driver and git commands, done together with Q7.

Q10 (a): an autoMode allow entry so sessions may remove needs-human and needs-session from melodic-software and kyle-sexton PRs whose design the owner decided, landed through a dotfiles PR.

The switch-on (merge.verify-capture.app: claude-code-plugins) goes ahead after PR #13707; Q7 and Q9 are hardening of the key-holding leg, with no known route from PR code to the key (Phase 5 security review). Phase 5b runner probe: runs 42222553868, 42223922880, 42224592725 (sandbox sealed, driven session reached the stub, screenshot path same inside and out after the font fix).

#12232 (comment)

This decision settles the open questions. The live-PR done-when items need a host run, which the lane can't do. The remaining work is a bounded, structural change to the pr-verify workflow. Related open issues: #8651 (parent), #11364, #11393, #13710 and #13711 (follow-ups).

_{Automated triage, applied by the lanes App.}

The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

Phase 6 run 1 graded (test PR #13719, capture run 42235412941, head 3c5dd0fb471).

Capture: capture.status: ran, count: 12, route drive, affected hooks and mods. The mod entry point ("the mod's command, typed at the prompt of the terminal session") passed; context-budget's mod opens a toast and a pane, so this is the UI-surface case. pr-verify read and judged it: "The capture ran with a count of 12. Its mod entry point passed" (fail-note on test(context-budget): reword a register.tsx comment for the #12232 capture run #13719; the FAIL is only the PR body's missing Remaining section, left as is so a re-judge does not replace this capture). This meets done-when 4.
Outer leg: 18 turns, 445 s, run-claude-cli on the key. Its typed command reached the driven session, so the wait-before-first-type risk from the Phase 5b probe does not hold.
Isolation: the sibling sandbox saw no credential and no outside process and reached only the key proxy; the reach probe found nothing head code or the driver's file tools can read; "Stop the key proxy" passed; the capture scanned clean (1 publishable file).
Key credit: the driven session's model calls answered "Credit balance too low" (proxy: 8 requests, 27,876 tokens, by status 200:2 400:2 404:2 499:2), so the hook entry points were not driven. Per Phase 6's run budget (stop after the first credit stop), runs 2 and 3 are not launched. Per the owner's key-only decision (issuecomment-6667440359), captures record blocked until PR_VERIFY_CLAUDE_API_KEY's promotional credit refreshes.
token-cap: not set from this run: 27,876 driven tokens stopped at the credit wall and are not a full capture. The provisional cap stays until a run on refreshed credit is measured.
Fixed on the way: fix(pr-verify): install the screenshot font and match the key proxy by command line #13707 (screenshot font, proxy stop by command line), fix(pr-verify): accept any font fc-list reports for U+23F8 #13720 (font check matched a family name the runner's fc-list does not print; every capture failed at that step from feat(pr-pipeline): switch the PR capture on for claude-code-plugins #13712's merge until fix(pr-verify): accept any font fc-list reports for U+23F8 #13720).

Remaining on #12232: measure a full capture on refreshed credit and set token-cap; hardening follow-ups #13710 and #13711.

Closing: every done-when item is met on main 6d9aee8, three of them under the design the owner decided in this thread (capture in its own workflow pr-verify-capture.yml, driven Claude Code in a sibling sandbox behind the key proxy, key-only). A fresh-context verifier checked each item against main:

Done when	Verdict	Evidence
1. `workflow.test.mjs` cases	met by the design change	76/76 on main. Named tests: "the capture workflow and its jobs can write nothing", "the three judge jobs carry identical capture steps", "the capture runs only for a same-repository head", "neither workflow has a job-level always()". `needs: capture`: with capture in its own workflow no job has it, which the suite asserts. The probe runs after isolation and before head code; it carries `continue-on-error` so a credit-stop handoff ends green (fallback decision issuecomment-6662742618), and "Check the probe" fails closed before the head checkout.
2. No credential to the app	met by the design change	The app starts in "Start the sibling sandbox" under an `env -i` allowlist (`PATH HOME TERM LANG CCD_RUNTIME_DIR`), asserted in the suite; `sibling-init.sh` probes for every pr-verify credential name inside the sandbox and fails closed. The driven claude holds only a dummy key (the broker refuses a real one).
3. `actionlint`	met	Clean on `pr-verify.yml` and `pr-verify-capture.yml`, apart from the `cache-mode` key `.github/actionlint.yaml` ignores.
4. A judged capture with the app set	met by the design change	`merge.verify-capture.app: claude-code-plugins` (this repository's app; its control skill drives each changed plugin through `control-claude-code`). Test PR #13719 (UI surface: context-budget's mod) captured with count 12, mod entry point passed, judged by pr-verify (grade).

Follow-ups: #13724 (set token-cap from a full capture once the key's promotional credit refreshes), #13710 (outer leg through the key proxy, narrowed Bash), #13711 (Bash in the driven session).

Navigation Menu

Repository navigation

ci(pr-verify): add the capture job that drives the affected app (S7 of #8651) #12232

Parent

Depends on

Outcome

Plan (phase S7)

Credential separation (Q10, Q11)

Done when

30 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

ci(pr-verify): add the capture job that drives the affected app (S7 of #8651) #12232

Description

Parent

Depends on

Outcome

Plan (phase S7)

Credential separation (Q10, Q11)

Done when

Activity

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

30 remaining items

github-actions commented on Nov 6, 2026

melodic-automation-lanes commented on Nov 6, 2026

kyle-sexton commented on Nov 6, 2026

kyle-sexton commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions