Skip to content

feat(pr-pipeline): run the project's app-control skill in pr-verify after the deterministic tests #8651

Description

@kyle-sexton

Refs: #7580
Refs: #7555
Decision: #7555 (comment)
Blocked by: #8816 (adds the app-control plugin and /app-control:drive, whose report.json this step reads).

Agent Brief

Type: Feature
Summary: After the deterministic tests pass, the verifier runs the project's own app-control skill against the PR head and folds what it finds into its verdict. Stack-neutral: the project names its skill; Docker is used only if that skill uses it.

Problem: The verifier judges a PR from its diff and text; nothing drives the running app the way a user would. The line that asked for this ("agent browser checks run after the deterministic tests, with no write token while PR code runs") sat in Phase 9, which closed with it unbuilt.

Current behavior:

  • .github/workflows/pr-verify.yml:3-27: runs on workflow_run after pr-require-checks completes; "No pull request code runs here"; job tokens can only read (:66-68, :384-386).
  • No app-control plugin exists yet; Phase 13 creates /app-control:drive and the generated control-<app> skill shape.

Desired behavior:

  • A setting in docs/conventions/pr-pipeline.yaml names the project's app-control skill. Unset: the step is skipped and the verdict says no app check ran.
  • When set: a separate job, after the deterministic checks pass, checks out the PR head and runs /app-control:drive --unattended, which finds the project's control skill; the verdict job judges drive's report.json (drive itself never judges; D54). It holds no write token and no secret beyond the model credential the run needs, on a fresh GitHub-hosted runner, under ADR 0049's hardening.
  • Its findings go to the verdict job as data, framed as untrusted (docs/conventions/untrusted-content/README.md); a blocking finding makes the verdict FAIL.
  • Nothing in the workflow names a stack. Docker appears only if the project's skill starts it.

Key interfaces:

Done when:

  • The diff adds the setting, the new job and the verdict input, and the job's permissions: grant no write scope (read from the diff).
  • With the setting unset, the verdict notes that no app check ran (a workflow.test.mjs case). CI's test-node passes on the PR (a PR check; this run does not run it).
  • CI's check-plugins passes on the PR (a PR check; this run does not run it).
  • Security & Cost's pass on the final head is recorded on the PR (.github/workflows changes; owner rule 2026-10-20).

Out of scope:

Origin:

  • Verifier agents that run the app and click around like a user, then fix and repeat: hers (Lauren, stream ~53:10), adapted.
  • Docker app stack plus browser checks in CI: the owner's (2026-10-16); moved to Phase 13 and made stack-neutral: ours, confirmed by the owner 2026-10-20.

Review: security (runs PR code in CI).

Do not run with: other pr-verify.yml changes.

Size Large, Complexity High, Uncertainty Knowable.


Slice of #7580 (Phase 13), filed from the owner's 2026-10-20 disposition round.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.

    Fields

    No fields configured for Feature.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions