feat(selfhost): execute generated E2E tests against the PR preview URL

[JSONbored]

Part of #4189. Depends on #4191. Independent of #4197.

Revised 2026-07-08 — deprioritized, NOT required for v1. CodeRabbit's own answer to "how do you safely handle executing untrusted generated test code" is that they don't — they push the commit and let the repo's own CI execute it, because CI is infrastructure the repo owner already trusts and already runs untrusted PR code in. gittensory's gate already requires CI to be green before merge, so #4197's commit mode alone already satisfies "the generated test gets validated before the PR merges," with zero new execution surface for gittensory itself. This issue — gittensory itself executing generated tests pre-commit, for a live pass/fail preview or for repos with weak/no CI — is a real but separate, higher-risk enhancement. Do not block shipping #4190-#4197/#4200-#4201 on this issue.

Context

If/when this ships, generated tests can be executed, but only in gittensory's self-host (real Node) runtime, never inside the Cloudflare Worker — this is structural, not a policy choice, verified directly against the code:

• Production hosted-mode browser access is @cloudflare/puppeteer against env.BROWSER (src/review/visual/shot.ts) — navigate-and-screenshot only, a single screenshot() call with a 10s timeout sized for one page render (SCREENSHOT_TIMEOUT_MS = 10000, shot.ts:72). There is no click/type/assert surface anywhere in this binding.
• A real Playwright test is a .spec.ts run by @playwright/test's own runner as a separate Node process, with its own ~300MB browser-binary install — Workers has no subprocess spawning and no bundled browser binaries.
• Self-host's existing browserless/chrome sidecar (scripts/build-selfhost.mjs:48, docker-compose.yml:368) is a long-lived, shared service used today only for navigate-and-screenshot (visual capture) — it is the wrong isolation model for running untrusted, AI-generated code, as opposed to just navigating an already-deployed URL. A shared, persistent browser service risks one PR's script affecting a subsequent PR's run (cookies, local storage, in-memory state) if not given a fresh, isolated context per run at minimum, and arguably needs a disposable container/microVM per run given it's executing generated code, not just rendering a page.

Requirements (if/when this is picked back up)

• A new, self-host-only, ephemeral, single-use execution sandbox per test run — not a shared long-lived service like browserless. Candidates: a disposable Docker container per invocation (the self-host stack is already Docker-based), or a microVM (Firecracker-class) if the operational overhead is acceptable.
• No secrets/credentials mounted into the sandbox at all.
• Network egress restricted to only the PR's preview-URL origin (resolved via src/review/visual/preview-url.ts's existing discovery chain) — no general internet access from inside the sandbox.
• Strict CPU/time/memory limits, sized for a multi-step test with network waits and assertions (explicitly not shot.ts's 10s SCREENSHOT_TIMEOUT_MS, sized for a single screenshot).
• Torn down immediately after each run — never reused across PRs or repos.
• Execution results (pass/fail/error) feed back as advisory annotations only, never a gate blocker.
• A documented plan for keeping the sandbox's Playwright/browser version in lockstep with what generated tests target.

Deliverables

• A documented sandboxing design reviewed before any implementation starts (this is security-sensitive enough to warrant a design review as its own step, separate from writing the execution code).
• Self-host-only execution module, ephemeral per-run, network-egress-restricted.
• Execution results rendered as advisory annotations only.

Expected outcome

On a self-host deployment that specifically wants pre-commit pass/fail preview (rather than relying on the repo's own CI), a generated Playwright test can run against the PR's live preview deployment inside a disposable, isolated sandbox — while v1 of the feature (#4190-#4197) ships and is useful without any of this.

Resources / examples

• src/review/visual/shot.ts:72, scripts/build-selfhost.mjs:48,54,56, docker-compose.yml:368 (why the existing browser infra is the wrong isolation model to reuse as-is)
• src/review/visual/preview-url.ts:83,169,210,242 (the preview-URL discovery chain, still reusable)
• CodeRabbit's actual execution model (delegates to the repo's own CI rather than executing itself) — the reason this issue is deprioritized, not the reason it's impossible

Effort

XL — new execution surface, new sandboxing surface, self-host-only; treat the design-review deliverable as a hard prerequisite to writing any code here.

[linear-code]

JSO-3230

[JSONbored]

Closing — confirmed with the maintainer that gittensory-side sandboxed execution isn't needed. The security goal this issue existed for (never run untrusted, LLM-generated test code without proper isolation) is satisfied without building anything here: the generate → comment/commit pipeline (#4190-#4197, #4201, #4196, all shipped) never executes the generated test itself. Generating test source and writing/committing it as a file isn't code execution — there's nothing to sandbox. Once a generated test lands via commit-delivery mode, the target repo's own CI runs it before merge (the same trust boundary CodeRabbit itself relies on, and the same one this repo's own gate already enforces via required CI checks).

This issue's actual scope — gittensory itself running the test pre-commit for a live pass/fail preview — remains a real, separate, higher-risk enhancement if ever wanted later (e.g. for a repo with weak/no CI), but it's not required for v1 and isn't planned. Re-open if that changes.