feat(review): LLM core to turn a PR diff into Playwright E2E test source - #4207
Merged
Conversation
Adds the one genuinely new piece of engineering the #4189 epic needs: a pure prompt builder + response parser, plus an async orchestrator that generates a complete Playwright test file from a PR's changed-file diffs. Mirrors ai-slop.ts's established shape exactly: BYOK-vs-default model selection (the maintainer's own frontier model when configured, else the free/default reviewer with bounded retry/fallback), the shared daily neuron budget + per-repo/day BYOK cap, and safety-feature-gated prompt-injection defanging (reusing defangReviewInput, not a second implementation). Parsed output is validated against a Playwright-shaped signature (a recognizable test call plus its own @playwright/test import) before being trusted -- malformed or off-topic model output is dropped, never surfaced. Fully fail-safe (disabled/unavailable/quota-exceeded/unparseable all degrade to a non-throwing result) and gated behind the e2eTests kill-switch from #4190, so it has zero effect until a future PR in the epic actually calls it. Part of #4189. Progresses #4191.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4207 +/- ##
==========================================
+ Coverage 93.74% 93.75% +0.01%
==========================================
Files 388 389 +1
Lines 36421 36495 +74
Branches 13342 13373 +31
==========================================
+ Hits 34143 34217 +74
Misses 1621 1621
Partials 657 657
🚀 New features to boost your workflow:
|
This was referenced Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/services/ai-e2e-test-gen.ts, the one genuinely new piece of engineering the Epic: AI-generated E2E test coverage for pull requests #4189 epic needs: given a PR's changed-file diffs, generate a complete Playwright E2E test file via LLM.buildE2eTestGenDiffText+buildE2eTestGenPromptare pure functions building the prompt (diff assembly capped at 20 files / 60,000 chars);parseE2eTestGenResponseis a pure, deliberately narrow validator (mirroring Boundary-safe test generation #1972's "false positives are worse than a narrow true-positive set" discipline) requiring both a recognizable Playwright test call and its own@playwright/testimport before trusting model output.runGittensoryE2eTestGenerationis the async orchestrator, mirroringai-slop.ts'srunGittensoryAiSlopAdvisoryshape exactly: the sameAI_SUMMARIES_ENABLED/AI_PUBLIC_COMMENTS_ENABLEDgates, the same shared daily-neuron-budget accounting (sumAiEstimatedNeuronsSince— the SAME counter every other AI-generated artifact draws from), the same BYOK-vs-default model selection and per-repo/day BYOK cap, and the samerecord()audit-event shape.defangReviewInput, gated on thesafetyconverged feature) rather than inventing a second one — whensafetyis on for a repo, the diff/title/body are defanged before they ever reach the model.isE2eTestGenerationEnabled(thee2eTestskill-switch from feat(review): register e2eTests as the sixth converged-feature key #4190/PR feat(review): register e2eTests as the sixth converged-feature key #4206) as a fourth, defense-in-depth layer, matchinggrounding-wire.ts's own internal re-check pattern.Closes #4191.
Scope
type(scope): short summaryConventional Commit format.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Validation
git diff --checknpm run actionlint— N/A, no workflow files touched (ran anyway, clean).npm run typechecknpm run test:coveragelocally (unsharded, full repo) — 100% statements/branches/functions/lines on the new file, confirmed both via the isolated coverage run and the rawcoverage/lcov.infodata (LF:67 LH:67,BRF:70 BRH:70) after merging with the full suite. All 12,140 existing tests still pass (2 pre-existing skips, PG-integration-gated).npm run test:workers— N/A, no Worker-entry code touched; not re-run separately this PR since the fulltest:cigate already validated the same tree for PR feat(review): register e2eTests as the sixth converged-feature key #4206 earlier this session and nothing here touches the Worker bundle boundary.npm run build:mcp/npm run test:mcp-pack— N/A, no MCP package surface touched.npm run ui:openapi:check— N/A, no API/OpenAPI surface touched.npm run ui:lint/npm run ui:typecheck/npm run ui:build— N/A, noapps/gittensory-ui/**files touched.npm audit --audit-level=moderate— N/A, no dependency changes.npm run docs:drift-check(clean — no new env vars introduced; this module only reuses existing sharedAI_*config already documented) and confirmedgit statusshows only the two new files (no accidental edits elsewhere).Safety
UI Evidencesection — N/A, no UI changes.instructionsfield this module accepts is wired to real config in a later PR, feat(config): repo/path-scoped test-coverage instructions for e2eTests generation #4200);CHANGELOG.mditself is untouched.Notes
runGittensoryE2eTestGenerationyet. It's gated behindGITTENSORY_REVIEW_E2E_TESTS(already default"false") as well, so there is no reachable path to it in production even once a future PR does wire it in, until that flag is explicitly set.instructionsinput field is accepted now but nothing populates it yet — feat(config): repo/path-scoped test-coverage instructions for e2eTests generation #4200 (repo/path-scoped test-coverage instructions) will wire real config into it without needing to touch this file's signature again.