Skip to content

feat(review): LLM core to turn a PR diff into Playwright E2E test source - #4207

Merged
JSONbored merged 1 commit into
mainfrom
feat/e2e-test-gen-llm-core
Jul 8, 2026
Merged

feat(review): LLM core to turn a PR diff into Playwright E2E test source#4207
JSONbored merged 1 commit into
mainfrom
feat/e2e-test-gen-llm-core

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Adds src/services/ai-e2e-test-gen.ts, the one genuinely new piece of engineering the Epic: AI-generated E2E test coverage for pull requests #4189 epic needs: given a PR's changed-file diffs, generate a complete Playwright E2E test file via LLM.
  • buildE2eTestGenDiffText + buildE2eTestGenPrompt are pure functions building the prompt (diff assembly capped at 20 files / 60,000 chars); parseE2eTestGenResponse is a pure, deliberately narrow validator (mirroring Boundary-safe test generation #1972's "false positives are worse than a narrow true-positive set" discipline) requiring both a recognizable Playwright test call and its own @playwright/test import before trusting model output.
  • runGittensoryE2eTestGeneration is the async orchestrator, mirroring ai-slop.ts's runGittensoryAiSlopAdvisory shape exactly: the same AI_SUMMARIES_ENABLED/AI_PUBLIC_COMMENTS_ENABLED gates, the same shared daily-neuron-budget accounting (sumAiEstimatedNeuronsSince — the SAME counter every other AI-generated artifact draws from), the same BYOK-vs-default model selection and per-repo/day BYOK cap, and the same record() audit-event shape.
  • Reuses the existing prompt-injection defense (defangReviewInput, gated on the safety converged feature) rather than inventing a second one — when safety is on for a repo, the diff/title/body are defanged before they ever reach the model.
  • Gated behind isE2eTestGenerationEnabled (the e2eTests kill-switch from feat(review): register e2eTests as the sixth converged-feature key #4190/PR feat(review): register e2eTests as the sixth converged-feature key #4206) as a fourth, defense-in-depth layer, matching grounding-wire.ts's own internal re-check pattern.

Closes #4191.

Scope

Validation

  • git diff --check
  • npm run actionlint — N/A, no workflow files touched (ran anyway, clean).
  • npm run typecheck
  • npm run test:coverage locally (unsharded, full repo) — 100% statements/branches/functions/lines on the new file, confirmed both via the isolated coverage run and the raw coverage/lcov.info data (LF:67 LH:67, BRF:70 BRH:70) after merging with the full suite. All 12,140 existing tests still pass (2 pre-existing skips, PG-integration-gated).
  • npm run test:workers — N/A, no Worker-entry code touched; not re-run separately this PR since the full test:ci gate already validated the same tree for PR feat(review): register e2eTests as the sixth converged-feature key #4206 earlier this session and nothing here touches the Worker bundle boundary.
  • npm run build:mcp / npm run test:mcp-pack — N/A, no MCP package surface touched.
  • npm run ui:openapi:check — N/A, no API/OpenAPI surface touched.
  • npm run ui:lint / npm run ui:typecheck / npm run ui:build — N/A, no apps/gittensory-ui/** files touched.
  • npm audit --audit-level=moderate — N/A, no dependency changes.
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries — 33 tests covering every gate (kill-switch off, both generic AI flags off, no-provider unavailable), both budget checks (shared neuron budget, BYOK daily cap), both call paths (BYOK success/malformed, free/default success/exhausted-retries/throwing), the AI Gateway passthrough, actor-null recording, and both sides of the safety-defanging branch (on vs. off).
  • Also ran npm run docs:drift-check (clean — no new env vars introduced; this module only reuses existing shared AI_* config already documented) and confirmed git status shows only the two new files (no accidental edits elsewhere).

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise — N/A directly (this module produces test code, not a public comment; the render/sanitization layer that decides what's public-safe is a separate, later PR in the epic), but the system prompt explicitly forbids the model from mentioning rewards/rankings/payouts/wallets/hotkeys/coldkeys/trust-scores, matching every other AI-generated artifact's system prompt in this codebase.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. — N/A, no auth/session/CORS surface touched.
  • API/OpenAPI/MCP behavior is updated and tested where needed — N/A, no new API/OpenAPI/MCP surface.
  • UI changes use live API data or real empty/error/loading states — N/A, no UI changes.
  • Visible UI changes include a UI Evidence section — N/A, no UI changes.
  • Public docs/changelogs are updated where needed — N/A, no new config surface to document (the instructions field this module accepts is wired to real config in a later PR, feat(config): repo/path-scoped test-coverage instructions for e2eTests generation #4200); CHANGELOG.md itself is untouched.

Notes

  • This PR ships dead code from the deployment's perspective: nothing calls runGittensoryE2eTestGeneration yet. It's gated behind GITTENSORY_REVIEW_E2E_TESTS (already default "false") as well, so there is no reachable path to it in production even once a future PR does wire it in, until that flag is explicitly set.
  • The instructions input field is accepted now but nothing populates it yet — feat(config): repo/path-scoped test-coverage instructions for e2eTests generation #4200 (repo/path-scoped test-coverage instructions) will wire real config into it without needing to touch this file's signature again.

Adds the one genuinely new piece of engineering the #4189 epic needs: a pure
prompt builder + response parser, plus an async orchestrator that generates a
complete Playwright test file from a PR's changed-file diffs.

Mirrors ai-slop.ts's established shape exactly: BYOK-vs-default model selection
(the maintainer's own frontier model when configured, else the free/default
reviewer with bounded retry/fallback), the shared daily neuron budget +
per-repo/day BYOK cap, and safety-feature-gated prompt-injection defanging
(reusing defangReviewInput, not a second implementation). Parsed output is
validated against a Playwright-shaped signature (a recognizable test call plus
its own @playwright/test import) before being trusted -- malformed or
off-topic model output is dropped, never surfaced.

Fully fail-safe (disabled/unavailable/quota-exceeded/unparseable all degrade to
a non-throwing result) and gated behind the e2eTests kill-switch from #4190, so
it has zero effect until a future PR in the epic actually calls it.

Part of #4189. Progresses #4191.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.75%. Comparing base (0cb6854) to head (f9ba70a).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4207      +/-   ##
==========================================
+ Coverage   93.74%   93.75%   +0.01%     
==========================================
  Files         388      389       +1     
  Lines       36421    36495      +74     
  Branches    13342    13373      +31     
==========================================
+ Hits        34143    34217      +74     
  Misses       1621     1621              
  Partials      657      657              
Files with missing lines Coverage Δ
src/services/ai-e2e-test-gen.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 8, 2026
@JSONbored
JSONbored merged commit 8048163 into main Jul 8, 2026
10 checks passed
@JSONbored
JSONbored deleted the feat/e2e-test-gen-llm-core branch July 8, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(review): LLM core to turn a PR diff into Playwright E2E test source

1 participant