Skip to content

feat(review): vision-verify screenshot-table PRs with the local VLM #4366

Description

@JSONbored

Part of #4325.

Context

src/review/screenshot-table-gate.ts (:91-143) only checks the MARKDOWN STRUCTURE of a PR's screenshot table (does a table exist, does each row contain ![]()/<img> markup) — it never looks at the actual image content. A contributor can satisfy this gate by pasting two identical images, an unrelated screenshot, or a screenshot with no visible before/after difference at all, and the gate has no way to catch it. This is a real gap in a check the team already treats as load-bearing for frontend PRs (see the "frontend PR screenshots non-negotiable" policy).

The self-host GPU now runs a vision-language model (qwen3-vl:8b-instruct, wired via env.AI_VISION, #4335) that's idle capacity outside of confirmed-visual-regression calls. The existing visual pipeline (src/review/visual/capture.ts, src/review/visual/pixel-diff.ts) already fetches and stores the before/after images this gate would need — no new capture infrastructure required.

Requirements

  • Add an ADVISORY-only vision check (mirror visual-findings.ts's existing pattern: pure decision logic separate from the live caller, never a gate blocker on its own) that verifies a PR's screenshot-table images are: (a) not pixel-identical to each other, (b) plausibly relevant to the stated change (not an obviously unrelated image).
  • Reuse env.AI_VISION (the binding Stretch: evaluate spare GPU headroom for a local vision-language model in visual-review #4335 already built) — do not build a second vision binding.
  • Must degrade gracefully exactly like the existing visual-vision advisory: no AI_VISION configured, a fetch failure, or an unparseable response all mean "no finding," never a thrown error or a false gate block.
  • Config-gated (global + per-repo, matching every other gate/settings field's precedence) — off by default until validated on real PRs.

Deliverables

  • New advisory check wired into the screenshot-table gate path, config-gated
  • Unit tests for the pure decision logic (mirroring visual-findings.test.ts's coverage bar) and the live caller wiring (mirroring visual-vision-wiring.test.ts)
  • Manual validation against a handful of real past PRs (both genuine and gamed screenshot tables) with results recorded in the PR description

Expected outcome

The screenshot-table gate can catch identical/irrelevant screenshot submissions it currently cannot see at all, using GPU capacity that's already idle between confirmed-regression vision calls.

Effort

M

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions