Skip to content

perf(review): downscale before/after screenshots before the local VLM - #4687

Merged
JSONbored merged 1 commit into
mainfrom
perf/vision-image-downscale-sharp
Jul 10, 2026
Merged

perf(review): downscale before/after screenshots before the local VLM#4687
JSONbored merged 1 commit into
mainfrom
perf/vision-image-downscale-sharp

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Vision-vision screenshots (shot.ts) are captured fullPage: true, so image HEIGHT scales with a page's full scrollable content even at a fixed 1440px viewport width — Qwen's dynamic-resolution vision encoder tokenizes proportional to pixel count, inflating vision prefill cost/latency on long pages.
  • Adds a downscale-to-1280px-longest-edge step (aspect ratio preserved, never upscales) scoped only to the bytes sent to the local VLM (fetchShotContentBlock in src/review/visual/capture.ts) — the stored/displayed screenshot (the same URL embedded verbatim in the PR comment) is completely untouched.
  • Uses the same Worker-safe-stub / self-host-real-implementation seam already established for pixel-diff.ts and scroll-gif.ts: sharp is a native binding that can't run on the Cloudflare Workers runtime, so capture.ts (Worker-reachable) imports only a no-op default (src/review/visual/image-downscale.ts); the real resize (src/selfhost/stubs/image-downscale.ts) is swapped in by scripts/build-selfhost.mjs's esbuild plugin only when bundling the self-host entry. sharp is added to test/unit/worker-entry-boundary.test.ts's forbidden-identifier list so a future accidental direct import trips CI.
  • sharp is marked external in the --all esbuild bundle (native binary, can't be inlined) and installed separately into the runtime Docker image (mirroring the existing puppeteer-core/INSTALL_VISUAL_REVIEW pattern, but unconditional — this is a core dependency, not an opt-in sidecar feature).

Closes #4370

A real bug caught by empirical verification, not by review

sharp initially went into package.json's devDependencies (mirroring pixelmatch/pngjs, which are also self-host-only). That's wrong for sharp specifically: unlike pixelmatch/pngjs (pure JS, fully bundled into dist/server.mjs, never touch the runtime image's node_modules), sharp has a native binary that must be installed as a real runtime dependency in the final image. With sharp under devDependencies, the Dockerfile's RUN npm install sharp@0.34.5 --ignore-scripts step silently no-opedNODE_ENV=production (set earlier in runtime-base) makes npm skip an explicitly-named package that's classified as a devDependency, even though it reports success ("up to date, audited N packages"). The resulting image had no node_modules/sharp at all; a real vision call would have crashed on ERR_MODULE_NOT_FOUND at the first invocation — no test or typecheck would have caught this, since local npm install and vitest don't run under NODE_ENV=production.

Caught by doing exactly what the issue and #4354 both required: a real local docker build + boot, on the target host, followed by an explicit runtime check that sharp genuinely resolves and resizes an image inside the built container (not just that npm install printed success). Fixed by moving sharp to regular dependencies. Full before/after verification trail:

  1. First build (sharp in devDependencies): image built successfully, npm install sharp@0.34.5 --ignore-scripts reported "up to date, audited 374 packages... found 0 vulnerabilities" — but node_modules/sharp was absent from the final image; a real resize script failed with ERR_MODULE_NOT_FOUND.
  2. Moved sharp to dependencies, rebuilt: node_modules/sharp present, npm ls sharp confirms sharp@0.34.5, a real resize script inside the container correctly downscaled a 1600×400 image to 1280×320.
  3. Booted the actual dist/server.mjs (not a synthetic script) against real Postgres/Redis on the box's compose network: clean startup, /ready returns {"ok":true,"checks":{"db":true,"migrations":true,"redis":true}} — no module-resolution errors anywhere in the boot path.

All verification ran in an isolated scratch clone + throwaway image tag on edge-nl-01, never touching the live production container; scratch artifacts were removed afterward.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries
  • Real local docker build + boot verification on the target architecture (the issue's own explicit requirement, mirroring fix(selfhost): stop .dockerignore from breaking the engine's own build #4354's diagnosis) — see the "real bug" section above.

If any required check was skipped, explain why:

  • No UI/MCP/OpenAPI/wrangler-binding surface touched (actionlint/test:workers/build:mcp/test:mcp-pack/ui:* not applicable). test:coverage was run targeted on the exact touched test files (image-downscale.test.ts, selfhost-image-downscale-stub.test.ts, worker-entry-boundary.test.ts, visual-capture.test.ts, visual-vision-wiring.test.ts — 133 tests, all green) rather than the full unsharded suite; both new files individually confirmed at ~100% statement coverage via the raw v8 JSON report (the CLI text table silently truncates past a few hundred files in this repo's size, a display quirk unrelated to actual coverage).

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth/session surface touched)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — internal vision-call image path only, no external API surface)
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — no UI changes)
  • Visible UI changes include a UI Evidence section below. (N/A — no visible UI change; stored/displayed screenshots are explicitly untouched)
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. (N/A)

UI Evidence

N/A — no UI/frontend/docs-visible change. The stored/displayed screenshot bytes are explicitly unaffected by this change (only the bytes sent to the vision model are resized).

Notes

  • Latency numbers for a real tall-page vision call weren't captured as a separate before/after benchmark in this PR — the empirical focus went into the build/boot correctness verification above, which surfaced a real shipping-blocking bug. The resize math itself is deterministic and directly bounds worst-case vision-call image size regardless of page length, which is the mechanism the latency win depends on.

…#4370)

Vision-vision screenshots are captured fullPage, so image HEIGHT scales
with a page's full scrollable content even at a fixed 1440px viewport
width -- Qwen's dynamic-resolution encoder tokenizes proportional to
pixel count, inflating vision prefill cost/latency on long pages.

Adds a downscale-to-1280px-longest-edge step scoped ONLY to the bytes
sent to the vision model (fetchShotContentBlock), via the same
Worker-safe-stub / self-host-real-implementation seam already used for
pixel-diff.ts and scroll-gif.ts -- sharp is a native binding and can't
run on the Cloudflare Workers runtime, so capture.ts (Worker-reachable)
imports only a no-op default; the real resize is swapped in by
build-selfhost.mjs's esbuild plugin for the self-host bundle only.
Stored/displayed screenshots (the same URL embedded in the PR comment)
are completely untouched.

Verified with a real local docker build + boot on the target
architecture: sharp must be a genuine `dependencies` entry (not
devDependencies) since NODE_ENV=production in the runtime image
silently skips an explicit `npm install` of a devDependency-classified
package -- caught only by actually building and booting the image, not
by static review, mirroring how #4354 was diagnosed.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored
JSONbored merged commit c3b10f2 into main Jul 10, 2026
8 checks passed
@JSONbored
JSONbored deleted the perf/vision-image-downscale-sharp branch July 10, 2026 12:20
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.14%. Comparing base (a2e5d3e) to head (0ac69ef).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4687   +/-   ##
=======================================
  Coverage   94.14%   94.14%           
=======================================
  Files         436      437    +1     
  Lines       38530    38531    +1     
  Branches    14049    14049           
=======================================
+ Hits        36273    36274    +1     
  Misses       1599     1599           
  Partials      658      658           
Files with missing lines Coverage Δ
src/review/visual/capture.ts 94.77% <100.00%> (ø)
src/review/visual/image-downscale.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

perf(review): downscale before/after screenshots before sending to the local VLM

1 participant