Skip to content

feat(selfhost): support an S3-compatible bucket for visual-capture storage - #4203

Merged
JSONbored merged 2 commits into
mainfrom
selfhost-visual-s3-bucket-storage
Jul 8, 2026
Merged

feat(selfhost): support an S3-compatible bucket for visual-capture storage#4203
JSONbored merged 2 commits into
mainfrom
selfhost-visual-s3-bucket-storage

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Advances fix(review): visual-capture screenshots render as broken images on self-host (private origin) #4184: PR test(ui): add AGPL-3.0 badge to footer to validate visual-capture pipeline #4180's "Visual preview" table rendered broken images because this self-host instance's PUBLIC_API_ORIGIN is a Tailscale-private hostname, unreachable by GitHub's own servers. fix(review): warn at boot when PUBLIC_API_ORIGIN/PUBLIC_SITE_ORIGIN look private #4185 (merged) added a boot-time advisory for this; its own "Remaining" section frames actually fixing reachability as an ops change outside this repo. This PR adds a genuine in-repo alternative: an operator's own publicly-readable S3-compatible bucket for screenshot storage, so PUBLIC_API_ORIGIN itself never needs to be public at all.
  • Related epic: Epic: extend automated visual-capture convergence (#577) to self-host + metagraphed #3607 (visual-capture convergence to self-host + metagraphed).
  • REVIEW_AUDIT (the visual-capture screenshot store) previously had one self-host backend: a local filesystem directory (REVIEW_AUDIT_DIR), always served back out through this instance's own /gittensory/shot proxy route.
  • Adds REVIEW_AUDIT_S3_BUCKET/_ENDPOINT/_ACCESS_KEY_ID/_SECRET_ACCESS_KEY/_REGION: an alternative backend (src/selfhost/s3-blob-store.ts, using aws4fetch to sign S3-compatible REST calls) storing screenshots in an operator's own bucket — their own Cloudflare R2 bucket, or any other S3-compatible provider. Takes priority over REVIEW_AUDIT_DIR when both are configured.
  • REVIEW_AUDIT_S3_PUBLIC_URL additionally makes served links point directly at the bucket's own public URL instead of proxying through /gittensory/shot — decoupling "does my review pipeline run on my own infrastructure" from "are the resulting public-facing images reachable by anyone." Unset (default) keeps today's proxy-based serving unchanged.
  • A bounded retry budget (3 attempts, not aws4fetch's resilience-first default of 10) keeps a persistently-misconfigured bucket from stalling the review pipeline for 30+ seconds on every best-effort read/write — caught this empirically in my own tests (a naive 500-response test took 30s to fail before I bounded it).
  • Also adds delete() to the filesystem-backed store for interface parity — actions-fallback.ts's dispatch-marker cleanup already calls .delete() and previously depended on that call silently throwing-and-being-caught on self-host, since the fs store never implemented it.

Scope

Validation

  • git diff --check
  • npm run test:ci (full local gate, green)
  • npm audit --audit-level=moderate (0 vulnerabilities)
  • npm run test:coverage — new/changed src/** lines covered, including both branches of every new conditional (S3-vs-fs backend priority, S3-public-url-vs-proxy URL construction, both configured-vs-unset combinations across all 4 call sites that construct a shot URL)
  • npm run selfhost:env-reference — regenerated, the 5 new process.env reads picked up
  • npm run ui:openapi — confirmed no drift from this change (ran last, after format, per this repo's own known ordering gotcha)

Safety

  • No secrets, wallet details, hotkeys, coldkeys, PATs, private keys, trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized and low-noise.
  • Auth/cookie/CORS/GitHub App/Cloudflare/session changes include negative-path tests. (N/A — no such changes; the new S3 client tests do cover the auth-header/region-signing behavior itself, including failure paths for every HTTP status class.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — no such surface touched.)
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. (Docs updated: .env.example, the self-hosting configuration docs page, generated env reference.)

Notes

  • This is a self-host-only feature (src/env.d.ts/server.ts/selfhost/**) — no hosted-mode behavior changes.
  • Follow-up (tracked separately, not part of this PR): actually provisioning a real R2 bucket + setting these env vars on edge-us-01, so its own live screenshots start rendering publicly.

…orage

REVIEW_AUDIT (the visual-capture screenshot store) previously had only one
self-host backend: a local filesystem directory (REVIEW_AUDIT_DIR), served
back out through this instance's own /gittensory/shot proxy route. That
means the resulting <img> URLs embedded in a public GitHub PR comment are
only reachable through this instance's own PUBLIC_API_ORIGIN -- if an
operator keeps their instance behind a private network (a VPN, a firewall,
no public DNS), GitHub's own servers can never fetch them, and every
screenshot renders as a broken image for every viewer.

Adds REVIEW_AUDIT_S3_BUCKET/_ENDPOINT/_ACCESS_KEY_ID/_SECRET_ACCESS_KEY/
_REGION: an alternative backend (src/selfhost/s3-blob-store.ts, using
aws4fetch to sign S3-compatible REST calls) that stores screenshots in an
operator's own bucket -- their own Cloudflare R2 bucket, or any other
S3-compatible provider -- instead of the local filesystem. Takes priority
over REVIEW_AUDIT_DIR when both are configured. A bounded retry budget (3
attempts, not aws4fetch's resilience-first default of 10) keeps a
persistently-misconfigured bucket from stalling the review pipeline for
30+ seconds on every best-effort read/write.

REVIEW_AUDIT_S3_PUBLIC_URL additionally makes served links point DIRECTLY
at the bucket's own public URL (an R2 public r2.dev bucket URL, or a
custom domain) instead of proxying through /gittensory/shot -- genuinely
decoupling "does my review pipeline run on my own infrastructure" from
"are the resulting public-facing images reachable by anyone." Unset (the
default) keeps today's proxy-based serving unchanged.

Also adds a delete() method to the filesystem-backed store (blob-store.ts)
for interface parity with the new S3 one -- actions-fallback.ts's dispatch-
marker cleanup already calls .delete() and previously depended on that
call silently throwing-and-being-caught on self-host, since the fs store
never implemented it.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui f889fe7 Commit Preview URL

Branch Preview URL
Jul 08 2026, 11:20 AM

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.75%. Comparing base (acc37e0) to head (5bf952e).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4203   +/-   ##
=======================================
  Coverage   93.74%   93.75%           
=======================================
  Files         387      388    +1     
  Lines       36420    36443   +23     
  Branches    13341    13348    +7     
=======================================
+ Hits        34142    34167   +25     
  Misses       1621     1621           
+ Partials      657      655    -2     
Files with missing lines Coverage Δ
src/review/visual/capture.ts 92.76% <100.00%> (+1.37%) ⬆️
src/selfhost/blob-store.ts 100.00% <100.00%> (ø)
src/selfhost/s3-blob-store.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 8, 2026
@loopover-orb

loopover-orb Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-08 17:15:36 UTC

13 files · 2 blockers · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Possible leaked secret in the diff (generic_secret_assignment) — Remove the secret from the diff, rotate the exposed credential, then re-run the gate.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Possible leaked secret in the diff (generic_secret_assignment) — Remove the secret from the diff, rotate the exposed credential, then re-run the gate.
Signal Result Evidence
Code review ❌ 2 blockers No AI review summary
Linked issue ✅ No-issue rationale PR body explains why no issue is linked.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 52 registered-repo PR(s), 43 merged, 518 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 52 PR(s), 518 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
  • Official Gittensor activity: 52 PR(s), 518 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
Visual preview
Route Viewport Before (production) After (this PR's preview) Diff
/docs/self-hosting-configuration desktop after /docs/self-hosting-configuration
/docs/self-hosting-configuration mobile after /docs/self-hosting-configuration (mobile)

Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 8, 2026
…kAfterShot

resolveShotUrl(env, key) || placeholder was only ever exercised with a
truthy resolveShotUrl result (PUBLIC_API_ORIGIN or REVIEW_AUDIT_S3_PUBLIC_URL
set), leaving the falsy/placeholder side of the fallback branch untested.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. manual-review Gittensor contributor context

Development

Successfully merging this pull request may close these issues.

1 participant