Skip to content

docs(self-host): add disk capacity planning and backup retention policy - #3289

Merged
JSONbored merged 2 commits into
mainfrom
docs/selfhost-capacity-and-backup-retention
Jul 5, 2026
Merged

docs(self-host): add disk capacity planning and backup retention policy#3289
JSONbored merged 2 commits into
mainfrom
docs/selfhost-capacity-and-backup-retention

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • docs.self-hosting-operations.tsx's Disk section had real measured numbers from one production instance but nothing to help an operator answer "how much disk do I need for N repos at M PRs/month." Added a capacity-planning subsection grounded in the actual growth/retention behavior of the disk-consuming subsystems: review_audit (one row per finalized gate decision/outcome, no retention policy, unbounded) and webhook_events (one row per inbound webhook delivery, also unbounded) grow with activity; audit_events is capped at 90 days via RETENTION_POLICY in src/db/retention.ts and the daily 03:00 UTC prune-retention job in src/index.ts; backup-volume usage scales as live DB size × BACKUP_RETAIN. Estimates are labeled as estimates; only the retention/pruning facts are stated as verified.
  • docs.self-hosting-backup-scaling.tsx never documented backup retention. Added a "Retention: how many backups are kept" section describing BACKUP_RETAIN (default 7, kept independently per target — postgres/sqlite/qdrant), normalize_backup_retain's fallback-to-7 and floor-of-1 guards, and a callout on the exact failure-path behavior in scripts/backup.sh: a failed SQLite backup is deleted, logged, and skips only the sqlite retention prune (Postgres/Qdrant still prune normally that run) so a known-good backup is never evicted by a broken one.

Advances #1819.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked an issue, or this is small enough that the summary explains why an issue is not needed. (Advances the roadmap(selfhost): production hardening and first release readiness #1819 self-host production-readiness roadmap; documentation-depth work, not a specific tracked bug/feature.)

Validation

  • git diff --check
  • npm run actionlint (not run — no workflow files touched)
  • npm run ui:typecheck
  • npm run test:coverage (not run — docs-only change, no src/** lines touched, no Codecov obligation)
  • npm run test:workers (not run — no worker code touched)
  • npm run build:mcp (not run — no MCP code touched)
  • npm run test:mcp-pack (not run — no MCP code touched)
  • npm run ui:openapi:check (not run — no API/schema changes)
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm run docs:drift-check
  • npm audit --audit-level=moderate (not run — no dependency changes)
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries (not applicable — no src/** code changed, prose-only docs routes)

If any required check was skipped, explain why:

  • This is a docs-only change confined to two apps/gittensory-ui/src/routes/docs.*.tsx files (prose, FeatureRow, Callout). No src/**, workflow, MCP, or dependency files changed, so the code-path checks (coverage, workers, MCP pack, OpenAPI, audit, actionlint) don't apply. Ran the full docs-relevant gate instead: ui:lint, ui:typecheck, ui:build, docs:drift-check — all green.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no such changes.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — no such changes.)
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — static docs prose only.)
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository. (No visual/layout change — new prose sections reuse existing FeatureRow/Callout components already styled elsewhere on these same pages; see Notes.)
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

Not applicable: this PR adds prose content (paragraphs, a FeatureRow, and a Callout) to two existing docs pages using components already rendered elsewhere on those same pages. There is no new layout, styling, or visual component to screenshot — the rendered result is indistinguishable in kind from the surrounding sections already on the page.

Notes

  • Both additions extend existing well-scoped sections rather than restructuring them: the Disk capacity-planning content is appended after "When a compose default might need to change" and before "Docker resource hygiene" in docs.self-hosting-operations.tsx; the backup retention content is appended after "Scheduled backups" and before "Multi-instance: Postgres and Redis" in docs.self-hosting-backup-scaling.tsx.
  • Verified against source, not restated from comments: src/db/retention.ts (RETENTION_POLICY, the 90-day audit_events rule, the daily prune job), src/index.ts (03:00 UTC prune-retention schedule), src/review/outcomes-wire.ts / src/review/parity-wire.ts (review_audit write sites), src/db/schema.ts (webhook_events, audit_events shape), and scripts/backup.sh (normalize_backup_retain, the per-target retention loop, and the exact SQLite-failure skip-prune branch).

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 5, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 2cba784 Commit Preview URL

Branch Preview URL
Jul 05 2026, 01:14 AM

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 5, 2026
@loopover-orb

loopover-orb Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-05 01:13:12 UTC

2 files · 1 AI reviewer · 1 blocker · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • Maintainer test expectations unmet — Add or update tests, or attach passing validation output that satisfies the maintainer's test expectations.

Review summary
This PR adds self-hosting docs for disk capacity planning and backup retention, and the visible backup-retention documentation matches the provided `scripts/backup.sh` behavior: `BACKUP_RETAIN` defaults to 7, is normalized to at least 1, retention is per target directory, and failed SQLite verification skips only SQLite pruning before exiting non-zero. The operations-page additions are documentation-only and clearly label sizing as estimates while tying the unbounded-table claims to retention behavior. I do not see a reachable code or documentation-contract defect in the provided diff.

Nits — 6 non-blocking
  • nit: `apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:337` says operators need to reason about tables and volumes but does not show a concrete sizing formula for the headline's `N repos at M PRs/month`, so the heading promises slightly more calculator-like guidance than the section delivers.
  • nit: `apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:350` says `review_audit` is roughly 2 rows per PR; if the gate can re-finalize or record multiple outcomes for unusual PR lifecycle paths, soften that claim or point to the exact producer that enforces it.
  • nit: `apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:358` gives `5-15x` as a common webhook multiplier without showing how it was measured, so I would either cite this as an observed range from your instance or phrase it as an operator example rather than a general expectation.
  • In `apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:337`, add a small worked estimate such as `monthly review_audit rows ~= 2 * PRs/month` and `monthly webhook_events rows ~= deliveries/month`, then tell operators to replace the multiplier with their measured delivery rate.
  • In `apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:350`, either link the row-count statement to the existing code path that writes `review_audit`, or change the wording to avoid implying an invariant that this docs page itself does not prove.
  • Code changes lack test evidence — Add focused regression tests or explain why existing coverage is sufficient.

Concerns raised — review before merging

  • Maintainer test expectations unmet — Add or update tests, or attach passing validation output that satisfies the maintainer's test expectations.
Signal Result Evidence
Code review ❌ 1 blocker 1 reviewer
Linked issue ✅ No-issue rationale PR body explains why no issue is linked.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 56 registered-repo PR(s), 46 merged, 423 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 56 PR(s), 423 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 56 PR(s), 423 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • No action.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

JSONbored added 2 commits July 4, 2026 18:12
The Disk section had real measured numbers from one production
instance but no way to extrapolate to a larger install, and the
backup-scaling doc never explained how many historical backups the
`backup` profile keeps before pruning. Ground both additions in the
actual growth/retention logic: review_audit and webhook_events have
no retention policy and grow unbounded per PR/webhook, audit_events
is capped at 90 days, and backup.sh keeps BACKUP_RETAIN (default 7)
newest copies per target independently, skipping the sqlite prune
entirely after a failed backup so a known-good copy is never lost.
The SQLite-failure callout only named the integrity_check failure
mode, missing the .backup-command-itself-fails and empty-output-file
cases the script also treats as failed verification. The review_audit
size estimate mixed a per-row byte figure with an aggregate MB
estimate without a clear derivation; points operators at measuring
their own instance instead of trusting a blanket ratio.
@JSONbored
JSONbored force-pushed the docs/selfhost-capacity-and-backup-retention branch from 5794603 to 2cba784 Compare July 5, 2026 01:12
@JSONbored
JSONbored merged commit ba1bb6f into main Jul 5, 2026
10 checks passed
@JSONbored
JSONbored deleted the docs/selfhost-capacity-and-backup-retention branch July 5, 2026 01:37
JSONbored added a commit that referenced this pull request Jul 5, 2026
…cy (#3289)

* docs(self-host): add disk capacity planning and backup retention policy

The Disk section had real measured numbers from one production
instance but no way to extrapolate to a larger install, and the
backup-scaling doc never explained how many historical backups the
`backup` profile keeps before pruning. Ground both additions in the
actual growth/retention logic: review_audit and webhook_events have
no retention policy and grow unbounded per PR/webhook, audit_events
is capped at 90 days, and backup.sh keeps BACKUP_RETAIN (default 7)
newest copies per target independently, skipping the sqlite prune
entirely after a failed backup so a known-good copy is never lost.

* fix(docs): tighten backup-retention and capacity-planning wording

The SQLite-failure callout only named the integrity_check failure
mode, missing the .backup-command-itself-fails and empty-output-file
cases the script also treats as failed verification. The review_audit
size estimate mixed a per-row byte figure with an aggregate MB
estimate without a clear derivation; points operators at measuring
their own instance instead of trusting a blanket ratio.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Development

Successfully merging this pull request may close these issues.

1 participant