Skip to content

chore(selfhost): automate Docker disk hygiene for the self-host stack - #2576

Merged
JSONbored merged 1 commit into
mainfrom
chore/selfhost-docker-disk-hygiene
Jul 2, 2026
Merged

chore(selfhost): automate Docker disk hygiene for the self-host stack#2576
JSONbored merged 1 commit into
mainfrom
chore/selfhost-docker-disk-hygiene

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Closes #2535.

Live production observation showed disk usage climbing to a majority-full state, with unused Docker images and build cache as the dominant reclaimable consumers, and no automated mitigation anywhere in the repo — no log rotation config on any of the 22 compose services, and no scheduled prune of Docker resources. An unaddressed disk-exhaustion path is a hard outage for a 24/7 stack: Postgres writes fail, log writes fail, and deploys fail once the volume fills.

What changed

  • Added a shared x-logging anchor (json-file, 10MB × 3 files) merged into every service in docker-compose.yml via <<: *default-logging, so none of the 22 services default to Docker's unbounded log driver.
  • Added scripts/selfhost-docker-prune.sh: age-filtered (7-day default) docker image prune + docker builder prune, paired with host-level systemd timer/service .example units (following this repo's existing .env.example/docker-compose.override.yml.example templating convention).
  • The prune mechanism runs on the host, not as a compose service, and does not touch or widen the existing docker-proxy service (which is deliberately read-only, POST: "0", for exactly this kind of blast-radius reason) or grant any container /var/run/docker.sock access — reclaiming images/build-cache needs real Docker daemon access, which this repo intentionally does not grant to any container (matching the documented security posture already explained in the docker-proxy and runner service comments).
  • Documented both as a required self-hosting-operations step, right after the existing Alerting section.

Correctness/safety notes

  • The logging cap is additive/config-only — no behavior change beyond bounding log file size.
  • The prune script only ever calls docker image prune/docker builder prune with an until=<hours> age filter — never a blind full wipe — so nothing built or pulled within the retention window (default 7 days) is ever a candidate, giving a safety buffer for rollback.

Validation

  • docker compose config with every optional profile active resolves a bounded logging config on all 22 services — verified programmatically (parsed the resolved config and asserted driver/max-size/max-file on every service).
  • test/unit/selfhost-compose-logging.test.ts — pure YAML structural check (with { merge: true } to resolve the <<: anchor the same way Docker Compose's own YAML 1.1 merge-key support does), no docker CLI invocation, matching the existing selfhost-compose-*.test.ts convention (the CI runner has no Docker-in-Docker access).
  • test/unit/selfhost-docker-prune-script.test.ts — actually executes the script with a stubbed docker binary on PATH (matching the existing selfhost-backup-metrics-script.test.ts convention), confirming both prune calls always carry an age filter (never a blind wipe) and the retention window is operator-tunable via GITTENSORY_DOCKER_PRUNE_RETAIN_HOURS.
  • Full local gate (npm run test:ci) green, npm audit --audit-level=moderate clean, git diff --check clean.
  • No src/** files touched, so no Codecov patch-coverage obligation for this change (config/docs/scripts only) — tests added anyway to match established repo convention for these file types.

Safety

  • No secrets or credentials introduced or exposed.
  • No hardcoded repo-specific paths — the systemd units ship as .example templates an operator copies and edits, matching .env.example's convention, since a real self-host deploy path can't be known in advance.

Live production observation showed disk usage climbing to a majority-full
state, with unused Docker images and build cache as the dominant reclaimable
consumers, and no automated mitigation anywhere in the repo -- no log
rotation config on any of the 22 compose services, and no scheduled prune of
Docker resources. An unaddressed disk-exhaustion path is a hard outage for a
24/7 stack: Postgres writes fail, log writes fail, and deploys fail once the
volume fills.

- Add a shared `x-logging` anchor (json-file, 10MB x 3 files) merged into
  every service in docker-compose.yml via `<<: *default-logging`, so none of
  them default to Docker's unbounded log driver.
- Add scripts/selfhost-docker-prune.sh: age-filtered (7-day default) `docker
  image prune` + `docker builder prune`, paired with host-level systemd
  timer/service .example units (following the repo's existing .env.example
  templating convention) -- this runs on the HOST, not as a compose service,
  because reclaiming images/build-cache needs real Docker daemon access,
  which this repo deliberately does not grant to any container (matching the
  existing docker-proxy/runner services' documented security posture).
- Document both as a required self-hosting-operations step.

Validation: docker compose config with every profile active resolves a
bounded logging config on all 22 services (verified programmatically); the
prune script is tested by execution with a stubbed `docker` binary on PATH,
confirming both prune calls always carry an age filter and the retention
window is operator-tunable via env var. Full local gate green; no src/**
files touched, so no Codecov patch-coverage obligation.
@dosubot dosubot Bot added the size:M label Jul 2, 2026
@loopover-orb

loopover-orb Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-02 10:56:49 UTC

7 files · 1 AI reviewer · no blockers · readiness 80/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • Touches a guarded path — held for manual review

Review summary
The change coherently addresses self-host disk growth by bounding Docker json-file logs on every compose service and adding a host-level, age-filtered prune path for images and build cache. The script uses Docker's own unused-resource pruning with an explicit retention filter, and the tests cover the default retention, override path, and compose logging resolution. I do not see a reachable correctness break in the visible diff; the main remaining gaps are operator hardening and documentation precision around what the prune job intentionally does and does not remove.

Nits — 7 non-blocking
  • nit: scripts/selfhost-docker-prune.sh:14 accepts any GITTENSORY_DOCKER_PRUNE_RETAIN_HOURS value, so an empty or non-numeric override turns into a Docker CLI error instead of a clear validation message.
  • nit: apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:128 says a recent deploy is never at risk, but the script only protects recently built or pulled unused images, not older rollback images that are no longer associated with a container.
  • scripts/selfhost-docker-prune.sh:14 should validate RETAIN_HOURS with a numeric positive-hours check before calling Docker so operator misconfiguration fails with a direct message.
  • apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:128 should soften the rollback claim to match Docker prune semantics, e.g. recent images are retained while older unused images/build cache may be removed.
  • systemd/gittensory-docker-prune.service.example:22 could add a short Environment= example for choosing a longer retention window on hosts that keep older rollback images locally.
  • PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
  • Touches a guarded path — held for manual review — A maintainer must review and merge this change.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #2535
Related work ⚠️ 2 scoped overlaps Top overlaps are listed below; lower-confidence bulk is hidden.
Change scope ❌ 8/20 High review scope from cached public metadata (size label size:M; 1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 65 registered-repo PR(s), 55 merged, 553 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 65 PR(s), 553 issue(s).
Gate result ⚠️ Not blocking Advisory; not blocking this PR.
Review context
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Review top overlaps.
  • Add a concise scope and risk note.
  • No action.
  • Check active issues and PRs before submitting.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 500dbcc Commit Preview URL

Branch Preview URL
Jul 02 2026, 09:36 AM

@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.98%. Comparing base (09aa0f1) to head (500dbcc).
⚠️ Report is 27 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2576      +/-   ##
==========================================
+ Coverage   95.96%   95.98%   +0.02%     
==========================================
  Files         226      229       +3     
  Lines       25470    25810     +340     
  Branches     9263     9389     +126     
==========================================
+ Hits        24442    24774     +332     
- Misses        417      425       +8     
  Partials      611      611              

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 81b14be into main Jul 2, 2026
15 checks passed
@JSONbored
JSONbored deleted the chore/selfhost-docker-disk-hygiene branch July 2, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Development

Successfully merging this pull request may close these issues.

chore(selfhost): automate Docker disk hygiene for the self-host stack

1 participant