chore(selfhost): automate Docker disk hygiene for the self-host stack - #2576
Conversation
Live production observation showed disk usage climbing to a majority-full state, with unused Docker images and build cache as the dominant reclaimable consumers, and no automated mitigation anywhere in the repo -- no log rotation config on any of the 22 compose services, and no scheduled prune of Docker resources. An unaddressed disk-exhaustion path is a hard outage for a 24/7 stack: Postgres writes fail, log writes fail, and deploys fail once the volume fills. - Add a shared `x-logging` anchor (json-file, 10MB x 3 files) merged into every service in docker-compose.yml via `<<: *default-logging`, so none of them default to Docker's unbounded log driver. - Add scripts/selfhost-docker-prune.sh: age-filtered (7-day default) `docker image prune` + `docker builder prune`, paired with host-level systemd timer/service .example units (following the repo's existing .env.example templating convention) -- this runs on the HOST, not as a compose service, because reclaiming images/build-cache needs real Docker daemon access, which this repo deliberately does not grant to any container (matching the existing docker-proxy/runner services' documented security posture). - Document both as a required self-hosting-operations step. Validation: docker compose config with every profile active resolves a bounded logging config on all 22 services (verified programmatically); the prune script is tested by execution with a stubbed `docker` binary on PATH, confirming both prune calls always carry an age filter and the retention window is operator-tunable via env var. Full local gate green; no src/** files touched, so no Codecov patch-coverage obligation.
|
Warning 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏸️ Gittensory review result - manual review recommendedReview updated: 2026-07-02 10:56:49 UTC
⏸️ Suggested Action - Manual Review
Review summary Nits — 7 non-blocking
Review context
Contributor next steps
Signal definitions
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
gittensory-ui | 500dbcc | Commit Preview URL Branch Preview URL |
Jul 02 2026, 09:36 AM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2576 +/- ##
==========================================
+ Coverage 95.96% 95.98% +0.02%
==========================================
Files 226 229 +3
Lines 25470 25810 +340
Branches 9263 9389 +126
==========================================
+ Hits 24442 24774 +332
- Misses 417 425 +8
Partials 611 611 🚀 New features to boost your workflow:
|
Summary
Closes #2535.
Live production observation showed disk usage climbing to a majority-full state, with unused Docker images and build cache as the dominant reclaimable consumers, and no automated mitigation anywhere in the repo — no log rotation config on any of the 22 compose services, and no scheduled prune of Docker resources. An unaddressed disk-exhaustion path is a hard outage for a 24/7 stack: Postgres writes fail, log writes fail, and deploys fail once the volume fills.
What changed
x-logginganchor (json-file, 10MB × 3 files) merged into every service indocker-compose.ymlvia<<: *default-logging, so none of the 22 services default to Docker's unbounded log driver.scripts/selfhost-docker-prune.sh: age-filtered (7-day default)docker image prune+docker builder prune, paired with host-level systemd timer/service.exampleunits (following this repo's existing.env.example/docker-compose.override.yml.exampletemplating convention).docker-proxyservice (which is deliberately read-only,POST: "0", for exactly this kind of blast-radius reason) or grant any container/var/run/docker.sockaccess — reclaiming images/build-cache needs real Docker daemon access, which this repo intentionally does not grant to any container (matching the documented security posture already explained in thedocker-proxyandrunnerservice comments).Correctness/safety notes
docker image prune/docker builder prunewith anuntil=<hours>age filter — never a blind full wipe — so nothing built or pulled within the retention window (default 7 days) is ever a candidate, giving a safety buffer for rollback.Validation
docker compose configwith every optional profile active resolves a boundedloggingconfig on all 22 services — verified programmatically (parsed the resolved config and asserteddriver/max-size/max-fileon every service).test/unit/selfhost-compose-logging.test.ts— pure YAML structural check (with{ merge: true }to resolve the<<:anchor the same way Docker Compose's own YAML 1.1 merge-key support does), nodockerCLI invocation, matching the existingselfhost-compose-*.test.tsconvention (the CI runner has no Docker-in-Docker access).test/unit/selfhost-docker-prune-script.test.ts— actually executes the script with a stubbeddockerbinary onPATH(matching the existingselfhost-backup-metrics-script.test.tsconvention), confirming both prune calls always carry an age filter (never a blind wipe) and the retention window is operator-tunable viaGITTENSORY_DOCKER_PRUNE_RETAIN_HOURS.npm run test:ci) green,npm audit --audit-level=moderateclean,git diff --checkclean.src/**files touched, so no Codecov patch-coverage obligation for this change (config/docs/scripts only) — tests added anyway to match established repo convention for these file types.Safety
.exampletemplates an operator copies and edits, matching.env.example's convention, since a real self-host deploy path can't be known in advance.