Skip to content

chore(selfhost): automate Docker disk hygiene for the self-host stack #2535

Description

@JSONbored

Parent: #1936

Problem

Live production observation showed disk usage climbing to a majority-full state, with Docker image and build-cache storage as the dominant reclaimable consumers (the large majority of both was reclaimable/unused). Separately, no service in docker-compose.yml configures container log rotation — every long-running service (the app, Postgres, Redis, and any self-hosted CI runners) is on Docker's default unbounded json-file log driver. Neither issue has any automated mitigation in the repository today: no docker system prune/builder prune script, cron, or systemd timer exists anywhere, and no self-hosting doc mentions periodic Docker maintenance.

An unaddressed disk-exhaustion path is a hard outage for a 24/7 self-hosted stack: Postgres writes fail, log writes fail, and deploys fail once the volume fills.

Requirements

  • Add explicit log-rotation config (logging: driver: json-file, options: max-size/max-file) to every long-running service in docker-compose.yml, at minimum the app, Postgres, and Redis.
  • Add an automated, scheduled Docker resource-pruning mechanism (dangling images, unused build cache, both bounded by an age filter so nothing in active use is touched) that does not require the app container to have Docker socket access (matching the existing security posture that intentionally omits Docker-in-Docker for CI runner containers).
  • A host-level cron/systemd timer is the right place for the prune step, not a change to what the app container itself can do.
  • Document the new maintenance step in the self-hosting operations docs.

Deliverables

  • logging: blocks added to the relevant docker-compose.yml services.
  • A prune script + scheduling mechanism (cron entry or systemd timer unit), documented as part of the standard self-host deployment.
  • A doc update explaining what gets pruned, on what cadence, and how to verify it's running.

Acceptance criteria

  • Fresh deployments have bounded log growth per container out of the box.
  • A scheduled job reliably reclaims unused Docker images/build cache without disrupting a running deployment or images still in use.
  • The mechanism does not require granting the app container Docker socket access.

Expected outcome

A self-hosted instance can run for months without a maintainer needing to manually intervene to prevent a disk-full outage.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions