Skip to content

fix(selfhost): add default memory/CPU limits to the runner compose service #3893

Description

@JSONbored

Context

Every other resource-risky service in docker-compose.yml (gittensory, redis, postgres, qdrant, ollama, browserless, rees, prometheus, grafana, loki, tempo) carries a deploy.resources.limits.memory default. The runner service (--profile runners) is the sole exception — no deploy.resources.limits block at all, and no RUNNER_MEM_LIMIT-style env var exists anywhere.

The surrounding comment in the compose file already documents the exact failure mode: a burst of CI jobs can starve the main app of CPU/memory with no limits set, "confirmed in production: 3 uncapped runner containers on an 8-vCPU box left the app starved under load." The only mitigation today is a cpu_shares/cpus snippet in docker-compose.override.yml.example that an operator must manually discover, copy, and edit — CPU only, never memory, and not applied by default.

test/unit/selfhost-compose-resource-limits.test.ts's EXPECTED_LIMITS map also omits runner, so there is no regression coverage that would catch this gap.

Requirements

  • Add deploy.resources.limits.memory: "${RUNNER_MEM_LIMIT:-2g}" (or a similarly conservative default) directly to the runner service in docker-compose.yml.
  • Document RUNNER_MEM_LIMIT in .env.example.
  • Extend test/unit/selfhost-compose-resource-limits.test.ts's EXPECTED_LIMITS map to include runner so this can't silently regress.
  • Consider moving a conservative cpu_shares/cpus default into docker-compose.yml itself rather than requiring the manual override-file copy, given the failure mode is already confirmed in production.

Acceptance criteria

  • runner has a default memory limit out of the box with no operator action required.
  • The resource-limits test fails if runner's limit is ever removed.

Parent: #1667

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions