Skip to content

fix(selfhost): add default memory limit to the runner compose service - #3913

Merged
JSONbored merged 1 commit into
mainfrom
fix/runner-compose-resource-limits
Jul 7, 2026
Merged

fix(selfhost): add default memory limit to the runner compose service#3913
JSONbored merged 1 commit into
mainfrom
fix/runner-compose-resource-limits

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • The runner service (--profile runners) was the sole service in docker-compose.yml with no deploy.resources.limits — every other resource-risky service (gittensory, redis, postgres, qdrant, ollama, browserless, rees, prometheus, grafana, loki, tempo) already has an operator-overridable memory default.
  • The file's own comment on this service documents a confirmed production incident from exactly this gap: "3 uncapped runner containers on an 8-vCPU box left the app starved under load." The only existing mitigation is a manually-copied docker-compose.override.yml CPU-priority pattern — opt-in and CPU-only, never memory.
  • Added deploy.resources.limits.memory: "${RUNNER_MEM_LIMIT:-2g}", documented RUNNER_MEM_LIMIT in .env.example, and extended the resource-limits regression test's EXPECTED_LIMITS map so this can't silently regress again.
  • Left the host-specific CPU-priority tuning (cpu_shares/cpus) as the existing opt-in override pattern — that genuinely needs per-host sizing (vCPU count, runner replica count), unlike memory, which can have a safe universal default like every sibling service.

Found via a fresh performance/scalability/accuracy hardening audit of the self-host ORB stack. Tracked under #1667.

Scope

  • docker-compose.yml — memory limit for runner
  • .env.example — document RUNNER_MEM_LIMIT
  • test/unit/selfhost-compose-resource-limits.test.ts — cover runner in EXPECTED_LIMITS

Validation

  • npm run typecheck
  • npm run selfhost:env-reference:check
  • npx vitest run test/unit/selfhost-compose-resource-limits.test.ts test/unit/docker-compose-override-example.test.ts — all green
  • git diff --check clean

Safety

  • No src/** change, no secrets. Default is additive-only (a cap that didn't exist before); existing deployments get a safe 2g ceiling they can raise via .env if needed.

Closes #3893

The runner service (--profile runners) was the sole service in
docker-compose.yml with no deploy.resources.limits, despite the file's
own comment documenting a confirmed production incident: uncapped
runner containers starving the main app under load. Give it the same
operator-overridable memory-limit default every other service already
has, and extend the resource-limits test so this can't regress.

Closes #3893
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.58%. Comparing base (685a39e) to head (6740077).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3913   +/-   ##
=======================================
  Coverage   93.58%   93.58%           
=======================================
  Files         358      358           
  Lines       34342    34342           
  Branches    12570    12570           
=======================================
  Hits        32138    32138           
  Misses       1580     1580           
  Partials      624      624           
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 7, 2026
@loopover-orb

loopover-orb Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-07 06:47:57 UTC

3 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR adds an operator-overridable memory limit (`RUNNER_MEM_LIMIT`, default 2g) to the previously-uncapped `runner` compose service, mirroring the existing pattern used by every other resource-risky service in `docker-compose.yml`, and extends the regression test's `EXPECTED_LIMITS` map plus `.env.example` docs accordingly. The change is small, additive-only (no existing behavior is removed since no limit existed before), and directly closes the documented incident referenced in the file's own comment about uncapped runner containers starving the host. The diff is internally consistent across all three touched files and CI is green.

Nits — 5 non-blocking
  • The `deploy.resources.limits` block only takes effect under Swarm mode or newer Compose versions with `docker compose up --compat`/v2 resource support — worth a one-line comment noting whether this repo's runner deployment path actually honors `deploy.resources` under plain `docker compose up` (as the sibling services presumably already rely on, so likely fine, but worth confirming once rather than per-PR).
  • The env.example comment says 'raise for memory-heavy CI jobs' but doesn't mention that with multiple runner replicas each container gets its own 2g cap (total host usage scales with replica count) — a brief note would help operators sizing the box.
  • docker-compose.yml: consider cross-referencing the replica count consideration in the new comment block, since the file already discusses '3 uncapped runner containers on an 8-vCPU box' as the incident context — clarifying that the new cap is per-container, not per-service-total, would prevent under-provisioning if an operator only accounts for one runner.
  • test/unit/selfhost-compose-resource-limits.test.ts: since this test only does structural YAML checks (no `docker compose config` invocation), consider a follow-up note or issue to eventually validate the effective resolved config in an environment where Docker-in-Docker is available, so silent syntax drift in the `deploy.resources` block is caught.
  • PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #3893
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 51 registered-repo PR(s), 43 merged, 343 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 51 PR(s), 343 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
  • Official Gittensor activity: 51 PR(s), 343 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 7, 2026
@JSONbored
JSONbored merged commit 6e829bb into main Jul 7, 2026
11 checks passed
@JSONbored
JSONbored deleted the fix/runner-compose-resource-limits branch July 7, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Development

Successfully merging this pull request may close these issues.

fix(selfhost): add default memory/CPU limits to the runner compose service

1 participant