Skip to content

chore: right-size admins/labs/ovk memory-limit quotas #534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mashkovd merged 1 commit into from
Jul 7, 2026
Merged

chore: right-size admins/labs/ovk memory-limit quotas #534

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion platform-gitops/tenants/admins/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,13 @@ tenant:
# mctl-agent/mctl-design/mctl-docs/mctl-web 1200m), leaving only 500m —
# not enough for the 1000m deploy-pipeline pods. Raised 8 to 10.
limits.cpu: "10"
limits.memory: "6Gi"
# Right-sized 2026-07: after the openclaw memory-limit trim (2560Mi ->
# 1536Mi, mctl-gitops#532), steady-state usage is 3712Mi/6Gi. Lowered to
# 5Gi, keeping ~1.3Gi headroom (same class of margin as the CPU buffer
# above) for deploy-pipeline/rollback workflow pods. Frees 1Gi of
# cluster-wide memory-limit quota ceiling for onboarding more tenants
# (docs/audit/PLATFORM_ASSESSMENT_2026-07.md capacity note).
limits.memory: "5Gi"
pods: "12"
persistentvolumeclaims: "2"
services: "10"
Expand Down
12 changes: 11 additions & 1 deletion platform-gitops/tenants/labs/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,17 @@ tenant:
requests.cpu: "3"
requests.memory: "5Gi"
limits.cpu: "12"
limits.memory: "11Gi"
# Right-sized 2026-07: after the openclaw memory-limit trim (2560Mi ->
# 2048Mi, mctl-gitops#532), current steady-state usage is 6560Mi/11Gi —
# but labs-openclaw is scaled to 0 right now (see
# services/labs/openclaw/values.yaml) and documented to come back
# online. Budgeted for that: 6560Mi + 2048Mi (openclaw's trimmed limit,
# once re-enabled) + ~2Gi headroom (same margin the 2026-05-10 bump
# validated for concurrent workflow + preview pods) ~= 10.5Gi. Only a
# 512Mi trim vs the un-adjusted 9Gi first estimate, on purpose — this
# tenant has the documented history of workflow pods stuck Pending when
# headroom got too tight.
limits.memory: "10.5Gi"

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Canonicalize the labs memory quota quantity

Kubernetes stores Quantity fields in canonical form; the Kubernetes Quantity docs note that 1.5Gi is re-emitted as 1536Mi, and the Argo CD FAQ calls out normalized resource quantities as an OutOfSync source. With this tenant Application using automated sync plus ApplyOutOfSyncOnly, applying 10.5Gi will come back from the API as 10752Mi, so tenant-labs can remain OutOfSync and keep self-healing even though the quota is numerically unchanged. Use the canonical 10752Mi (or an integral Gi value) in git.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include openclaw sidecars in the labs quota budget

The budget here adds only 2048Mi for bringing labs-openclaw back, but that pod's app containers include the 2048Mi main container plus the 256Mi s3-sync, 64Mi skills-fanout, and 64Mi mctl-token-refresh sidecars, so re-enabling it consumes 2432Mi of limits.memory. With the stated current usage, the new quota leaves 10752Mi - 6560Mi - 2432Mi = 1760Mi, below the ~2Gi workflow/preview headroom this comment says is preserved; the same concurrent workflow/preview case that drove the May bump can still be denied by quota. Raise this to at least 11040Mi (or keep 11Gi) if that headroom is intended.

Useful? React with 👍 / 👎.

pods: "15"
persistentvolumeclaims: "2"
services: "15"
Expand Down
8 changes: 7 additions & 1 deletion platform-gitops/tenants/ovk/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,13 @@ tenant:
# skills-fanout 300m) brings pod limits to 5300m; bumped from 5 to 6
# to accommodate new sidecar limits with headroom.
limits.cpu: "6"
limits.memory: "4Gi"
# Right-sized 2026-07: after the openclaw memory-limit trim (2560Mi ->
# 1536Mi, mctl-gitops#532), steady-state usage is 1920Mi/4Gi (only
# openclaw runs in this tenant currently). Lowered to 3Gi, keeping
# ~1.1Gi headroom for workflow/rollout pods. Frees 1Gi of cluster-wide
# memory-limit quota ceiling for onboarding more tenants
# (docs/audit/PLATFORM_ASSESSMENT_2026-07.md capacity note).
limits.memory: "3Gi"
pods: "5"
persistentvolumeclaims: "2"
services: "5"
Expand Down
Loading