Repository navigation
chore: right-size admins/labs/ovk memory-limit quotas - #534
Conversation
Follow-up to mctl-gitops#532 (openclaw/claude-remote memory-limit trim). Re-measured live tenant quota usage after that trim and lowered each tenant's limits.memory to steady-state usage plus a buffer sized the same way each tenant's own quota-bump history justified past increases: - admins: 6Gi -> 5Gi (used 3712Mi, ~1.3Gi headroom) - ovk: 4Gi -> 3Gi (used 1920Mi, ~1.1Gi headroom) - labs: 11Gi -> 10.5Gi (used 6560Mi, but labs-openclaw is currently scaled to 0 and documented to come back online; budgeted for its 2048Mi limit plus the ~2Gi headroom labs' own 2026-05-10 bump validated for concurrent workflow + preview pods, so only a modest 512Mi trim here instead of matching the other two tenants' cut) Frees 2.5Gi of cluster-wide memory-limit quota ceiling for onboarding more tenants, per the 2026-07 capacity audit (docs/audit/PLATFORM_ASSESSMENT_2026-07.md).
|
Claude encountered an error after 0s —— View job I'll analyze this and get back to you. |
|
Claude finished @mashkovd's task in 41s —— View job Review: Approved ✓This is a trivial config-only change to tenant memory-limit quotas. All three changes are correct, well-documented, and operationally sound: admins (6Gi → 5Gi): Frees 1Gi. Measured usage 3712Mi + 1.3Gi headroom for deploy/rollback pods. labs (11Gi → 10.5Gi): Frees 512Mi (conservative trim). Accounts for 6560Mi current usage + 2048Mi future openclaw restart + 2Gi headroom (validates concurrent workflow/preview pods). ovk (4Gi → 3Gi): Frees 1Gi. Measured usage 1920Mi + 1.1Gi headroom for workflow pods. All quota adjustments follow the capacity audit rationale (#532 follow-up), preserve adequate headroom, and reference the context clearly in comments. YAML syntax is valid; no security, correctness, or operational safety issues. P1 findings: 0 ✅ No issues, good to merge. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 032bac40e0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # 512Mi trim vs the un-adjusted 9Gi first estimate, on purpose — this | ||
| # tenant has the documented history of workflow pods stuck Pending when | ||
| # headroom got too tight. | ||
| limits.memory: "10.5Gi" |
There was a problem hiding this comment.
Kubernetes stores Quantity fields in canonical form; the Kubernetes Quantity docs note that 1.5Gi is re-emitted as 1536Mi, and the Argo CD FAQ calls out normalized resource quantities as an OutOfSync source. With this tenant Application using automated sync plus ApplyOutOfSyncOnly, applying 10.5Gi will come back from the API as 10752Mi, so tenant-labs can remain OutOfSync and keep self-healing even though the quota is numerically unchanged. Use the canonical 10752Mi (or an integral Gi value) in git.
Useful? React with 👍 / 👎.
| # 512Mi trim vs the un-adjusted 9Gi first estimate, on purpose — this | ||
| # tenant has the documented history of workflow pods stuck Pending when | ||
| # headroom got too tight. | ||
| limits.memory: "10.5Gi" |
There was a problem hiding this comment.
The budget here adds only 2048Mi for bringing labs-openclaw back, but that pod's app containers include the 2048Mi main container plus the 256Mi s3-sync, 64Mi skills-fanout, and 64Mi mctl-token-refresh sidecars, so re-enabling it consumes 2432Mi of limits.memory. With the stated current usage, the new quota leaves 10752Mi - 6560Mi - 2432Mi = 1760Mi, below the ~2Gi workflow/preview headroom this comment says is preserved; the same concurrent workflow/preview case that drove the May bump can still be denied by quota. Raise this to at least 11040Mi (or keep 11Gi) if that headroom is intended.
Useful? React with 👍 / 👎.
Summary
Follow-up to #532 (openclaw/claude-remote memory-limit trim, merged). Re-measured live tenant quota usage after that trim via
mctl_get_resource_usageand lowered each tenant'squotas.limits.memory:labsgets a smaller cut than the other two on purpose:labs-openclawis currentlyreplicaCount: 0but documented as intended to come back online. Its 6560Mi "used" today doesn't include openclaw's 2048Mi (post-trim) limit, so the new quota is budgeted forcurrent used + openclaw's future limit + the ~2Gi headroom labs' own 2026-05-10 bump comment validated for concurrent workflow/preview pods— not just today's snapshot.adminsandovkdon't have a similar "currently scaled down" service, so their buffers are sized directly off live usage.Why
Part of the 2026-07 platform capacity audit follow-up (
docs/audit/PLATFORM_ASSESSMENT_2026-07.md): by memory limits the cluster only had headroom for ~1 morenfc-sized tenant. This is step 2 of the plan (step 1 was #532); freeing the quota ceiling itself is what actually returns memory to the shared cluster pool, since the ResourceQuota ceiling — not the underlying pod specs — is what reserves capacity for planning purposes.Test plan
helm templaterenderedplatform-gitops/helm-charts/tenantfor all three tenants, confirmed the newlimits.memoryvalue in the renderedResourceQuotamctl_get_resource_usagefor admins/labs/ovk to confirmusedstays comfortably under the newallocated@claude reviewand CI green before merge