Repository navigation
chore: trim CPU limits on vmagent, backstage, mctl-api, admintools - #146
Merged
Merged
Conversation
added 2 commits
May 7, 2026 01:45
…ntools Second pass after PR #145. After bucket A landed, sum(limits.cpu) sat at 25.50c (91% of 28-core cluster). These four containers all have 11-22x limit/actual ratios and contribute another ~0.85c of dead headroom: - vmagent: 500m -> 250m (steady-state 123m, 2x cushion) - backstage: 500m -> 300m (steady-state 26m, 11x cushion) - mctl-api: 500m -> 250m (steady-state 11m, 22x cushion) - temporal admintools: 200m -> 100m (idle CLI sidecar, ~2m) Expected new total: ~24.65c (88% of 28). Postgres skip: shared-pg Application sets ignoreDifferences on .spec.resources to accommodate CNPG operator normalization, so resource trims via gitops won't propagate here.
Phase 4 of the incident auto-cleanup track. Adds Prometheus metrics
wiring so the cleanup activity from Phases 1/2/3 (stale TTL GC,
orphan pruning, AM fingerprint reconciliation) becomes visible in
Grafana and alertable, instead of slog-only.
Six metrics, low cardinality (status / source / reason / outcome —
no tenant or service labels):
- mctl_agent_stale_ttl_resolved_total{status} — Phase 1
- mctl_agent_orphan_pruned_total — Phase 3
- mctl_agent_am_reconcile_resolved_total — Phase 2
- mctl_agent_cleanup_skipped_total{reason} — guards across all phases
- mctl_agent_am_request_duration_seconds{outcome} — Phase 2 latency
- mctl_agent_open_tickets{status, source} — gauge updated each cycle
Adds a sibling ServiceMonitor in
bootstrap/templates/mctl-platform/mctl-agent-monitor.yaml mirroring
mctl-api-monitor.yaml (port http, path /metrics, 30s interval) — no
base-service chart changes required.
Status: accepted (operator-seeded, skipping mentor digest).
Member
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Nice work! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This was referenced May 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second-pass trim after #145. With #145 merged, cluster now at
sum(limits.cpu) = 25.50c(91% of 28). These four containers all have 11-22× limit/actual ratios and contribute ~0.85c of dead headroom.bootstrap/templates/observability/monitoring.yamlbootstrap/templates/mctl-platform/mctl-portal.yamlbootstrap/templates/mctl-platform/mctl-api.yamlbootstrap/templates/data/temporal.yamlExpected new total: ~24.65c (88%).
Skipped
shared-pgApplication setsignoreDifferences: .spec.resourceson the CNPGClusterto accommodate operator-side normalization. Trims via gitops won't propagate here — would require either removing the ignore (risky — operator drift will surface as OutOfSync), or a directkubectl patch cluster(out-of-band). Both deferred.Test plan
monitoring,mctl-portal,mctl-api,temporalApplications green./api/healthcontinues to respond./healthzcontinues to respond.CPUThrottlingHighalerts over the next 24h.