Skip to content

fix(k8s): set admin Go soft memory limit in prod and prod-us (hotfix) - #1272

Merged
edukmx merged 1 commit into
mainfrom
hotfix/admin-gomemlimit
Nov 6, 2026
Merged

edukmx merged 1 commit into
mainfrom
hotfix/admin-gomemlimit

Conversation

@edukmx

@edukmx edukmx commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Hotfix to main with only the memory fix from #1270 (already on develop, cherry-picked as-is): GOMEMLIMIT=1536MiB on the admin container in the prod and prod-us overlays (75% of the 2Gi limit, same ratio as dev).

Nothing else from develop is included; Store Everyone (#1268) stays out of this hotfix.

Why

On 2026-11-06 the platform-pod-saturation alert fired for the admin in saas-prod (memory at 95% of 2Gi). It is not a leak: a stream of consumer/auth writes on one gateway drove about 210 snapshot compiles per hour, and without GOMEMLIMIT the Go heap swung between 1.1Gi and 1.95Gi. The soft limit makes the GC collect before the container limit.

Test plan

  • kubectl kustomize k8s/overlays/{prod,prod-us} renders; the admin env adds only GOMEMLIMIT, and resources and image pins are unchanged
  • After merge: Auto Release bumps both overlays, and the healthz version updates in EU and US
  • Admin pods stay below about 1.6Gi in kubectl top under write load

Without GOMEMLIMIT the admin heap grows to about twice the live set before
GC runs, ignoring the 2Gi container limit. During a write burst the admin
recompiles the config snapshot on every write, and prod admin pods swung
between 1.1Gi and 1.95Gi, tripping the memory saturation alert at 95%.

Use the same 75% ratio as dev (768MiB of 1Gi): 1536MiB of 2Gi.
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Trivy code scan — HIGH/CRITICAL

No HIGH or CRITICAL vulnerabilities found. (run)

@edukmx
edukmx merged commit 320431b into main Nov 6, 2026
21 checks passed
@edukmx
edukmx deleted the hotfix/admin-gomemlimit branch November 6, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants