Skip to content

chore: promote develop to main (admin GOMEMLIMIT, Store Everyone) (ENG-3054) - #1271

Merged
edukmx merged 16 commits into
mainfrom
develop
Nov 6, 2026
Merged

edukmx merged 16 commits into
mainfrom
develop

Conversation

@edukmx

@edukmx edukmx commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Refs ENG-3054

What this promotes (develop → main)

PR What
#1270 GOMEMLIMIT=1536MiB on the admin container in prod and prod-us (75% of the 2Gi limit, same ratio as dev). The only change to the prod/prod-us overlays.
#1268 Store Everyone: GET|PUT /v1/gateways/{id}/store/everyone, new store_everyone table (migration 20261105120000), snapshot field 18. Additive: with no Everyone object every caller resolves exactly as before, and planes built before field 18 ignore it.

Why now

On 2026-11-06 the platform-pod-saturation alert fired for the admin in saas-prod (memory at 95% of 2Gi). It is not a leak: a stream of consumer/auth writes on one gateway drove about 210 snapshot compiles per hour, and without GOMEMLIMIT the Go heap swung between 1.1Gi and 1.95Gi. The soft limit makes the GC collect before the container limit.

Rollout

  • git merge-tree --write-tree origin/main origin/develop is clean.
  • The only prod/prod-us overlay change is the GOMEMLIMIT env on admin; image pins are unchanged.
  • One new migration (additive table).

After merging

  • Auto Release bumps both overlays, and the healthz version updates in EU and US
  • Admin pods report GOMEMLIMIT and stay below about 1.6Gi in kubectl top under the write stream
  • ENG-3054: save Everyone from the console in DEV and check that a hybrid plane picks it up (still pending from feat: add Everyone access to the MCP Store (ENG-3054) #1268)

edukmx and others added 12 commits November 5, 2026 12:19
Exploration, proposal, specs, design and tasks for a per-gateway Everyone
Store access and bulk Assign / Remove on Access > Groups.
A gateway can hold one Everyone object: a Store level (open or curated)
and the servers every group passes on, minus excluded groups. This adds
the domain type, the store_everyone table, its repository with an outbox
marker, the service with grant validation, and GET/PUT
/v1/gateways/{gateway_id}/store/everyone. Evaluation and the config
snapshot come next.
A caller reaches Everyone when one of its Store groups is neither excluded
nor on a none policy. The level becomes the most permissive of today's
result and Everyone's mode, so Everyone never lowers anyone's access and
changes nothing when absent. Grant decisions add Everyone's servers for
callers who reach it, through the shared loader; the approver keeps
writing explicit grants. On the control plane the Store module serves
grants, policies and Everyone from one AccessReader.
…s (ENG-3054)

The config snapshot carries each gateway's Everyone object in a new field
18, so hybrid and DB-less planes evaluate it like grants and policies; an
older plane ignores the field. Deleting a registry drops it from
Everyone's grants, and removes the object when a curated one is left
without grants. A functional test covers group keys, exclusions, groups
on none, install without approval and removal.
- Install honours instance-level Everyone grants, named or picked from the
  usable instances, like search and the scoper already did.
- PUT store/everyone requires mode: a body without it is a 422, not a
  removal; an empty mode still removes Everyone.
- Pruning a deleted registry from Everyone is one locked read-modify-write
  in the repository, and runs only after the registry delete commits, so a
  refused delete leaves Everyone untouched.
- The registry cleaner and the Everyone DTOs move to their own files; two
  narrative test comments go.
feat: add Everyone access to the MCP Store (ENG-3054)
Without GOMEMLIMIT the admin heap grows to about twice the live set before
GC runs, ignoring the 2Gi container limit. During a write burst the admin
recompiles the config snapshot on every write, and prod admin pods swung
between 1.1Gi and 1.95Gi, tripping the memory saturation alert at 95%.

Use the same 75% ratio as dev (768MiB of 1Gi): 1536MiB of 2Gi.
fix(k8s): set admin Go soft memory limit in prod and prod-us
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Trivy code scan — HIGH/CRITICAL

No HIGH or CRITICAL vulnerabilities found. (run)

edukmx and others added 3 commits November 6, 2026 12:07
Every compile appended a full copy of the encoded catalog to each scoped
snapshot and hashed the concatenation. In prod that was 613 copies of a
0.71 MiB catalog, about 435 of the 445 MiB of scoped bytes held per compile,
which kept the admin near its 2Gi memory limit during write bursts.

A scoped snapshot now holds its tenant bytes and a reference to the catalog
slice shared by the whole compile. The gRPC service streams the two segments
in order, so data planes receive the same bytes and version as before, and
the version is hashed over the segments without concatenating them. A scope
whose tenant bytes and catalog are unchanged keeps its previous slices and
version instead of being hashed and held again.
GetSnapshot now sends gzip-compressed chunks when the data plane advertises
gzip in grpc-accept-encoding. Registering the gzip codec makes every data
plane on this build advertise it; data planes never compress their own
requests, so mixed versions keep working in either rollout order.
…atalog

fix(configsnapshot): share one encoded catalog across scoped snapshots; gzip snapshot streams
@edukmx
edukmx merged commit 3eda48f into main Nov 6, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants