Repository navigation
Render the PodDisruptionBudget for every two-replica portal and spread replicas across nodes (#3772) - #3774
Conversation
…d replicas across nodes (#3772) memex.systemorph.com answered 503 for ~90 s on 2026-09-09 01:05Z: an AKS node drain evicted both portal pods in the same second. Nothing in the memex namespace could have said why — same ReplicaSet, same image, no crash — the cluster-wide Loki read did: pods in three namespaces on one node stopped together, then the next node four minutes later. Ours in that: pdb.yaml was gated on keda.enabled, and memex's helm-release lane renders the vault values plus values.memex.public.yaml, never deploy/aks/values.aks.yaml where KEDA is on — so replicas.portal: 2 rendered two pods and no budget for the eviction API to honour. The pod template also declared no anti-affinity, so both replicas shared one node. - pdb.yaml renders whenever the replica floor is two or more (keda.minReplicas under autoscaling, replicas.portal otherwise), maxUnavailable: 1 as before. - deployment.yaml declares a preferred podAntiAffinity on kubernetes.io/hostname: two nodes when two are schedulable, still schedules when only one is; the budget throttles the next eviction so the survivor keeps serving. - check-chart-invariants: 14 (a floor > 1 implies a budget) and 15 (a floor > 1 implies anti-affinity or a spread constraint) — the converse of 6, which was green throughout; plus the memex shape as a fixture combination (two plain replicas, KEDA off). Negative control: with the chart fix reverted the gate reds on exactly that shape (both invariants) and on the shared AKS overlay (15). - DeploymentAKS: what a drain does to a two-replica namespace with and without a budget; a What's New entry (Fix). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Documentation references invariant numbers inconsistently (and the invariant list header is currently misleading), which will make it harder to correlate gate failures with the documented invariants.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR hardens the AKS Helm chart so two-replica portals survive node drains without simultaneous eviction, and adds chart-gate coverage + documentation to prevent regressions.
Changes:
- Render
PodDisruptionBudgetwhenever the effective replica floor is ≥ 2 (KEDA or fixed replicas). - Add preferred pod anti-affinity on
kubernetes.io/hostnamefor the portal pods to encourage replica spreading. - Extend
check-chart-invariantswith new invariants + a memex-shaped fixture, and document the operational behavior in AKS docs / What’s New.
File summaries
| File | Description |
|---|---|
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-09-two-replicas-no-longer-leave-together-on-a-node-drain.md | Adds a What’s New entry describing the incident and the chart-level fix. |
| src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md | Documents node drain behavior and the chart protections (PDB + spreading). |
| deploy/helm/templates/memex-portal/pdb.yaml | Changes PDB rendering condition to key off replica floor ≥ 2 instead of keda.enabled. |
| deploy/helm/templates/memex-portal/deployment.yaml | Adds preferred pod anti-affinity to spread replicas across nodes. |
| deploy/aks/scripts/testdata/values.two-replicas-no-keda.yaml | Adds a fixture representing the memex “2 replicas, no KEDA” shape for invariant gating. |
| deploy/aks/scripts/check-chart-invariants.sh | Adds the memex fixture combination and documents the new invariants. |
| deploy/aks/scripts/check-chart-invariants.py | Adds invariant checks ensuring floor>1 implies both a PDB and spreading directives. |
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # 14. a replica floor > 1 implies a PDB or one node drain evicts every replica at once | ||
| # 15. a replica floor > 1 implies anti-affinity / spread or every replica shares one node |
There was a problem hiding this comment.
Done in the follow-up commit: the header lists 11–13 and the doc says 14/15, matching the gate output.
Test Results (shard 0)254 tests 254 ✅ 2m 12s ⏱️ Results for commit 1197303. ♻️ This comment has been updated with latest results. |
Test Results (shard 1)496 tests 496 ✅ 35s ⏱️ Results for commit 1197303. ♻️ This comment has been updated with latest results. |
Test Results (shard 3)389 tests 389 ✅ 50s ⏱️ Results for commit 1197303. ♻️ This comment has been updated with latest results. |
… new checks 14/15 as the gate does Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Test Results (shard 4)1 857 tests 1 857 ✅ 2m 31s ⏱️ Results for commit 1197303. |
Test Results (shard 2)1 141 tests 1 139 ✅ 3m 46s ⏱️ Results for commit 1197303. |
Test Results (shard 5) 6 files 6 suites 4m 7s ⏱️ Results for commit 1197303. |
Test Results 17 files 17 suites 14m 5s ⏱️ Results for commit 1197303. |
Closes #3772
Incident
memex.systemorph.com answered 503 for ~90 s from 01:05:45Z on 2026-09-09. Both portal pods logged
Application is shutting down...at 01:05:01Z simultaneously; the replacements came from the same ReplicaSet on the same image (no rollout, no crash). Read cluster-wide through Loki, the cluster was draining its nodes one by one: at 01:09:08Zloki-0,loki-alertmanager-0, the pushgateway,memex-cloud/memex-websiteand bothk8s-dashboardpods stopped within 15 s, then the next memex pod and a memex-cloud pod at 01:09:23Z. An AKS node-pool operation — not an application event. (Modules:AutoRecycleOnStaleBuildis a hubDisposeRequest, never a process exit, and is excluded.)Defect
pdb.yamlwas gated onkeda.enabled. memex's helm-release lane renders-f vault-values.yaml -f values.memex.public.yaml(neverdeploy/aks/values.aks.yaml, where KEDA is on) and its overlay never setskeda, soreplicas.portal: 2rendered two pods and no PodDisruptionBudget — the eviction API took both.podAntiAffinity/topologySpreadConstraints: both replicas on one node.check-chart-invariants.pyasserted "a PDB implies a floor > 1" but not the converse.Change
keda.minReplicasorreplicas.portal),maxUnavailable: 1.podAntiAffinityonkubernetes.io/hostnamein the portal pod template.Verification
check-chart-invariants.sh: 7/7 combinations self-consistent.two plain replicas, KEDA offfor BOTH invariants and on the shared AKS overlay for 15.helm templateof the memex shape:PodDisruptionBudgetwithmaxUnavailable: 1,podAntiAffinityon hostname,replicas: 2.check-values-are-read.sh: 119/119 keys read.MeshWeaver.Documentation.Test(built Release-warnaserrorafter the doc edits): WhatsNewEntryIntegrityTest, DocumentationLinkIntegrityTest, NoConflictMarkersGuard — 3/3 passed.No C# change; no public surface touched. Pairs-with: none — chart + scripts + docs only.
Follow-up filed separately: #3773 (Loki drops unflushed chunks when
loki-0is drained — the 01:05Z lines were readable at 01:08Z and gone at 01:11Z).🤖 Generated with Claude Code