memex 503 on node drain: 2 replicas, no PodDisruptionBudget (PDB gated on keda.enabled), no anti-affinity #3772

What happened

memex.systemorph.com answered 503 for ~90 s from 01:05:45Z on 2026-09-09. Both portal pods of one ReplicaSet (memex-portal-deployment-76c99bdc6-9stvn, -g7f89) logged Application is shutting down... at 01:05:01Z simultaneously; replacements -kcb4d / -mwnnb reached PortalReady at 01:05:58Z / 01:06:01Z. No image change, no rollout (same ReplicaSet), no crash (createdump/Unhandled absent).

Read cluster-wide through Loki (all namespaces, 70 min): at 01:09:08Z monitoring/loki-0, loki-alertmanager-0, loki-prometheus-pushgateway, memex-cloud/memex-website and both k8s-dashboard pods stopped/restarted within 15 s; at 01:09:23Z memex/…-kcb4d and memex-cloud/…-276kb shut down together; replacements Ready 01:10:35Z / 01:10:36Z; a further memex-cloud replacement at 01:13:30Z. That is a node drain rolling node by node (an AKS node-pool operation — planned maintenance / node image upgrade / autoscaler consolidation; which one needs az, which is break-glass under the operate-through-memex directive). Pods across three namespaces on one node stopping together is not an application event.

Modules:AutoRecycleOnStaleBuild is excluded: the converge branch posts a hub DisposeRequest (NodeTypeEnrichmentHelpers.cs:2122), never a process exit. The operator-image patch on the record and the CD run that started at 01:02Z were coincidental.

The defect that is ours

A single drain took BOTH memex replicas because nothing stopped it:

• deploy/helm/templates/memex-portal/pdb.yaml is gated on keda.enabled. memex's helm-release lane renders -f vault-values.yaml -f values.memex.public.yaml (Memex helm-release.yml:530) — not deploy/aks/values.aks.yaml, where KEDA is on — and the memex overlay never mentions keda. So memex runs replicas.portal: 2 with no PodDisruptionBudget: the eviction API had nothing to honour and evicted both pods at once.
• The pod template has no podAntiAffinity / topologySpreadConstraints, so the scheduler is free to co-locate both replicas on one node (memex also has no nodeSelector — the silos pool is min 2 / max 4 nodes, so spreading was possible). cluster-autoscaler.kubernetes.io/safe-to-evict: "false" only speaks to the autoscaler's bin-packing; a drain ignores it.
• check-chart-invariants.py asserts "a PDB implies a floor > 1" (invariant 6) but not the converse — a floor > 1 with no PDB rendered green.

Fix

• Render the PDB whenever the replica floor is ≥ 2 — keda.enabled OR replicas.portal >= 2 — maxUnavailable: 1 as today.
• Add podAntiAffinity (preferred, kubernetes.io/hostname) to the portal pod template so two replicas land on two nodes when two are available, without ever leaving a pod Pending.
• Invariants: a replica floor > 1 implies a PDB is rendered, and implies the pod template spreads replicas across hosts.
• Doc: DeploymentAKS → what a drain does to a 2-replica namespace with and without a budget.

Secondary (separate issue): Loki lost the 01:05Z lines when loki-0 itself was drained at 01:09:08Z — read at 01:08, gone at 01:11 — so the monitoring stack has no WAL / persistence for unflushed chunks.