wait-for-postgres gates only MEMEX_HOST — an explicit ConnectionStrings__orleans naming another host is gated by nothing #4173

What is wrong

The portal's wait-for-postgres init container exists to make one thing deterministic — "Postgres and the portal start concurrently on a fresh install; the portal's hub init FAILS (and the host aborts) if Postgres isn't yet accepting connections. Gate startup on Postgres TCP readiness so the ordering is deterministic." It probes one host, and that host comes from a different input than the connection strings the process actually opens at boot.

deploy/helm/templates/memex-portal/deployment.yaml:104-111:

- name: "wait-for-postgres"
  image: "busybox:1.36"
  command:
    - "sh"
    - "-c"
    {{- $pgHost := .Values.postgres.enabled | ternary "memex-postgres-service" (.Values.config.memex_portal.MEMEX_HOST | default "memex-postgres-service") }}
    {{- $pgPort := .Values.postgres.enabled | ternary "5432" (.Values.config.memex_portal.MEMEX_PORT | default "5432") }}
    - "until nc -z {{ $pgHost }} {{ $pgPort }}; do echo 'waiting for postgres'; sleep 2; done"

The probe's host is config.memex_portal.MEMEX_HOST — a config value, on AKS derived from the deployment record (DeploymentPortalConfig.DatabaseHost, src/MeshWeaver.Deployment.Contract/DeploymentPortalConfig.cs:69-73: explicit FQDN → Azure server FQDN → in-cluster service).

The connections the boot actually opens are secrets, rendered from the Key Vault values half that no repository holds (deploy/helm/templates/memex-portal/secrets.yaml):

Nothing renders, asserts or logs that those hosts equal MEMEX_HOST. grep -rn MEMEX_HOST deploy/helm/templates/ scripts/ returns four hits: the two probes and the two ConfigMaps that publish the value — no comparison anywhere.

So:

• Explicit ConnectionStrings__orleans naming another host (a dedicated orleans server, or the same server under another name) — that host is gated by nothing. The init container can pass on a host that is up while the silo then opens a connection to one that is not.
• Derived ConnectionStrings__orleans (the AKS default for an external mesh database) inherits ConnectionStrings__memex's host — covered by the probe only if that host string equals MEMEX_HOST, which nothing checks. They are set from two different halves of the values.
• The migration Job has the same shape: deploy/helm/templates/memex-migration/job.yaml:70 gates on config.memex_migration.MEMEX_HOST and then migrates against ConnectionStrings__memex.

What an ungated host produces at boot

Exactly the fingerprint of #3780: the pod passes Init:1/1, the silo reaches lifecycle stage RuntimeGrainServices, and MembershipTableManager fails on the orleans connection —

'MembershipTableManager' failed to start due to errors at stage 'RuntimeGrainServices (8000)'.
Npgsql.NpgsqlException: Name or service not known
 ---> System.Net.Sockets.SocketException: Name or service not known
   at System.Net.Dns.GetHostEntryOrAddressesCore(...)

— and the container exits, because the portal fails closed. The chart rolls surge-first (maxUnavailable: 0), so the cost of one occurrence is a delayed roll rather than an outage; a persistent one stalls every roll of the instance. The init container's passing tells you nothing about the connection that failed, which is what makes this read as a transient resolver blip rather than as a gate that was never covering that host.

🚨 What this is NOT

It is not established as the cause of #3780. That incident's values live in Key Vault (helm-values-memex), which I cannot read, so I do not know whether memex's ConnectionStrings__orleans is explicit or derived, nor whether its host string equals MEMEX_HOST. #3780's own measurement — two later boots of the same deployment through the same lifecycle stage on the same secret — excludes a permanently wrong host there. This issue is the hole in the guard, filed on its own because it is true whatever #3780's cause turns out to have been: a deployment whose orleans host differs from MEMEX_HOST has an init container that is not gating the connection the silo makes.

What would close this

The gate covers every host the boot connects to, or says loudly that it cannot:

1. Derive the probe's host(s) from the rendered connection strings rather than from a parallel config value — probe each distinct Host= the pod's secret carries (mesh + orleans), so "the gate passed" means "every database this pod will open is accepting connections"; or
2. Assert agreement at render time — fail in the template when an explicit ConnectionStrings__orleans (or ConnectionStrings__memex) names a host other than MEMEX_HOST, so a divergence is a red render naming both values instead of a silent hole. The chart already uses fail for the values-shape traps it refuses to guess at.

Either way the verification is the same and does not need a cluster: render the chart with postgres.enabled=false, MEMEX_HOST=a, and an explicit ConnectionStrings__orleans naming host b, and assert that the output either probes b or fails the render. Today it probes a and says nothing about b.

Found while re-measuring #3780 (2026-09-13); read-only, nothing changed.