Repository navigation
Record-driven Reconcile of memex rolls back — the new portal pod cannot resolve its Postgres host at boot #3786
Description
What is failing
The 02:35Z record-driven Reconcile of the memex control instance (memex-reconcile-20260909-final) — the first record-driven run to reach a full helm upgrade --wait — minted revision 44, and its surge pod crash-looped: every boot died at startup because a Postgres host named in the pod's connection configuration did not resolve (the DNS lookup failed before any TCP attempt was made). Helm's 15-minute wait expired and --atomic rolled the release back to revision 45 (the pre-upgrade state, 19 managed objects); the previous pods served throughout, so this was a stalled roll, not an outage.
The same crash loop surfaced on two sibling fingerprints at other log sites of the same boots — MembershipTableManager aborting the silo at stage RuntimeGrainServices, already filed as #3780, and the Host site carrying the CancelledStartup frame. This ticket covers the same defect and can be closed as a duplicate of #3780.
Probable cause — medium confidence
Name resolution of a Postgres host failed on the new ReplicaSet only, while the previous pods kept serving on already-pooled connections. The image is ruled out — revision 44 pinned the same 3.0.0-ci.8131 the healthy pods run — so it is config or DNS. Two candidates, and the evidence does not cleanly discriminate:
- Transient external-DNS failure in the boot window. The four consecutive failed boots span ~3.5 minutes, and an in-cluster operator job resolved service names normally ~10 minutes later; the Postgres server is an Azure Flexible Server reached through the VNet's private DNS — a different resolution path from in-cluster names, and one the old pods never exercise again once their connections are pooled. Consistent, not proven: no new pod started after the rollback, so nothing re-tested the app path.
- A wrong or default-derived host in the connection string revision 44 rendered — the chart's documented trap class (a template
defaultmanufacturing a plausible URI that points at a host that does not exist; DeploymentAKS → "Secrets the chart renders that no in-cluster Postgres backs", Memex#204). Revision 44 was the first upgrade whose values came purely from the record:ConnectionStrings__memex/ConnectionStrings__orleansarrive viavaultValuesKeys(rendered from the Key Vault values half no repository holds) and thememex-kvCSI class. The orleans clustering database is the leading suspect host, because the sibling fingerprint shows the silo's membership table as the first connect that died.
Discriminating checks: resolve the Postgres host from inside the namespace now (resolves → cause 1; does not → cause 2), and diff revision 44's rendered memex-portal-secrets against revision 43's (compare by shape/length only — the values are secrets).
Impact
Judged from the numbers, contained: a rolled-back release, service continuously available on the previous revision, no recurrence after the rollback. The real exposure is the record-driven reconcile lane on the control instance — this was the third failed reconcile of the overnight sequence (01:59 upgrade failure, 02:27 preflight refusal, 02:35 wait-timeout), so memex cannot complete a Reconcile from its record until the boot resolves or the rendered string names a resolvable host.
Where to look
- Deployments/memex-reconcile-20260909-final — the failed run; Deployments/memex-audit-20260909-rev45 — the post-rollback release state.
- Deployments/memex — the instance record:
vaultValuesKeys,keyVaultSecretClasses, pinned image tag. - Doc/Architecture/DeploymentAKS — diagnose through the Hosting API (
Logs/SampleonDeployments/memex) before break-glass kubectl. - Siblings covering the same boots: Admin/_LogIncident/0b1877ae6c51353f (Portal silo fails to start — orleans clustering database host does not resolve (memex) #3780), Admin/_LogIncident/1c9c655198118379.
Evidence
| Fingerprint | 4fafb2a896dc71c2 |
| Category | Microsoft.Extensions.Hosting.Internal.Host |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-85b6cfc9c9-p5d7p |
| Occurrences | 2 |
| First seen | 2026-09-09 02:37:22Z |
| Last seen | 2026-09-09 02:38:35Z |
Recent log lines
2026-09-09 02:37:22Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Microsoft.Extensions.Hosting.Internal.Host[11]
Hosting failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.P…[truncated]
2026-09-09 02:38:35Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Microsoft.Extensions.Hosting.Internal.Host[11]
Hosting failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.P…[truncated]
Opened automatically from Admin/_LogIncident/4fafb2a896dc71c2. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 9, 2026
Duplicate of #3780, which now carries the consolidated record of this incident (the 02:35Z record-driven Reconcile, helm rev 44, portal pod unable to resolve its Postgres host) and a table saying what each report added — nothing from here is lost.
Closing as a duplicate only. The defect is NOT fixed: memex serving 200 today is explained by the rollback, not by anyone establishing why a record-driven Reconcile rendered an unresolvable Postgres hostname. #3780 stays open for that.
Note #3782 was deliberately NOT folded in — the
--atomic --wait 15mrollback of an upgrade that had already landed is a separate defect in the deploy path that would bite a healthy rollout too.systemorph-com commentedon Sep 14, 2026
Reopened — this fault is happening again.
Still occurring: 28 more since the last update (30 total).
memex-portal-deployment-85b6cfc9c9-p5d7p,memex-portal-deployment-69999d86b8-8hsbs,memex-portal-deployment-5b967589bf-5j6g2,memex-portal-deployment-78d795bf59-87c7xRecent log lines
rbuergi commentedon Sep 15, 2026
Closing as the duplicate this ticket's own body names.
#3780 is fixed and closed. The cause was established on the 2026-09-14 recurrence and it is code, not DNS: a record-driven deploy rendered the chart without the Key Vault values half, so
memex.orleansConnectionStringfell back to the chart defaultHost=memex-postgres-service;…;Database=orleans— a Service no AKS release renders (postgres.enabled: false) — and the silo died at name resolution before any TCP attempt. Every lane-driven deploy layered that half (-f vault-values.yaml -f <overlay>) and rolled the same image healthy, which is why the image was correctly ruled out here.ConnectionStrings__memexsurvived only because thememex-kvCSI class shadows it later inenvFrom; nothing shadowed the orleans string.Both halves merged: core #4356 (
hosting-deploy --vault <name>layershelm-values-<release>first, and the chart now REFUSES to invent the host instead of defaulting to a dead one) and MeshWeaver.Plugins #1892 (the plan passes--vaultexactly when the record declaresvaultValuesKeys). Operator imagehosting-operator:b63310acarries it.Live confirmation: the lane deploy that restored the vault half rolled memex onto 3.0.0-ci.8612 and every probe of
memex.systemorph.com/api/versionanswered the new image at 22:28Z; the incident recorded nothing after 21:59:25Z.