Repository navigation
memex portal pods crash-loop on boot — orleans clustering database host does not resolve from the pod #3784
Description
What is failing
During the 02:35Z record-driven re-apply of the memex control instance, the new portal pod (new ReplicaSet, surge pod) fails to boot: the Orleans silo's MembershipTableManager — cluster membership on the orleans PostgreSQL database reached via ConnectionStrings__orleans — aborts silo start because the database hostname does not resolve (SocketException: Name or service not known out of Dns.GetHostEntryOrAddressesCore; the failure is at name resolution, before any TCP connect). The portal fails closed, the container exits, and the pod restarts into the same failure with growing backoff.
Probable cause — medium confidence
The orleans host on the freshly rendered memex-portal-secrets does not resolve deterministically — not a one-off DNS blip:
- The same pinned image (
3.0.0-ci.8131) was 2/2 ready on the previous rollout (helm rev 43, ~2 minutes before the first failure). The image is not the variable; the config render is. - The failure repeated on every restart over ~12 minutes (the
Host-site sibling below still shows it failing at 02:47Z). A host that resolves intermittently does not fail every attempt. - The instance runs
postgres.enabled=false— there is no in-cluster Postgres behind a service name; the databases are external (memexaks-pg). A connection string pointing at an in-cluster service name this namespace never had produces exactly this error.
Most likely the re-render from the record produced a wrong ConnectionStrings__orleans host. That value renders from the Key Vault values half (helm-values-memex) held in no repository — the record carries only the key name in vaultValuesKeys — so nothing diffs it. Second possibility: the surge pod landed on a node with broken DNS (container restarts stay on the same node). One command discriminates: getent hosts <orleans-host> from the pod (read the host by name/shape only — it is a secret). Also compare the operator job's rendered /tmp/hosting-values.yaml against the values rev 43 rendered, and check the memex-migration job from the same rollout — if its orleans phase connected, the bad value is portal-side only.
The 2026-08-17 pre-start gate (A silo checks its Orleans database before starting) closes the "orleans DB never provisioned" case by checking OrleansQuery rows; an unresolvable host is not that case — the boot proceeded and died at MembershipTableManager with a message naming nothing to fix. That is the gap.
Impact
One pod, one instance — but memex is the control instance (infra control, accounting, home of the Deployments space). The roll strategy is surge-first (maxSurge: 1, maxUnavailable: 0 per the record), so the previous ReplicaSet kept serving: a stalled roll, not an outage. Helm ran --atomic --wait --timeout 15m, so the upgrade rolls back to rev 43 and the record→cluster path this run was meant to prove "end to end" cannot complete while the rendered host is wrong. Precedent: the 01:59Z upgrade's rollback itself errored (audit), so the release state after this rollback should be audited too.
Where to look
Deployments/memex— instance record;ConnectionStrings__orleansinvaultValuesKeys, rendered intomemex-portal-secrets.Deployments/memex-reconcile-20260909-final— the in-flight re-apply whose helm upgrade produced this pod; its rendered values are the artifact to diff.Doc/WhatsNew/2026-08-17-a-silo-checks-its-orleans-database-before-starting— the orleans pre-start gate; the natural home for a host-resolution check.
Same event, other fingerprints — merge or close as duplicates:
Admin/_LogIncident/0b1877ae6c51353f—SiloLifecycleSubjectwrapper of this same failure (same pod, milliseconds apart); already triaged, filing attempted as Portal silo fails to start — orleans clustering database host does not resolve (memex) #3780 but errored ("Resource not accessible by integration") — verify that issue exists before merging.Admin/_LogIncident/1c9c655198118379— theHost: Hosting failed to startsite, same pod and exception; carries the later restarts.Admin/_LogIncident/2936e22a9e7f286b— earlier Npgsql incident, different failure mode (connect timeout to a resolved IP, Aug 23); not a duplicate, same surface.
Evidence
| Fingerprint | c34ccbddb5c4b011 |
| Category | Orleans.Runtime.MembershipService.MembershipTableManager |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-85b6cfc9c9-p5d7p |
| Occurrences | 7 |
| First seen | 2026-09-09 02:36:24Z |
| Last seen | 2026-09-09 02:47:36Z |
Recent log lines
2026-09-09 02:36:24Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Orleans.Runtime.MembershipService.MembershipTableManager[100646]
Membership failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)…[truncated]
2026-09-09 02:37:22Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Orleans.Runtime.MembershipService.MembershipTableManager[100646]
Membership failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)…[truncated]
2026-09-09 02:38:35Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Orleans.Runtime.MembershipService.MembershipTableManager[100646]
Membership failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)…[truncated]
2026-09-09 02:39:53Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Orleans.Runtime.MembershipService.MembershipTableManager[100646]
Membership failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)…[truncated]
2026-09-09 02:41:31Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Orleans.Runtime.MembershipService.MembershipTableManager[100646]
Membership failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)…[truncated]
2026-09-09 02:43:51Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Orleans.Runtime.MembershipService.MembershipTableManager[100646]
Membership failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)…[truncated]
2026-09-09 02:47:36Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Orleans.Runtime.MembershipService.MembershipTableManager[100646]
Membership failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)…[truncated]
Opened automatically from Admin/_LogIncident/c34ccbddb5c4b011. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 9, 2026
Duplicate of #3780, which now carries the consolidated record of this incident (the 02:35Z record-driven Reconcile, helm rev 44, portal pod unable to resolve its Postgres host) and a table saying what each report added — nothing from here is lost.
Closing as a duplicate only. The defect is NOT fixed: memex serving 200 today is explained by the rollback, not by anyone establishing why a record-driven Reconcile rendered an unresolvable Postgres hostname. #3780 stays open for that.
Note #3782 was deliberately NOT folded in — the
--atomic --wait 15mrollback of an upgrade that had already landed is a separate defect in the deploy path that would bite a healthy rollout too.systemorph-com commentedon Sep 14, 2026
Reopened — this fault is happening again.
Still occurring: 28 more since the last update (35 total).
memex-portal-deployment-85b6cfc9c9-p5d7p,memex-portal-deployment-69999d86b8-8hsbs,memex-portal-deployment-5b967589bf-5j6g2,memex-portal-deployment-77476b68-l4gw8Recent log lines
rbuergi commentedon Sep 15, 2026
Duplicate of #3780 — the cause, the two merged fixes (#4356, Systemorph/MeshWeaver.Plugins#1892) and the measurements are in #3780 (comment).
systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 27 more since the last update (62 total).
memex-portal-deployment-85b6cfc9c9-p5d7p,memex-portal-deployment-69999d86b8-8hsbs,memex-portal-deployment-5b967589bf-5j6g2,memex-portal-deployment-77476b68-l4gw8,memex-portal-deployment-78d795bf59-87c7xRecent log lines
rbuergi commentedon Sep 17, 2026
Re-closed — the reopen above was on stale evidence (#4577).
The recurrence bot reopened this at
2026-09-17T07:53:07Zunder the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:Last seen:2026-09-14 21:59:25Z2026-09-15T02:24:38ZSo the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix:
LogIncidentFiler.Reopenreopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen requireincident.LastSeen > issue.ClosedAt).This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.
One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.