Portal silo fails to start — orleans clustering database host does not resolve (memex) #3780

Activity

1. rbuergi commented on Sep 9, 2026

   

   rbuergi

   on Sep 9, 2026

   Contributor

   More actions

   Consolidating five duplicate reports into this one

   Six issues were filed for this single incident within 16 minutes by different sessions, all
   describing the 02:35Z record-driven Reconcile of memex (operator 1a075bb, helm revision 44)
   whose new portal pod could not resolve its Postgres host. Closing the other five against this one so
   the incident has one thread. Nothing here is a claim that the defect is fixed — see the status
   at the end.

   issue	what it added	disposition
   #3780 (this one, 02:45:11Z)	the INNER failure: Orleans MembershipTableManager aborts silo start at lifecycle stage RuntimeGrainServices, SocketException: Name or service not known on ConnectionStrings__orleans	canonical
   #3781 (02:45:46Z)	the CAUSE framing — the record-driven Reconcile rendered the unresolvable host into rev 44 (Deployments/memex-reconcile-20260909-final)	duplicate
   #3783 (02:54:33Z)	the same boot failure at a second, outer log site; self-declared duplicate of this issue	duplicate
   #3784 (02:59:37Z)	crash-loop framing of the same surge pod	duplicate
   #3785 (02:59:37Z)	byte-identical to #3784 (same second, same body md5 ceccd028…)	duplicate
   #3786 (03:00:53Z)	the Reconcile subsequently rolled back	duplicate

   The actionable statement, which #3781 and #3786 carry and this issue's own title does not: the
   inner SocketException is the symptom; the defect is that a record-driven Reconcile rendered a
   Postgres hostname that does not resolve from inside the pod. Fixing the DNS name in one revision
   does not close that — the question is why the record produced an unresolvable host.

   Explicitly NOT consolidated: #3782 stays open on its own. It is a different defect that this
   incident merely revealed — helm upgrade --atomic --wait 15m rolled back an upgrade that had
   already landed, because the wait time-boxes instead of observing a gated rollout. That is a bug in
   the deploy path independent of any hostname, and it would have bitten a healthy rollout too.

   Status at the time of writing (2026-09-09): https://memex.meshweaver.cloud/health, /alive and
   / all answer 200, so the portal is serving. That is recovery, not evidence of a fix — the
   rollback in #3786 is a sufficient explanation for a healthy portal, and this issue stays OPEN until
   someone establishes why the rendered host did not resolve.
2. rbuergi

   closed this as not plannedon Sep 10, 2026
3. rbuergi

   reopened this on Sep 10, 2026
4. rbuergi commented on Sep 10, 2026

   

   rbuergi

   on Sep 10, 2026

   Contributor

   More actions

   Reopened — the 2026-09-10 sweep was not a triage decision

   This issue was closed not_planned at 2026-09-10T14:30:36Z by a bulk sweep that PATCHed every open issue in all 13 Systemorph repositories — 311 issues in twelve minutes, no comment on any of them — after a session was asked to "close all issues in all repos". Nothing on this thread was read, and no judgement was made about this issue.

   All 82 issues the sweep closed in the MeshWeaver fleet were reviewed afterwards. This one is unresolved: Consolidated report for the 2026-09-09 silo boot failure (five duplicates folded in); no fix landed.

   Sixteen of the 82 were left closed — their fix had actually landed, or the author had withdrawn them; those should have read completed, not not planned.
5. rbuergi commented on Sep 10, 2026

   

   rbuergi

   on Sep 10, 2026

   Contributor

   More actions

   Cause 2 is EXCLUDED — measured through the Hosting API, no break-glass needed

   The issue names its own discriminator:

   Check the pod's restart count on the new ReplicaSet — still crash-looping means cause 2 and should be escalated.

   That reading is available without resolving anything from inside a pod. Deployments/memex-sample-20260910-1207-settled, a read-only Sample on Deployments/memex, executed 2026-09-10T12:05:44→12:05:49Z:

   Degraded: 2/2 ready, 0 restart(s), running meshweaver.azurecr.io/memex-portal-ai:3.0.0-ci.8238
     memex-portal-deployment-59d8fd958f-b2z6v  ready  …:3.0.0-ci.8238  restarts=0  health=Degraded
     memex-portal-deployment-59d8fd958f-sszkb  ready  …:3.0.0-ci.8238  restarts=0  health=Degraded
   

   Two replicas ready, zero restarts, both on the same image. Not crash-looping.

   The positive control: the operation that failed on 09-09 has since SUCCEEDED

   Absence of a recurrence would be weak on its own — a negative that could mean "fixed", "not attempted" or "not observed". It is not the only evidence here. Deployments/memex-restart-20260910-1140-activate-ai16 restarted this same deployment on 2026-09-10 at 11:33:56Z, and its rollout is what the 12:05Z sample above reads back: both replicas replaced and Ready on 3.0.0-ci.8238.

   So a fresh portal pod on memex booted through the Orleans MembershipTableManager lifecycle stage, on the same ConnectionStrings__orleans rendered from the same memex-portal-secrets, twice.

   That is what excludes cause 2. A connection string naming a host that does not exist — the default-derived-URI trap class — fails consistently: every pod, every boot, from the same secret. These pods resolved it and started. Cause 1 (a transient resolver failure inside the boot window) is what survives, exactly as the issue's own reasoning predicted from the wait-for-postgres initContainer having passed seconds earlier.

   What that leaves, and it is not nothing

   The cause is a CoreDNS/resolver blip outside this repository, so there is no code defect here to fix. What remains is a design question this repo does own, and I am not deciding it unilaterally:

   Should a silo fail closed and exit the container on a transient NAME-RESOLUTION failure at boot?

   The arguments are genuinely balanced, and the issue's own impact paragraph contains both:

   • For fail-closed: it is the documented posture, it is what DbVersionGate does, and a portal that starts without cluster membership is worse than one that does not start. The chart rolls surge-first (maxUnavailable: 0), so the previous ReplicaSet keeps serving and the cost of the refusal is a delayed roll, not an outage.
   • Against: a resolver blip is not a misconfiguration, and the failure mode it produces on the control instance is "safe but stuck" — the same shape flagged for DbVersionGate rollouts. A bounded retry at the resolution step specifically (not at connect, not at auth) would distinguish "the host does not exist" from "DNS did not answer this instant", which is precisely the distinction SocketException: Name or service not known with no TCP attempt already makes visible.

   🚨 That is a maintainer call about failure posture, not a patch. I am leaving this issue open on that narrowed question rather than closing it on "did not recur" — a cause identified is not a decision taken, and closing here would lose the question.

   Nothing was changed and no cluster command was run; every reading above came from Hosting/InstanceAction records through the Hosting API.
6. rbuergi commented on Sep 13, 2026

   

   rbuergi

   on Sep 13, 2026

   Contributor

   More actions

   Status 2026-09-13 — still the narrowed question, plus one chart fact worth having on the thread

   Not recurred, and not a fix. Since the 09-09 boot both control-instance replicas have re-booted through MembershipTableManager several times on the same ConnectionStrings__orleans — most recently 2026-09-12 16:49:06Z and 16:51:56Z on c84c6c055 (/health bake-report timestamps), 2/2 ready. Cause 2 stays excluded as measured on 09-10; cause 1 (a resolver blip inside the boot window) stays the reading.

   One fact for whoever takes the posture decision. The init container that is supposed to make "the database is reachable before the portal starts" deterministic gates on one host: deploy/helm/templates/memex-portal/deployment.yaml:104-111 runs until nc -z $MEMEX_HOST $MEMEX_PORT, and MEMEX_HOST is a config value. The orleans string is a secret (secrets.yaml:74-96): explicit (ConnectionStrings__orleans, "it always wins") or derived from ConnectionStrings__memex + Search Path=orleans. When it is explicit and names a different host from MEMEX_HOST — a dedicated server, or the same server under another name — that host is gated by nothing; when it is derived, nc -z on MEMEX_HOST only covers it if MEMEX_HOST and the connection string's Host= agree, which nothing checks. I could not read which shape memex uses (the values live in Key Vault), so this is a gap in the guard, not a claim about the incident.

   What is left is the same maintainer call as on 09-10 — fail-closed on a transient name-resolution failure at boot (today's posture; safe-but-stuck on the control instance) versus a bounded resolution retry that separates "no such host" from "DNS did not answer this instant". Not a code task until that is decided; leaving open on it. Read-only throughout.
7. rbuergi commented on Sep 13, 2026

   

   rbuergi

   on Sep 13, 2026

   Contributor

   More actions

   The chart fact from the 08:2xZ comment is now its own issue — #4173 — so it is not read as a claim about this incident: wait-for-postgres probes config.memex_portal.MEMEX_HOST while the boot opens ConnectionStrings__memex/__orleans from the secrets half, and nothing checks they name the same host. That hole is true whatever this incident's cause turns out to be; this issue stays on its own narrowed question (fail-closed vs a bounded retry at name resolution).
8. rbuergi commented on Sep 13, 2026

   

   rbuergi

   on Sep 13, 2026

   Contributor

   More actions

   There is no posture decision left to make — the retry belongs in the init container, and that half is now fixed (#4219)

   Re-measured today. Two separable questions were tangled here, and only one of them was ever a
   posture call.

   1. The half that was a real defect, and is fixable without deciding anything

   wait-for-postgres probed one host, taken from config.memex_portal.MEMEX_HOST — a config
   value — while this boot opened ConnectionStrings__orleans, a secret rendered from the Key
   Vault values half. grep -rn MEMEX_HOST deploy/helm/templates/ scripts/ returns four hits: the two
   probes and the two ConfigMaps that publish the value. Nothing compares them. That is #4173, and
   it is exactly the shape that produces this fingerprint: Init:1/1, the silo reaches
   RuntimeGrainServices, MembershipTableManager fails on a name that never resolved, the container
   exits.

   🚨 The reading that cost the time here is the init container's PASS. A gate that was never
   covering that host is indistinguishable, from the outside, from a resolver blip — which is why this
   issue's own "probable cause" leans to the blip at medium confidence and needs a cluster command to
   settle. A gate whose passing says nothing about the connection that failed is the same defect class
   as a CI job that skips wearing a green tick.

   #4219 closes it: both wait-for-postgres init containers now probe every distinct host:port
   the process will open, derived from the connection strings themselves rather than from a parallel
   config value, and they name each endpoint in the log as they wait. The strings move into one
   templates/_database.tpl so the portal Secret, the migration Secret and both probes cannot drift.
   A render that can derive no host now fails rather than emitting a probe that waits for nothing.

   Two negative controls, both measured, in the PR: invariant 16 fails on two combinations without
   the fix — including two plain replicas, KEDA off, which is the memex.systemorph.com shape, so
   a combination this repo already shipped was un-gated — and every rendered Secret is byte-identical
   before and after, so the refactor moved no connection string.

   2. The posture question — it dissolves, it is not decided

   Fail closed, or a bounded resolution retry in the silo? The question presupposes that the silo is
   where the waiting has to happen. It is not, and once the gate covers the right host it is the wrong
   place for it:

   	fail-closed silo (today, kept)	bounded retry in the silo (not added)
   transient resolver blip	pod exits, k8s restarts, maxUnavailable: 0 keeps the old ReplicaSet serving → a delayed roll	survives the blip — but only by making every failure slower
   permanently wrong host	fails immediately, loudly, at a named lifecycle stage	fails later, quieter, after N attempts that all had the same answer
   where a human looks	silo lifecycle error	a retry log, then the same silo error

   A retry inside the silo buys the first row and pays for it in the second — and the first row is
   already bought, for free, one layer down: the init container is the retry. until nc -z … waits
   as long as it takes, before the silo starts, and the pod shows Init:0/1 while it does. A blip now
   costs a slower pod rather than a crash loop; a wrong host now holds at Init:0/1 with the host
   named in the pod's own log, instead of passing and dying on an error that names nothing.

   So the silo's fail-closed stays correct because by the time it runs, every host it will open has
   already resolved and accepted TCP. A failure there is then a real answer rather than a race — which
   is what fail-closed is for. No timer, no watchdog, no bounded retry added to the silo. There is
   nothing here for a maintainer to arbitrate.

   Why this stays open for now

   Whichever of this issue's two candidate causes it actually was, #4219 changes the failure mode:

   • blip → the pod waits instead of dying;
   • wrong host → the pod holds at Init:0/1 naming the host, instead of passing and dying.

   But the cause was never established (the values live in Key Vault, which neither this issue's author
   nor I can read), and it has not recurred, so there is no measurement that would confirm the
   cure — only its absence, which proves nothing on a defect that occurred once. Closing it now would be
   settling by silence. It should be closed after #4219 ships, on the next roll of this instance, if the
   boot shape is what the change predicts.

   Read-only on the deployment; nothing was changed on the cluster or the control instance.
9. rbuergi commented on Sep 13, 2026

   

   rbuergi

   on Sep 13, 2026

   Contributor

   More actions

   The guard half is merged (#4219, 16e18828) — leaving this open only for the confirmation a single un-recurred incident cannot give

   #4173 is closed: both wait-for-postgres init containers now probe every distinct host the
   process will open, derived from the connection strings themselves rather than from
   config.<half>.MEMEX_HOST, and they name each endpoint in the log as they wait. The deploy.sh
   path was fixed in the same change — it used to hand the external connection string over by patching
   the Secrets after helm upgrade, so a value arriving after the render could not reach the init
   container the render produced; it now goes in through helm.

   As set out above, there is no posture decision left: the retry lives in the init container,
   where waiting is free and visible as Init:0/1, and the silo's fail-closed stays correct because
   by the time it runs every host it will open has already resolved and accepted TCP. Nothing was added
   to the silo — no timer, no watchdog, no bounded retry.

   Why this is not being closed with it. The cause here was never established — the two candidates
   (a resolver blip in the boot window vs. a ConnectionStrings__orleans naming a host that does not
   exist) are separated only by values held in Key Vault, and it has not recurred. An absence of
   recurrence on a defect that occurred once proves nothing, so closing now would be settling by
   silence. The next roll of this instance is the measurement: with the fix in, a blip costs a slower
   pod and a wrong host holds at Init:0/1 naming the host — either is a different, diagnosable
   shape from Init:1/1 followed by MembershipTableManager dying on an error that names nothing.
   Close it then.
10. rbuergi commented on Sep 14, 2026

    

    rbuergi

    on Sep 14, 2026

    Contributor

    More actions

    Measured on the incident node: it recurred 67 more times, it crash-looped, and this issue was never told

    Read from Admin/_LogIncident/0b1877ae6c51353f on memex.systemorph.com, 2026-09-14 ~06:1xZ.

    field	value
    occurrences	69
    firstSeen	2026-09-09T02:36:24Z
    lastSeen	2026-09-12T07:59:49Z
    pods	3 distinct (…-85b6cfc9c9-p5d7p, …-69999d86b8-8hsbs, …-5b967589bf-5j6g2)
    occurrencesAtLastComment	2

    1. The "transient resolver blip" reading is falsified for the 2026-09-12 episode

    The issue rates cause 1 (a CoreDNS/resolver blip inside the boot window) above cause 2 (a wrong or
    default-derived ConnectionStrings__orleans), and offers a discriminator: "Check the pod's restart
    count on the new ReplicaSet — still crash-looping means cause 2 and should be escalated."

    The samples answer that without needing the cluster. Five bursts, all on one pod
    (memex-portal-deployment-5b967589bf-5j6g2), each a MembershipTableManager failure immediately
    followed by Lifecycle start canceled:

    07:35:53.413 / 07:35:53.420
    07:41:52.185 / 07:41:52.192
    07:47:53.130 / 07:47:53.137
    07:53:47.657 / 07:53:47.663
    07:59:49.954 / 07:59:49.961
    

    Five failed boots at a ~6-minute cadence on a single pod is the crash loop the issue names as
    the escalation trigger. A transient resolver failure does not repeat on a fixed interval five times
    against one pod while two sibling pods run. So for that episode the evidence points at cause 2,
    not cause 1 — and the issue's stated confidence should be read the other way round.

    2. It has not recurred since — so the bad pod is gone, not the cause established

    Nothing since 2026-09-12T07:59:49Z, roughly 46 hours. The three pod hashes show the deployment
    rolled through. So whatever carried the bad value went away with that ReplicaSet; that is a
    disappearance, not a diagnosis, and it would return on the next re-render of the same values.

    This is why I am not closing it. Closing on "no recurrence in 46 h" would repeat the #1391
    mistake — a symptom's absence read as a cause's absence. What would actually close it is confirming
    which of the two ConnectionStrings__orleans shapes that pod received; per AGENTS.md that is a
    Hosting-API read on Deployments/memex, and the values half lives in Key Vault (helm-values-memex),
    which no repository holds — so it needs the maintainer, not an agent.

    3. 🚨 The reason this issue still reads "a single failed boot, no recurrence"

    occurrencesAtLastComment: 2 against occurrences: 69. This issue stopped being updated after the
    second occurrence of sixty-nine. The incident node says why, in its own error field:

    GitHub App installation-token request failed (401): {"message":"A JSON web token could not be decoded", … }
    

    So the incident could not comment on its own ticket. Everything after occurrence 2 — including the
    entire crash-loop episode that settles the cause — never reached this thread. The "every red log
    gets exactly one ticket" guarantee held for filing and then silently stopped holding for
    updating, which is the half that carries the evidence.

    That is not specific to this incident and I am tracking it separately rather than here. Noting it on
    this thread because it explains why the issue body's impact assessment ("a single failed boot … no
    recurrence within the watch window") is not wrong so much as frozen at occurrence 2, and anyone
    triaging from the body alone would rate this far lower than it deserves.

    ---

    🤖 Generated with Claude Code
11. rbuergi commented on Sep 14, 2026

    

    rbuergi

    on Sep 14, 2026

    Contributor

    More actions

    The mechanism that froze this ticket is fixed — but the fix is roll-gated, so this thread stays manually current until then

    Root cause of this issue's body reading "occurrence 2 of 69" was MeshWeaver.Plugins#1836: a log incident's comment debt was settled only by a new occurrence, so a fault that stopped left its ticket frozen at whatever count the last successful comment saw. Fixed in Plugins#1841, merged 7222fef934 — a 15-minute reconcile over incidents where Occurrences > OccurrencesAtLastComment and CommentDue holds, on an injectable scheduler, with the 401 that created the debt deliberately not retried (it recovered on its own; the debt was the defect).

    It is not running anywhere yet. The reconcile lives in the MeshWeaver.Observability module on the portal, and every portal is Held at 627fb3cd pending a Roll (the delivery freeze — core #4293 + #4295 unfreeze the seal; the Roll is the maintainer's). Until a portal carries 7222fef934, incident 0b1877ae6c51353f keeps its debt of 67 and this thread stays as I left it above — current by hand, not by the automation.

    When the roll lands, the observable that says it worked: this thread receives one automated comment reading roughly "Still occurring: 67 more since the last update (69 total)", and occurrencesAtLastComment on the incident node moves from 2 to 69. If that comment never arrives after the roll, #1836's fix did not reach the running module, and that is a new finding.

    ---

    🤖 Generated with Claude Code
12. systemorph-com commented on Sep 14, 2026

    

    systemorph-comBot

    on Sep 14, 2026

    Contributor

    More actions

    Still occurring: 69 more since the last update (71 total).

    • Last seen: 2026-09-14 20:15:23Z
    • Pods: memex-portal-deployment-85b6cfc9c9-p5d7p, memex-portal-deployment-69999d86b8-8hsbs, memex-portal-deployment-5b967589bf-5j6g2, memex-portal-deployment-77476b68-l4gw8

    Recent log lines

    2026-09-12 07:59:49Z memex-portal-deployment-5b967589bf-5j6g2 fail: Orleans.Runtime.SiloLifecycleSubject[100450]
          Lifecycle start canceled due to errors at stage 'RuntimeGrainServices (8000)'.
          Npgsql.NpgsqlException (0x80004005): Name or service not known
           ---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
             at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
             at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
             at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
             at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
          --- End of stack trace from previous location ---
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
             at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
          --- End of stack trace from previous location ---
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, Canc…[truncated]
    2026-09-14 20:15:23Z memex-portal-deployment-77476b68-l4gw8 fail: Orleans.Runtime.SiloLifecycleSubject[100450]
          'MembershipTableManager' failed to start due to errors at stage 'RuntimeGrainServices (8000)'.
          Npgsql.NpgsqlException (0x80004005): Name or service not known
           ---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
             at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
             at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
             at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
             at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
          --- End of stack trace from previous location ---
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
             at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
          --- End of stack trace from previous location ---
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boo…[truncated]
    2026-09-14 20:15:23Z memex-portal-deployment-77476b68-l4gw8 fail: Orleans.Runtime.SiloLifecycleSubject[100450]
          Lifecycle start canceled due to errors at stage 'RuntimeGrainServices (8000)'.
          Npgsql.NpgsqlException (0x80004005): Name or service not known
           ---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
             at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
             at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
             at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
             at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
          --- End of stack trace from previous location ---
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
             at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
          --- End of stack trace from previous location ---
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, Canc…[truncated]
    
13. rbuergi commented on Sep 14, 2026

    

    rbuergi

    on Sep 14, 2026

    Last edited by rbuergi

    Contributor

    More actions

    The 2026-09-14 20:15Z recurrence, measured — and it is the same defect as 09-09, in code

    Read-only throughout, through the Hosting API on memex.systemorph.com and the repositories; no cluster command, no InstanceAction written, the record untouched.

    1. What rolled the control instance

    Ops/Actions/reconcile-memex-20260914-nav-rail — a record-driven Reconcile (requested 20:13:57Z, operator image hosting-operator:4c99ec2), whose step 4 "Re-apply the record" ran

    helm upgrade memex /opt/hosting/chart --install --namespace memex --create-namespace -f /tmp/hosting-values.yaml --set ingress.tlsSecret=memex-tls --set …MEMEX_DATABASENAME=memex
    

    — one values file, logged as 10306 bytes, rendered from the record — producing helm revision 55 at 20:14:17Z. The surge pod memex-portal-deployment-77476b68-l4gw8 failed MembershipTableManager at 20:15:23Z and kept failing (incident node 0b1877ae6c51353f: 85 occurrences by 20:30:36Z, all on that pod). On 09-09 it was revision 44, from Deployments/memex-reconcile-20260909-final — also a Reconcile — and the portal recovered only because --atomic rolled it back onto the lane's revision 43.

    2. The rendered host vs the healthy pods' host

    The record's render is secret-free by construction (HelmValues header: "the Key Vault half is layered separately and this file never names a secret's value"). The chart's memex-portal-secrets renders from secrets.memex_portal.*, and for memex those values live only in Key Vault helm-values-memex — the record's own vaultValuesKeys lists ConnectionStrings__orleans among the 11 keys that Secret carries. With no secrets: block, memex.orleansConnectionString in deploy/helm/templates/_database.tpl falls through to its default:

    Host=memex-postgres-service;Port=5432;Username=postgres;Password=;Database=orleans
    

    memex-postgres-service is the chart's in-cluster Service, rendered only under postgres.enabled: true; every AKS namespace runs false. So the host in the new pod's ConnectionStrings__orleans was a name that exists nowhere — SocketException: Name or service not known from Dns.GetHostEntryOrAddressesCore, before any TCP attempt, exactly the fingerprint. The healthy pods were last rendered by the config repo's helm-release.yml lane, which runs helm upgrade … -f vault-values.yaml -f <overlay> with vault-values.yaml = helm-values-memex, so their Secret carries the flexible server. Every Roll and Restart between the two incidents was healthy because neither touches the Secret — which is what made "did not recur" read as a resolver blip.

    Why ConnectionStrings__memex did not fail the same way: the record's keyVaultSecretClasses[0] (memex-kv, later in envFrom) shadows it with the vault object memex-connectionstring. Nothing shadows the orleans string anywhere in the fleet. Why the init container passed: operator image 4c99ec2 predates #4219 (16e18828 is not its ancestor), so its baked chart still probed config.MEMEX_HOST, which the record renders correctly (memexaks-pg.postgres.database.azure.com).

    The other session's remediation confirms it from the operational side: Ops/Actions/helm-memex-20260914-deploy-8612-b — "Deploy memex through the helm-release lane — restore the Key Vault connection strings helm revision 55 dropped" — dispatched Memex run 34901699934 at 21:58Z.

    3. The code path, and the defect

    deploy/aks/operator/bin/hosting-deploy (core, baked into the operator image): files=( -f "$values" ) — it never read the vault half, while Hosting/Configuration.md documents that half as "layered FIRST at deploy". Two deploy paths, two answers to one question. Not the record, not Key Vault, not DNS: the operator path replaced layer 2 of the pod's environment with chart defaults on every record-driven deploy of an instance on an external database.

    The fix (root cause, both halves)

    • Core #4356 — hosting-deploy --vault <name> layers helm-values-<release> FIRST, then the record's render (the lane's order; the record still wins on every key it declares); read by the operator's own identity, never echoed, size-only in the log; required where the flag is given (a vault holding none refuses before helm, naming helm-release capture). And the chart refuses to invent the host: memex.orleansConnectionString now fails the render on an external database when neither connection string is supplied, instead of manufacturing memex-postgres-service; check-chart-invariants.sh carries a REFUSALS control with a fixture of exactly this render, which must fail to template. Operator suite 464/464 (11 new; 9 red against the pre-fix script); chart harness 8 combinations + 1 refusal (the refusal reds against the pre-fix template).
    • Plugins MeshWeaver.Plugins#1892 — the plan passes --vault <keyVault> on Reconcile and Provision exactly when the record declares vaultValuesKeys; a record that declares the half but names no keyVault is refused by name; a provisioned instance (vaultValuesKeys: [], connection string on a CSI class) keeps its one-layer shape. Hosting 1.20.

    Outcome (2026-09-15 02:2xZ)

    • Both halves are merged: core fix(#3780): a record-driven deploy layers the Key Vault values half, and the chart refuses to invent a database host #4356 → b63310a7 (queue-merged 22:2xZ), MeshWeaver.Plugins#1892 → d9d314a5 (Hosting 1.20, merged 22:41:48Z; the two new cases are registered in the Tests area, so they run rather than only compile).
    • The operator image carrying the fix is published: the Hosting Operator lane on b63310a7 (run 34904032153) pushed meshweaver.azurecr.io/hosting-operator:b63310a (+ :main) and verified both tags in the registry.
    • The control instance is green on the lane render. Ops/Actions/helm-memex-20260914-deploy-8612-b dispatched Memex run 34901699934 ("restore the Key Vault connection strings helm revision 55 dropped, and apply the committed pin 3.0.0-ci.8612"): its Upgrade the release step succeeded at 22:2xZ with -f vault-values.yaml -f <overlay>; only the bounded Observe the rollout until it completes step ran out — the rollout itself completed: https://memex.systemorph.com/api/version answers b8143398a7 (set 8612's core commit) on every probe, and incident 0b1877ae6c51353f recorded nothing after 21:59:25Z (125 occurrences total; the last ones on …-78d795bf59-87c7x, the rollback-to-8411 pod, which still carried revision 55's Secret — the same mechanism, not a third cause).

    That is the positive control the diagnosis predicted: the same image that crash-looped under the record-only render boots under the render that carries the vault half. Closing as completed.

    Still owed, outside these two repositories (not done here — read-only on the record)

    1. Deployments/memex → operator.image: hosting-operator:4c99ec2 → hosting-operator:b63310a (Systemorph/Memex, mesh/Deployments/memex). Until that pin moves, a record-driven Reconcile of memex runs the old script; once Hosting 1.20 is live on the control instance the plan emits --vault and the old image refuses with unknown argument '--vault' — loud, where it used to be a silent wrong render. Neither intermediate state can reproduce this incident.
    2. Hosting 1.20 live on the control instance (module landed + restart) — that is what makes the plan emit --vault.
    3. The measurement that would reopen this: the next record-driven Reconcile of memex on hosting-operator:b63310a must show vault helm-values-memex (N bytes) — layered FIRST in the operator job log and a new pod passing MembershipTableManager. A pod dying there again on that image is a new finding.
    4. Stated in the doc and deliberately open: with the wait-for-postgres gates only MEMEX_HOST — an explicit ConnectionStrings__orleans naming another host is gated by nothing #4173 probe on the new image, an instance with no vault half (every provisioned record) holds at Init:0/1 naming memex-postgres-service — a diagnosable stall, not a crash loop; the record path should feed databaseConnectionSecret into secrets.<half>.ConnectionStrings__memex as well.
14. rbuergi

    closed this as completedon Sep 15, 2026
15. systemorph-com

    reopened this on Sep 17, 2026
16. systemorph-com commented on Sep 17, 2026

    

    systemorph-comBot

    on Sep 17, 2026

    Contributor

    More actions

    Reopened — this fault is happening again.

    Still occurring: 54 more since the last update (125 total).

    • Last seen: 2026-09-14 21:59:25Z
    • Pods: memex-portal-deployment-85b6cfc9c9-p5d7p, memex-portal-deployment-69999d86b8-8hsbs, memex-portal-deployment-5b967589bf-5j6g2, memex-portal-deployment-77476b68-l4gw8, memex-portal-deployment-78d795bf59-87c7x

    Recent log lines

    2026-09-14 21:53:30Z memex-portal-deployment-78d795bf59-87c7x fail: Orleans.Runtime.SiloLifecycleSubject[100450]
          Lifecycle start canceled due to errors at stage 'RuntimeGrainServices (8000)'.
          Npgsql.NpgsqlException (0x80004005): Name or service not known
           ---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
             at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
             at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
             at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
             at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
          --- End of stack trace from previous location ---
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
             at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
          --- End of stack trace from previous location ---
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, Canc…[truncated]
    2026-09-14 21:59:25Z memex-portal-deployment-78d795bf59-87c7x fail: Orleans.Runtime.SiloLifecycleSubject[100450]
          'MembershipTableManager' failed to start due to errors at stage 'RuntimeGrainServices (8000)'.
          Npgsql.NpgsqlException (0x80004005): Name or service not known
           ---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
             at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
             at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
             at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
             at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
          --- End of stack trace from previous location ---
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
             at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
          --- End of stack trace from previous location ---
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boo…[truncated]
    2026-09-14 21:59:25Z memex-portal-deployment-78d795bf59-87c7x fail: Orleans.Runtime.SiloLifecycleSubject[100450]
          Lifecycle start canceled due to errors at stage 'RuntimeGrainServices (8000)'.
          Npgsql.NpgsqlException (0x80004005): Name or service not known
           ---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
             at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
             at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
             at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
             at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
          --- End of stack trace from previous location ---
             at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
             at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
          --- End of stack trace from previous location ---
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
             at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, Canc…[truncated]
    
17. rbuergi commented on Sep 17, 2026

    

    rbuergi

    on Sep 17, 2026

    Contributor

    More actions

    Re-closed — the reopen above was on stale evidence (#4577).

    The recurrence bot reopened this at 2026-09-17T07:51:32Z under the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:

    the bot's Last seen:	the close it reverted
    2026-09-14 21:59:25Z	2026-09-15T02:25:11Z

    So the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix: LogIncidentFiler.Reopen reopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen require incident.LastSeen > issue.ClosedAt).

    This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.

    _{One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.}
18. rbuergi

    closed this as completedon Sep 17, 2026