Repository navigation
DynamicTypePreWarmerHostedService fails readiness when Admin/Build hub is slow to respond at startup #1635
Description
What is failing
The DynamicTypePreWarmerHostedService hosted service fails its NodeType warm-up sweep and refuses readiness when the Admin/Build hub does not respond to a subscription request within 60 seconds. The pod cannot join the service, stalling the rollout on the previous image.
Probable cause
A SubscribeRequest targeting the Admin/Build hub timed out after 60 seconds with System.TimeoutException — the target hub was either unavailable, slow to initialize, or the request was undeliverable during pod startup. Given this is a single occurrence on one pod during the deployment window, this looks like a startup race condition: the warm-up sweep starts before the Admin/Build hub grain is fully available.
Confidence: High — the exception type and message are unambiguous about the timeout and the target. The low occurrence count (1) and the pod-level isolation point strongly at a transient race rather than a code defect in the pre-warmer itself.
Impact
Single pod (memex-portal-deployment-f96d46ddf-mrgqf) failed to pass readiness during a single deployment attempt. The rollout stalls on the previous image. Lazy compilation still works, so existing running pods are unaffected. This is a low-severity incident — one pod, one event, self-clears on restart — but repeated occurrences across pods would block a full rollout.
Where to look
MeshWeaver.Hosting.DynamicTypePreWarmerHostedService(namespaceMeshWeaver.Hosting)- The warm-up logic that subscribes to the
Admin/Buildhub for NodeType verification - Consider retry logic or a backoff before marking the sweep as faulted; the
Admin/Buildhub may not be fully initialized when the pre-warmer starts.
Evidence
| Fingerprint | 3cd719ae33cd1e19 |
| Category | MeshWeaver.Hosting.DynamicTypePreWarmerHostedService |
| Severity | Critical |
| Exception | System.TimeoutException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-f96d46ddf-mrgqf |
| Occurrences | 1 |
| First seen | 2026-08-15 07:44:25Z |
| Last seen | 2026-08-15 07:44:25Z |
Recent log lines
2026-08-15 07:44:25Z memex-portal-deployment-f96d46ddf-mrgqf crit: MeshWeaver.Hosting.DynamicTypePreWarmerHostedService[0]
DynamicTypePreWarmer: REFUSING READINESS — the warm-up stream FAULTED, so this pod has NOT verified that its NodeTypes build on this image. A sweep that errored is not a sweep that passed. The rollout will stall with the previous image still serving; a restart re-runs the sweep, so a transient cause clears itself. Lazy compilation still works, so if this deployment must roll forward regardless, set 'PreWarm:AllowUnprovenBake'=true — that serves on an UNPROVEN bake and says so in the health payload.
System.TimeoutException: No response received in hub cache/mQTsN2ScEEKbD8eWzwc_YA within 00:01:00 for request SubscribeRequest (id=h_N2s7uJVEu2PJxmsF--Cw) → target Admin/Build. The request may have been undeliverable or the target hub was not found.
Opened automatically from Admin/_LogIncident/3cd719ae33cd1e19. Recurrences are folded into this issue rather than opening new ones.
Activity
Merge pull request #2035 from Systemorph/fix/1635-prewarm-coordinatio…
systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 9 more since the last update (10 total).
memex-portal-deployment-f96d46ddf-mrgqf,memex-portal-deployment-5d6744fcc-jf7x8,memex-portal-deployment-68cd75c658-x588b,memex-portal-deployment-7b59b7d657-7d5t6,memex-portal-deployment-cb69ff6-zx7l4,memex-portal-deployment-7f9d94b9d4-vjqv4,memex-portal-deployment-565b67998c-spnqv,memex-portal-deployment-565b67998c-wnpvl,memex-portal-deployment-65fbcf6cd5-9t64t,memex-portal-deployment-5f9479f848-57t9hRecent log lines
rbuergi commentedon Sep 17, 2026
Closing — fixed twice over (#2035, then #3457), both live on the running portals, and 20 days silent
1. The original fix is core PR #2035 — "fix(#1635): a coordination node that is still activating is not a failed bake" — merged
3734a26067a5on 2026-08-22T06:24:45Z, one second before the close this reopen reverted. Its rule is still in the code:src/MeshWeaver.Hosting/DynamicTypePreWarmerHostedService.cs:493— "Name the ABSENCE of a verdict apart from a bad one (#1635). Both refuse readiness … but an unreachable coordination node is a startup race that a restart clears, while a faulted sweep is usually this image" — withBuildProtocolDriver.DescribesUnreachableCoordinationandRetryUnreachableCoordination(BuildProtocolDriver.cs:52, 795, 870, 933).2. The residual — a rollout actually held by that silence — was fixed by core PR #3457 (#3404), merged
87f929d088on 2026-09-06T17:48:01Z: when theSubscribeRequesttoAdmin/Buildgoes unanswered, the pre-warmer now reads the durable witness and honours a GO that was already written, withUndeterminedkept distinct fromNoGo("THE THIRD STATE, and why this door needed a third one (#3404)",BuildProtocolDriver.cs:184). RESTcomparesays that commit is an ancestor of both images running today (c84c6c05…on memex.meshweaver.cloud,afde4eab…on memex.systemorph.com —behind_by = 0).That ordering also explains the one occurrence after #2035: the reopen's own
Last seen:is 2026-08-28 21:05:16Z, which is after #2035 but before #3457 — the class #3457 closed. Nothing in the 20 days since.3. The reopen itself is stale-wave. Reopened
2026-09-17T07:52:53Zunder "this fault is happening again" against aLast seen:from 2026-08-28 — the pattern measured in #4577 (counter moved, clock did not); reopen rule fixed in Systemorph/MeshWeaver.Plugins#2014.Closing. A future reopen needs an occurrence newer than this close.