Repository navigation
FleetWatch sweep stalled ≥37 min on memex-cloud, leaving stale Ops/Status readings rendering as live (recurrence) #5259
Description
What is failing
The FleetWatch sweep stopped completing passes on the memex-cloud control instance for at least 37 minutes: at 2026-09-22 12:35:45Z the heartbeat's census found 0 of 1 (and 0 of 4 in the same burst) fleet readings current, with client-b, memex-cloud and build all ~37 min past the staleAfter deadline their own writer declared (last written 11:57:56–11:57:57Z). Every Ops/Status reading on the Fleet Console therefore rendered frozen data as live — the exact failure mode this guard exists to catch, and it did. The client-a reading named in the census (3 days old, no cadence) is a separate, known and documented artifact: its Hosting/Deployment record was deleted, so nothing writes or retires its status node ("A status node outlives the record that justified it" in /Hosting/FleetWatch).
Probable cause
Cause unclear; evidence points at the sweep on the control instance not completing passes (a pass faulted, hung, or never re-armed) rather than at the heartbeat — the heartbeat is behaving as designed. Confidence: high that the sweep was silent for ≥37 min; low on why. The log carries no exception and no stack. Prior art: MeshWeaver#4471 (total stop via a config lambda that never ran) and MeshWeaver#4611 (the earlier manifestation of this same fingerprint, Sep 17–19). The pass record at {space}/Watch/fleet-watch and the version history of Ops/Status/client-b around 11:57Z–12:35Z will separate hung from stopped from alive-and-blind.
Impact
A single detection burst at 12:35:45 (3 log lines within 60 ms on one portal pod, memex-portal-deployment-7cc85f47c-9j7hd). During the blind window the fleet's observability verdict was wrong — history rendered as live on the Fleet Console — but the watcher outage was itself detected and alarmed, so no condition went unreported by design. This is a recurrence of the intermittent sweep-stop pattern already seen Sep 17–19 and 12:33–12:34 the same day (across all three sibling pods of the same ReplicaSet), so it is repeating, not a one-off.
Where to look
/Hosting/FleetWatch— the sweep's design, budgets (PassBudget,WriteTimeout), and known blind spots./Hosting/FleetWatchPass//Hosting/PassLifecycleRecord— pass records showing how far the last pass got.FleetWatchHeartbeatinMeshWeaver.SelfUpdate.Aks(compiled half) — the reporting component, working as intended.
Not filed separately: the sibling incident Admin/_LogIncident/4467a01247b67c2d covers the same 2026-09-22 12:33–12:34 window on the other pods of this ReplicaSet; its draft failed to file, so this ticket covers the whole event. The client-a frozen-status cleanup (retire {space}/Status/{id} when its Hosting/Deployment record is deleted) is an open documented limitation, not re-filed here.
Evidence
| Fingerprint | 1864b6f67972b4fd |
| Category | MeshWeaver.SelfUpdate.Aks.FleetWatchHeartbeat |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-7cc85f47c-9j7hd |
| Occurrences | 3 |
| First seen | 2026-09-22 12:35:45Z |
| Last seen | 2026-09-22 12:35:45Z |
| Routing | not determined — no configured route matches the category MeshWeaver.SelfUpdate.Aks.FleetWatchHeartbeat. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 12:35:45Z memex-portal-deployment-7cc85f47c-9j7hd fail: MeshWeaver.SelfUpdate.Aks.FleetWatchHeartbeat[0]
[FleetWatchHeartbeat] 0 of 1 fleet reading(s) are current, 1 declare a cadence — and 1 past the deadline its own writer declared (client-b last written 37 minute(s) ago). Those readings are HISTORY rendering exactly like live ones. From a status node alone a deliberate set-aside (a record that is Planned, Provisioning or Suspended) and an abandoned reading (a record that has been deleted, so its row left the roster) are indistinguishable — this names them rather than guessing which
2026-09-22 12:35:45Z memex-portal-deployment-7cc85f47c-9j7hd fail: MeshWeaver.SelfUpdate.Aks.FleetWatchHeartbeat[0]
[FleetWatchHeartbeat] the fleet watch is OVERDUE: its newest observation is client-b at 2026-09-22 11:57:57Z, 37 minute(s) ago, and it promised another by 2026-09-22 12:12:57Z. Every Ops/Status reading is now HISTORY — the fleet is not being observed, and a frozen reading renders exactly like a live one
2026-09-22 12:35:45Z memex-portal-deployment-7cc85f47c-9j7hd fail: MeshWeaver.SelfUpdate.Aks.FleetWatchHeartbeat[0]
[FleetWatchHeartbeat] 0 of 4 fleet reading(s) are current, 3 declare a cadence — and 3 past the deadline its own writer declared (build last written 37 minute(s) ago, memex-cloud last written 37 minute(s) ago, client-b last written 37 minute(s) ago); 1 declaring no refresh cadence at all and older than anything in this space promises (client-a last written 3 day(s) ago). Those readings are HISTORY rendering exactly like live ones. From a status node alone a deliberate set-aside (a record that is Planned, Provisioning or Suspended) and an abandoned reading (a record that has been deleted, so its row left the roster) are indistinguishable — this names them rather than guessing which
Re-addressed by the current identity function: this incident inherited 93710ed097873d0b (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/1864b6f67972b4fd. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 22, 2026
This ticket is in the wrong repository, and the cluster has TWO roots, not one
1. The subject is entirely in
MeshWeaver.Plugins. Core's tree contains no.csimplementing any of it —grep -rl FleetWatchHeartbeat --include=*.csover core matches nothing, and only doc pages mentionFleetWatch. The implementation issrc/MeshWeaver.SelfUpdate.Aks/FleetWatchHeartbeat.cs(the compiled guard that emits the OVERDUE line),FleetWatchHealthCheck.cs(the/healthfleet_watchentry), andHosting/Issue/Source/FleetWatch.cs— the sweep itself, an in-mesh source compiled at runtime, which is why nodotnet buildin either repo ever sees it. Denominator: 2 of 2 core tickets in this cluster carry the banner "Routing: not determined … This repository is the configured fallback", and the emitting logger's assembly is a Plugins assembly. Both belong in Plugins beside #2132.2. A new measurement neither ticket contains: the sweep's hub re-activates, repeatedly. Read live off
Ops/Watch/fleet-watch: 14 versions in 27 minutes, every onepass 1, with two consecutive versions carryingarmedAt19 s apart. That is not an interpretation — inAddFleetWatch,armedAtis a local captured in the initialization closure and each pass number isliveness.Value.Passes + 1off a seed of 0, sopass 1with a fresharmedAtis a new activation by construction (the code comment says so, citing #2132). So: ≥14 re-activations in 27 minutes of the per-node hub ofHosting/PlatformBuildInbox— the node whose own description calls it "the always-activated hub … kept activated for the life of the portal (#777)".AddFleetWatch()has exactly two hits in the Plugins tree: its declaration and one call site, insidecontent.configurationof that node's JSON.3. It does not mean "no activation ever survives".
Ops/Status/memex-cloudat 18:34Z readpass 4, 20 minutes after thepass 1series. Those reconcile rather than conflict: the pass number is per-activation and the interval is 300 s, so an activation surviving ~20 min reaches pass 4 exactly. The honest reading is re-activation is frequent but not uniform.4. Two roots, and this refutes the single-root grouping.
memex-cloud) and Plugins#2132 (~15 min,memex). Strongest duplicate candidate in the cluster: same shape, different namespaces, 5 days apart, different fingerprints, but both core tickets carry the footer "inherited93710ed097873d0b(Plugins#1796)", which is PostgreSqlCrossSchemaQueryProvider surfaces transient Npgsql errors (EndOfStream, timeout, 40P01 deadlock) and blanks dashboard areas #2132's own fingerprint. The re-activation churn above is a strong candidate cause.memex at 2026-09-19 08:47:41Z) reported from two different pods of one ReplicaSet, 21 min then 1438 min overdue. Re-activation churn cannot produce that — a fresh activation re-arms and would emit a pass. Fleet watch stops observing the fleet, freezing every Ops/Status reading as silently stale history #5011's own body points at a shared blocker instead (aHosting/PlatformBuildshub read; a cross-hub write conflict atOps/Issue/build__Unobserved). Fingerprints differ:ab4f533a0d848402vs1864b6f67972b4fd.5. Plugins#2258 is not an occurrence — it is the merged instrument built to tell slow from gone, and it does not belong in the bug queue. 🚨 It merged 2026-09-22T14:15:04Z (
938ca3284f55), and every occurrence in all three tickets pre-dates it. So the whole cluster's evidence was gathered blind to the discriminator, which is a caveat on root (B): B could be A seen without the instrument.What is NOT established
Why the hub re-activates — none of the obvious candidates was excluded: a subscription re-creating the activation on each emission, a write to the node the sweep itself owns re-entering through its own-node subscription, a scheduled dispose, a timer re-armed from a handler that also disposes. The cheapest unrun check is one read: does
SpaceOf(hub)put{space}/Watch/fleet-watchandOps/Status/*inside the sweep's own hub's subtree? If it does, the own-node re-entry candidate is live. Also unread: #5011's 3 comments and #2132's 14 — the latter is the likeliest place this work already exists. No control was written and nothing was built or fixed.🤖 Generated with Claude Code
rbuergi commentedon Sep 22, 2026
Part of this is the same root as #5011, fixed in Systemorph/MeshWeaver.Plugins#2287 — and part is still open
Read-only on memex.meshweaver.cloud, 2026-09-22 ~19:15Z.
What #2287 fixes. Fresh activations of the sweep on this portal sometimes conclude pass 1
inactive — 0 of 0 row(s)(Ops/Watch/fleet-watchv63 at 18:39:25Z, v66 at 18:50:26Z) while pass 2 of the same activation opens over 4 rows (v67). The roster read carried no identity on a hub whose ambient identity is nobody; read as nobody, RLS answers empty and the pass writes nothing. Details and the negative control are on #5011 and the PR.What it does NOT fix, and this ticket's 45 minutes may be it.
Ops/Status/buildon this portal has no version between 11:57:55Z and 12:42:53Z today; the pass record did not exist yet (its first version is 16:02:53Z), so whether that window was an inactive activation or no activation at all cannot be read back. Worth recording for the next occurrence: fresh arming instants onOps/Watch/fleet-watchcluster on writes to theHostingpackage record (Hostingv111/v112 at 17:43:47Z/17:44:49Z and 18:37:21Z; re-arms at ~17:45Z and 18:39:25Z), i.e. package installs recycleHosting/PlatformBuilds; and12:40:38Z/12:40:55Z(v107/v108) sit two minutes before the end of this ticket's gap. That is a correlation, not a mechanism.Next discriminator (after #2287 rolls): on the next OVERDUE, read
Ops/Watch/fleet-watch— a last recordclosedwith no successor means no activation existed; a run ofinactivemeans #2287 did not cover it.🤖 Generated with Claude Code
rbuergi commentedon Sep 22, 2026
Folding into #5011 — same defect, and this issue's own title calls it a recurrence. One tracking issue for one fault.
The measurement is recorded on #5011: the fix (#5152,
33e3cfa40b) is an ancestor of3da3423c, the commit both portals serve now, and the watch is cadencing correctly (all four live instances written 20:52:15–20:52:21Z, memex version 2536 → 2540 over 21 minutes).🚨 Note for anyone scoring the fix: this issue's 12:35Z stall cannot be used as evidence either way. It happened after #5152 merged but before today's pods started at 17:28Z, so it ran on an image I did not identify. #5011 stays open for a real observation window.