Repository navigation
A CreateNodeRequest addressed to the sending hub itself is never answered (nodeops, empty queue, RunLevel=Started) — activity tracking and every downstream read then time out #1174
Description
What is failing
The MeshWeaver.Graph.ActivityTracking service failed to track user activity because the underlying mesh hub for the _UserActivity node never delivered its initial state within the 30-second timeout.
Probable cause
Two occurrences hit the same user (rbuergi) on different paths (Underwriting and rbuergi) in a 1-second window on a single pod, then stopped entirely. The exception message lists four likely causes:
- RLS silently rejected the prior
CreateNodecall - The node path is misspelled or points at a namespace no
NodeTypeclaims - The node was deleted between create and update
- The per-node hub activated but its
MeshDataSourcedidn't load the node from persistence
The pattern (2 bursts, same user, same pod, no recurrence) most strongly suggests cause (1) — the user rbuergi may lack permission to create _UserActivity nodes under their path — or a transient storage hiccup (cause 4).
Confidence: Low–medium — the evidence is thin and the root cause could be any of the four listed possibilities.
Impact
Very low. Two activity-tracking updates silently failed for one user on one pod within a single second, with no recurrence in the hours since. No customer-facing feature is blocked; activity history for those two paths/events was simply not recorded.
Where to look
- Component:
MeshWeaver.Graph.ActivityTracking— specifically the code path that creates or updates a_UserActivitynode and awaits its hub initialization. - Check: Whether the
_UserActivitynode type is registered in the mesh and claims therbuergi/path namespace. - Check: RLS rules applied to the
memex-cloudnamespace — does the activity-tracking service identity (orrbuergithemselves) have permission to create nodes underrbuergi/_UserActivity/? - Check: The 30-second hub-initialization timeout value and whether the persistence layer (
MeshDataSource) was experiencing latency at the time.
Evidence
| Fingerprint | 7473a637daf1e169 |
| Category | MeshWeaver.Graph.ActivityTracking |
| Severity | Error |
| Exception | System.TimeoutException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-6bd9c46f8b-z9t6r |
| Occurrences | 7 |
| First seen | 2026-08-10 13:49:50Z |
| Last seen | 2026-08-10 14:20:27Z |
Recent log lines
2026-08-10 13:49:50Z memex-portal-deployment-6bd9c46f8b-z9t6r fail: MeshWeaver.Graph.ActivityTracking[0]
Failed to track activity for user=rbuergi path=Underwriting
System.TimeoutException: Update aborted: no initial state arrived for 'rbuergi/_UserActivity/Underwriting' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence.
2026-08-10 13:49:51Z memex-portal-deployment-6bd9c46f8b-z9t6r fail: MeshWeaver.Graph.ActivityTracking[0]
Failed to track activity for user=rbuergi path=rbuergi
System.TimeoutException: Update aborted: no initial state arrived for 'rbuergi/_UserActivity/rbuergi' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence.
2026-08-10 14:03:24Z memex-portal-deployment-6bd9c46f8b-z9t6r fail: MeshWeaver.Graph.ActivityTracking[0]
Failed to track activity for user=carson path=carson
System.TimeoutException: Update aborted: no initial state arrived for 'carson/_UserActivity/carson' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence.
2026-08-10 14:13:59Z memex-portal-deployment-6bd9c46f8b-z9t6r fail: MeshWeaver.Graph.ActivityTracking[0]
Failed to track activity for user=rsalzmann path=rsalzmann/DataModeling/Formatting/Exercise
System.TimeoutException: Update aborted: no initial state arrived for 'rsalzmann/_UserActivity/rsalzmann_DataModeling_Formatting_Exercise' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence.
2026-08-10 14:15:21Z memex-portal-deployment-6bd9c46f8b-z9t6r fail: MeshWeaver.Graph.ActivityTracking[0]
Failed to track activity for user=rsalzmann path=rsalzmann/DataModeling/Formatting/Exercise/FormatTheMoney
System.TimeoutException: Update aborted: no initial state arrived for 'rsalzmann/_UserActivity/rsalzmann_DataModeling_Formatting_Exercise_FormatTheMoney' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence.
2026-08-10 14:16:56Z memex-portal-deployment-6bd9c46f8b-z9t6r fail: MeshWeaver.Graph.ActivityTracking[0]
Failed to track activity for user=rsalzmann path=rsalzmann/DataModeling/Formatting/Exercise/FormatTheMoney
System.TimeoutException: Update aborted: no initial state arrived for 'rsalzmann/_UserActivity/rsalzmann_DataModeling_Formatting_Exercise_FormatTheMoney' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence.
2026-08-10 14:20:27Z memex-portal-deployment-6bd9c46f8b-z9t6r fail: MeshWeaver.Graph.ActivityTracking[0]
Failed to track activity for user=rsalzmann path=ThreeBody
System.TimeoutException: Update aborted: no initial state arrived for 'rsalzmann/_UserActivity/ThreeBody' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence.
Opened automatically from Admin/_LogIncident/7473a637daf1e169. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 10, 2026
Triage verdict: not a defect in
MeshWeaver.Graph.ActivityTracking, and none of the four suspected causes in the exception text. This is a prod manifestation of the deploy-roll / NodeType-bake starvation family (#1172, PR #1161, PR #1168, PR #1173), pinned by the roll timeline.Evidence (all times 2026-08-10 UTC, namespace
memex-cloud, gathered read-only viaaz aks command invoke):memex-portal-deployment-6bd9c46f8b-z9t6rbooted with the 11:44:38 RS.memex-portal-deployment-59479dc7d9-kmwvr) joined the Orleans cluster at ~13:39 but became Ready only at 15:59:37 (Ready.lastTransitionTimeon the still-running pod) — 2 h 20 min held Unhealthy by thenodetype_bakestartup gate while it reran the full dynamic-NodeType rebuild (every roll invalidates the whole compile cache — PR Portals roll once a day — every roll pays a full NodeType rebuild #1173's own rationale).rbuergi,carson,rsalzmann), five distinct paths — zero occurrences before or after. Orleans places grain activations by membership, not k8s readiness, so per-node-hub activations land on the joining, mid-bake silo and starve behind the compile sweep: RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172 records route dispatch slots exhausting behind non-terminating_Activity/compiledispatches on every freshly booted silo (z9t6ritself at 11:45:09,…67464cbcfc-jlmpwat 12:56:14 — each within ~a minute of its own join), and the enrichment slow path additionally waits on stale NodeTypes queued behind the sweep. The caller's 30 s initial-state bound inUpdateRemotethen loses deterministically — producing exactly this signature. The identical boot-window signature is dissected in PR The boot plugin install no longer races the NodeType bake it depends on #1161 (boot install racing the bake) and PR A parked NodeType no longer makes every instance of it un-writable #1168 (parked type burning a 60 s budget against the caller's 30 s).memex-portal-deployment-748bfcc86b-2m6vghad been Running/not-Ready for 80+ min at 20:45Z withHealth check nodetype_bake … Unhealthy ('NodeType bake in progress'),DynamicTypePreWarmer: Ifrs17/PartnerRating → TimedOut, and 16[ROUTE] 64 route dispatches are in flight … latest target …/_Activity/compile-…crit lines in its recent log — the same statekmwvrwas in during this issue's window.Falsifying the exception's own suspects: (1) RLS — these same users' tracking succeeds at every other time of day; an RLS denial is time-invariant, this failure is strictly roll-correlated. (2)/(3) typo'd path / deleted node — three unrelated users, five paths, failing only inside one roll window. (4) is the closest but one level down: the per-node hub never finished activating on the starved silo (same conclusion PR #1161 reached for the boot-install victim of this state).
Resolution:
_Activity/compiledispatches stalling and starving routing/activation on a freshly joined, baking silo — is RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172. Recommend closing this issue as a duplicate manifestation of RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172 (or letting the RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172 fix PR carryFixes #1174).rbuergi commentedon Aug 10, 2026
Closing as duplicate of #1172 — same starvation, different victim.
Decisive evidence (read-only cluster forensics, full detail in the comment above): every occurrence of this signature (13:49:50–14:20:27Z, three unrelated users) lies inside the 13:38:56 roll's join-to-Ready window — pod
kmwvrjoined Orleans membership immediately but passed thenodetype_bakestartup gate only at 15:59:37 (2 h 20 min later). Grain placement follows Orleans membership, not k8s readiness, so_UserActivityactivations landed on the mid-bake silo where route dispatches sit saturated at 64 in-flight behind stalled_Activitycompiles (#1172's exact state, reconfirmed live on the 19:28 roll's pod). Zero occurrences outside such windows; the RLS/typo/deleted-node hypotheses are falsified.The fix lands with #1172 (route-slot starvation root cause); exposure is already reduced by #1173 (daily roll cadence). Recurrences on an image containing the #1172 fix would be a new fact — reopen then.
systemorph-com commentedon Sep 14, 2026
Reopened — this fault is happening again.
Still occurring: 407 more since the last update (414 total).
memex-portal-deployment-6bd9c46f8b-z9t6r,memex-portal-deployment-59479dc7d9-kmwvr,memex-portal-deployment-686bdb74cb-vj9xl,memex-portal-deployment-cc58dc7c6-ckll8,memex-portal-deployment-bdc9f8dc4-fdkgw,memex-portal-deployment-6bf5d86dc4-zpgpl,memex-portal-deployment-648c756f74-smlsb,memex-portal-deployment-f96d46ddf-zn9nl,memex-portal-deployment-7d54845c86-brdng,memex-portal-deployment-79cddfb6b7-l7gvv,memex-portal-deployment-79cddfb6b7-pjrvx,memex-portal-deployment-74c7d69bf5-l2xtd,memex-portal-deployment-68795ccb45-4dpdk,memex-portal-deployment-68795ccb45-hzj6p,memex-portal-deployment-5d6f77b68c-wc9pw,memex-portal-deployment-5d6f77b68c-vsmvd,memex-portal-deployment-5d6f77b68c-srtqg,memex-portal-deployment-c89ddfbb6-vzqm7,memex-portal-deployment-7b59b7d657-nmtbz,memex-portal-deployment-f47f8cdd-f8fmq,memex-portal-deployment-f47f8cdd-rx8xj,memex-portal-deployment-7cb5549c57-xxg55,memex-portal-deployment-76b967b5fb-pkw5p,memex-portal-deployment-868d765df7-mwmhj,memex-portal-deployment-745df44fc8-drjtp,memex-portal-deployment-999c9c79d-g9wps,memex-portal-deployment-df7ddfdf4-xb9bk,memex-portal-deployment-775748f44d-7xlkm,memex-portal-deployment-6f76b9c79b-9spms,memex-portal-deployment-7c6b48957d-9zc78,memex-portal-deployment-7c6b48957d-xl99h,memex-portal-deployment-799c674567-p5lr8,memex-portal-deployment-b846c4cc8-v77vj,memex-portal-deployment-856f765d66-47gs2,memex-portal-deployment-7b747dbdb-5526x,memex-portal-deployment-7b747dbdb-sshxj,memex-portal-deployment-9bf79b774-tk88b,memex-portal-deployment-9bf79b774-s9sr9,memex-portal-deployment-9bf79b774-bwgqp,memex-portal-deployment-9bf79b774-b8dl6,memex-portal-deployment-9bf79b774-nvnzz,memex-portal-deployment-7f6bf4c7cc-2649s,memex-portal-deployment-67dbc7bc9c-455hx,memex-portal-deployment-85859db964-q5zcm,memex-portal-deployment-85859db964-7sqbk,memex-portal-deployment-85859db964-mgxj7,memex-portal-deployment-55547495d7-c8z9r,memex-portal-deployment-85859db964-xk5gw,memex-portal-deployment-85859db964-ghsbw,memex-portal-deployment-85859db964-wgtgw,memex-portal-deployment-85859db964-rfsd7,memex-portal-deployment-77c854cfcf-rcbtw,memex-portal-deployment-77c854cfcf-ns2r5,memex-portal-deployment-77cfc55bfc-ptd5l,memex-portal-deployment-77cfc55bfc-txpk7,memex-portal-deployment-77cfc55bfc-q4f9r,memex-portal-deployment-776b97b9fb-9xq2k,memex-portal-deployment-7fd9668b69-jwzzlRecent log lines
rbuergi commentedon Sep 14, 2026
2026-09-14 — live, the 2026-08-10 verdict no longer covers the population, and the failing write is pinned at the queue level
This was closed on 2026-08-10 as a duplicate of #1172 (roll-window route starvation), with "Recurrences on an image containing the #1172 fix would be a new fact — reopen then." That condition is met, and the evidence has moved.
The incident, read today on the control instance
get @Admin/_LogIncident/7473a637daf1e169(memex.systemorph.com), 2026-09-14 18:1xZ:The fold is honest here: all ten retained samples are the same category, the same exception type and the same message shape, differing only in user and path. This is one defect, and it is live.
The roll-window explanation does not cover it
The 08-10 argument was that every occurrence then known lay inside the 13:38:56 roll's join-to-Ready window. The retained ring now spans 2026-09-02T19:22Z → 2026-09-14T17:45Z across many pods, and the last one landed on
memex-portal-deployment-7fd9668b69-jwzzl— a pod that already appears in another incident's samples at 2026-09-13T22:03Z, i.e. up for ~20 h at the time. It may still be a contributor; it is no longer the explanation.The failing write, pinned
A
Logsinstance action on the control instance (read-only; no cluster access) over the 17:45Z window returned, from the same pod:Same
seq. So the write had its queue slot from ~17:45:03.83 and its base read simply never completed — it was not sitting behind a predecessor, and the owner had not acknowledged it even at +5 s.Two more readings worth having:
{user}/_UserActivity/{user}— the login node written on every cold page load; mine is at version 6498 — or a page that user revisits (Store,AgenticOffice/01-Willkommen). Cold paths do not appear.[ROUTE] NotFound, noAccess deniedand no[SYNC_STREAM] OnErrornaming any_UserActivitypath. (Positive control on the same query: the 17:45:33Z abort line itself came back, so the zero is readable.)Why it could not be attributed, and what just landed
The exception lists four suspects. There are really only two shapes of silence, and they are opposite defects:
MeshDataSourcethat did not load it🚨 Row 1 rules the node's content and its grants OUT, not in: an absent node and one the reader may not see both still produce a change item that carries no node. So suspects (1)–(3) of the message's own list are impossible in that half — and until now the log could not say which half you were in, because both raised the same bare
TimeoutException("The operation has timed out.").#4342 closes that gap: the base read counts what the mirror produced and says so. No bound changed, no retry, nothing swallowed; three negative controls watched red. Conserved as
Doc/Architecture/BaseStateTimeoutCensus.Staying open, and on what
#4342 is an INSTRUMENT, not a fix — the fault is still live at 414 occurrences. Closing this needs, in order:
memex-cloudcarrying fix(#1174): a base-state timeout says WHICH silence it waited on #4342 (deploy decision, not mine);All reads read-only (
get/searchon the control instance, oneLogsinstance action,curl /api/version). No mesh data was changed; no cluster access was used.review(#4342): the zero branch does NOT establish that the owner neve…
rbuergi commentedon Sep 14, 2026
Correction to my comment above — the zero branch does NOT rule the node's content out
The review on #4342 caught a real error in the fork I posted 25 minutes ago, and it is the direction that matters, so correcting it here rather than leaving the wrong table standing.
What I wrote: that "NO change item at all" means the owner never answered the subscribe, and therefore rules the node's content and its grants OUT.
Why that is wrong. This seam observes
ChangeItemFRAMES, not the subscribe acknowledgement — and a fresh stream acknowledges IMMEDIATELY, deliberately, before its firstFullcomes out of the reduce's own hydration.JsonSynchronizationStreamsays so verbatim:So zero frames means no snapshot ever hydrated, and an owner that acked and then produced nothing — a cold data source, a per-node hub still activating, an initial read that yielded nothing — is indistinguishable from one that never answered. My table would have sent a reader away from exactly the
MeshDataSource/initial-load causes they must still consider.The corrected fork:
MeshDataSourcethat did not load itBranch two is the one that excludes something. Branch one narrows nothing away — and the shipped message now says that in its own sentence rather than implying a verdict.
Everything else in my comment above stands as measured: 414 occurrences,
lastSeen 2026-09-14T17:45:33Z, theseq=30538pair, the hot-path concentration, and the fact that the 2026-08-10 roll-window verdict no longer covers the population. Corrected in #4342 at35b5243, in the message, the XML doc, the doc page and the What's New entry.rbuergi commentedon Sep 14, 2026
#4342 merged — and this issue STAYS OPEN
Landed on
mainas48a3bee143363cabf55c01dd44f990aa5426138f(PR #4342, head35b524302b00ca649e5a2de3259ee2b57ba77d29, both verified as ancestors oforigin/main). The shipped text is the post-review one, not my first draft.🚨 This is a better DIAGNOSTIC, not a fix. The write still fails. #4342 makes the base-state timeout name what it waited on; it does nothing to make the write succeed, and the 414 occurrences are untouched by it. Do not read the merge as a resolution.
What changed
BaseStateSourcenow counts what the mirror produced inside the bound and says so:MeshDataSourcethat did not load itNo bound moved, nothing retries, nothing is swallowed. Conserved as
Doc/Architecture/BaseStateTimeoutCensus.What still has to happen, in order
memex-cloudcarrying48a3bee. Not imminent, and that is now measured rather than assumed: the fleet's publication has not SEALED since 12:53Z (Systemorph/MeshWeaver.Plugins#1872), and at 19:01Z both portals still served3.0.0+c84c6c055…. Until that moves, every new sample the watcher folds here comes from an image without the census — a recurrence is not a new fact while that is true.The measurements that stand
Unchanged from my comment above, and they are what the next session should start from rather than re-deriving:
Admin/_LogIncident/7473a637daf1e169: 414 occurrences since 2026-08-10 across 57 pods,lastSeen 2026-09-14T17:45:33Z.…7fd9668b69-jwzzl:[UpdateQueue] ADVANCE_WITHOUT_HANDOFF … seq=30538 bound=5000msat17:45:08.832Zand[UpdateQueue] FAILED … seq=30538 elapsedMs=30004at17:45:33.831Z— same seq, so it was dispatched and never got a base; it was not queued behind a predecessor.{user}/_UserActivity/{user}(one at version 6498) and repeatedly-visited pages. Cold paths do not appear. This is a correlation, not an established cause; the census is what will decide it.rbuergi commentedon Sep 14, 2026
Triage 2026-09-14: #4342 is an instrument, not a fix (confirmed), and there IS a shipped mechanism that predicts the hot-path concentration the thread calls unexplained
48a3bee143(#4342) touched the message, docs and tests only —BaseStateWaitBoundis still 30 s (MeshNodeStreamExtensions.cs:147),BaseStateSource(:232) has the same.Timeoutand no retry, and the activity writer (MeshWeaver.Graph/MeshNodeExtensions.cs:380) still dies at the bound.The correlation "hot paths only" has a mechanism in this repo:
Workspace.EvictForPath(src/MeshWeaver.Data/Workspace.cs:337) fires on EVERY change-feedCreated/Updated/Deletedand evicts the mirror unconditionally — including the one the writer itself just used — so the next write resolves a fresh mirror: anotherSubscribeRequest, another initial-state round trip, another pair ofsync/hubs, per write (the remarks atWorkspace.cs:315-336measure six progress writes minting 7 client + 11 ownersync/hubs;Doc/Architecture/LiveMirrorsAndTheChangeFeeddocuments it and records why gatingEvictForPathon liveness LOST writes).{user}/_UserActivity/{user}at version 6498 has paid ~6498 evict → dispose → re-subscribe cycles; a cold path pays one. Every hot write therefore hydrates a brand-new mirror inside the 30 s bound while the one it just destroyed is being torn down on the same owner address — which predicts census branch one ("no change item reached the mirror") for the next occurrence. Falsifiable in one line once a replica carrying48a3beeis rolled.Fix shape, per the doc: make the write's base version-aware (wait for the mirror to reach the
VersiontheMeshChangeEventcarries) soEvictForPathstops being the freshness barrier; not a liveness gate on the eviction, which is measured to lose writes.review(#4428): the freshness floor is a field of the mirror, so it ca…
systemorph-com commentedon Sep 15, 2026
Reopened — this fault is happening again.
Still occurring: 2 more since the last update (416 total).
memex-portal-deployment-6bd9c46f8b-z9t6r,memex-portal-deployment-59479dc7d9-kmwvr,memex-portal-deployment-686bdb74cb-vj9xl,memex-portal-deployment-cc58dc7c6-ckll8,memex-portal-deployment-bdc9f8dc4-fdkgw,memex-portal-deployment-6bf5d86dc4-zpgpl,memex-portal-deployment-648c756f74-smlsb,memex-portal-deployment-f96d46ddf-zn9nl,memex-portal-deployment-7d54845c86-brdng,memex-portal-deployment-79cddfb6b7-l7gvv,memex-portal-deployment-79cddfb6b7-pjrvx,memex-portal-deployment-74c7d69bf5-l2xtd,memex-portal-deployment-68795ccb45-4dpdk,memex-portal-deployment-68795ccb45-hzj6p,memex-portal-deployment-5d6f77b68c-wc9pw,memex-portal-deployment-5d6f77b68c-vsmvd,memex-portal-deployment-5d6f77b68c-srtqg,memex-portal-deployment-c89ddfbb6-vzqm7,memex-portal-deployment-7b59b7d657-nmtbz,memex-portal-deployment-f47f8cdd-f8fmq,memex-portal-deployment-f47f8cdd-rx8xj,memex-portal-deployment-7cb5549c57-xxg55,memex-portal-deployment-76b967b5fb-pkw5p,memex-portal-deployment-868d765df7-mwmhj,memex-portal-deployment-745df44fc8-drjtp,memex-portal-deployment-999c9c79d-g9wps,memex-portal-deployment-df7ddfdf4-xb9bk,memex-portal-deployment-775748f44d-7xlkm,memex-portal-deployment-6f76b9c79b-9spms,memex-portal-deployment-7c6b48957d-9zc78,memex-portal-deployment-7c6b48957d-xl99h,memex-portal-deployment-799c674567-p5lr8,memex-portal-deployment-b846c4cc8-v77vj,memex-portal-deployment-856f765d66-47gs2,memex-portal-deployment-7b747dbdb-5526x,memex-portal-deployment-7b747dbdb-sshxj,memex-portal-deployment-9bf79b774-tk88b,memex-portal-deployment-9bf79b774-s9sr9,memex-portal-deployment-9bf79b774-bwgqp,memex-portal-deployment-9bf79b774-b8dl6,memex-portal-deployment-9bf79b774-nvnzz,memex-portal-deployment-7f6bf4c7cc-2649s,memex-portal-deployment-67dbc7bc9c-455hx,memex-portal-deployment-85859db964-q5zcm,memex-portal-deployment-85859db964-7sqbk,memex-portal-deployment-85859db964-mgxj7,memex-portal-deployment-55547495d7-c8z9r,memex-portal-deployment-85859db964-xk5gw,memex-portal-deployment-85859db964-ghsbw,memex-portal-deployment-85859db964-wgtgw,memex-portal-deployment-85859db964-rfsd7,memex-portal-deployment-77c854cfcf-rcbtw,memex-portal-deployment-77c854cfcf-ns2r5,memex-portal-deployment-77cfc55bfc-ptd5l,memex-portal-deployment-77cfc55bfc-txpk7,memex-portal-deployment-77cfc55bfc-q4f9r,memex-portal-deployment-776b97b9fb-9xq2k,memex-portal-deployment-7fd9668b69-jwzzlRecent log lines
rbuergi commentedon Sep 15, 2026
Triage 2026-09-15 17:1xZ: the reopen at 16:33Z is expected, because no pod runs the fix yet.
13ac6a0210. It stopsWorkspace.EvictForPathfrom evicting the writer's own mirror on every change-feed event.memex-portal-deployment-*pods. memex.systemorph.com runsb8143398a(3.0.0-ci.8612), which does not carry13ac6a0210. The first sealed sets that do are 3.0.0-ci.8684/8685/8686 (core9cad958f6e).So this stays open until memex rolls past both. The closing check then is: this incident's
occurrencesstop advancing on pods running a set that carries #4428.🤖 Generated with Claude Code
9 remaining items
systemorph-com commentedon Sep 18, 2026
Still occurring: 1 more since the last update (425 total).
memex-portal-deployment-6bd9c46f8b-z9t6r,memex-portal-deployment-59479dc7d9-kmwvr,memex-portal-deployment-686bdb74cb-vj9xl,memex-portal-deployment-cc58dc7c6-ckll8,memex-portal-deployment-bdc9f8dc4-fdkgw,memex-portal-deployment-6bf5d86dc4-zpgpl,memex-portal-deployment-648c756f74-smlsb,memex-portal-deployment-f96d46ddf-zn9nl,memex-portal-deployment-7d54845c86-brdng,memex-portal-deployment-79cddfb6b7-l7gvv,memex-portal-deployment-79cddfb6b7-pjrvx,memex-portal-deployment-74c7d69bf5-l2xtd,memex-portal-deployment-68795ccb45-4dpdk,memex-portal-deployment-68795ccb45-hzj6p,memex-portal-deployment-5d6f77b68c-wc9pw,memex-portal-deployment-5d6f77b68c-vsmvd,memex-portal-deployment-5d6f77b68c-srtqg,memex-portal-deployment-c89ddfbb6-vzqm7,memex-portal-deployment-7b59b7d657-nmtbz,memex-portal-deployment-f47f8cdd-f8fmq,memex-portal-deployment-f47f8cdd-rx8xj,memex-portal-deployment-7cb5549c57-xxg55,memex-portal-deployment-76b967b5fb-pkw5p,memex-portal-deployment-868d765df7-mwmhj,memex-portal-deployment-745df44fc8-drjtp,memex-portal-deployment-999c9c79d-g9wps,memex-portal-deployment-df7ddfdf4-xb9bk,memex-portal-deployment-775748f44d-7xlkm,memex-portal-deployment-6f76b9c79b-9spms,memex-portal-deployment-7c6b48957d-9zc78,memex-portal-deployment-7c6b48957d-xl99h,memex-portal-deployment-799c674567-p5lr8,memex-portal-deployment-b846c4cc8-v77vj,memex-portal-deployment-856f765d66-47gs2,memex-portal-deployment-7b747dbdb-5526x,memex-portal-deployment-7b747dbdb-sshxj,memex-portal-deployment-9bf79b774-tk88b,memex-portal-deployment-9bf79b774-s9sr9,memex-portal-deployment-9bf79b774-bwgqp,memex-portal-deployment-9bf79b774-b8dl6,memex-portal-deployment-9bf79b774-nvnzz,memex-portal-deployment-7f6bf4c7cc-2649s,memex-portal-deployment-67dbc7bc9c-455hx,memex-portal-deployment-85859db964-q5zcm,memex-portal-deployment-85859db964-7sqbk,memex-portal-deployment-85859db964-mgxj7,memex-portal-deployment-55547495d7-c8z9r,memex-portal-deployment-85859db964-xk5gw,memex-portal-deployment-85859db964-ghsbw,memex-portal-deployment-85859db964-wgtgw,memex-portal-deployment-85859db964-rfsd7,memex-portal-deployment-77c854cfcf-rcbtw,memex-portal-deployment-77c854cfcf-ns2r5,memex-portal-deployment-77cfc55bfc-ptd5l,memex-portal-deployment-77cfc55bfc-txpk7,memex-portal-deployment-77cfc55bfc-q4f9r,memex-portal-deployment-776b97b9fb-9xq2k,memex-portal-deployment-7fd9668b69-jwzzl,memex-portal-deployment-fb6488b5f-x7bv5Recent log lines
rbuergi commentedon Sep 19, 2026
Roll-gated close review, 2026-09-19 — HELD OPEN
This issue was on a list of twelve bot-filed incidents whose fixes merged after the 2026-09-12 freeze commit
c84c6c05and which were believed to be open only because memex.meshweaver.cloud had not rolled. The roll has happened. I checked three clauses before closing anything. This one is not closable, and the reason that dominates is the same for all twelve.(a) Ancestry — PASSES
memex.meshweaver.cloud is running commit
c25f86ae85b7050f0fffa45d81444cc69cbb64d8— image3.0.0-ci.8969, 3 of 3 replicas on one generation,converged: true, pods started 08:34:53 / 08:39:58 / 08:39:59Z, read fromOps/Status/memex-cloudsampled 10:24:54Z. Verified locally withgit merge-base --is-ancestor:141e5fdc91— ancestor ofc25f86ae85b713ac6a0210— ancestor ofc25f86ae85b748a3bee143— ancestor ofc25f86ae85b7The local clone is shallow (grafted at
bc4b1bd892, 2026-08-27), which can return a false "no" for a pre-graft commit. Every commit above is dated 2026-09-13 or later, i.e. well inside the grafted history, so none of these answers is affected.(c) Thread does not call the fix partial
(c) FAILS. Your 2026-09-16 comment on #4468 is explicit: "🚨 This is a better instrument plus two silent-drop fixes. It does NOT make the #1174 write succeed, and the occurrence count is untouched by it." The 2026-09-18 triage adds: "stays open. The blocker ... is now HALF cleared, and the half that matters is not."
Last seen for this fingerprint (
Admin/_LogIncident/7473a637daf1e169, read on the control instance):2026-09-18T17:04:23.980Z, 425 occurrences,namespace: memex-cloud. That is before the 08:00Z settle line, and the newest of its retained samples carries the same timestamp. The node was readable — this is a real reading, not an absent one.🚨 (b) No recurrence since the roll — CANNOT BE ESTABLISHED. The instrument is dark.
A silence is only evidence if the thing that would have broken it is still working. It is not.
No incident anywhere in the population has folded a sighting since
2026-09-19T07:28:51Z— before the current replicas even started — and the single node written since 08:00Z has itslastSeenfrozen at05:43:46Z, roughly 4 h 45 m before this comment. I checked the whole readableAdmin/_LogIncidentpopulation on the control instance, sorted bylastModifieddescending. Since a fold is what writes the node,lastSeen <= lastModified, solastModifiedis a sound upper bound: only one node in the entire population has been written since 08:00Z —e4a97855ab595beb, theRoutingGrainback-pressure incident, which is the highest-volume fingerprint in the fleet at 13,108,032 lifetime occurrences and severityCritical.I probed it directly as the positive control. Its
versionadvanced across my reads (51427 → 51430 → 51434,lastModified10:12:29Z → 10:26:28Z → 10:27:29Z), so something is still writing the node — but across twogets 4 m 35 s apart:A fingerprint that has fired 13.1 million times has folded zero sightings in 4 h 44 m. The writes that do land are triage bookkeeping — the node carries a standing
content.error: "Cannot comment: the incident has no repository."since2026-09-17T19:09:17Z. Rewrites without an occurrence advance are a known shape here: fingerprint70a700c6cfe6c28ahas alastModified11.6 days later than itslastSeenwith its counter unmoved. A highversionis not evidence of a recent recurrence.The control that settles it — the same event class on both sides of the change. memex-cloud rolled or restarted six times this morning:
3.0.0-ci.8955(+ 05:08Z migration reconcile)3.0.0-ci.89633.0.0-ci.89663.0.0-ci.8969(pods up 08:34:53 / 08:39:58 / 08:39:59Z)The 05:03Z roll produced a textbook roll signature —
DbVersionGate: DB migration incomplete,Host: Hosting failed to start,MemexConfiguration: Startup was cancelled, anObjectDisposedExceptionwave acrossSiloHealthMonitor/OrleansRoutingService/Messaging/Grain/MessageService,IoPoolSiloTeardown,nodetype_bake Unhealthy, and the Orleansmemory-*queue timeouts — 28 folds between 05:07:58Z and 05:47:25Z. The four later rolls produced none. Same event class, same instrument, opposite result.To state the bound exactly: the last incident-node write of any kind across the population was
07:28:51Z, and the only node written after that is the bookkeeping one above. A sighting is what writes the node — so nothing has been recorded anywhere in the ~3 h since 07:28:51Z, a window that wholly contains the post-roll period (replicas up 08:34–08:40Z). Whether ingestion died at 05:47Z or at 07:28Z does not change this issue's answer: either way there is no observation window after the roll.That the drop is uniform across fingerprints these twelve fixes cannot touch — Orleans
MemoryAdapterFactorytimeouts,DbVersionGate,Hosting failed to start— rules out "we fixed everything". The mesh itself is demonstrably alive:Admin/UpdatePolicyandAdmin/_Build/*were written at 10:28Z,Ops/Status/*at 10:24Z.So the required positive control fails: across the entire readable population I could not find one incident that fired after the settle line. Per the rule I was working to, that reading is indistinguishable from "the watcher stopped", and it would be wrong to convert it into twelve closures.
(Honest limit: I could not find a health entry naming log ingestion as failed. Every
/healthhealthDetailonOps/Status/memex-cloudis stored truncated mid-source-discovery— ~7 of 16 checks are visible — so an ingestion check past the cut would be invisible. An absent entry is not a clean one. The ingestion stall is inferred from the fold record above, not from a health verdict.)Verdict
Stays open. Clause (b) is unmet for every one of the twelve. This issue also fails clause (c) above, so it would stay open even with a working watcher.
Nothing here is a statement that the defect is still live — it may well be fixed. It is a statement that the evidence required to say so does not currently exist, and that the incident-ingestion stall is itself worth a ticket: the portal has been rolling all morning with its recurrence watcher recording nothing.
No labels, state or assignees were changed. Read-only apart from this comment.
rbuergi commentedon Sep 19, 2026
Source-level mechanism, 2026-09-19 — the write takes its branch from an eventually-consistent query's POSITIVE, and this repo already documents that shape as retired-by-design
Every prior comment here concluded the next step is a production reading. This one does not need one: the branch that produces the timeout is decidable from source, and the platform already contains the verb that exists to remove it.
What the failing path actually does
MeshWeaver.Graph/MeshNodeExtensions.cs:286var stream = workspace.GetMeshNodeStream(activityPath);320-323workspace.GetQuery("UserActivity|<path>", "path:<path> nodeType:UserActivity …")— the eventually-consistent index379if (existing != null)392return AsCaller(() => stream.Update(FoldOntoLive));← the 30 s wait430CreateNode→NodeAlreadyExistsrace →stream.Update(FoldOntoLive)The two branches do not establish existence the same way. At
430the owner has just refused the create withNode already exists:— durable truth, serialised by the hub that owns the row. At392the only evidence is the query index.The code already knows that index is stale-prone: its own comment at
378saysrecord.AccessCountis "the count derived from the EVENTUALLY-CONSISTENT query snapshot above". It handles the stale negative (create → already-exists → fold to update). Nothing handles the stale or ghost positive, and that is the branch that lands onstream.Update.The repo already states this rule — for the delete side
MeshWeaver.Mesh.Contract/MeshExtensions.cs:3315, written by #4668:CreateOrUpdateNodeRequest's own summary says the same in the affirmative — it is the "single global verb … so callers don't replay the existence dance". TrackActivity replays it.Why that verb is the one that matters here — it carries the repair for this exact condition
The upsert does not consult the query index. It establishes existence from a persistence read (
MeshExtensions.cs:5851-5868, trail stagesUPSERT_READ absent → create/UPSERT_READ existing → update). And its update leg floors the version on that durable row, for a reason it states verbatim (~6270):"A hub that could not hydrate the stored node" is, word for word, the condition the newest sample on this issue reports:
So the one code path in the platform that repairs a node whose owner cannot hydrate it is the path this caller does not take.
🚨 Honest limits — what this does and does not establish
BaseStateTimeoutException's own text says a cold data source, a still-activating hub and an initial read yielding nothing are indistinguishable there. This names a mechanism the code can produce deterministically; it does not close the other three.UPSERT_READ absentvsexisting, which is durable truth. The query cannot answer that question at all.🚧 The blocker — and it is a real gap in the verb, not in this caller
Routing TrackActivity through
CreateOrUpdateNodeRequestloses the counter increment, and neither payload mode can express it:UpdateAccordingToSourceNode, which takesContent = sourceNode.Content ?? state.Content— content wholesale.FoldOntoLive's whole purpose (:356, guarding A no-op stream.Update never completes its writer — one of five concurrent activity tracks never LEAVEs, and its increment is lost #3001) is thatAccessCountis re-read off the live node inside the owner-serialised lambda. A full-instance upsert would write the stale query-derived count instead.AccessCountbump is a view-count bump — but JSON Patch has no increment operation, so the value still has to be computed from a stale read before the request is sent. The documented use case is not actually served by either mode.MeshNodeStreamHandlehas threeUpdateoverloads (MeshNodeStreamExtensions.cs:1246,1268,1355) and none takes a create-if-missing seed, so there is no third route that keeps the fold owner-side.Proposed next step
Give the upsert an owner-side fold — the merge lambda shape
stream.Updatealready has — so a counter bump can go through the one verb that decides existence against durable truth and carries theAdoptDurableTruthrepair. Then move TrackActivity's:379/:392branch onto it and delete the query's role in choosing the write.That is a framework change with a real design surface, so I am not landing it unilaterally on a cold-login hot path. Filing it as the concrete follow-up rather than leaving this issue resting on "needs a roll".
Nothing closed. No labels, state or assignees changed.
Source-level triage 2026-09-19 by Claude Opus 5 — read-only; every line/quote above is from the current
main.rbuergi commentedon Sep 20, 2026
The blocker named on 2026-09-19 has a shape now —
Doc/Architecture/ExpressingAWrite(PR #4970)The 19:22Z comment localised this to one line (
MeshNodeExtensions.cs:379/392decides create-vs-updatefrom the eventually-consistent query index) and filed the framework gap as #4928, stating the reason it
could not simply be fixed: "a delegate cannot cross the wire."
That is true of a delegate and not of an expression tree. A tree is analysable, and what it
lowers to is JSON Patch operations — which do cross the wire, and which the owner applies inside the
Updatelambda the upsert's update leg already runs (MeshExtensions.cs:~6325). So the fix needsno new execution point: every field except
Contentis already folded owner-side there;Contentisthe one taken wholesale.
The design is committed in #4970. It is design only — nothing is implemented, and this issue stays
blocked on #4928.
One thing the design changes about the eventual fix here
FoldOntoLive(:356) does three different things, and only one of them is an increment:AccessCountlive + 1FirstAccessedAtLastAccessedAtSo whatever lands for #4928 has to carry more than "increment". A lambda expresses all three
naturally, which is the argument for making the lambda the authoring surface rather than hand-writing
patch ops at the call site.
Status unchanged
Still open, still 425 occurrences, still gated on #4928. Nothing here is a production reading, and the
19:22Z comment's honest-limits section still stands: this names a mechanism the code can produce
deterministically, and does not close the other three causes
BaseStateTimeoutExceptioncannotdistinguish.
Cross-reference 2026-09-20 by Claude Opus 5 — read-only.
review(#4970): the lowering example lost the count, and a cross-hub l…
systemorph-com commentedon Sep 21, 2026
Still occurring: 3 more since the last update (428 total).
memex-portal-deployment-75b545bbd-nssvp,memex-portal-deployment-6bd9c46f8b-z9t6r,memex-portal-deployment-59479dc7d9-kmwvr,memex-portal-deployment-686bdb74cb-vj9xl,memex-portal-deployment-cc58dc7c6-ckll8,memex-portal-deployment-bdc9f8dc4-fdkgw,memex-portal-deployment-6bf5d86dc4-zpgpl,memex-portal-deployment-648c756f74-smlsb,memex-portal-deployment-f96d46ddf-zn9nl,memex-portal-deployment-7d54845c86-brdng,memex-portal-deployment-79cddfb6b7-l7gvv,memex-portal-deployment-79cddfb6b7-pjrvx,memex-portal-deployment-74c7d69bf5-l2xtd,memex-portal-deployment-68795ccb45-4dpdk,memex-portal-deployment-68795ccb45-hzj6p,memex-portal-deployment-5d6f77b68c-wc9pw,memex-portal-deployment-5d6f77b68c-vsmvd,memex-portal-deployment-5d6f77b68c-srtqg,memex-portal-deployment-c89ddfbb6-vzqm7,memex-portal-deployment-7b59b7d657-nmtbz,memex-portal-deployment-f47f8cdd-f8fmq,memex-portal-deployment-f47f8cdd-rx8xj,memex-portal-deployment-7cb5549c57-xxg55,memex-portal-deployment-76b967b5fb-pkw5p,memex-portal-deployment-868d765df7-mwmhj,memex-portal-deployment-745df44fc8-drjtp,memex-portal-deployment-999c9c79d-g9wps,memex-portal-deployment-df7ddfdf4-xb9bk,memex-portal-deployment-775748f44d-7xlkm,memex-portal-deployment-6f76b9c79b-9spms,memex-portal-deployment-7c6b48957d-9zc78,memex-portal-deployment-7c6b48957d-xl99h,memex-portal-deployment-799c674567-p5lr8,memex-portal-deployment-b846c4cc8-v77vj,memex-portal-deployment-856f765d66-47gs2,memex-portal-deployment-7b747dbdb-5526x,memex-portal-deployment-7b747dbdb-sshxj,memex-portal-deployment-9bf79b774-tk88b,memex-portal-deployment-9bf79b774-s9sr9,memex-portal-deployment-9bf79b774-bwgqp,memex-portal-deployment-9bf79b774-b8dl6,memex-portal-deployment-9bf79b774-nvnzz,memex-portal-deployment-7f6bf4c7cc-2649s,memex-portal-deployment-67dbc7bc9c-455hx,memex-portal-deployment-85859db964-q5zcm,memex-portal-deployment-85859db964-7sqbk,memex-portal-deployment-85859db964-mgxj7,memex-portal-deployment-55547495d7-c8z9r,memex-portal-deployment-85859db964-xk5gw,memex-portal-deployment-85859db964-ghsbw,memex-portal-deployment-85859db964-wgtgw,memex-portal-deployment-85859db964-rfsd7,memex-portal-deployment-77c854cfcf-rcbtw,memex-portal-deployment-77c854cfcf-ns2r5,memex-portal-deployment-77cfc55bfc-ptd5l,memex-portal-deployment-77cfc55bfc-txpk7,memex-portal-deployment-77cfc55bfc-q4f9r,memex-portal-deployment-776b97b9fb-9xq2k,memex-portal-deployment-7fd9668b69-jwzzl,memex-portal-deployment-fb6488b5f-x7bv5Recent log lines
Re-addressed by the current identity function: this incident inherited
7473a637daf1e169(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 22, 2026
Two loops this thread left open, and one of them falsifies its surviving sub-hypothesis
Grouped as part of the
area:meshtimeout cluster. Nothing here changes the disposition — this stays open, gated on #4928 — but three facts are derivable from what is already on the thread and no comment states them.1. The
EvictForPathsub-hypothesis is falsified by the thread's own later readingsThe 2026-09-14 21:44Z comment named a mechanism that predicts the hot-path concentration:
Workspace.EvictForPathfires on every change-feed event and evicts unconditionally, including the mirror the writer just used, so every hot write re-hydrates a brand-new mirror inside the 30 s bound — and it predicted census branch one for the next occurrence. It was offered as "falsifiable in one line once a replica carrying48a3beeis rolled".That falsification has now happened three times and nobody closed the loop:
13ac6a0210) stopsEvictForPathevicting the writer's own mirror. The 2026-09-19 10:34Z comment established bygit merge-base --is-ancestorthat it is an ancestor ofc25f86ae85b7/3.0.0-ci.8969, the commit memex-cloud was running,converged: trueacross 3 of 3 replicas.So branch one was read on images carrying the fix that was predicted to remove its cause.
EvictForPathis not what produces it. That eliminates the one competing hypothesis this thread had, and leaves the ghost/stale query positive as the only mechanism still standing.2. The 2026-09-18 10:25Z premise is superseded within the same day
That comment states memex-cloud carries none of #4342/#4428/#4468 and is still on
c84c6c05. The 17:04:23Z sample later that day, on a new pod (fb6488b5f-x7bv5), prints the #4342 census text verbatim — so memex-cloud rolled onto a #4342-carrying image between those two timestamps, and the "do not read the silence as a fix, no pod runs it" reasoning does not cover the samples that followed.3. The
Trail:instrument's silence has a second, better explanationThe 2026-09-16 09:01Z comment attributes the unread
RequestFateLedgertrail to "no portal runs141e5fdc". But no sample after 2026-09-15T18:56:54Z is the self-addressed shape (b) — every later one is shape (a), theBaseStateTimeoutException. The trail instrument #4468 shipped exists to discriminate shape (b)'s intake-refusal-versus-detached-chain fork. It has had no shape-(b) occurrence to print on, which is a different reason for its silence than image lag, and it means the (a)-vs-(b) question the thread lists as open may no longer be reachable from production at all.Disposition
Leave open, gated on #4928. The remedy the thread converged on — the caller must not decide create-vs-update from an eventually-consistent index positive — is now also the platform's written rule (
AGENTS.mdnames this issue by number as the fallback that rule forbids), and it cannot be implemented whileCreateOrUpdateNodeRequestcannot express a fold. Naming the gap is the correct state; I did not routeTrackActivitythrough the upsert, because doing so loses the counter increment and that is exactly what #4928 is.sev:Mstands.On its relationship to #5115
Kept separate, and the 2026-09-21 19:52Z reasoning on #5115 is right: this ticket is a diagnosed root, #5115 is an undiagnosed symptom family over the same log site, and folding the family into the root buries whichever of the two is wrong. Same log site + different diagnostic state ⇒ keep both. Recorded on #5115 too, with the fingerprint-lineage evidence that they ARE the same site (#5115 inherited
7473a637daf1e169, which is this ticket's fingerprint).What I did NOT establish
75b545bbdreplica set runs. The three 2026-09-20 22:25Z samples are on pod…-75b545bbd-nssvp; the fix(#1174): a write's base is held to the announced version, so a hot path stops rebuilding its mirror per write #4428 ancestry reading is forc25f86ae85b7, taken 2026-09-19 against a different generation. So point 1 rests on the 2026-09-18 17:04Z sample (whose image carried fix(#1174): a base-state timeout says WHICH silence it waited on #4342, established) plus the 09-19 ancestry reading for fix(#1174): a write's base is held to the announced version, so a hot path stops rebuilding its mirror per write #4428 — not on a single reading that proves one replica carried both and still produced branch one. That reading is the one measurement that would make the falsification airtight, and it needs aSample//api/versionread I did not take: live portals were read-only for this session.Grouping method committed as
Doc/Architecture/OneLogSiteManyTerminals(PR #5170).rbuergi commentedon Sep 22, 2026
Grouping verdict: NOT the same root as the
MeshNodeHubFactorypair (#5008 / #2231) — and the blocker this thread has been gated on since 2026-09-19 is GONE. The remedy it converged on is inmain. Full verdict: #5008 (comment)This ticket was put to me as possibly one root with #5008, #2231 and #4597. It is not: that cluster is a NodeType resolution defect at the activation/write boundary; this one is a caller deciding create-vs-update from an eventually-consistent index. Nothing in either touches the other.
🚨 The gate is lifted and the caller has moved
The 2026-09-19 19:22Z comment localised this to one line and filed the framework gap as #4928, stating the reason it could not simply be fixed: "a delegate cannot cross the wire." #4970 then designed the declarative alternative. Both halves have landed.
CreateOrUpdateNodeRequest.Foldsexists, withContentFold/ContentFolds/WithFolds<T>and the rulesSum/Max/Min/KeepExisting(src/MeshWeaver.Mesh.Contract/ContentFold.cs), applied owner-side inside theUpdatelambda the upsert already runs (MeshExtensions.cs:~6398). AndMeshWeaver.Graph/MeshNodeExtensions.csno longer branches on the query index at all —HandleTrackActivitynow issues one upsert (commit2b514860dc, "Fix activity tracking with owner-side upsert folds"):The three folds are exactly the three things
FoldOntoLivedid, in the order the 2026-09-20 11:35Z comment tabulated them —AccessCountsum,FirstAccessedAtkeep-live,LastAccessedAttake-newer. That comment's objection ("whatever lands for #4928 has to carry more than 'increment'") is answered rather than sidestepped. The code carries this issue by number: "The query index can retain a positive row while its node has no live base; using that row to choose stream.Update selected the unsafe update path implicated in #1174."So this thread's own acceptance criterion — "
MeshNodeExtensions.cs:379-392no longer chooses its write from a query result" — is met, and there are no lines 379-392 of that shape left to point at.What that does NOT settle, stated so nobody reads this as done
BaseStateTimeoutExceptioncannot distinguish a cold data source, a still-activating hub and an initial read yielding nothing. The 19:22Z comment's honest-limits section stands: the ghost/stale-positive is a mechanism the code can produce deterministically, not a proven cause of any given occurrence.ActivityTrackingerror.I have not closed this, changed its labels, or touched its state — the remaining work is a post-roll reading, which is not mine to take. Recording the code state so the next reader does not re-derive a blocker that no longer exists.
rbuergi commentedon Sep 23, 2026
Post-roll status 2026-09-23: #5187 is live on both portals, and the new path has one timeout sample. It is on #5230
ad17f6e5c9) is contained in both running builds: memex.systemorph.com runs3da3423c4band memex.meshweaver.cloud runs39f49494eb, both checked withgit merge-base --is-ancestor.HandleTrackActivity. Activity tracking now issues oneCreateOrUpdateNodeRequestwith owner-side folds.HANDLER_ENTER→HANDLER_EXI…onportal/nodeops-…and then goes silent for 60 s. It is not this issue's shape: the request is aCreateOrUpdateNodeRequest, not a self-addressedCreateNodeRequest, and nothing decides create-vs-update from the index. The triage there shows every leg of the upsert's detached work answering except the existence read, which has no own bound. The truncated trail hides which leg it was.Not closing: this issue's own verification is a clean window after the roll on the instrument that would show the old shape, and I could not take a
Logsread (the control instance's MCP answers 503). #5230 carries the new shape separately.rbuergi commentedon Sep 23, 2026
Closing: the localised defect is fixed and deployed, and the code path it described no longer exists
ad17f6e5c9).HandleTrackActivityno longer decides create-vs-update from the query index. It issues oneCreateOrUpdateNodeRequestwith owner-side folds (Sum AccessCount,KeepExisting FirstAccessedAt,Max LastAccessedAt). Existence is decided by the owner's persistence read, never by a query positive.9a61fed964, and memex.systemorph.com serves39f49494eb(/api/version, 2026-09-23T20:15Z).git merge-base --is-ancestor ad17f6e5c9 …is true for both.TimeoutException: Update aborted …from a query-chosenstream.Update, nor a self-addressedCreateNodeRequestfromTrackActivity. The self-addressed shape (b) has had no sample since 2026-09-15.What remains is a different terminal, tracked where it belongs:
HANDLER_EXIT.OwnerUnreachable).A new
ActivityTrackingerror reopens by fingerprint on its own ticket, not here.