ActivityTracking fails to write user activity on optimistic concurrency conflict

What is failing

The ActivityTracking component fails to record a user's activity when the mesh's optimistic concurrency guard detects that 3 fields on the _UserActivity/{userId} node changed between the writer's read and write. The write is refused with a MeshNodeStreamException and silently dropped.

Probable cause

Optimistic concurrency collision on the user's activity node (rbuergi/_UserActivity/rbuergi). The tracking code reads the current state, modifies it, and writes back — but if another concurrent request modifies the same node in between, the write is rejected ("cross-hub write refused: 3 field(s) changed on the owner since the writer's base"). The ActivityTracking code does not appear to retry or merge in this scenario. Confidence: high — the exception message is explicit about the conflict.

Impact

Minimal. 1 occurrence on 1 pod for 1 user. The activity entry for this user at this moment was not persisted, but no crash or user-visible error occurred. With higher concurrent load on popular users this could escalate into silent data loss on activity streams.

Where to look

MeshWeaver.Graph.ActivityTracking — the component that writes activity records. The trace path goes through ActivityTracking attempting to write to the _UserActivity/{userId} node.
The mesh's MeshNodeStreamException handler or the concurrency resolution logic for _UserActivity nodes — consider adding a retry-with-refresh loop for concurrency conflicts, or switching to an append-only/event-sourced pattern for activity tracking.

Evidence

The exception detail reads: "MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write refused: 3 field(s) changed on the owner since the writer's base and nothing was applied — re-read and re-apply."

Evidence


Fingerprint	`686060703d06c1b8`
Category	`MeshWeaver.Graph.ActivityTracking`
Severity	Error
Exception	`MeshWeaver.Mesh.MeshNodeStreamException`
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-868d46474d-w8dc2`
Occurrences	1
First seen	2026-08-19 20:31:53Z
Last seen	2026-08-19 20:31:53Z

Recent log lines

2026-08-19 20:31:53Z memex-portal-deployment-868d46474d-w8dc2 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write refused: 3 field(s) changed on the owner since the writer's base and nothing was applied — re-read and re-apply

_{Opened automatically from Admin/_LogIncident/686060703d06c1b8. Recurrences are folded into this issue rather than opening new ones.}

Merge pull request #1961 from Systemorph/fix/1910-activity-conflict-r…

Reopened — this fault is happening again.

Still occurring: 75 more since the last update (76 total).

Last seen: 2026-09-12 08:28:48Z
Pods: memex-portal-deployment-868d46474d-w8dc2, memex-portal-deployment-59ccff47c5-58zt4, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7858d9f8cc-h6gp2, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-799c674567-v8l2n, memex-portal-deployment-75597864c8-vllh5, memex-portal-deployment-9bf79b774-b8dl6, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-c7f5ff965-vk9zq, memex-portal-deployment-c7f5ff965-7kj5l, memex-portal-deployment-85859db964-ghsbw, memex-portal-deployment-85859db964-rfsd7, memex-portal-deployment-58c88885f8-lhgp7, memex-portal-deployment-67b654495f-d8g8n, memex-portal-deployment-67b654495f-6xjg9, memex-portal-deployment-85859db964-bldn9, memex-portal-deployment-77c854cfcf-ns2r5, memex-portal-deployment-77c854cfcf-rcbtw, memex-portal-deployment-77c854cfcf-clz9l, memex-portal-deployment-5c8c476d54-h22p6, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-58c859d4cc-tqhzx, memex-portal-deployment-5f455dbc49-bzt6n, memex-portal-deployment-5f455dbc49-kcmxm, memex-portal-deployment-6457bb9544-8plns, memex-portal-deployment-6457bb9544-qpdjd, memex-portal-deployment-59955df69b-t2pbb, memex-portal-deployment-5777bff985-n98c5, memex-portal-deployment-59d8fd958f-sszkb, memex-portal-deployment-685fb69888-r8wl9, memex-portal-deployment-7ffbd9b949-rvqk8, memex-portal-deployment-75dff475b6-dg5r9

Recent log lines

2026-09-10 19:57:31Z memex-portal-deployment-7ffbd9b949-rvqk8 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 2 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.
2026-09-10 20:02:45Z memex-portal-deployment-7ffbd9b949-rvqk8 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 2 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.
2026-09-12 08:28:48Z memex-portal-deployment-75dff475b6-dg5r9 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 1 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.

Closing — the defect this issue reports (a Conflict retry that could not converge) is fixed in main, and the reopen carries a different, later message.

Fixed in current origin/main. Closed by PR #1961 (fix/1910-activity-conflict-reread). In a read-only checkout at /Users/roland/code/MW-current-main:

git grep -n "#1910" origin/main -- 'src/**/*.cs' 'test/**/*.cs'

→ src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs:318, :2035, :2658 and test/MeshWeaver.Graph.Test/HotPathWriteBaseIsVersionAwareTest.cs:244.

:318 names the fix outright — "The emission a (re)attempt rebuilds its patch from — and the whole of the #1910 fix" — and states the defect exactly as this issue described it:

it re-ran the lambda against mirror.Take(1) … which, immediately after a NACK, is very often still the version the owner just refused. The lambda then recomputes the same values from the same base … and the owner refuses identically. Three attempts, three identical patches, then a terminal MeshNodeStreamException and the caller's write is gone. A retry whose input cannot change is not a retry.

The correction is live at :2658: on MeshNodeErrorCode.Conflict the re-enqueue now passes rebaseFrom = current.Version (the version the owner refused) so the re-attempt rebases on state newer than the refusal, instead of passing 0. RebaseSource implements the wait, and HotPathWriteBaseIsVersionAwareTest pins it.

The reopen evidence is a different, later message. The incident Admin/_LogIncident/686060703d06c1b8 fingerprints on category + exception type + the outer log line ("Failed to track activity for user=rbuergi path=rbuergi"), not on the inner conflict detail — so two different inner faults share one bucket. Read read-only on memex.systemorph.com (get @Admin/_LogIncident/686060703d06c1b8), all 10 stored samples (2026-09-08 → 2026-09-12) carry:

cross-hub write PARTIALLY refused: N field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.

That string is #2463's deliberate partial-refusal NACK, introduced after this fix at src/MeshWeaver.Data/DataExtensions.cs:2324 / :2773, where the comment records it as intentional: "A PARTIAL refusal is still a refusal (#2463) … nacking here and letting the commit proceed is deliberate." Not one sample carries this issue's filed message — cross-hub write refused: 3 field(s) changed … and nothing was applied — which is the total-refusal shape the fix removed.

Last seen: 2026-09-12 08:28:48Z — 6 days quiet as of 2026-09-18, and log ingest is live (incidents updated 2026-09-18T05:42Z), so the silence is measured, not an ingest gap.

Closing a dormant incident is safe by design: the incident record stays live, and the recurrence bot reopens this issue automatically with fresh Last seen evidence the moment the fingerprint fires again. Nothing is lost by closing; what is lost by leaving it open is the meaning of "open".

_{Read-only triage, 2026-09-18. Nothing was run, re-run or changed on any deployment.}

Reopened — this fault is happening again.

Still occurring: 1 more since the last update (77 total).

Last seen: 2026-09-18 16:31:58Z
Pods: memex-portal-deployment-868d46474d-w8dc2, memex-portal-deployment-59ccff47c5-58zt4, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7858d9f8cc-h6gp2, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-799c674567-v8l2n, memex-portal-deployment-75597864c8-vllh5, memex-portal-deployment-9bf79b774-b8dl6, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-c7f5ff965-vk9zq, memex-portal-deployment-c7f5ff965-7kj5l, memex-portal-deployment-85859db964-ghsbw, memex-portal-deployment-85859db964-rfsd7, memex-portal-deployment-58c88885f8-lhgp7, memex-portal-deployment-67b654495f-d8g8n, memex-portal-deployment-67b654495f-6xjg9, memex-portal-deployment-85859db964-bldn9, memex-portal-deployment-77c854cfcf-ns2r5, memex-portal-deployment-77c854cfcf-rcbtw, memex-portal-deployment-77c854cfcf-clz9l, memex-portal-deployment-5c8c476d54-h22p6, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-58c859d4cc-tqhzx, memex-portal-deployment-5f455dbc49-bzt6n, memex-portal-deployment-5f455dbc49-kcmxm, memex-portal-deployment-6457bb9544-8plns, memex-portal-deployment-6457bb9544-qpdjd, memex-portal-deployment-59955df69b-t2pbb, memex-portal-deployment-5777bff985-n98c5, memex-portal-deployment-59d8fd958f-sszkb, memex-portal-deployment-685fb69888-r8wl9, memex-portal-deployment-7ffbd9b949-rvqk8, memex-portal-deployment-75dff475b6-dg5r9, memex-portal-deployment-6b68cf8588-rqt25

Recent log lines

2026-09-10 20:02:45Z memex-portal-deployment-7ffbd9b949-rvqk8 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 2 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.
2026-09-12 08:28:48Z memex-portal-deployment-75dff475b6-dg5r9 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 1 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.
2026-09-18 16:31:58Z memex-portal-deployment-6b68cf8588-rqt25 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 1 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.

Reopened — this fault is happening again.

Still occurring: 1 more since the last update (77 total).

Last seen: 2026-09-18 16:31:58Z
Pods: memex-portal-deployment-868d46474d-w8dc2, memex-portal-deployment-59ccff47c5-58zt4, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7858d9f8cc-h6gp2, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-799c674567-v8l2n, memex-portal-deployment-75597864c8-vllh5, memex-portal-deployment-9bf79b774-b8dl6, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-c7f5ff965-vk9zq, memex-portal-deployment-c7f5ff965-7kj5l, memex-portal-deployment-85859db964-ghsbw, memex-portal-deployment-85859db964-rfsd7, memex-portal-deployment-58c88885f8-lhgp7, memex-portal-deployment-67b654495f-d8g8n, memex-portal-deployment-67b654495f-6xjg9, memex-portal-deployment-85859db964-bldn9, memex-portal-deployment-77c854cfcf-ns2r5, memex-portal-deployment-77c854cfcf-rcbtw, memex-portal-deployment-77c854cfcf-clz9l, memex-portal-deployment-5c8c476d54-h22p6, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-58c859d4cc-tqhzx, memex-portal-deployment-5f455dbc49-bzt6n, memex-portal-deployment-5f455dbc49-kcmxm, memex-portal-deployment-6457bb9544-8plns, memex-portal-deployment-6457bb9544-qpdjd, memex-portal-deployment-59955df69b-t2pbb, memex-portal-deployment-5777bff985-n98c5, memex-portal-deployment-59d8fd958f-sszkb, memex-portal-deployment-685fb69888-r8wl9, memex-portal-deployment-7ffbd9b949-rvqk8, memex-portal-deployment-75dff475b6-dg5r9, memex-portal-deployment-6b68cf8588-rqt25

Recent log lines

2026-09-10 20:02:45Z memex-portal-deployment-7ffbd9b949-rvqk8 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 2 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.
2026-09-12 08:28:48Z memex-portal-deployment-75dff475b6-dg5r9 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 1 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.
2026-09-18 16:31:58Z memex-portal-deployment-6b68cf8588-rqt25 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 1 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.

Still occurring: 26 more since the last update (103 total).

Last seen: 2026-09-18 20:04:02Z
Pods: memex-portal-deployment-868d46474d-w8dc2, memex-portal-deployment-59ccff47c5-58zt4, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7858d9f8cc-h6gp2, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-799c674567-v8l2n, memex-portal-deployment-75597864c8-vllh5, memex-portal-deployment-9bf79b774-b8dl6, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-c7f5ff965-vk9zq, memex-portal-deployment-c7f5ff965-7kj5l, memex-portal-deployment-85859db964-ghsbw, memex-portal-deployment-85859db964-rfsd7, memex-portal-deployment-58c88885f8-lhgp7, memex-portal-deployment-67b654495f-d8g8n, memex-portal-deployment-67b654495f-6xjg9, memex-portal-deployment-85859db964-bldn9, memex-portal-deployment-77c854cfcf-ns2r5, memex-portal-deployment-77c854cfcf-rcbtw, memex-portal-deployment-77c854cfcf-clz9l, memex-portal-deployment-5c8c476d54-h22p6, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-58c859d4cc-tqhzx, memex-portal-deployment-5f455dbc49-bzt6n, memex-portal-deployment-5f455dbc49-kcmxm, memex-portal-deployment-6457bb9544-8plns, memex-portal-deployment-6457bb9544-qpdjd, memex-portal-deployment-59955df69b-t2pbb, memex-portal-deployment-5777bff985-n98c5, memex-portal-deployment-59d8fd958f-sszkb, memex-portal-deployment-685fb69888-r8wl9, memex-portal-deployment-7ffbd9b949-rvqk8, memex-portal-deployment-75dff475b6-dg5r9, memex-portal-deployment-6b68cf8588-rqt25, memex-portal-deployment-6b68cf8588-bgrjp, memex-portal-deployment-6b68cf8588-tqjxt, memex-portal-deployment-6b68cf8588-zzs46

Recent log lines

2026-09-18 19:58:01Z memex-portal-deployment-6b68cf8588-zzs46 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 2 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.
2026-09-18 20:02:20Z memex-portal-deployment-6b68cf8588-tqjxt fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 2 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.
2026-09-18 20:04:02Z memex-portal-deployment-6b68cf8588-zzs46 fail: MeshWeaver.Graph.ActivityTracking[0]
      Failed to track activity for user=rbuergi path=rbuergi
      MeshWeaver.Mesh.MeshNodeStreamException: MeshNode Conflict at 'rbuergi/_UserActivity/rbuergi': cross-hub write PARTIALLY refused: 2 field(s) changed on the owner since the writer's base. What did not conflict was kept — re-read and re-apply so the refused field(s) converge.

Closing on its own subject, with the live half split out to #4912 so it is not lost.

This issue's filed defect is fixed. #1961 landed it and MeshNodeStreamExtensions.cs:318 names it outright — a Conflict re-attempt that rebuilt its patch from a base the owner had just refused, producing three identical patches and then a terminal exception. "A retry whose input cannot change is not a retry."

What keeps reopening this is a different fault, and the 2026-09-18 analysis on this thread established it: the incident fingerprints on category + exception type + the outer line (Failed to track activity for user=… path=…), never the inner conflict detail, so two inner faults share one bucket. Every one of the incident's ten stored samples carries cross-hub write **PARTIALLY** refused … What did not conflict was kept — #2463's deliberate partial-refusal NACK, introduced after this fix — and not one carries this issue's filed refused … and nothing was applied.

I have not simply re-closed it on that reasoning, because doing so twice more is what produced this loop. The live half now has its own ticket: #4912, which carries the three 2026-09-18 occurrences, why the shared retry budget deliberately excludes a partial refusal (it is scoped to NACKs where the owner stated the patch never applied, and a partial one did partly apply), and the two defensible remedies at the call site.

If the bot reopens this again it will be on that same partial-refusal message, which is now #4912's subject — and the durable fix for the bouncing is to include the inner conflict kind in the fingerprint, noted there.

Navigation Menu

Repository navigation

ActivityTracking fails to write user activity on optimistic concurrency conflict #1910

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ActivityTracking fails to write user activity on optimistic concurrency conflict #1910

Description

Evidence

Activity

systemorph-com commented on Sep 17, 2026

rbuergi commented on Sep 18, 2026

systemorph-com commented on Sep 18, 2026

systemorph-com commented on Sep 18, 2026

systemorph-com commented on Sep 18, 2026

rbuergi commented on Sep 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions