Repository navigation
Read-after-delete can serve the deleted entity: the shared read stream (#3432) returns a snapshot older than an acknowledged write #3971
Description
What failed
MeshWeaver.AI.Test.UnifiedContentAccessTest.DeleteUnifiedReferenceRequest_DataEntity_DeletesEntity failed inside core CD, which is what stopped the platform publishing:
Continuous Delivery (main) run 34517839623, sha d540b0dc
job "Plugins: pack the module bundles the bake composes / Module tests (MeshWeaver.AI)"
MeshWeaver.AI.Test.UnifiedContentAccessTest.DeleteUnifiedReferenceRequest_DataEntity_DeletesEntity [FAIL]
AssertionException : Expected value to be <null>, but found
TestPricing { Id = delete-test-025ce4f7…, Name = To Delete, Status = Active }
Failed: 1, Passed: 1940 — 170 ms
The test creates an entity, reads it, deletes it (asserting Success == true and Error == null — both held), then reads again and expects null. The delete was acknowledged and the next read still returned the entity.
It is a RACE, not a deterministic break — measured both ways
| run | same test |
|---|---|
core CD 34517839623 (d540b0dc) |
FAIL |
MeshWeaver.Plugins#1630, Module bundles (floor) / Module tests (MeshWeaver.AI) |
PASS |
Same test, same lane, same core. Not in .github/known-flakes.json, and no prior issue names it.
Mechanism
4a79c371 ("fix(#3432): a read shares one stream instead of minting a permanent hub", on main since 2026-09-10T17:59:56Z) replaced three call sites in src/MeshWeaver.Data/DataExtensions.cs:
-var stream = workspace.GetStream(reference, x => x.ReturnNullWhenNotPresent());
+var stream = workspace.GetNullableStream(reference);in GetDataResponseObservable, GetDataFromWorkspaceCore and DeleteDataPath, and made ReturnNullWhenNotPresent part of the local stream cache key so all three now share ONE ISynchronizationStream per (reference, nullReturn).
That is the right fix for #3432 — the measured 1:1 sync/ hub minting per read, ≈985 hubs/h — but it changes the read's freshness contract. Before, each read minted its own reduce and initialized from the owner, so a read issued after an acknowledged write observed that write. Now the read takes .Take(1) off a shared, already-materialised stream, whose latest emission can predate a RequestChange that has already reported Committed. DeleteDataPath itself reads from that same shared stream (.Timeout(30s).Take(1)) before issuing its DataChangeRequest, so the delete and the verifying read are now coupled to one snapshot source.
The window is small, which is why it is intermittent, and why 1940 sibling assertions in the same suite pass.
Why no gate caught it
The test lives in MeshWeaver.Plugins (src/MeshWeaver.AI.Test), so core's own CI never runs it — this is the documented shape-7 blind spot in AGENTS.md: "a method's BEHAVIOUR changing behind an unchanged signature, which no surface detector can see by construction", with no Dependent suites (MeshWeaver.Plugins) check (#2689). It surfaced only because MeshWeaver#3933 routed core CD's four composed Plugins modules through the reusable module packer, which runs each module's own suite. That is CD catching it at delivery rather than a gate catching it at merge.
What a fix has to establish
A read issued after a write that has reported Committed must not observe a state older than that write. The shared stream is the right structure; what is missing is that the read either waits for the stream to reach at least the written version, or the write invalidates/advances the shared entry before it answers. Not a .Delay, a retry, or a widened .Timeout — those hide the window rather than closing it.
Reverting 4a79c371 would trade this correctness defect back for #3432's measured ≈385 MB/h leak, so it is not the answer either.
Activity
rbuergi commentedon Sep 11, 2026
Completed by #3985. The fix establishes a versioned causal boundary between the acknowledged owner write and the shared read stream, including updates, deletes, repeated no-ops, same-ID recreation, identity mismatches, and deferred AccessContext. Core CD 8312 ran the previously failing downstream MeshWeaver.AI module suite on the exact merge commit and passed, then sealed the bundle. The later production image at 45306a3 includes that merge; both production replicas converged and the exact live NodeType error sweep returned zero.