Skip to content

Read-after-delete can serve the deleted entity: the shared read stream (#3432) returns a snapshot older than an acknowledged write #3971

Description

@rbuergi

What failed

MeshWeaver.AI.Test.UnifiedContentAccessTest.DeleteUnifiedReferenceRequest_DataEntity_DeletesEntity failed inside core CD, which is what stopped the platform publishing:

Continuous Delivery (main) run 34517839623, sha d540b0dc
job "Plugins: pack the module bundles the bake composes / Module tests (MeshWeaver.AI)"
  MeshWeaver.AI.Test.UnifiedContentAccessTest.DeleteUnifiedReferenceRequest_DataEntity_DeletesEntity [FAIL]
  AssertionException : Expected value to be <null>, but found
    TestPricing { Id = delete-test-025ce4f7…, Name = To Delete, Status = Active }
  Failed: 1, Passed: 1940 — 170 ms

The test creates an entity, reads it, deletes it (asserting Success == true and Error == null — both held), then reads again and expects null. The delete was acknowledged and the next read still returned the entity.

It is a RACE, not a deterministic break — measured both ways

run same test
core CD 34517839623 (d540b0dc) FAIL
MeshWeaver.Plugins#1630, Module bundles (floor) / Module tests (MeshWeaver.AI) PASS

Same test, same lane, same core. Not in .github/known-flakes.json, and no prior issue names it.

Mechanism

4a79c371 ("fix(#3432): a read shares one stream instead of minting a permanent hub", on main since 2026-09-10T17:59:56Z) replaced three call sites in src/MeshWeaver.Data/DataExtensions.cs:

-var stream = workspace.GetStream(reference, x => x.ReturnNullWhenNotPresent());
+var stream = workspace.GetNullableStream(reference);

in GetDataResponseObservable, GetDataFromWorkspaceCore and DeleteDataPath, and made ReturnNullWhenNotPresent part of the local stream cache key so all three now share ONE ISynchronizationStream per (reference, nullReturn).

That is the right fix for #3432 — the measured 1:1 sync/ hub minting per read, ≈985 hubs/h — but it changes the read's freshness contract. Before, each read minted its own reduce and initialized from the owner, so a read issued after an acknowledged write observed that write. Now the read takes .Take(1) off a shared, already-materialised stream, whose latest emission can predate a RequestChange that has already reported Committed. DeleteDataPath itself reads from that same shared stream (.Timeout(30s).Take(1)) before issuing its DataChangeRequest, so the delete and the verifying read are now coupled to one snapshot source.

The window is small, which is why it is intermittent, and why 1940 sibling assertions in the same suite pass.

Why no gate caught it

The test lives in MeshWeaver.Plugins (src/MeshWeaver.AI.Test), so core's own CI never runs it — this is the documented shape-7 blind spot in AGENTS.md: "a method's BEHAVIOUR changing behind an unchanged signature, which no surface detector can see by construction", with no Dependent suites (MeshWeaver.Plugins) check (#2689). It surfaced only because MeshWeaver#3933 routed core CD's four composed Plugins modules through the reusable module packer, which runs each module's own suite. That is CD catching it at delivery rather than a gate catching it at merge.

What a fix has to establish

A read issued after a write that has reported Committed must not observe a state older than that write. The shared stream is the right structure; what is missing is that the read either waits for the stream to reach at least the written version, or the write invalidates/advances the shared entry before it answers. Not a .Delay, a retry, or a widened .Timeout — those hide the window rather than closing it.

Reverting 4a79c371 would trade this correctness defect back for #3432's measured ≈385 MB/h leak, so it is not the answer either.

Refs #3432, #3933.

Activity

  1. rbuergi commented on Sep 11, 2026

    @rbuergi
    ContributorAuthor

    Completed by #3985. The fix establishes a versioned causal boundary between the acknowledged owner write and the shared read stream, including updates, deletes, repeated no-ops, same-ID recreation, identity mismatches, and deferred AccessContext. Core CD 8312 ran the previously failing downstream MeshWeaver.AI module suite on the exact merge commit and passed, then sealed the bundle. The later production image at 45306a3 includes that merge; both production replicas converged and the exact live NodeType error sweep returned zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions