Flake: InstanceSyncPushTest.A_pending_content_change_pushes_a_DELETE_when_the_local_node_is_being_deleted times out #1617

Observation

MeshWeaver.InstanceSync.Test.InstanceSyncPushTest.A_pending_content_change_pushes_a_DELETE_when_the_local_node_is_being_deleted failed on run 31835443168 (shard 0):

System.TimeoutException : The operation has timed out.
  at InstanceSyncTestBase.WaitForRemote(...) InstanceSyncTestBase.cs:98
  at InstanceSyncPushTest...() InstanceSyncPushTest.cs:136

Line 136 is the final await WaitForRemote(r => r.Node("theta/doc") is null) — the remote never saw the DELETE.

Not a timing-margin flake

The obvious explanation — exponential backoff outrunning the 30 s wait — is refuted. InstanceSyncTestBase overrides every interval to be tight:

DrainDebounce = 50ms   PullInterval = 200ms
RetryInitial  = 100ms  RetryMax     = 400ms

So the 30 s wait covers roughly 150 pull attempts and 75 retry probes. The drain did not merely arrive late; it never produced a DELETE.

Not attributable to the PR it appeared on

Observed on a branch whose diff is MeshWeaver.Plugin.Build (a build-time CLI) plus a workflow file — no source path to InstanceSync. main is green on 82cd60a81 with all 12 shard jobs passing, including this project.

Hypothesis (NOT verified)

The test's sequence is: go offline → accumulate a content change → RecentlyDeletedRegistry.MarkDeleted("theta/doc") (synchronous, emits no change event) → reconnect → expect the drain to convert the pending push into a DELETE.

A drain attempt that read the local node before MarkDeleted landed could complete after Remote.Unreachable = false, push a stale UPDATE, and clear the pending entry — leaving the remote node present with nothing left to drain. That matches the symptom (no DELETE, no further attempts) and is load-sensitive, since it needs an in-flight attempt straddling the flip.

Stated as a hypothesis: I have not reproduced it or instrumented the drain path.

Suggested next step

flake-repro.yml at ≥100 iterations on this single test would settle both the rate and the hypothesis, and would show whether the drain pushes an UPDATE rather than nothing.