fix(#2543): a synchronous fault while building the flush is a NACK with a name, never silence

[rbuergi]

#2543: a synchronous fault on the flush path is a NACK with a name, never silence

The observation (CD 7937's bake host, core 85ab1dfbe, which carries #3464): 13 OwnerUnreachable … no verdict within 31s stalls; every trail ends at PATCH_ECHO_SEEN; zero PATCH_FLUSH_SUBSCRIBED; every [STALE-CALLBACK] line reads pendingWork=0. The starved-pool arm is refuted, and IoPool.Run/Invoke both SubscribeOn(TaskPoolScheduler.Default), so the storage write cannot park the subscribing thread. The only code between the two stages is the ack watcher's own onNext: flush(committed) — StoragePostCommitFlush.Flush, which resolves three services from the hub's provider — then Subscribe. An exception thrown there escapes the commit-echo subscription's onNext (Rx routes an onNext throw to the producer, not to onError): no stage, no ack, no bound, silence. A closed lifetime scope makes GetService throw exactly there — the incident #3196 documents for the sibling lateVerdicts resolution — and a bake host recycles hubs en masse. Reading on the issue: #2543 (comment)

What changes

• ArmPatchAckWatcher: building the flush is inside try/catch; a throw records PATCH_FLUSH_FAULTED_SYNC building <Exception>, logs [PatchAck] FLUSH_FAULTED_SYNC, and NACKs once with the classified error — the verdict the async fault arm already posts, so the writer retries instead of waiting out its bound. (A raw Subscribe throw is not a case: Take(1) subscribes through SubscribeSafe, which forwards it into the fault arm — measured, and the doc says so.)
• StoragePostCommitFlush.Flush: the three service resolutions are guarded; ObjectDisposedException becomes the observable's fault instead of a synchronous escape.
• DebuggingMessageFlow.md: the new stage row.

Verified

PatchAckTotalityTest.AFlushFactoryThatThrows_NacksOnceWithTheClassifiedError_AndNamesTheException — before the change the throw surfaced out of ArmPatchAckWatcher (measured on the first run of this test); after: one NACK, the named stage, nothing subscribed. Ack suites 22/22; Documentation + guards 304/304; -c Release -warnaserror clean on Data.Test, Memex.Portal.Shared.Test (Hosting closure), Documentation.Test.

This does not yet prove the closed-scope cause — the next occurrence's trail names the exception, which is the proof. It does close the silence: a stall in this arm is now a retryable NACK. Refs #3355.

[Copilot]

🟢 Approval recommended

The change is narrowly scoped, aligns with the documented failure mode, and is covered by a targeted regression test plus updated diagnostic documentation.

Pull request overview

This PR hardens the mesh-node patch ack pipeline so that synchronous exceptions thrown while constructing the durable flush can no longer escape an Rx onNext and leave writers waiting in silence; instead, the owner produces a single retryable NACK and records a diagnostic stage naming the exception.

Changes:

• Wraps ArmPatchAckWatcher’s flush construction in a try/catch so synchronous faults become a classified NACK and a PATCH_FLUSH_FAULTED_SYNC … stage instead of an unobservable stall.
• Guards StoragePostCommitFlush.Flush against ObjectDisposedException during service resolution by surfacing it as an observable fault.
• Adds a regression test for the “flush factory throws” case and documents the new trail stage in DebuggingMessageFlow.md.

File summaries

File	Description
test/MeshWeaver.Data.Test/PatchAckTotalityTest.cs	Adds a test proving a synchronous flush-factory throw yields exactly one classified NACK and records the fault stage.
src/MeshWeaver.Hosting/Persistence/StoragePostCommitFlush.cs	Converts disposed-scope service-resolution throws into an observable fault rather than a synchronous escape.
src/MeshWeaver.Documentation/Data/Architecture/DebuggingMessageFlow.md	Documents the new PATCH_FLUSH_FAULTED_SYNC … trail meaning for #2543 debugging.
src/MeshWeaver.Data/DataExtensions.cs	Catches flush-construction exceptions inside the commit-echo onNext, records a stage, logs, and NACKs once to avoid silence.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results (shard 0)

251 tests   251 ✅  2m 12s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 7d55681.

[github-actions]

Test Results (shard 3)

1 016 tests   1 016 ✅  1m 4s ⏱️
    2 suites      0 💤
    2 files        0 ❌

Results for commit 7d55681.

[github-actions]

Test Results (shard 1)

488 tests   488 ✅  45s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 7d55681.

[github-actions]

Test Results (shard 5)

    5 files      5 suites   1m 52s ⏱️
1 416 tests 1 416 ✅ 0 💤 0 ❌
1 417 runs  1 417 ✅ 0 💤 0 ❌

Results for commit 7d55681.

[github-actions]

Test Results (shard 4)

1 597 tests   1 403 ✅  2m 5s ⏱️
    4 suites    194 💤
    4 files        0 ❌

Results for commit 7d55681.

[github-actions]

Test Results (shard 2)

1 048 tests   1 048 ✅  3m 24s ⏱️
    3 suites      0 💤
    3 files        0 ❌

Results for commit 7d55681.

[github-actions]

Test Results

   16 files     16 suites   11m 23s ⏱️
5 816 tests 5 622 ✅ 194 💤 0 ❌
5 817 runs  5 623 ✅ 194 💤 0 ❌

Results for commit 7d55681.