[rbuergi]

#6125: the root cause

memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:108 (before this change) awaited WebhookInbox.Deliver(...) with no mapping for availability faults. Deliver's target-existence read (path:{target}, WebhookEventNodeType.cs Store) can end in QueryProviderStalledException when a provider misses the fan-in's Initial budget. That exception escaped the handler, so ASP.NET answered a bare 500.

The fix: the delivery verdict is mapped to an IResult in a Select, then passed through InstanceAuthResponses.UnavailableOnAStalledRead, the same convention the plugin-bundle routes use (#5345). Only what AreaErrorClassifier.IsStorageUnavailable classifies becomes 503 + Retry-After. Every other fault still escapes. ReadUnavailable and UnavailableOnAStalledRead gain an optional error text, so the 503 body names what this route could not read ("nothing was stored; retry shortly"). The bundle routes keep their wording because the default is unchanged.

The underlying pg-read starvation (#5315 / QueryFanInStallTerminal) is untouched, and QueryInitialBudget is not changed.

Test and negative control

test/Memex.Portal.Shared.Test/WebhookInboxStalledReadTest.cs uses a real TestServer host over the real mesh. A test IMeshQueryProvider, the same harness as PluginBundleStalledReadTest, stalls or faults only the target read. The stall fires at the production rung. There are three cases:

• a stalled read answers 503, with Retry-After set to 5 s and the route's own error text;
• an answered read still answers 200 accepted;
• an ordinary fault still escapes and is not answered 503.

Negative control, run locally: with the UnavailableOnAStalledRead line commented out, AStalledTargetRead_Answers503WithRetryAfter FAILS with the production exception verbatim (QueryProviderStalledException … for query 'path:Hosting/StalledInbox' (user 'system-security') at WebhookInboxEndpoints.Deliver). With the line in, all 8 Webhook* tests pass, and PluginBundleStalledReadTest passes 4/4.

#6039: investigated, no code change; findings committed to Doc/Architecture/WebhookInbox

• The issue's candidate is refuted. No INodePostCreationHandler in core or MeshWeaver.Plugins matches WebhookEvent. So RunPostCreationHandlersObs is Observable.Empty, and CreateNodeResponse.Ok is posted synchronously right after the Node created at … line (MeshExtensions.cs ~1540–1589).
• What remains is the reply's dispatch on portal/nodeops-{meshId}. I tried issuing the create from portal/reads-{meshId} (hub.ReadIssuingHub(), the A freshly uploaded content file 503s for minutes (10s timeout per request) before it starts serving — /api/content settle window #2901 cure). A deterministic repro held the nodeops block after the commit and before the reply. The delivery was still not answered. The request-fate trail shows why: the responder enqueues its own outgoing reply on its own block (RESPONSE_POSTED target=portal/reads-… ↩ reply#1: RECEIVED @portal/nodeops-… → ENQUEUED → QUEUED queue=main). No endpoint-local change separates the answer from that block, so that change was dropped. No timeout was added or raised.
• Not established: which turn held nodeops at the time of the incident (Bake wedge: portal/nodeops stops draining mid-bake — one shared node-CRUD hub seen from eight places (queue latency bimodal: <=3.2s vs 33-49s) #2543). The two framework-level options are recorded on the doc page.

Closes #6125 is deliberately not used, because sev:M issues close after deployed verification. Refs #6125. Refs #6039.

Pairs-with: none — no public type or member removed (one optional parameter added to two internal methods)
Mirror-sync: none — no i18n catalog change

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The new test uses an unsafe reactive impersonation scope that can leak System identity across threads.

1 open finding

• Avoid cross-thread AsyncLocal impersonation disposal · New

What changed in this PR

Maps stalled webhook target reads to retryable HTTP 503 responses and documents related delivery behavior.

Changes:

• Adds 503/Retry-After mapping with route-specific error text.
• Adds integration coverage for stalled, successful, and ordinary-fault cases.
• Documents #6125 and the #6039 investigation.

File	Description
memex/​Memex.Portal.Shared/​Api/​WebhookInboxEndpoints.cs	Maps availability failures to 503 responses.
memex/​Memex.Portal.Shared/​Api/​InstanceAuthResponses.cs	Supports route-specific 503 error text.
test/​Memex.Portal.Shared.Test/​WebhookInboxStalledReadTest.cs	Tests webhook target-read outcomes.
src/​MeshWeaver.Documentation/​Data/​Architecture/​WebhookInbox.md	Documents behavior and investigation findings.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[rbuergi]

Correct — fixed in 2c1b4ed. The target seed now goes through access.RunAsSystem(() => …CreateOrUpdateNode(...)), the sealed boundary WebhookInbox.Store itself uses (#1790), so the impersonation enters and leaves inside one synchronous Subscribe and cannot latch the test thread as System. Rebuilt Release with -warnaserror (0 errors / 0 warnings); all 8 Webhook* tests in Memex.Portal.Shared.Test pass.

[github-actions]

Test Results

    22 files  +    1      22 suites  +1   48m 17s ⏱️ + 2m 47s
11 144 tests +1 330  10 951 ✅ +1 330  193 💤 ±0  0 ❌ ±0 
11 157 runs  +1 333  10 964 ✅ +1 333  193 💤 ±0  0 ❌ ±0 

Results for commit 2c1b4ed. ± Comparison against base commit 3fc485c.

This pull request removes 29 and adds 1341 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
 ---> System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432)
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "fec53f59-6c80-436a-ae1c-90decad9ab50")
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
Memex.Portal.Shared.Test.WebhookInboxStalledReadTest ‑ AStalledTargetRead_Answers503WithRetryAfter
Memex.Portal.Shared.Test.WebhookInboxStalledReadTest ‑ AnAnsweredTargetRead_StillAccepts
Memex.Portal.Shared.Test.WebhookInboxStalledReadTest ‑ AnOrdinaryFault_OnTheTargetRead_IsNotConvertedTo503
MeshWeaver.Compiler.Pipeline.Test.AFailedCompileCarriesTheSetItConsumedTest ‑ TheFailureResult_FoldsTheConsumedSetUnderTheSameRuleAsTheLiveSnapshot
MeshWeaver.Compiler.Pipeline.Test.ALateReleaseIsAdoptedWhenItLandsTest ‑ APrebuiltAdoption_RetiresThePreviousBuildsPendingRelease
MeshWeaver.Compiler.Pipeline.Test.ALateReleaseIsAdoptedWhenItLandsTest ‑ AReleaseAtAnotherPath_IsNotAdopted_AndTheStampStands
MeshWeaver.Compiler.Pipeline.Test.ALateReleaseIsAdoptedWhenItLandsTest ‑ ARelease_LandingAtTheStampedPath_IsAdopted
MeshWeaver.Compiler.Pipeline.Test.ALateReleaseIsAdoptedWhenItLandsTest ‑ AReplayOfThePendingBuild_PreservesItsMarker
…

♻️ This comment has been updated with latest results.