Repository navigation
Webhook inbox delivery answers a bare 500 on a fan-in stall instead of 503, and is starved by the known pedestrian-walk pg-read head-of-line blocking #6125
Description
What is failing
The webhook inbox delivery route (WebhookInboxEndpoints.Deliver, Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:108) faults when its path:Hosting/PlatformBuilds read misses the query fan-in's 15 s Initial budget — a one-row exact probe issued under the system identity. The probe itself is not the defect: it is the known victim of the pedestrian satellite walk's head-of-line starvation of the process-wide pg-read:Postgres pool (cap 16), documented 2026-10-04 in Doc/Architecture/QueryFanInStallTerminal → "Head-of-line starvation of pg-read by the pedestrian satellite walk". A second, endpoint-local defect: the route lets QueryProviderStalledException escape as an unhandled exception, so ASP.NET answers a bare 500 instead of the instance-key 503 + Retry-After convention the plugin-bundle routes adopted after #5345 (InstanceAuthResponses.UnavailableOnAStalledRead, classifier AreaErrorClassifier.IsStorageUnavailable).
Probable cause
High confidence on the mechanism, because the architecture page records this exact query shape and provider as one of the two recurring stall shapes between 2026-09-27 and 2026-10-04: ThreadSupervisor.Scan's newest/Admin thread query drove StorageAdapterMeshQueryProvider's unbounded pedestrian walk (~2×N pg-read admissions, unbounded SelectMany), and every read issued after it — including the one-row Hosting/PlatformBuilds probe — waited behind the walk's whole remainder. A two-part fix (defer scoped satellite reads to the native provider; bound the walk to WalkConcurrency = 4, pinned by PedestrianWalkHeadOfLineTest) is recorded as made but its rollout to the memex portal pods is not established; today's recurrence suggests these pods do not yet carry it, or the walk is not the whole cause (a per-query pool-admission census still does not exist). The endpoint's bare-500 mapping is a separate, certain defect: the fan-in's terminal is doing exactly what it is for, and this consumer — unlike the plugin-bundle routes — was never converted.
Impact
Small: a handful of occurrences across two portal pods over ~12 hours, so an intermittent 500 on webhook delivery, retryable by the sender. The stall itself is a fleet-wide availability risk (routing gates on the same fan-in) tracked under #5315; this ticket is about this endpoint's share of the blast radius.
Where to look
Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs(theDeliverroute) — mapQueryProviderStalledExceptionto 503 +Retry-AfterviaInstanceAuthResponses.UnavailableOnAStalledRead, asPluginBundleEndpointsdoes; any other fault still escapes so real defects surface.MeshWeaver.Hosting.StorageAdapterMeshQueryProvider— confirm the 2026-10-04 walk fix (defer scoped satellite reads,WalkConcurrencybound) has actually rolled to thememexportal images; the doc pins the mechanism but records no production timing data.MeshWeaver.Plugins ThreadSupervisor.Scan— the sweep that issues the Admin/Thread walk.- Context:
Doc/Architecture/QueryFanInStallTerminal, issue PostgreSQL query provider stalls past the fan-in Initial budget on plugin-bundle listing for anonymous portal users #5345 (the same escape-as-500 defect in a sibling route), and MessageHubGrain activations fault when query providers stall past the 15s fan-in bound (access/entitlements queries unavailable) #5315 (the open provider-stall investigation this instance belongs to). Do not bumpQueryInitialBudget; policyquery-fanin-stall-terminalis fixed and correct here.
Evidence
| Fingerprint | b1942b90104d63ed |
| Category | Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware |
| Severity | Error |
| Exception | MeshWeaver.Mesh.QueryProviderStalledException |
| Top frame | Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) |
| Namespace | memex |
| Pods | memex-portal-deployment-76dd69c7fc-xxql9, memex-portal-deployment-6d8c9665cc-jltnc |
| Occurrences | 3 |
| First seen | 2026-10-04 19:25:46Z |
| Last seen | 2026-10-05 07:44:45Z |
| Routing | not determined — no configured route matches the category Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-10-04 19:25:46Z memex-portal-deployment-76dd69c7fc-xxql9 fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Hosting/PlatformBuilds' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 108
at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
at Memex.Portal.Shared.Authentication.OnboardingMiddleware.Continue(HttpContext context, IObservable`1 pendingDecision) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 191
at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 319
at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass12_0.<<UsePublicHostRedirect>b__0>d.MoveNext() in /home/runner/work/MeshWeav…[truncated]
2026-10-04 19:25:46Z memex-portal-deployment-76dd69c7fc-xxql9 fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HNP27FHC82EM", Request id "0HNP27FHC82EM:00000001": An unhandled exception was thrown by the application.
MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Hosting/PlatformBuilds' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 108
at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
at Memex.Portal.Shared.Authentication.OnboardingMiddleware.Continue(HttpContext context, IObservable`1 pendingDecision) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 191
at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 319
at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass12_0.<<UsePublicHostRedirect>b__0>d.MoveNe…[truncated]
2026-10-05 07:44:45Z memex-portal-deployment-6d8c9665cc-jltnc fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Hosting/PlatformBuilds' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 108
at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
at Memex.Portal.Shared.Authentication.OnboardingMiddleware.Continue(HttpContext context, IObservable`1 pendingDecision) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 191
at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 319
at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass12_0.<<UsePublicHostRedirect>b__0>d.MoveNext() in /home/runner/work/MeshWeav…[truncated]
Opened automatically from Admin/_LogIncident/b1942b90104d63ed. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 4660764b5f543068: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
systemorph-com commentedon Oct 5, 2026
Still occurring: 2 more since the last update (5 total).
memex-portal-deployment-76dd69c7fc-xxql9,memex-portal-deployment-6d8c9665cc-jltnc,memex-portal-deployment-6485bffdff-cgd5rRecent log lines
Merge pull request #6322 from Systemorph/fix/6125-6039-webhook-inbox