A starved permission fold parks every [RequiresPermission] delivery until the caller's own timeout — AccessControlPipeline has no terminal, and the bounded twin attributes the same event correctly #2742
Description
What is failing
PermissionEvaluator.GetEffectivePermissions is a CombineLatest fold over the grant and policy reads of the target's scope and every ancestor scope. Per Access Control:
A leg that starves never emits, never completes and never errors … The fold therefore has no terminal at all, and a
.Take(1)around it bounds the number of emissions rather than the wait. This is not hypothetical — it is the ordinary cross-silo shape, where the owning activation lives on a peer silo that is busy or has just gone away.
There are two gates in front of that fold and they behave differently:
| gate | bounded? | what the caller sees |
|---|---|---|
RlsNodeValidator (node operations) |
✅ PermissionEstablishmentBudget, 20 s |
Unavailable — attributed to the starved read |
AccessControlPipeline (every [RequiresPermission] message, incl. SubscribeRequest) |
❌ none | the caller's timeout, 60 s later, naming the caller rather than the cause |
So one event produces two different-looking failures, and the more common one is the uninformative one.
Evidence
Reported from a satellite deployment (private repo, so the detail stays there): intermittent first-read-after-idle failures, some surfacing as an attributed Unavailable, others as No response received in hub cache/… within 00:01:00 for request SubscribeRequest → target …, plus portal pages hanging on "Subscribing to …". The two shapes are the two rows above. A cold fold has legs that must reach a peer activation; a warm one emits synchronously from the process-wide cached queries — which is exactly a first-read-starves / retry-succeeds shape, and it needs no replica straddle to explain it.
🚨 The fix is NOT a timeout on the gate
AccessControlPipeline.cs:233 says so outright:
// No Timeout here: the access cache must always be a reactive Subscribe … A 10s wait was a
// workaround for a wedged cache — fix the cache, don't ceiling-block here.and that is the house rule (AGENTS.md → No band-aids): raising or adding a bound to make a symptom pass is not the fix. The defect is that a leg of the fold can starve — never emitting, completing, or erroring — so the work is to give those legs a terminal (an owner that cannot answer must answer something: the TargetUnserved/ShuttingDown verdicts the routing layer now produces are the model), not to ceiling-block the gate that waits on them.
Note the doc's own caveat, which is why this is filed as a decision and not just a patch: giving the shared gate a terminal "changes the behaviour of every gated message and wants its own argument". If the conclusion is that the gate should also carry a terminal as defence-in-depth, that is a deliberate platform decision to be argued here — not something to slip in under a symptom fix.
Where to look
src/MeshWeaver.Mesh.Contract/Security/PermissionEvaluator.cs—GetEffectivePermissions, theCombineLatestfold and its legs. (PermissionEvaluator and the layout error placeholder resolve services from the hub's DI scope after it has been disposed #2679 / PR fix(#2679): the permission fold resolves its services once; a render on a disposed scope is a teardown race, not two errors #2728 reworked this fold today so every service resolves once at entry rather than per emission, and a scope-teardown fault now terminates it as a classified benign teardown. That removes a different hazard on the same path — it does not address leg starvation, but any measurement should be taken on a build containing it.)src/MeshWeaver.Hosting/Security/AccessControlPipeline.cs:233— the deliberate absence of a terminal, and:80-105for the synchronous prologue that IS guarded (Three more sites report an availability failure as a definitive negative (permission-fold fault → "Access denied", read timeout → "Not found", cold cache → "user unknown") #974) and answersUnavailableon a disposing hub.RlsNodeValidator'sPermissionEstablishmentBudget— the bounded twin, and the model for what an attributed answer looks like.
The measurement that discriminates
On a current build, hit an idle node and check which of the two shapes comes back: the attributed Unavailable (bounded path) or the 60 s SubscribeRequest timeout (unbounded path). If it is always the latter, the caller-side symptom is the gate's missing terminal; either way the starving leg is the root.
🤖 Generated with Claude Code
Activity
fix(#2742): close the SAME silent-terminal hole in RlsNodeValidator —…
fix(#2742): build both silent-terminal fallbacks LAZILY — DefaultIfEm…
test(#2742): drop the 4 `.ToTask(` bridges these new tests introduced…
fix(security): three more permission swallows — and the guard blind s…
fix: gate PreserveAuthorship, and refuse rather than fall back when p…
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-08-30)
The original deployment-side report, ported here in full (maintainer approved making it public)
This issue was filed with the reporting detail withheld because the report lived in a private repo and GitHub refuses a private → public transfer. The maintainer has since confirmed the content is fine to publish, so the original is reproduced here and the private issue is closed against this one.
Originally reported 2026-08-28 by @carson-client-g in
Systemorph/MeshWeaver.SocialMedia#90, "Intermittent read failures: SubscribeRequest timeouts and unreachable-hub errors on existing nodes":How the report maps onto the mechanism above
RlsNodeValidator's 20 sPermissionEstablishmentBudget). Working as designed: it fired, attributed the starved read, returned fast.cache/…within 00:01:00 for requestSubscribeRequest" and the pages stuck on "Subscribing to…" — the unbounded path. Same underlying event; nothing bounded it, so it surfaced 60 s later as the caller's impatience, naming the caller rather than the read that starved. That is also why the message hedges with "may have been undeliverable or the target hub was not found" — the gate never reached a verdict to report.One reporter hypothesis that does not hold
The replica-identity straddle ("two pods on ci.5670, one crash-looping on ci.6011") was true when observed but is not the current state: both portals have since been measured serving one uniform build on 6/6 samples each. If the intermittent reads persist on a uniform build, replica straddle can be struck from the list — which is what points at the fold rather than at routing.
The 3–6 retry loops are the real cost
Worth keeping in view when this is sized: every automation against this mesh carries backoff retries to survive a first-read failure. That is the user-visible price of the missing terminal, and it is paid on every read path, not only the ones that eventually error.