PermissionEvaluator and the layout error placeholder resolve services from the hub's DI scope after it has been disposed

What is failing

While a layout area is rendering, the reactive chain resolves services from the hub's Autofac scope. Two paths fault with ObjectDisposedException from the dead LifetimeScope when that scope is disposed mid-render:

The permission fold: PermissionEvaluator.GetEffectivePermissions calls GetRequiredService inside a Reactive SelectMany selector (src/MeshWeaver.Mesh.Contract/Security/PermissionEvaluator.cs, lines 126 and 237) — so the first stream item arriving after disposal throws straight into the render chain.
The fallback for that very failure: LayoutAreaHost.CreateRenderErrorControl (line 838) → LayoutAreaLocalizationExtensions.Localize (line 26) calls GetService<T> through the same disposed scope, so the error placeholder for the area fails too. One disposal therefore produces two fail entries, and the user gets a dead area with no error message.

Probable cause

High confidence on the mechanism, medium on the trigger. The hub's DI scope was disposed while the area-render chain (permission observable + render) was still subscribed. Permission observables are long-lived by design — they re-emit whenever an AccessAssignment changes (Permission API) — so a chain built during a render keeps resolving services until it is cancelled. If the scope is disposed before that chain is drained or cancelled, the next emission resolves from the dead scope. This is exactly the DI-scope half of the documented teardown contract: Controlled I/O Pooling ("The mesh teardown drains THREE things, not one") names this precise exception as the failure mode of disposing the service scope while reactive work is still in flight.

The pool-primitives half of this family was already fixed (#2134 / #2135, plus the admission-count regions and #2146 — the pool now answers OperationCanceledException, never ObjectDisposedException), but the analogous guard is missing on the DI-resolution paths in PermissionEvaluator and the layout error path. Not a duplicate of #2135 despite the similar LayoutAreaHost surface: that one's top frame was the IoPool's SemaphoreSlim.Release; here it is LifetimeScope.ThrowDisposedException.

What the evidence does not show: whether the disposal that raced this render was pod shutdown (silo mesh teardown), hub deactivation, or area-level disposal. A single-instant event cannot distinguish these; the fix — making both paths disposal-tolerant — is the same in all three cases.

Impact

Low as observed: a one-shot pair of failures at a single instant, at most one degraded render. But the failure mode is structural — every hub disposal that races a live area render can reproduce it (rolling deploys and hub deactivations included), and when it fires the user gets neither the area nor the error placeholder. Worth fixing on the same grounds as the #2135 family.

Where to look

src/MeshWeaver.Mesh.Contract/Security/PermissionEvaluator.cs:126 and :237 — the GetEffectivePermissions fold. Tie service resolution to the hub's disposal state (capture the needed service before subscribe, or resolve from a scope-stable provider) and let a disposing hub terminate the stream as cancellation — the same contract the IoPool now honours ("a leaf the pool will not run is a CANCELLATION, never an ObjectDisposedException").
src/MeshWeaver.Layout/Composition/LayoutAreaHost.cs:838 (CreateRenderErrorControl) and src/MeshWeaver.Layout/Composition/LayoutAreaLocalizationExtensions.cs:26 — make the error-placeholder path disposal-tolerant: when the hub/scope is disposing, log the original render exception and skip localisation rather than throwing a second ObjectDisposedException.
The teardown boundary that raced the render — verify the three-phase drain in Controlled I/O Pooling and Mesh Lifecycle also cancels long-lived permission/render subscriptions before the service scope is disposed. Permission API already documents "a hub whose DI scope is disposing" as a fault source for permission folds — this incident is that hazard realised.

Evidence


Fingerprint	`e98f19a126f6aa4b`
Category	`MeshWeaver.Layout.Composition.LayoutAreaHost`
Severity	Error
Exception	`System.ObjectDisposedException`
Top frame	`Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()`
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-7c6b48957d-n5vkr`
Occurrences	2
First seen	2026-08-29 06:54:11Z
Last seen	2026-08-29 06:54:11Z

Recent log lines

2026-08-29 06:54:11Z memex-portal-deployment-7c6b48957d-n5vkr fail: MeshWeaver.Layout.Composition.LayoutAreaHost[0]
      Rendering failed for area Overview
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.Security.PermissionEvaluator.GetEffectivePermissions(IMessageHub hub, String nodePath, String userId) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Security/PermissionEvaluator.cs:line 126
         at MeshWeaver.Mesh.Security.PermissionEvaluator.<>c__DisplayClass11_0.<GetEffectivePermissions>b__2(ValueTuple`3 state) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Security/PermissionEvaluator.cs:line 237
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
2026-08-29 06:54:11Z memex-portal-deployment-7c6b48957d-n5vkr fail: MeshWeaver.Layout.Composition.LayoutAreaHost[0]
      Failed to render the error placeholder for area Overview
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Layout.Composition.LayoutAreaLocalizationExtensions.Localize(LayoutAreaHost host, String key, Object[] args) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaLocalizationExtensions.cs:line 26
         at MeshWeaver.Layout.Composition.LayoutAreaHost.CreateRenderErrorControl(Exception ex) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaHost.cs:line 838
         at MeshWeaver.Layout.Composition.LayoutAreaHost.RenderRenderingError(RenderingContext context, EntityStoreAndUpdates cleared, Exception ex) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaHost.cs:line 771

_{Opened automatically from Admin/_LogIncident/e98f19a126f6aa4b. Recurrences are folded into this issue rather than opening new ones.}

Reopened — this fault is happening again.

Still occurring: 6 more since the last update (8 total).

Last seen: 2026-08-30 09:59:10Z
Pods: memex-portal-deployment-7c6b48957d-n5vkr, memex-portal-deployment-767d8d6684-gdl2g, memex-portal-deployment-554545c8cc-xw7xn, memex-portal-deployment-66cfbf759-svxh9

Recent log lines

2026-08-30 08:16:40Z memex-portal-deployment-554545c8cc-xw7xn fail: MeshWeaver.Layout.Composition.LayoutAreaHost[0]
      Failed to render the error placeholder for area Overview
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Layout.Composition.LayoutAreaLocalizationExtensions.Localize(LayoutAreaHost host, String key, Object[] args) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaLocalizationExtensions.cs:line 26
         at MeshWeaver.Layout.Composition.LayoutAreaHost.CreateRenderErrorControl(Exception ex) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaHost.cs:line 838
         at MeshWeaver.Layout.Composition.LayoutAreaHost.RenderRenderingError(RenderingContext context, EntityStoreAndUpdates cleared, Exception ex) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaHost.cs:line 771
2026-08-30 09:59:10Z memex-portal-deployment-66cfbf759-svxh9 fail: MeshWeaver.Layout.Composition.LayoutAreaHost[0]
      Rendering failed for area Overview
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Graph.NodeTypeLayoutAreas.<>c__DisplayClass15_0.<CompileProgressView>b__0(Exception ex) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/NodeTypeLayoutAreas.cs:line 158
         at System.Reactive.Linq.ObservableImpl.Catch`2._.OnError(Exception error)
2026-08-30 09:59:10Z memex-portal-deployment-66cfbf759-svxh9 fail: MeshWeaver.Layout.Composition.LayoutAreaHost[0]
      Failed to render the error placeholder for area Overview
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Layout.Composition.LayoutAreaLocalizationExtensions.Localize(LayoutAreaHost host, String key, Object[] args) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaLocalizationExtensions.cs:line 26
         at MeshWeaver.Layout.Composition.LayoutAreaHost.CreateRenderErrorControl(Exception ex) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaHost.cs:line 838
         at MeshWeaver.Layout.Composition.LayoutAreaHost.RenderRenderingError(RenderingContext context, EntityStoreAndUpdates cleared, Exception ex) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Layout/Composition/LayoutAreaHost.cs:line 771

Closing — BOTH paths this issue named are fixed in current `origin/main`, and quiet 19 days

Dormancy. Newest occurrence Last seen: 2026-08-30 09:59:10Z (the bot's reopen comment above). Measured 2026-09-18T05:52Z that is 19 days quiet, with live log ingest (incidents updated 2026-09-18T05:42Z) and no portal roll in between — the fault stopped while the same image kept running.

Checked against origin/main at 229930bf2f29845323e0f350e73829b1bdb4928e.

1. The permission fold — `PermissionEvaluator.cs`

The GetRequiredService inside the Rx SelectMany selector is gone. Every dependency is now resolved once on the caller's thread and carried:

$ git -C <core> show origin/main:src/MeshWeaver.Mesh.Contract/Security/PermissionEvaluator.cs | sed -n '126,152p'
// 🚨 Every service the fold needs is resolved HERE, ONCE, on the caller's thread — never
// inside a selector (#2679). ... Every dependency travels in `services` from here on — the
// Public leg and GetRole included — so a subscribed fold never touches the scope again.
var services = ResolveFoldServices(hub);

and the fold terminates as cancellation, which is precisely the contract this ticket asked for ("let a disposing hub terminate the stream as cancellation — the same contract the IoPool now honours"):

.Catch<Permission, Exception>(ex => hub.IsTerminatedByScopeTeardown(ex)
    ? Observable.Throw<Permission>(new HubDisposingException(...))
    : Observable.Throw<Permission>(ex));

Probe-gated on purpose, so an ObjectDisposedException from an unrelated disposed dependency on a live scope still faults as the real defect it is.

2. The error placeholder — `LayoutAreaHost.cs`

RenderRenderingError now probes the scope once per failure and refuses to build a placeholder on a dead one, so the second fail line this issue reported cannot be produced:

$ git -C <core> show origin/main:src/MeshWeaver.Layout/Composition/LayoutAreaHost.cs | sed -n '817,875p'
var ownScopeDisposed = Hub.IsServiceScopeDisposed();
...
if (AreaErrorClassifier.IsHubDisposalRace(ex, () => ownScopeDisposed))
    logger.LogDebug(ex, "Area {Area} render raced a hub disposal — transient, ...");
...
// 🚨 No placeholder on a dead scope (#2679). CreateRenderErrorControl localises its copy
// through this host's scope ... so on a disposed scope the placeholder itself faulted with a
// SECOND ObjectDisposedException — one disposal produced two fail lines ...
if (ownScopeDisposed)
    return cleared;

The first log line also drops from Error to Debug for this shape, so a routine teardown race no longer files an incident at all.

Pinned by test/MeshWeaver.Layout.Test/ScopeTeardownRenderTest.cs.

Not the same defect as #2358, despite both being in this triage batch: that one is a cache holding a NodeTypeDefinition under a User key (fixed at the writer, in MeshWeaver.Graph), this one is DI resolution after scope disposal. Different subsystems, different fixes, no duplication either way.

Closing as redundant. If the fault recurs, the recurrence bot reopens this issue with fresh evidence — that is the designed behaviour, and it is why closing a dormant incident is safe: a real recurrence returns carrying a Last seen that postdates this close.

Navigation Menu

Repository navigation

PermissionEvaluator and the layout error placeholder resolve services from the hub's DI scope after it has been disposed #2679

What is failing

Probable cause

Impact

Where to look

Evidence

Closing — BOTH paths this issue named are fixed in current `origin/main`, and quiet 19 days

1. The permission fold — `PermissionEvaluator.cs`

2. The error placeholder — `LayoutAreaHost.cs`

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PermissionEvaluator and the layout error placeholder resolve services from the hub's DI scope after it has been disposed #2679

Description

What is failing

Probable cause

Impact

Where to look

Evidence

Activity

systemorph-com commented on Sep 17, 2026

rbuergi commented on Sep 18, 2026

Closing — BOTH paths this issue named are fixed in current origin/main, and quiet 19 days

1. The permission fold — PermissionEvaluator.cs

2. The error placeholder — LayoutAreaHost.cs

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Closing — BOTH paths this issue named are fixed in current `origin/main`, and quiet 19 days

1. The permission fold — `PermissionEvaluator.cs`

2. The error placeholder — `LayoutAreaHost.cs`