Skip to content

Orleans message delivery fails with ObjectDisposedException when DI scope is torn down mid-resolution #1540

Closed as duplicate
#1573
Closed as duplicate
Orleans message delivery fails with ObjectDisposedException when DI scope is torn down mid-resolution#1540
#1573
Labels
area:orleansCore subsystem: the Orleans silo - grains, placement, routingbugSomething isn't workingfeature:shutdown-disposalFeature: shutdown disposalsev:LLow - cosmetic, rare edge case, or developer-only

Description

@meshweaver-cloud
meshweaver-cloud
bot
opened on Aug 14, 2026

What is failing

When Orleans tries to forward a message to IMessageHubGrain.DeliverMessage (targeting the PlatformVersion hub), the PlacementDirectorResolver in the placement service calls into Autofac to resolve a placement director — but the DI LifetimeScope has already been disposed, producing an ObjectDisposedException that aborts the delivery.

Probable cause (high confidence)

A lifetime-scope race during shutdown or grain-deactivation cycling. The stack trace shows Orleans's PlacementDirectorResolver.GetPlacementDirector (in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:18) resolving via Autofac, whose LifetimeScope.ThrowDisposedException() fires because the scope was disposed before Orleans finished its message-addressing pipeline. This is a known class of race in Orleans+Autofac hosts when a request scope is disposed while in-flight resolutions are still outstanding.

Impact (low — ephemeral)

A single occurrence on one pod (memex-portal-deployment-75d78d7d95-svhr4), not repeated. The message was lost, but no broader cascade is evident. Likely coincided with a pod recycle or rolling update.

Where to look

  • PlacementDirectorResolver.cs in Orleans.Runtime (upstream, line 18) — this is where the DI resolve is attempted.
  • The mesh's Autofac container registration for placement strategies — specifically how the ILifetimeScope is captured and whether it is a child scope that may outlive its parent.
  • The pod's lifecycle events around 2026-08-14T10:05:40Z — confirm whether a deployment or health-check recycle was in progress.

Evidence
Fingerprint ebf16fe17d683fb4
Category Orleans.Messaging
Severity Error
Exception System.ObjectDisposedException
Top frame Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
Namespace memex
Pods memex-portal-deployment-75d78d7d95-svhr4
Occurrences 1
First seen 2026-08-14 10:05:40Z
Last seen 2026-08-14 10:05:40Z
Recent log lines 
2026-08-14 10:05:40Z memex-portal-deployment-75d78d7d95-svhr4 fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145706738 sys.client/hosted-127.0.0.1:11111@145706738]->[ messagehub/Admin/PlatformVersion] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #656CB6C5D7FB2140[ForwardCount=1]
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Or…[truncated]

Opened automatically from Admin/_LogIncident/ebf16fe17d683fb4. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. rbuergi
    closed this as completedin #1573on Aug 14, 2026
  2. systemorph-com
    reopened this on Sep 17, 2026

  3. systemorph-com commented on Sep 17, 2026

    @systemorph-com
    systemorph-comBot
    on Sep 17, 2026
    Contributor

    Reopened — this fault is happening again.

    Still occurring: 71 more since the last update (72 total).

    • Last seen: 2026-09-03 17:21:25Z
    • Pods: memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf
    Recent log lines 
    2026-09-03 17:18:21Z memex-portal-deployment-7fd44b7695-xvwnf fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.234:11111:147460621 sys.client/hosted-10.244.2.234:11111@147460621]->[ pubsubrendezvous/Memory/null/import/dvb51E2yXUa3rstngu3ngw] Orleans.Streams.IPubSubRendezvousGrain.RegisterConsumer(Orleans.Runtime.GuidId, Orleans.Runtime.QualifiedStreamId, Orleans.Runtime.GrainId, System.String) #779722574D8B0B3A
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 535
      --- End of stack trace from previous location ---
         at Orleans.Runtime…[truncated]
2026-09-03 17:21:23Z memex-portal-deployment-7fd44b7695-xvwnf fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.234:11111:147460841 sys.client/hosted-10.244.2.234:11111@147460841]->[ messagehub/Admin/PlatformVersion] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #450ADC0D1559AAC4[ForwardCount=1]
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 535
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivation…[truncated]
2026-09-03 17:21:25Z memex-portal-deployment-7fd44b7695-xvwnf fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.234:11111:147460841 sys.client/hosted-10.244.2.234:11111@147460841]->[ pubsubrendezvous/Memory/null/import/lCyi6FfL6kCGlTOCulQQgw] Orleans.Streams.IPubSubRendezvousGrain.RegisterConsumer(Orleans.Runtime.GuidId, Orleans.Runtime.QualifiedStreamId, Orleans.Runtime.GrainId, System.String) #450ADC0D1559AAB8
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 535
      --- End of stack trace from previous location ---
         at Orleans.Runtime…[truncated]

  4. rbuergi commented on Sep 18, 2026

    @rbuergi
    rbuergi
    on Sep 18, 2026
    Contributor

    Closing the family root too — its fix is in origin/main and core says so in source.

    src/MeshWeaver.Hosting/MeshTeardownHostedService.cs names this issue and its siblings explicitly:

    "Once the host then disposed the container, every one of those in-flight paths threw ObjectDisposedException from a disposed LifetimeScope — issues #1540, #1541, #1542, #1544, #1545, #1546, #1547, #1548 and #1560, all one root."

    The repair is ordering, not a catch: the drain moved to IHostedLifecycleService.StoppedAsync, which runs only after every IHostedService.StopAsync has returned — the web host's included — and still strictly before the host disposes the root provider. Draining in StopAsync was what tore the mesh down while Kestrel was still serving, so late requests re-entered the disposed root hub, re-created Admin/PlatformVersion, and re-subscribed Orleans streams on a stopped silo.

    Pinned by two tests that exist on disk today: MeshHostBuilderTeardownOrderingTest.cs and MeshTeardownRunsAfterEveryOtherHostedServiceTest.cs (plus HubScopeClosesAfterTeardownDrainsTest.cs, HubWatcherStopsAtTeardownStartTest.cs).

    All eight siblings in this family are now closed on that same evidence. Keeping the lowest number open as a "tracker" would track work that is finished, so this closes with them.

    🚨 If the fault recurs, the bot reopens this issue with fresh evidence — that is the designed behaviour, and it is why closing a dormant incident is safe rather than optimistic. Two live faults from the same 2026-08-14 window are deliberately NOT covered here and stay open on their own tickets.

  5. rbuergi
    closed this as not plannedon Sep 18, 2026
  6. systemorph-com
    reopened this on Sep 19, 2026

  7. systemorph-com commented on Sep 19, 2026

    @systemorph-com
    systemorph-comBot
    on Sep 19, 2026
    Contributor

    Reopened — this fault is happening again.

    Still occurring: 1 more since the last update (73 total).

    • Last seen: 2026-09-19 05:07:01Z
    • Pods: memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f
    Recent log lines 
    2026-09-03 17:21:23Z memex-portal-deployment-7fd44b7695-xvwnf fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.234:11111:147460841 sys.client/hosted-10.244.2.234:11111@147460841]->[ messagehub/Admin/PlatformVersion] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #450ADC0D1559AAC4[ForwardCount=1]
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 535
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivation…[truncated]
2026-09-03 17:21:25Z memex-portal-deployment-7fd44b7695-xvwnf fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.234:11111:147460841 sys.client/hosted-10.244.2.234:11111@147460841]->[ pubsubrendezvous/Memory/null/import/lCyi6FfL6kCGlTOCulQQgw] Orleans.Streams.IPubSubRendezvousGrain.RegisterConsumer(Orleans.Runtime.GuidId, Orleans.Runtime.QualifiedStreamId, Orleans.Runtime.GrainId, System.String) #450ADC0D1559AAB8
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 535
      --- End of stack trace from previous location ---
         at Orleans.Runtime…[truncated]
2026-09-19 05:07:01Z memex-portal-deployment-65468bfccf-68g7f fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.179:11111:148799218 sys.client/hosted-10.244.5.179:11111@148799218]->[ pubsubrendezvous/Memory/null/import/o4r5HukLYkqFn92XCLWAXw] Orleans.Streams.IPubSubRendezvousGrain.RegisterConsumer(Orleans.Runtime.GuidId, Orleans.Runtime.QualifiedStreamId, Orleans.Runtime.GrainId, System.String) #385E42BF9CD45674
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext co…[truncated]

  8. rbuergi commented on Sep 19, 2026

    @rbuergi
    rbuergi
    on Sep 19, 2026
    Contributor

    Why this family keeps reopening: "dormant" was measured across a period with no rolls

    Working the backlog oldest-first. #1540, #1547, #1548 (and #1449, #1422 next door) were all closed on 2026-09-18 partly on dormancy — "15 days with no occurrence" — and all were reopened within a day. The reopens are correct, and it is worth saying so precisely, because the obvious reading is that the bot is misbehaving:

    issue closed reopened newest occurrence
    #1449 09-18 06:07:05 09-18 15:36:25 09-18 15:35:29Z
    #1540 09-18 05:58:27 09-19 05:08:26 09-19 05:07:01Z
    #1547 09-18 05:57:08 09-19 05:07:26 09-19 05:10:25Z
    #1548 09-18 05:57:09 09-19 05:05:27 09-19 05:12:06Z
    #1422 09-17 11:27:46 09-19 05:36:25 09-19 05:35:41Z

    Every occurrence postdates its close, which is exactly what Reopen requires since #4577. Nothing here is a false reopen.

    🚨 What the dormancy actually measured. All five fired inside the 05:05–05:36Z window on 2026-09-19 — the roll storm. These are teardown faults: they fire when pods stop, and they are silent when nothing is stopping them. Fifteen quiet days meant fifteen days without a roll, not fifteen days of health. A rate measured under one regime does not survive ending it, and closing on that silence is what guaranteed the reopen.

    The fix may not cover the path that is actually firing

    MeshTeardownHostedService names this issue and its siblings as "all one root", and its drain is real: StoppedAsync runs after every StopAsync and strictly before the root provider is disposed, so the mesh's in-flight paths are drained inside that window.

    But the line that reopened #1547 on 09-19 is not a mesh path:

    fail: Orleans.Grain[1528537479]
  Failed to unregister activation '[Activation: …/messagehub/Admin/_Build/… State=Deactivating]' from directory
  System.ObjectDisposedException: … this LifetimeScope … has already been disposed.
     at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type)
     at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.CreateInstance(IServiceProvider, Type, Object[])

    That is Orleans' own grain-directory unregistration, on Orleans' own logger, resolving a service out of the container while deactivating. The mesh drain does not obviously reach it: it is not mesh in-flight work, and it runs on the silo's deactivation path rather than through any hosted service the drain waits on.

    I am stating that as the open question rather than a conclusion — establishing whether Orleans' deactivation can outlive StoppedAsync needs the silo lifecycle traced, not log lines.

    One confounder before anyone measures the current image

    The pod that logged it, memex-portal-deployment-65468bfccf-68g7f, was running at 05:04:58Z, and the image memex-cloud runs now (c25f86ae85b7) was built at 07:32Z the same day. So that occurrence says nothing about the current image — it is evidence about whatever that pod was running. merge-base --is-ancestor <teardown fix> c25f86ae85b7 is what would settle whether the fix is even on the image, and the next roll is what would test it.

    Suggested handling: leave these open until a roll happens on an image provably carrying the teardown fix and produces no occurrence. Closing them on quiet between rolls will reopen them on the next roll, which is now the third time.

  9. rbuergi
    added a commit that references this issue on Sep 19, 2026

    docs: rest the section on the three teardown faults, not five issues

    41eb8de
  10. rbuergi
    added
    bugSomething isn't working
    area:orleansCore subsystem: the Orleans silo - grains, placement, routing
    feature:shutdown-disposalFeature: shutdown disposal
    sev:LLow - cosmetic, rare edge case, or developer-only
    on Sep 20, 2026

  11. rbuergi commented on Sep 21, 2026

    @rbuergi
    rbuergi
    on Sep 21, 2026
    Contributor

    Cluster triage: this symptom has five distinct roots

    Reviewed together with #1540 #1547 #1548 #3243 #3593 #2833 #2480 #5064 #4859 #5136 #4847 #4883 #4888 #4961 — every one of them reports ObjectDisposedException with Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException() on top, or a teardown that did not finish. They are not one bug. The grouping, the discriminators and the evidence are committed as Doc/Architecture/DisposedScopeAndDyingHubs (added in #5159).

    Root R1 — the host container went down ahead of the mesh. Duplicate of #1548's root; already fixed.

    This frame is upstream Orleans code (PlacementDirectorResolver.GetPlacementDirector) resolving from
    our Autofac scope, so there is nothing to patch at the frame — only the ordering around it.

    MeshTeardownHostedService (src/MeshWeaver.Hosting/MeshTeardownHostedService.cs) names this issue
    by number in its own doc comment, together with #1541, #1542, #1544, #1545, #1546, #1547, #1548 and
    #1560, as one root. It drains the mesh root hub in StoppedAsync — after every other hosted
    service's StopAsync has returned, the web host's included, and still strictly before the host disposes
    the root provider. StopAsync was the wrong hook: hosted services stop in reverse registration order,
    but GenericWebHostService is registered by WebApplication.CreateBuilder, so Kestrel — and every live
    request and Blazor circuit — stops after anything MeshHostApplicationBuilder registers. Pinned by
    MeshHostBuilderTeardownOrderingTest and MeshTeardownRunsAfterEveryOtherHostedServiceTest.

    Evidence this is settled: last seen 2026-08-14 10:05:40Z, a single occurrence on one pod during a
    rolling deploy; the fix landed 2026-09-15 and both portals have rolled since, with no recurrence.

    🚨 One gap named rather than glossed: the ordering fix does not cover an aborted startup, where
    Host.StartAsync throws and RunAsync's finally goes straight to DisposeAsync() with the silo still
    stopping. An R1-shaped fault on a pod that never finished booting is a different condition.

    Closing as a duplicate of #1548, which is the survivor of the three in this cluster.

  12. rbuergi
    closed this as duplicateon Sep 21, 2026

  13. systemorph-com commented on Sep 23, 2026

    @systemorph-com
    systemorph-comBot
    on Sep 23, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 5 more since the last update (78 total).

    • Last seen: 2026-09-23 14:20:19Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f
    Recent log lines 
    2026-09-23 14:20:19Z memex-portal-deployment-6968cb99df-jrlb9 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.238:11111:149178013 sys.client/hosted-10.244.2.238:11111@149178013]->[ pubsubrendezvous/Memory/null/portal/reads-LX6x4IAIBUaim9NIzuvBtw] Orleans.Streams.IPubSubRendezvousGrain.RegisterConsumer(Orleans.Runtime.GuidId, Orleans.Runtime.QualifiedStreamId, Orleans.Runtime.GrainId, System.String) #0C172C57E05EA387
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOut…[truncated]
2026-09-23 14:19:10Z memex-portal-deployment-6968cb99df-jrlb9 crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for import/SUjN5D8nXUGTt2eXWQHeWg — cross-process routing for this hub is DISABLED (in-process routing remains active)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
         at Orleans.Runtime.Place…[truncated]
2026-09-23 14:20:19Z memex-portal-deployment-6968cb99df-jrlb9 crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/reads-LX6x4IAIBUaim9NIzuvBtw — cross-process routing for this hub is DISABLED (in-process routing remains active)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveKeyed(IComponentContext context, Object serviceKey, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService(Type serviceType, Object serviceKey)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions.GetRequiredKeyedService[T](IServiceProvider provider, Object serviceKey)
         at Orleans.Runtime.Placement.PlacementDirectorResolver.GetPlacementDirector(PlacementStrategy placementStrategy) in /_/src/Orleans.Runtime/Placement/PlacementDirectorResolver.cs:line 18
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         at Polly.Outcome…[truncated]

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  14. systemorph-com commented on Sep 23, 2026

    @systemorph-com
    systemorph-comBot
    on Sep 23, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 62 more since the last update (140 total).

    • Last seen: 2026-09-23 15:46:43Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j
    Recent log lines 
    2026-09-23 15:46:43Z memex-portal-deployment-786d86cf54-cfg4j fail: MeshWeaver.Messaging.MessageService[0]
      Unhandled exception in delivery pipeline for hub mesh/sdJHsMdvKkOE45QSvujWMQ
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshBuilder.<>c__DisplayClass56_1.<Register>b__8(IMessageDelivery delivery) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshBuilder.cs:line 1078
         at MeshWeaver.Messaging.RouteConfiguration.<>c__DisplayClass13_0.<WithHandler>b__0(IMessageDelivery delivery, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/RouteConfiguration.cs:line 37
         at MeshWeaver.Messaging.HierarchicalRouting.RouteMessageAsync(IMessageDelivery delivery, CancellationToken cancellationToken) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/HierarchicalRout…[truncated]
2026-09-23 15:46:43Z memex-portal-deployment-786d86cf54-cfg4j fail: MeshWeaver.Messaging.MessageService[0]
      Unhandled exception in delivery pipeline for hub mesh/sdJHsMdvKkOE45QSvujWMQ
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshBuilder.<>c__DisplayClass56_1.<Register>b__8(IMessageDelivery delivery) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshBuilder.cs:line 1078
         at MeshWeaver.Messaging.RouteConfiguration.<>c__DisplayClass13_0.<WithHandler>b__0(IMessageDelivery delivery, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/RouteConfiguration.cs:line 37
         at MeshWeaver.Messaging.HierarchicalRouting.RouteMessageAsync(IMessageDelivery delivery, CancellationToken cancellationToken) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/HierarchicalRout…[truncated]
2026-09-23 15:46:43Z memex-portal-deployment-786d86cf54-cfg4j fail: MeshWeaver.Messaging.MessageService[0]
      Unhandled exception in delivery pipeline for hub mesh/sdJHsMdvKkOE45QSvujWMQ
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshBuilder.<>c__DisplayClass56_1.<Register>b__8(IMessageDelivery delivery) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshBuilder.cs:line 1078
         at MeshWeaver.Messaging.RouteConfiguration.<>c__DisplayClass13_0.<WithHandler>b__0(IMessageDelivery delivery, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/RouteConfiguration.cs:line 37
         at MeshWeaver.Messaging.HierarchicalRouting.RouteMessageAsync(IMessageDelivery delivery, CancellationToken cancellationToken) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/HierarchicalRout…[truncated]

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  15. systemorph-com commented on Oct 5, 2026

    @systemorph-com
    systemorph-comBot
    on Oct 5, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 4 more since the last update (144 total).

    • Last seen: 2026-10-05 08:45:48Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j, memex-portal-deployment-6485bffdff-cgd5r
    Recent log lines 
    2026-10-05 08:45:48Z memex-portal-deployment-6485bffdff-cgd5r fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2788-review-1/ad95d6ef/do-one-write-on-the-existing-node-hostin-7c18b526c9d382673471/0299ec75 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 08:45:48Z memex-portal-deployment-6485bffdff-cgd5r fail: MeshWeaver.Mesh.CreateNode[0]
      Post-creation handler chain errored at Hosting/Triage/_Thread/pr-systemorph-meshweaver-plugins-2793/75517c74 — rolling the create back (#638)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass97_0.<RunPostCreationHandlersObs>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 6008
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 08:45:48Z memex-portal-deployment-6485bffdff-cgd5r fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Triage/_Thread/pr-systemorph-meshweaver-plugins-2793/75517c74 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  16. systemorph-com commented on Oct 5, 2026

    @systemorph-com
    systemorph-comBot
    on Oct 5, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 696 more since the last update (840 total).

    • Last seen: 2026-10-05 14:05:17Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j, memex-portal-deployment-6485bffdff-cgd5r, memex-portal-deployment-848f664fd5-zzqll, memex-portal-deployment-848f664fd5-zqfrh, memex-portal-deployment-6cfcf78898-gkkdv
    Recent log lines 
    2026-10-05 14:04:23Z memex-portal-deployment-6cfcf78898-gkkdv fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Babysitter/_Thread/1-heal-proposal-s-on-open-fleet-pull-req-d70681df2bd32bf01326/ff838904 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 14:05:17Z memex-portal-deployment-6cfcf78898-gkkdv fail: MeshWeaver.Mesh.CreateNode[0]
      Post-creation handler chain errored at Hosting/Babysitter/_Thread/1-heal-proposal-s-on-open-fleet-pull-req-d70681df2bd32bf01326/d6e6234e — rolling the create back (#638)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass97_0.<RunPostCreationHandlersObs>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 6008
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 14:05:17Z memex-portal-deployment-6cfcf78898-gkkdv fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Babysitter/_Thread/1-heal-proposal-s-on-open-fleet-pull-req-d70681df2bd32bf01326/d6e6234e — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  17. systemorph-com commented on Oct 5, 2026

    @systemorph-com
    systemorph-comBot
    on Oct 5, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 1182 more since the last update (2022 total).

    • Last seen: 2026-10-05 20:05:39Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j, memex-portal-deployment-6485bffdff-cgd5r, memex-portal-deployment-848f664fd5-zzqll, memex-portal-deployment-848f664fd5-zqfrh, memex-portal-deployment-6cfcf78898-gkkdv, memex-portal-deployment-6cfcf78898-kss8q
    Recent log lines 
    2026-10-05 20:05:26Z memex-portal-deployment-6cfcf78898-kss8q fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-49fa2a3af9be9d29823c/e7f6a50c — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 20:05:39Z memex-portal-deployment-6cfcf78898-kss8q fail: MeshWeaver.Mesh.CreateNode[0]
      Post-creation handler chain errored at Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-8d97aedf030f152ea356/1a6fcc69 — rolling the create back (#638)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass97_0.<RunPostCreationHandlersObs>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 6008
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 20:05:39Z memex-portal-deployment-6cfcf78898-kss8q fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-8d97aedf030f152ea356/1a6fcc69 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  18. systemorph-com commented on Oct 6, 2026

    @systemorph-com
    systemorph-comBot
    on Oct 6, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 432 more since the last update (2454 total).

    • Last seen: 2026-10-05 21:06:52Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j, memex-portal-deployment-6485bffdff-cgd5r, memex-portal-deployment-848f664fd5-zzqll, memex-portal-deployment-848f664fd5-zqfrh, memex-portal-deployment-6cfcf78898-gkkdv, memex-portal-deployment-6cfcf78898-kss8q, memex-portal-deployment-6cfcf78898-lsklh
    Recent log lines 
    2026-10-05 21:06:47Z memex-portal-deployment-6cfcf78898-lsklh fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2787/2dcba3d9/one-patch-then-read-back-node-hosting-tr-a6a67d3537e2c7879657/6d2b7ca6 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 21:06:52Z memex-portal-deployment-6cfcf78898-lsklh fail: MeshWeaver.Mesh.CreateNode[0]
      Post-creation handler chain errored at Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2787/2dcba3d9/one-patch-then-read-back-node-hosting-tr-a6a67d3537e2c7879657/6d2b7ca6 — rolling the create back (#638)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass97_0.<RunPostCreationHandlersObs>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 6008
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 21:06:52Z memex-portal-deployment-6cfcf78898-lsklh fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2787/2dcba3d9/one-patch-then-read-back-node-hosting-tr-a6a67d3537e2c7879657/6d2b7ca6 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  19. systemorph-com commented on Oct 7, 2026

    @systemorph-com
    systemorph-comBot
    on Oct 7, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 1 more since the last update (2455 total).

    • Last seen: 2026-10-07 03:02:37Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j, memex-portal-deployment-6485bffdff-cgd5r, memex-portal-deployment-848f664fd5-zzqll, memex-portal-deployment-848f664fd5-zqfrh, memex-portal-deployment-6cfcf78898-gkkdv, memex-portal-deployment-6cfcf78898-kss8q, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-5d74648df-ht6qx
    Recent log lines 
    2026-10-05 21:06:52Z memex-portal-deployment-6cfcf78898-lsklh fail: MeshWeaver.Mesh.CreateNode[0]
      Post-creation handler chain errored at Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2787/2dcba3d9/one-patch-then-read-back-node-hosting-tr-a6a67d3537e2c7879657/6d2b7ca6 — rolling the create back (#638)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass97_0.<RunPostCreationHandlersObs>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 6008
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 21:06:52Z memex-portal-deployment-6cfcf78898-lsklh fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2787/2dcba3d9/one-patch-then-read-back-node-hosting-tr-a6a67d3537e2c7879657/6d2b7ca6 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-07 03:02:37Z memex-portal-deployment-5d74648df-ht6qx fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 03:02:37.186 threadPath=Hosting/Triage/pull-request/systemorph-meshweaver-plugins-3014/_Thread/owner-systemorph-meshweaver-plugins-3014
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.AI.RoundTranscript.IsEnabled(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/RoundTranscript.cs:line 56
         at MeshWeaver.AI.RoundTranscript.Record(IMessageHub hub, String threadPath, String responseMessageId, IObservable`1 accumulatedText, AccessContext identity, ILogger logger) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/RoundTranscript.cs:line 77
         at MeshWeaver.AI.ThreadExecution.<>c__DisplayClass32_8.<<ExecuteMessageAsync>b__46>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ThreadExecution.cs:line 2786

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  20. systemorph-com commented on Oct 7, 2026

    @systemorph-com
    systemorph-comBot
    on Oct 7, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 1 more since the last update (2455 total).

    • Last seen: 2026-10-07 03:02:37Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j, memex-portal-deployment-6485bffdff-cgd5r, memex-portal-deployment-848f664fd5-zzqll, memex-portal-deployment-848f664fd5-zqfrh, memex-portal-deployment-6cfcf78898-gkkdv, memex-portal-deployment-6cfcf78898-kss8q, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-5d74648df-ht6qx
    Recent log lines 
    2026-10-05 21:06:52Z memex-portal-deployment-6cfcf78898-lsklh fail: MeshWeaver.Mesh.CreateNode[0]
      Post-creation handler chain errored at Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2787/2dcba3d9/one-patch-then-read-back-node-hosting-tr-a6a67d3537e2c7879657/6d2b7ca6 — rolling the create back (#638)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass97_0.<RunPostCreationHandlersObs>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 6008
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-05 21:06:52Z memex-portal-deployment-6cfcf78898-lsklh fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2787/2dcba3d9/one-patch-then-read-back-node-hosting-tr-a6a67d3537e2c7879657/6d2b7ca6 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass34_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 1825
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-07 03:02:37Z memex-portal-deployment-5d74648df-ht6qx fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 03:02:37.186 threadPath=Hosting/Triage/pull-request/systemorph-meshweaver-plugins-3014/_Thread/owner-systemorph-meshweaver-plugins-3014
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.AI.RoundTranscript.IsEnabled(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/RoundTranscript.cs:line 56
         at MeshWeaver.AI.RoundTranscript.Record(IMessageHub hub, String threadPath, String responseMessageId, IObservable`1 accumulatedText, AccessContext identity, ILogger logger) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/RoundTranscript.cs:line 77
         at MeshWeaver.AI.ThreadExecution.<>c__DisplayClass32_8.<<ExecuteMessageAsync>b__46>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.AI/ThreadExecution.cs:line 2786

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  21. systemorph-com commented on Oct 13, 2026

    @systemorph-com
    systemorph-comBot
    on Oct 13, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 3 more since the last update (2458 total).

    • Last seen: 2026-10-13 19:53:00Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j, memex-portal-deployment-6485bffdff-cgd5r, memex-portal-deployment-848f664fd5-zzqll, memex-portal-deployment-848f664fd5-zqfrh, memex-portal-deployment-6cfcf78898-gkkdv, memex-portal-deployment-6cfcf78898-kss8q, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-5d74648df-ht6qx, memex-portal-deployment-68f94fffc4-mjl9v, memex-portal-deployment-f96cb5fd6-ckwqm
    Recent log lines 
    2026-10-13 19:52:52Z memex-portal-deployment-68f94fffc4-mjl9v fail: MeshWeaver.Mesh.CreateNode[0]
      Post-creation handler chain errored at Admin/_LogIncident/4796c7ec425516c3/_Thread/triage-the-log-incident-at-admin-loginci-9b96faa0b44b49698c40 — rolling the create back (#638)
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at MeshWeaver.Mesh.MeshExtensions.BuildPostCreationHandlers(IMessageHub hub, MeshNode node, String createdBy, ILogger logger, Action`1 onStage) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 6890
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-13 19:52:52Z memex-portal-deployment-68f94fffc4-mjl9v fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Admin/_LogIncident/4796c7ec425516c3/_Thread/triage-the-log-incident-at-admin-loginci-9b96faa0b44b49698c40 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass56_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 2157
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-13 19:53:00Z memex-portal-deployment-f96cb5fd6-ckwqm fail: Microsoft.AspNetCore.Components.Server.Circuits.CircuitHost[111]
      Unhandled exception in circuit 'taAmt7sknwcDfbICWhr6jM58tO4ZOn9x9SxfrkrET-Y'.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.AspNetCore.Components.ComponentFactory.<>c__DisplayClass11_0.<CreatePropertyInjector>g__Initialize|1(IServiceProvider serviceProvider, IComponent component)
         at Microsoft.AspNetCore.Components.ComponentFactory.InstantiateComponent(IServiceProvider serviceProvider, Type componentType, IComponentRenderMode callerSpecifiedRenderMode, Nullable`1 parentComponentId)
         at Microsoft.AspNetCore.Components.RenderTree.Renderer.InstantiateChildComponentOnFrame(RenderTreeFrame[] frames, Int32 frameIndex, Int32 parentComponentId)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewComponentFrame(DiffContext& diffContext, Int32 frameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewSubtree(DiffContext& diffContext, Int32 frameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.AppendDiffEntriesForRange(DiffContext& …[truncated]

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  22. systemorph-com commented on Oct 22, 2026

    @systemorph-com
    systemorph-comBot
    on Oct 22, 2026
    Contributor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 1 more since the last update (2459 total).

    • Last seen: 2026-10-22 14:40:49Z
    • Pods: memex-portal-deployment-6968cb99df-jrlb9, memex-portal-deployment-75d78d7d95-svhr4, memex-portal-deployment-7d89fc7947-s7s9k, memex-portal-deployment-7bcd994c74-vwn96, memex-portal-deployment-6dfbdb4d69-tzghx, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-65468bfccf-68g7f, memex-portal-deployment-786d86cf54-cfg4j, memex-portal-deployment-6485bffdff-cgd5r, memex-portal-deployment-848f664fd5-zzqll, memex-portal-deployment-848f664fd5-zqfrh, memex-portal-deployment-6cfcf78898-gkkdv, memex-portal-deployment-6cfcf78898-kss8q, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-5d74648df-ht6qx, memex-portal-deployment-68f94fffc4-mjl9v, memex-portal-deployment-f96cb5fd6-ckwqm, memex-portal-deployment-c5dbd6d7b-56p5s
    Recent log lines 
    2026-10-13 19:52:52Z memex-portal-deployment-68f94fffc4-mjl9v fail: MeshWeaver.Mesh.CreateNode[0]
      [CreateNode] ROLLBACK FAILED at Admin/_LogIncident/4796c7ec425516c3/_Thread/triage-the-log-incident-at-admin-loginci-9b96faa0b44b49698c40 — whether the partially-created node remains is undetermined
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Mesh.MeshExtensions.<>c__DisplayClass56_0.<CompensateFailedCreate>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:line 2157
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()
2026-10-13 19:53:00Z memex-portal-deployment-f96cb5fd6-ckwqm fail: Microsoft.AspNetCore.Components.Server.Circuits.CircuitHost[111]
      Unhandled exception in circuit 'taAmt7sknwcDfbICWhr6jM58tO4ZOn9x9SxfrkrET-Y'.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.AspNetCore.Components.ComponentFactory.<>c__DisplayClass11_0.<CreatePropertyInjector>g__Initialize|1(IServiceProvider serviceProvider, IComponent component)
         at Microsoft.AspNetCore.Components.ComponentFactory.InstantiateComponent(IServiceProvider serviceProvider, Type componentType, IComponentRenderMode callerSpecifiedRenderMode, Nullable`1 parentComponentId)
         at Microsoft.AspNetCore.Components.RenderTree.Renderer.InstantiateChildComponentOnFrame(RenderTreeFrame[] frames, Int32 frameIndex, Int32 parentComponentId)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewComponentFrame(DiffContext& diffContext, Int32 frameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewSubtree(DiffContext& diffContext, Int32 frameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.AppendDiffEntriesForRange(DiffContext& …[truncated]
2026-10-22 14:40:49Z memex-portal-deployment-c5dbd6d7b-56p5s fail: Microsoft.AspNetCore.Components.Server.Circuits.CircuitHost[111]
      Unhandled exception in circuit 'VgKebtsd3i5vkxYMS-lE_zzh1W5eSMVpZrf3m1JWr1c'.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.AspNetCore.Components.ComponentFactory.<>c__DisplayClass11_0.<CreatePropertyInjector>g__Initialize|1(IServiceProvider serviceProvider, IComponent component)
         at Microsoft.AspNetCore.Components.ComponentFactory.InstantiateComponent(IServiceProvider serviceProvider, Type componentType, IComponentRenderMode callerSpecifiedRenderMode, Nullable`1 parentComponentId)
         at Microsoft.AspNetCore.Components.RenderTree.Renderer.InstantiateChildComponentOnFrame(RenderTreeFrame[] frames, Int32 frameIndex, Int32 parentComponentId)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewComponentFrame(DiffContext& diffContext, Int32 frameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewSubtree(DiffContext& diffContext, Int32 frameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.AppendDiffEntriesForRange(DiffContext& …[truncated]

    Re-addressed by the current identity function: this incident inherited ebf16fe17d683fb4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:orleansCore subsystem: the Orleans silo - grains, placement, routingbugSomething isn't workingfeature:shutdown-disposalFeature: shutdown disposalsev:LLow - cosmetic, rare edge case, or developer-only

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions