Microsoft.Extensions.Hosting: Application stop throws exception if host is still starting

Description

If the application is stopped while one or more hosted services are still starting, a misleading TaskCanceledException is logged and bubbles up to Program. This is often due to an exception thrown in BackgroundService, in which case it obscures the real problem.

Reproduction Steps

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var builder = new HostApplicationBuilder();
builder.Services.AddHostedService<ServiceThatStops>();
builder.Services.AddHostedService<ServiceThatWaits>();

using var host = builder.Build();

try
{
    host.Run();
}
catch (Exception exception)
{
    Console.WriteLine($"'Run()' threw exception: {exception}");
}

class ServiceThatStops : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await Task.Delay(10, stoppingToken);
        throw new Exception("My environment is bad! This is the exception I want to see.");
    }
}

class ServiceThatWaits : IHostedService
{
    public Task StartAsync(CancellationToken cancellationToken)
    {
        return Task.Delay(100, cancellationToken);
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        return Task.CompletedTask;
    }
}

Expected behavior

Only the triggering exception should be logged. Cancellation of the remaining service startup should be handled by the host without throwing or logging an error.

Actual behavior

fail: Microsoft.Extensions.Hosting.Internal.Host[9]
      BackgroundService failed
      System.Exception: My environment is bad! This is the exception I want to see.
         at ServiceThatStops.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\KMcClellan\Desktop\HelloHosting\Program.cs:line 22
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
crit: Microsoft.Extensions.Hosting.Internal.Host[10]
      The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
      System.Exception: My environment is bad! This is the exception I want to see.
         at ServiceThatStops.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\KMcClellan\Desktop\HelloHosting\Program.cs:line 22
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
         at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
'Run()' threw exception: System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>g__LogAndRethrow|14_3(<>c__DisplayClass14_0&)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Program.<Main>$(String[] args) in C:\Users\KMcClellan\Desktop\HelloHosting\Program.cs:line 10

Regression?

No response

Known Workarounds

Catch OperationCanceledException in any implementation of StartAsync(...) instead of throwing to acknowledge.

Configuration

.NET runtime 9.0
.NET SDK 9.0.101
Microsoft.Extensions.Hosting 9.0.0

Other information

Similar to #98935.

In the repro, there are two HostedServices instances and they share a cancellation token to ensure they are cancelled together. They also have a guarantee that they all have a chance to start and thus that causes TaskCancelledException across the other threads\services.

Since the sync portion of each service is run sequentially before they are all awaited via Task.Run(), any validation logic before an await is run and should get the result you are looking for. i.e. in the repro, don't have the await Task.Delay(10, stoppingToken) before the validation logic.

However, a likely better alternative is to use IStartupValidator service which runs before Start():

...
services.AddSingleton<IStartupValidator, ValidationService>();
...

class ValidationService : IStartupValidator
{
    public void Validate()
    {
        throw new Exception("My environment is bad! This is the exception I want to see.");
    }
}

or the newer IHostedLifecycleService.StartingAsync which can be implemented on the service:

class ServiceThatStops : BackgroundService, IHostedLifecycleService
{
...
    public Task StartingAsync(CancellationToken cancellationToken) => throw new Exception("My environment is bad!!!!!!!!!!!!!!! This is the exception I want to see.");
    public Task StartedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StoppedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StoppingAsync(CancellationToken cancellationToken) => Task.CompletedTask;

Sure, throwing before the await does not reproduce the issue. But it's not the case that all exceptions will be thrown in the synchronous section of the implementation. In my case, it is a failed HTTP response on startup that indicates the bad environment.

You can't convince me that throwing TaskCanceledException from IHost.Run() with no cancellation token passed is correct behavior.

In my case, it is a failed HTTP response on startup that indicates the bad environment.

The TaskCanceledException is actually from ServiceThatWaits.StartAsync() when it calls Task.Delay() which throws here due to the shared cancellation token:

runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs

Line 146 in c9f0d04

throw new TaskCanceledException(task);

Change StartAsync() to this to see that you can have a graceful shutdown without TaskCanceledException:

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        while (cancellationToken.IsCancellationRequested == false)
        {
        }
    }

The TaskCanceledException is actually from ServiceThatWaits.StartAsync() when it calls Task.Delay() which throws here due to the shared cancellation token:

Yes, I'm aware. Throwing TaskCanceledException (or any OperationCanceledException) in a method that accepts a cancellation token is known as "acknowledging" cancellation. This is what implementers of StartAsync(...) should do if the token is triggered and I am not done with my startup work. The host should not log this as an error, nor allow the exception to bubble up to Program (unless Program passed a cancellation token - not the case in my repro).

Again, this is very similar to #98935, where it's granted that BackgroundService.ExecuteAsync(...) should be able to acknowledge cancellation without this representing an error condition. The same should be true for other members and other types of hosted service.

@eerhardt

Again, this is very similar to #98935, where it's granted that BackgroundService.ExecuteAsync(...) should be able to acknowledge cancellation without this representing an error condition. The same should be true for other members and other types of hosted service.

The difference is that both #98935 and the original #56032 have a single hosted service.

To not throw TCE for other hosted services when a background service is still starting (which is itself also a hosted service) could be considered a breaking change.

Also, since there can be more than one background service we would have to define the rules around that such as "if any background service fails during startup, then ignore TCE for all other hosted services including other background services". I believe this can be done simply by checking if the host is shutting down already like the other two issues linked above.

You can't convince me that throwing TaskCanceledException from IHost.Run() with no cancellation token passed is correct behavior.

It should act the same with or without a specified cancellation token since one is created if not passed to host.RunAsync(). In general, and as mentioned before, the shared cancellation token is created in order to support stopping all threads\hosted services in a timely manner:

runtime/src/libraries/Microsoft.Extensions.Hosting/src/Internal/Host.cs

Line 76 in 4f16361

    
           using (var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, _applicationLifetime.ApplicationStopping))

So the TCE exception will still occur, but it can be ignored. I put up a prototype at https://github.com/steveharter/runtime/tree/Issue-98935 to do that.

@eerhardt @halter73 any thoughts here?

We still seem to be talking past each other :/ To be clear, the host is correct to trigger cancellation for all services if any of them throws an exception. What is not correct is the logging of acknowledgement TCEs as errors and allowing them to bubble up higher in the call stack than where the token source is created.

Granted, this is a fairly minor issue. But fairly often someone complains to me that an app is crashing and sends me the exception details for the TCE, and I have to explain to this bug to them and that they need to look for the actual error earlier in the logs.

Triage: Application stop during host starting is edge case. We might consider not logging cancellation exceptions.

fix(daemon): guard host start, share the real teardown steps, retire …

[AI-1707] Make daemon shutdown crash-free: idempotent dispose + conta…

Navigation Menu

Repository navigation

`Microsoft.Extensions.Hosting`: Application stop throws exception if host is still starting #111013

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Microsoft.Extensions.Hosting: Application stop throws exception if host is still starting #111013

Description

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Activity

steveharter commented on Jan 10, 2025

kmcclellan commented on Jan 10, 2025

steveharter commented on Jan 13, 2025

kmcclellan commented on Jan 14, 2025

steveharter commented on Jan 15, 2025

kmcclellan commented on Jan 17, 2025

karelz commented on Jan 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`Microsoft.Extensions.Hosting`: Application stop throws exception if host is still starting #111013