AzureAIFoundry RunAsFoundryLocal() stuck in 'Downloading' state even when model is already cached

[Voice3]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When using builder.AddAzureAIFoundry(...).RunAsFoundryLocal().AddDeployment(...) in an Aspire AppHost, the resource for the deployment (e.g., "chat") gets stuck in the "Downloading model..." state in the Aspire Dashboard.

This happens even when the model is already downloaded, cached, and runnable via the Foundry Local CLI.

Because the model resource never transitions to "Running", any dependent resources (like a web project) using .WaitFor(chat) will hang indefinitely, and their associated health checks will remain in the "Waiting for health data..." state, effectively deadlocking the application

Expected Behavior

To download the model if it doesn't existing locally and start it.

Steps To Reproduce

Install Foundry Local (winget install Microsoft.FoundryLocal) and verify the version (0.8.94.38902_x64).

I was adding Aspire to my existing OrchardCore project. But to rule if it was just my project. I download https://github.com/nubiquest-blogs/orchard-core-with-aspire which I know was working. Update Aspire packages to 9.5.2 and installed Aspire.Hosting.Azure.AIFoundry (v9.5.2-preview.1.25522.3).

Update the AppHost to include the following:

var localFoundry = builder.AddAzureAIFoundry("foundry-service")
    .RunAsFoundryLocal();

localFoundry.AddDeployment("chat", modelId, "1", "Microsoft");

builder.AddProject<OrchardApp>("App")
.WithEnvironment("OrchardCore__OrchardCore_DataProtection_Azure__ConnectionString", blobs)
.WithEnvironment("OrchardCore__OrchardCore_Shells_Azure__ConnectionString", blobs)
.WithEnvironment("OrchardCore__OrchardCore_AutoSetup__Tenants__0__DatabaseConnectionString", mysqldb)
.WithEnvironment("OrchardCore__OrchardCore_AutoSetup__Tenants__0__AdminPassword", adminPassword)
    .WithReference(localFoundry)
    .WaitFor(container)
    .WaitFor(mysqldb);

Exceptions (if any)

No exceptions was thrown in the log but I have noticed that these two errors are added each time the project is started. Not sure if they are related ?
Exception:
System.OperationCanceledException: The operation was canceled.
at System.Threading.Channels.AsyncOperation1.GetResult(Int16 token) at System.Threading.Channels.ChannelReader1.ReadAllAsync(CancellationToken cancellationToken)+MoveNext()
at Aspire.Dashboard.ServiceClient.DashboardClient.<>c__DisplayClass54_0.<g__StreamUpdatesAsync|0>d.MoveNext() in //src/Aspire.Dashboard/ServiceClient/DashboardClient.cs:line 639
--- End of stack trace from previous location ---
at Aspire.Dashboard.ServiceClient.DashboardClient.<>c__DisplayClass54_0.<g__StreamUpdatesAsync|0>d.MoveNext() in //src/Aspire.Dashboard/ServiceClient/DashboardClient.cs:line 639
--- End of stack trace from previous location ---
at Aspire.Dashboard.ServiceClient.DashboardClient.<>c__DisplayClass54_0.<g__StreamUpdatesAsync|0>d.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
at Aspire.Dashboard.Components.Interactions.InteractionsProvider.WatchInteractionsAsync() in //src/Aspire.Dashboard/Components/Interactions/InteractionsProvider.cs:line 331
at Aspire.Dashboard.Components.Interactions.InteractionsProvider.WatchInteractionsAsync() in //src/Aspire.Dashboard/Components/Interactions/InteractionsProvider.cs:line 331
at Aspire.Dashboard.Components.Interactions.InteractionsProvider.b__44_1() in /_/src/Aspire.Dashboard/Components/Interactions/InteractionsProvider.cs:line 120

Category: Aspire.Dashboard.ServiceClient.DashboardClient
EventId: 0
ConnectionId: 0HNGT45JH7B12
RequestId: 0HNGT45JH7B12:0000001D
RequestPath: /_blazor
TransportConnectionId: 6-QccYTm6eBZ95-rEL2yvA

Error #1 watching resources.

Exception:
Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error reading next message. IOException: The request was aborted. IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. SocketException: An existing connection was forcibly closed by the remote host.", DebugException="System.IO.IOException: The request was aborted.")
---> System.IO.IOException: The request was aborted.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at Grpc.Net.Client.Balancer.Internal.StreamWrapper.ReadAsync(Memory1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory1 buffer, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span1 buffer, Boolean partOfSyncRead) at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken) at Grpc.Net.Client.Internal.GrpcCall2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.HttpContentClientStreamReader2.MoveNextCore(CancellationToken cancellationToken) --- End of inner exception stack trace --- at Grpc.Net.Client.Internal.HttpContentClientStreamReader2.MoveNextCore(CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.Retry.RetryCallBaseClientStreamReader2.MoveNext(CancellationToken cancellationToken) at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader1 streamReader, CancellationToken cancellationToken)+MoveNext()
at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader1 streamReader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at Aspire.Dashboard.ServiceClient.DashboardClient.WatchResourcesAsync(RetryContext retryContext, CancellationToken cancellationToken) in /_/src/Aspire.Dashboard/ServiceClient/DashboardClient.cs:line 350 at Aspire.Dashboard.ServiceClient.DashboardClient.WatchResourcesAsync(RetryContext retryContext, CancellationToken cancellationToken) in /_/src/Aspire.Dashboard/ServiceClient/DashboardClient.cs:line 350 at Aspire.Dashboard.ServiceClient.DashboardClient.WatchWithRecoveryAsync(Func3 action, CancellationToken cancellationToken) in /_/src/Aspire.Dashboard/ServiceClient/DashboardClient.cs:line 321

.NET Version info

Host:
Version: 10.0.0-rc.2.25502.107
Architecture: x64
Commit: 89c8f6a112

.NET SDKs installed:
9.0.305 [C:\Program Files\dotnet\sdk]
10.0.100-rc.1.25451.107 [C:\Program Files\dotnet\sdk]
10.0.100-rc.2.25502.107 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 10.0.0-rc.2.25502.107 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 10.0.0-rc.2.25502.107 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.21 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 10.0.0-rc.2.25502.107 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Anything else?

No response

[GDWhitehead]

Hi, I also had this issue today. I have managed to resolve it by ensuring I had the latest NUGET packages installed, I am now using these:

• Aspire.Hosting.AppHost version = "13.0.0"
• Aspire.Microsoft.Azure.AIFoundry version="13.0.0-preview-1.25560.3"

I hope this maybe resolve the issues for you if you haven't already resolved them. You should be able to see "Downloading model [model] x%" logs in the console

[matt-goldman]

Not sure if this is related but I've found it downloads the model at startup every time. It's not stuck - it actually does download - but it's not using the model from cache. Running on Windows with latest versions:

<Project Sdk="Aspire.AppHost.Sdk/13.1.0">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Aspire.Hosting.Azure.AIFoundry" Version="13.1.1-preview.1.26105.8" />
  </ItemGroup>
</Project>

[Kumima]

Hope the latest CLI integration from 13.5 will fix this.