Skip to content

Fix duplicate resource stopped events during restart - #20279

Merged
Karol Zadora-Przylecki (karolz-ms) merged 1 commit into
mainfrom
fix/resource-stopped-dcp-state
Sep 21, 2026
Merged

Karol Zadora-Przylecki (karolz-ms) merged 1 commit into
mainfrom
fix/resource-stopped-dcp-state

Conversation

@JamesNK

Copy link
Copy Markdown
Member

Description

Prevent duplicate OnResourceStopped callbacks when a handler initiates a restart. Restart preparation can change the published resource snapshot to Waiting or Starting while DCP still reports the same terminal state. Comparing against that snapshot incorrectly treats a later notification as another stop.

Carry PreviousState on OnResourceChangedContext, using the DCP watcher's existing resource cache. ApplicationOrchestrator compares the supplied DCP states without maintaining its own tracking collection. Endpoint/service refreshes report no state transition, and replacement DCP objects have no previous state.

Fixes #17745

User-facing scenario

An OnResourceStopped handler requests a restart. Another notification for the same stopped process should not invoke the handler again; a subsequent real run/stop cycle should invoke it once more. No AppHost API changes are required.

Validation

  • Reproduced the original failure in a deterministic unit test: changing the snapshot from a stopped handler resulted in two callbacks instead of one.
  • All 41 orchestrator and DCP resource-watch tests pass, including six regression cases covering snapshot mutation, repeated terminal notifications, later run/stop cycles, endpoint refreshes, recreated objects with and without delete notifications, and replica isolation.
  • Live DCP stop/start verification has not been run.
dotnet test --project tests/Aspire.Hosting.Tests/Aspire.Hosting.Tests.csproj --no-launch-profile -- --filter-method "Aspire.Hosting.Tests.Orchestrator.ApplicationOrchestratorTests.*" --filter-method "Aspire.Hosting.Tests.Dcp.DcpExecutorTests.ResourceWatch_*" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 20279

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 20279"

@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

50 / 99 PR test projects · 4 PR jobs, from 6 changed files.

Selected PR test projects (50 / 99)

Aspire.Cli.EndToEnd.Tests, Aspire.Hosting.Analyzers.Tests, Aspire.Hosting.Azure.Kubernetes.Tests, Aspire.Hosting.Azure.Kusto.Tests, Aspire.Hosting.Azure.Provisioning.Tests, Aspire.Hosting.Azure.Tests, Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Browsers.Tests, Aspire.Hosting.CodeGeneration.Go.Tests, Aspire.Hosting.CodeGeneration.Java.Tests, Aspire.Hosting.CodeGeneration.Python.Tests, Aspire.Hosting.CodeGeneration.Rust.Tests, Aspire.Hosting.CodeGeneration.TypeScript.Tests, Aspire.Hosting.Containers.Tests, Aspire.Hosting.DevTunnels.Tests, Aspire.Hosting.Docker.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.DotnetTool.Tests, Aspire.Hosting.EntityFrameworkCore.Tests, Aspire.Hosting.Foundry.Tests, Aspire.Hosting.Garnet.Tests, Aspire.Hosting.Go.Tests, Aspire.Hosting.Java.Tests, Aspire.Hosting.JavaScript.Tests, Aspire.Hosting.Kafka.Tests, Aspire.Hosting.Keycloak.Tests, Aspire.Hosting.Kubernetes.Tests, Aspire.Hosting.Maui.Tests, Aspire.Hosting.Milvus.Tests, Aspire.Hosting.MongoDB.Tests, Aspire.Hosting.MySql.Tests, Aspire.Hosting.Nats.Tests, Aspire.Hosting.OpenAI.Tests, Aspire.Hosting.Oracle.Tests, Aspire.Hosting.Orleans.Tests, Aspire.Hosting.PostgreSQL.Tests, Aspire.Hosting.Python.Tests, Aspire.Hosting.Qdrant.Tests, Aspire.Hosting.RabbitMQ.Tests, Aspire.Hosting.Radius.Tests, Aspire.Hosting.Redis.Tests, Aspire.Hosting.RemoteHost.Tests, Aspire.Hosting.Rust.Tests, Aspire.Hosting.Seq.Tests, Aspire.Hosting.SqlServer.Tests, Aspire.Hosting.Testing.Tests, Aspire.Hosting.Tests, Aspire.Hosting.Valkey.Tests, Aspire.Hosting.Yarp.Tests, Aspire.Playground.Tests

Selected PR jobs (4)

cli-starter-validation, extension-e2e, polyglot, typescript-api-compat


How these were chosen — grouped by what changed

⚠️ 45 of the 50 selected test projects come from a single change — src/Aspire.Hosting/Dcp/DcpExecutor.cs.

🔧 src/Aspire.Hosting/Dcp/DcpExecutor.cs (changed source)
45 via the project graph

show 45

Aspire.Hosting.Analyzers.Tests (2 hops), Aspire.Hosting.Azure.Kubernetes.Tests (2 hops), Aspire.Hosting.Azure.Kusto.Tests (2 hops), Aspire.Hosting.Azure.Provisioning.Tests (3 hops), Aspire.Hosting.Azure.Tests, Aspire.Hosting.Browsers.Tests (2 hops), Aspire.Hosting.CodeGeneration.Go.Tests, Aspire.Hosting.CodeGeneration.Java.Tests, Aspire.Hosting.CodeGeneration.Python.Tests, Aspire.Hosting.CodeGeneration.Rust.Tests, Aspire.Hosting.CodeGeneration.TypeScript.Tests, Aspire.Hosting.Containers.Tests (2 hops), Aspire.Hosting.DevTunnels.Tests (2 hops), Aspire.Hosting.Docker.Tests (2 hops), Aspire.Hosting.DotnetTool.Tests (2 hops), Aspire.Hosting.EntityFrameworkCore.Tests (2 hops), Aspire.Hosting.Foundry.Tests (2 hops), Aspire.Hosting.Garnet.Tests (2 hops), Aspire.Hosting.Go.Tests (2 hops), Aspire.Hosting.Java.Tests (2 hops), Aspire.Hosting.JavaScript.Tests (2 hops), Aspire.Hosting.Kafka.Tests (2 hops), Aspire.Hosting.Keycloak.Tests (2 hops), Aspire.Hosting.Kubernetes.Tests (2 hops), Aspire.Hosting.Maui.Tests, Aspire.Hosting.Milvus.Tests (2 hops), Aspire.Hosting.MongoDB.Tests (2 hops), Aspire.Hosting.MySql.Tests (2 hops), Aspire.Hosting.Nats.Tests (2 hops), Aspire.Hosting.OpenAI.Tests (2 hops), Aspire.Hosting.Oracle.Tests (2 hops), Aspire.Hosting.Orleans.Tests (2 hops), Aspire.Hosting.PostgreSQL.Tests (2 hops), Aspire.Hosting.Python.Tests (2 hops), Aspire.Hosting.Qdrant.Tests (2 hops), Aspire.Hosting.RabbitMQ.Tests (2 hops), Aspire.Hosting.Redis.Tests (2 hops), Aspire.Hosting.RemoteHost.Tests, Aspire.Hosting.Rust.Tests (2 hops), Aspire.Hosting.Seq.Tests (2 hops), Aspire.Hosting.SqlServer.Tests (2 hops), Aspire.Hosting.Testing.Tests (2 hops), Aspire.Hosting.Valkey.Tests (2 hops), Aspire.Hosting.Yarp.Tests (2 hops), Aspire.Playground.Tests

🧪 tests/Aspire.Hosting.Tests/Dcp/DcpExecutorTests.cs (changed test)
1 directly: Aspire.Hosting.Tests
3 via the project graph: Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.Radius.Tests

📦 affected project Aspire.Hosting
1 test: Aspire.Cli.EndToEnd.Tests

🧪 tests/Aspire.Hosting.Tests/Orchestrator/ApplicationOrchestratorTests.cs (changed test)
1 directly: Aspire.Hosting.Tests

Job reasons

Job Triggered by
cli-starter-validation affected project Aspire.Hosting.PostgreSQL
extension-e2e src/Aspire.Hosting/Dcp/DcpExecutor.cs, src/Aspire.Hosting/Dcp/DcpExecutorEvents.cs, src/Aspire.Hosting/Dcp/DcpResourceWatcher.cs, src/Aspire.Hosting/Orchestrator/ApplicationOrchestrator.cs
• affected project Aspire.Hosting
polyglot • affected project Aspire.Hosting
• affected project Aspire.Hosting.Azure.Provisioning.KeyVault
typescript-api-compat affected project Aspire.Hosting

Selection computed for commit 82bfd61.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The transition logic is consistent across watcher and orchestrator paths and has focused regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Prevents duplicate resource-stopped callbacks by tracking transitions from the DCP watcher’s cached state.

Changes:

  • Adds PreviousState to resource-change events.
  • Suppresses repeated terminal-state notifications while preserving later stop cycles.
  • Adds regression coverage for refreshes, replacements, and replicas.
File Description
src/​Aspire.Hosting/​Dcp/​DcpExecutor.cs Supplies previous-state metadata for initial updates.
src/​Aspire.Hosting/​Dcp/​DcpExecutorEvents.cs Extends the internal change context.
src/​Aspire.Hosting/​Dcp/​DcpResourceWatcher.cs Derives previous state from cached DCP objects.
src/​Aspire.Hosting/​Orchestrator/​ApplicationOrchestrator.cs Detects terminal transitions using DCP states.
tests/​Aspire.Hosting.Tests/​Dcp/​DcpExecutorTests.cs Covers state tracking, refreshes, replacements, and replicas.
tests/​Aspire.Hosting.Tests/​Orchestrator/​ApplicationOrchestratorTests.cs Covers snapshot mutation and repeated stop notifications.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@karolz-ms
Karol Zadora-Przylecki (karolz-ms) merged commit 32db91d into main Sep 21, 2026
564 of 569 checks passed
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 13.6 milestone Sep 21, 2026
aspire-repo-bot Bot added a commit to microsoft/aspire.dev that referenced this pull request Sep 21, 2026
Documents microsoft/aspire#20279

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

Pull request created: #1705

Generated by PR Documentation Check · copilot · auto · 67.4 AIC · ⌖ 10.2 AIC · ⊞ 18.5K

@aspire-repo-bot

Copy link
Copy Markdown
Contributor

📝 Documentation has been drafted in microsoft/aspire.dev#1705 targeting release/13.6.

Added a clarifying note to the "Resource stopped event" section of app-host/eventing.mdx explaining that ResourceStoppedEvent fires once per stop, and that a restart triggered from within a handler does not cause a duplicate firing for the same stop. This documents the fix in #20279 (duplicate OnResourceStopped callbacks during restart).

  • Files updated: src/frontend/src/content/docs/app-host/eventing.mdx

Note

This draft PR needs human review before merging.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ CI Failure Analysis: Possible Flaky Test(s)

The CI build failed due to test failure(s) that appear unrelated to the PR changes. These may be flaky tests.

Suspected flaky test(s):

  • Aspire.Hosting.Maui.Tests.MauiBuildQueueTests.CancelQueuedResource_CompletesGracefullyAndDoesNotAcquireSemaphore in job Tests / No-package tests (regular, Aspire.Hosting.Maui.Tests, Hosting.Maui, Hosting.Maui, tests/Aspire.Ho... / Hosting.Maui (windows-latest)
    • Error: Assert.IsType() Failure: Value is not the exact type
      Expected: typeof(System.OperationCanceledException)
      Actual: typeof(System.Threading.Tasks.TaskCanceledException)
    • Stack Trace (first frames):
at Aspire.Hosting.Maui.Tests.MauiBuildQueueTests.CancelQueuedResource_CompletesGracefullyAndDoesNotAcquireSemaphore() in D:\a\aspire\aspire\tests\Aspire.Hosting.Maui.Tests\MauiBuildQueueTests.cs:line 543
   at Aspire.Hosting.Maui.Tests.MauiBuildQueueTests.CancelQueuedResource_CompletesGracefullyAndDoesNotAcquireSemaphore() in D:\a\aspire\aspire\tests\Aspire.Hosting.Maui.Tests\MauiBuildQueueTests.cs:line 552
--- End of stack trace from previous location ---
  • Why likely flaky: The test asserts an exact exception type (OperationCanceledException) but the code path can non-deterministically throw the subclass TaskCanceledException depending on timing of the cancellation source. The PR does not touch any Maui build queue code, so this failure is unrelated to the PR's DCP/Orchestrator changes.
  • Aspire.Templates.Tests.XUnit_V3_NewUpAndBuildSupportProjectTemplatesTests.CanNewAndBuild(templateName: "aspire-xunit", extraTestCreationArgs: "--xunit-version v3", sdk: Net8, tfm: Net8, error: null) in job Tests / Package tests - macOS (class, Aspire.Templates.Tests, Templates-XUnit_V3_NewUpAndBuildSupportProj... / Templates-XUnit_V3_NewUpAndBuildSupportProjectTemplatesTests (macos-latest)
    • Error: Aspire.Templates.Tests.ToolCommandException : Expected 0 exit code but got 1: /Users/runner/work/aspire/aspire/artifacts/bin/dotnet-8/dotnet build ... CSC : error CS1504: Source file '.../AssemblyAttributes.cs' could not be opened -- Method not found: 'Boolean Microsoft.CodeAnalysis.EncodingExtensions.TryGetMaxCharCount(System.Text.Encoding, Int64, Int32 ByRef)'.
    • Stack Trace (first frames):
at Aspire.Templates.Tests.CommandResult.EnsureExitCode(Int32 expectedExitCode, String messagePrefix, Boolean suppressOutput) in /_/tests/Shared/TemplatesTesting/CommandResult.cs:line 36
   at Aspire.Templates.Tests.CommandResult.EnsureSuccessful(String messagePrefix, Boolean suppressOutput) in /_/tests/Shared/TemplatesTesting/CommandResult.cs:line 20
   at Aspire.Templates.Tests.AspireProject.BuildAsync(String[] extraBuildArgs, CancellationToken token, String workingDirectory) in /_/tests/Shared/TemplatesTesting/AspireProject.cs:line 347
   at Aspire.Templates.Tests.NewUpAndBuildSupportProjectTemplatesBase.CanNewAndBuildActual(String templateName, String extraTestCreationArgs, TestSdk sdk, TestTargetFramework tfm, String error) in /_/tests/Aspire.Templates.Tests/NewUpAndBuildSupportProjectTemplatesTests.cs:line 56
  • Why likely flaky: A Roslyn compiler 'Method not found' error on macOS during template project build, indicating an SDK/Roslyn binary version mismatch on the macOS runner image. This is a known class of environmental issue (a similar HashAlgorithmToPal variant is already tracked) and is unrelated to the PR's DCP/Orchestrator changes.

Suggested actions:

  • Re-run the failed CI jobs to confirm if the failure is intermittent
  • If the test continues to fail, consider quarantining it using /quarantine-test <test name> <issue URL>
  • Search existing issues to see if this test is already known to be flaky

You can re-run the failed jobs from the workflow run page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race condition in resource eventing - OnResourceStopped fires twice and breaks lifecycle handlers

3 participants