fix(local-ai): add non-destructive recovery

[shanselman]

Problem

Replaces #1357 (Add non-destructive Local AI recovery), supersedes #1342 (Add recovery flow for local AI setup), and fixes #1307.

Local AI Retry setup or download could enter the destructive-capable onboarding pipeline, recreate an existing app-owned WSL gateway, or dead-end when the gateway was exact but the Local AI receipt was missing or incomplete.

Fix

• route Local AI retry through an owner-aware setup command
• select non-destructive recovery only when exactly one setup-managed local gateway, exact registry ID, effective managed distro, and canonical loopback gateway endpoint/port are proven
• carry the exact gateway ID, distro, gateway port, receipt model, and requested Local AI port into recovery
• keep receipt-less retries for an exact managed owner in the non-destructive recovery pipeline while allowing the current catalog recommendation to be selected
• pin the receipt model and port when a structurally valid receipt exists
• validate ownership before acquisition and again immediately before WSL/provider mutation
• exclude gateway cleanup, distro creation/configuration, CLI install, and service install from recovery
• retain the original receipt as the rollback baseline while repairing incomplete runtime/model assets
• rewrite repaired receipts conservatively, preserving schema-4 cache migration fields, requested port, fallback model, and original install timestamp while clearing the unverified endpoint
• preserve the first provider/primary rollback snapshot across retries and fix(local-ai): keep routing safe across endpoint restarts #1390 endpoint-restart lifecycle states
• restore the original provider, primary model, and receipt only when compensation proves the original live route
• retain the replacement receipt when provider rollback is unproven
• arm WSL restart recovery before shutdown, force rollback-on-failure, and provide recovery-sized rollback budgets
• disable the Local AI toggle during recovery and restore every recovery-only config override when Back or the gateway-installed milestone exits to normal setup

Integrated dependencies

• fix(local-ai): keep routing safe across endpoint restarts #1390 (fix(local-ai): keep routing safe across endpoint restarts) is integrated from 621f4100f8b40ac84e844bcc39b01fe03a2238bb.
• feat(local-ai): add verified legacy model cache migration #1388 (feat(local-ai): add verified legacy model cache migration) is integrated from 3ce56b8902d558a2d43a1308b4d61ccbf6856274.
• feat(local-ai): safely use the standard Hugging Face hub cache #1397 (feat(local-ai): safely use the standard Hugging Face hub cache) is integrated from current origin/main 7dab793673ff00a43fd74259ed83321894ef601e.
• The expected LocalAiGatewayProviderCoordinatorTests overlap was resolved without weakening either lifecycle contract.

Current head: c9dee4102e2a8f22bb0d7edea081c3d52f357abc, including current origin/main 7dab793673ff00a43fd74259ed83321894ef601e.

Required proof pools

• windows-wsl-dgx-blackwell: Not verified / blocked. The available RTX 4080 SUPER exposes 16,376 MiB, below the approximately 22.2 GiB minimum for the smallest current new-install catalog recipe.
• windows-winui-interactive: Not verified / blocked. The current-catalog hardware gate correctly prevents an enabled recovery review on this host, so no current-head enabled recovery-page screenshot/video is claimed.
• windows-wsl-gateway-e2e: Not verified / blocked for the real recovery transaction. Hyper-V firewall policy blocks WSL-to-Windows loopback without elevation, so no real existing-gateway provider replacement, restart, inference, and receipt-compensation transaction is claimed.

Validation

Current-head local validation:

• .\build.ps1: passed
• dotnet test .\tests\OpenClaw.Shared.Tests\OpenClaw.Shared.Tests.csproj --no-restore: 3,991 passed, 32 skipped
• dotnet test .\tests\OpenClaw.Tray.Tests\OpenClaw.Tray.Tests.csproj --no-restore: 2,908 passed
• dotnet test .\tests\OpenClaw.SetupEngine.Tests\OpenClaw.SetupEngine.Tests.csproj --no-restore: 1,186 passed, 1 environment-dependent cross-volume test skipped
• focused LocalAiInstallRecoveryTests: 44 passed, 1 environment-dependent cross-volume test skipped
• focused Local AI Connection migration/lifecycle tests: 110 passed, 1 environment-dependent cross-volume test skipped
• dotnet test .\tests\OpenClaw.Connection.Tests\OpenClaw.Connection.Tests.csproj --no-restore: 793 passed, 1 environment-dependent cross-volume test skipped

Reviews

• Final autoreview: clean, no accepted/actionable findings. One valid finding was fixed so schema-4 reconciliation verifies the retained legacy compatibility copy and routes a missing or corrupt copy through safe rematerialization. One cold-refresh finding was rejected after confirming a new runtime service returns before probing and a live managed process already owns a verified model lease.
• Final rubber-duck review: clean, no actionable issues. It independently confirmed the compatibility-copy fix, exact pre-recovery receipt preservation, schema-4 active cache identity, independent runtime/model repair, and the rejected cold-refresh invariant.
• ClawSweeper remains advisory rather than the merge authority; local tests and independent code review are the landing evidence.

Real behavior proof

Current-head automated coverage establishes:

• exact-owner routing for receipt-backed and receipt-less recovery
• visible fail-closed handling for ambiguous, stale, or drifted gateway ownership
• two ownership checks before mutation and absence of destructive provisioning steps
• receipt model/port propagation and current-recommendation selection only when no receipt exists
• incomplete receipt reconciliation through repair and persistence
• schema-4 cache-field and original timestamp preservation
• exact provider replacement, lost-acknowledgement handling, first-snapshot pinning, endpoint-cycle rollback, and conservative receipt restoration
• WSL restart arming, per-step rollback budgets, forced rollback, recovery toggle/model gating, and full config restoration on both recovery exits

Prior host evidence is intentionally not counted as current-catalog recovery proof: pinned legacy Qwen3.5 9B and llama.cpp b10655 loaded successfully, but that does not satisfy the current catalog or enabled WinUI recovery gate.

Not verified / blocked: current-catalog native inference, enabled recovery-page interactive proof, real WSL-to-Windows reachability, and a live existing-gateway provider/restart/receipt recovery transaction. Keep this PR draft with status: 📣 needs proof until windows-wsl-dgx-blackwell and interactive proof are available or a maintainer explicitly accepts the scoped proof gap.

Ownership notes

• old owner: Local AI Retry entered general onboarding and its destructive-capable default pipeline
• new owner: WindowManager.ShowLocalAiSetupAsync selects provision/recovery/blocked routing; BuildLocalAiRecoverySteps owns the exact-owner non-destructive transaction
• preserved invariant: gateway/provider lifecycle remains compatible with fix(local-ai): keep routing safe across endpoint restarts #1390, and legacy cache migration receipts remain compatible with feat(local-ai): add verified legacy model cache migration #1388

Security impact

No new credentials or permissions. Recovery fails closed on ambiguous ownership and binds WSL/provider mutation to one revalidated registry owner, distro, canonical loopback endpoint/port, and retained receipt.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed September 11, 2026, 2:32 PM ET / 18:32 UTC (Revision 9).

ClawSweeper review

What this changes

Adds a dedicated Local AI retry flow that preserves the existing WSL gateway while repairing installation assets, updating provider routing, and coordinating rollback.

Merge readiness

⛔ Blocked before merge - 13 items remain

The recovery fix remains necessary on main. Both prior rollback blockers remain, and integration with the busy-GPU change exposes another installation blocker.

Likely related people: shanselman and joelagnel (routing candidates, medium confidence).

Priority: P0
Reviewed head: 3763c195434120fb8d3566f92d3ec5137163551c
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The recovery design and extensive tests are useful, but three blocking defects and missing authority-boundary proof prevent readiness.
Proof confidence	🦪 silver shellfish (2/6)	Needs real behavior proof before merge: Authority-chain proof required: ConfigureLocalAiGatewayStep and runtime cleanup are exercised through fake Gateway commands, injected health results, and disposal counters. No supplied trace shows allowed recovery plus rejection of a foreign or reassigned listener before prompt delivery, including same-port rollback. Collaborator status exempts unrelated contributor proof; the separately declared GPU and WinUI gaps remain a maintainer choice. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🦪 silver shellfish (2/6)	Security review found an item that needs attention.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs real behavior proof before merge: Authority-chain proof required: ConfigureLocalAiGatewayStep and runtime cleanup are exercised through fake Gateway commands, injected health results, and disposal counters. No supplied trace shows allowed recovery plus rejection of a foreign or reassigned listener before prompt delivery, including same-port rollback. Collaborator status exempts unrelated contributor proof; the separately declared GPU and WinUI gaps remain a maintainer choice. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	10 items	Applicable repository policy: Read the complete root AGENTS.md and the applicable proof-validation skill. No ancestor-scoped policy was found under the changed source, test, or docs paths; .agents/maintainer-notes is absent. Applied the window ownership, recovery validation, and explicit proof-blocker guidance. Central fix is still absent from main: Main still routes RetrySetup through ShowOnboarding. Its default setup pipeline includes gateway/distro cleanup and provisioning; the dedicated recovery pipeline belongs to this PR. Provider restoration precedes listener verification: Rollback builds and applies the original provider at lines 282-294 without checking its live listener. The new probe runs later in PreserveLocalAiRecoveryGatewayStep, after provider rollback and runtime cleanup, and only controls receipt restoration.
Findings	3 actionable findings	[P1] Verify the original listener before restoring its provider [P1] Preserve replacements that reuse the original endpoint [P1] Keep model-loading diagnostics out of recovery installation
Security	Needs attention	Stale endpoint restoration can authorize a foreign listener: The retained receipt supplies the original provider URL, but rollback applies it before any live ownership check. A different process occupying that port can receive subsequent Gateway prompts; the later health probe cannot undo that exposure.

How this fits together

Local AI setup connects the Windows model runtime to the Gateway running in WSL. Recovery uses gateway records and installation receipts to select the existing installation, repair its assets, and update where the Gateway sends inference requests.

flowchart TD
  A[Local AI retry] --> B[Check gateway ownership]
  B --> C[Existing gateway recovery]
  B --> D[Fresh setup or visible warning]
  C --> E[Repair model and runtime]
  E --> F[Update Gateway provider]
  F --> G[Restart and complete]
  F --> H[Coordinate route and receipt rollback]

Loading

Decision needed

Question	Recommendation
After the code blockers and authority-chain proof are resolved, must the declared GPU and interactive recovery pools complete before landing, or may those specific gaps be accepted?	Complete the declared recovery proof: Require the relevant hardware, existing-gateway recovery, and enabled WinUI evidence before landing.

Why: The PR explicitly leaves these capacity-dependent checks blocked and requests maintainer acceptance; source inspection cannot supply that acceptance.

Before merge

• Add real behavior proof - Needs real behavior proof before merge: Authority-chain proof required: ConfigureLocalAiGatewayStep and runtime cleanup are exercised through fake Gateway commands, injected health results, and disposal counters. No supplied trace shows allowed recovery plus rejection of a foreign or reassigned listener before prompt delivery, including same-port rollback. Collaborator status exempts unrelated contributor proof; the separately declared GPU and WinUI gaps remain a maintainer choice. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• Verify the original listener before restoring its provider (P1) - If recovery moves from endpoint A to B and A exits or its port is reassigned, this branch restores A's provider without validating its live owner. The added probe in PreserveLocalAiRecoveryGatewayStep runs later in reverse rollback and only gates receipt restoration, so it cannot prevent Gateway traffic reaching a dead or foreign listener. This prior blocker remains: establish live managed-listener ownership before applying the restore batch, and preserve a safe route when that cannot be proven.
• Preserve replacements that reuse the original endpoint (P1) - When the original process has stopped and recovery starts its replacement on the same configured port, the provider matches the current install, so ConfigureLocalAiGatewayStep never sets LocalAiRecoveryProviderTransition. A later failure therefore bypasses this guard and disposes the replacement while rollback retains the matching provider. The setup runtime uses the no-op Gateway lifecycle, so disposal does not withdraw that route. This prior blocker remains: coordinate cleanup from actual route/process ownership, including equal endpoints.
• Keep model-loading diagnostics out of recovery installation (P1) - The newly integrated main accepts supported-but-busy GPUs and defers model loading, but this recovery pipeline still requires inference and GPU-load verification. With another application occupying VRAM, asset repair and router startup can succeed before VerifyLocalAiInferenceStep fails and forces rollback, leaving Retry unable to complete. Remove these mandatory diagnostics from recovery and retain them in the existing opt-in proof path, with a busy-GPU recovery regression test.
• Resolve security concern: Stale endpoint restoration can authorize a foreign listener - The retained receipt supplies the original provider URL, but rollback applies it before any live ownership check. A different process occupying that port can receive subsequent Gateway prompts; the later health probe cannot undo that exposure.
• Resolve merge risk (P1) - Live rollback under listener reassignment remains unproven; restoring a stale local endpoint can expose prompts to its new occupant.
• Resolve merge risk (P1) - Receipt-backed upgrades and receipt-less recovery lack an observed complete transaction demonstrating consistent provider, process, and receipt state after failure.
• Resolve merge risk (P1) - The declared hardware and interactive proof pools remain blocked; their acceptance requires an explicit maintainer decision.
• Complete next step (P2) - Resolve all three findings, provide final-effect authority proof with private details redacted, and obtain a decision on the declared capacity-dependent proof gaps.
• Improve patch quality - Fix provider restoration ordering, same-port cleanup, and mandatory recovery model loading with transaction-level regression coverage.
• Improve patch quality - Supply redacted production-boundary logs showing authorized recovery and rejection before prompt delivery to foreign or reassigned ports; update the PR body to trigger review, or request @clawsweeper re-review if needed.
• Improve patch quality - Complete the declared capacity-dependent proof or record explicit maintainer acceptance of the remaining GPU and interactive gaps.
• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Findings

• [P1] Verify the original listener before restoring its provider — src/OpenClaw.SetupEngine/LocalAiGatewayConfiguration.cs:282-286
• [P1] Preserve replacements that reuse the original endpoint — src/OpenClaw.SetupEngine/LocalAiSetupSteps.cs:823-830
• [P1] Keep model-loading diagnostics out of recovery installation — src/OpenClaw.SetupEngine/SetupPipeline.cs:80-82
• [high] Stale endpoint restoration can authorize a foreign listener — src/OpenClaw.SetupEngine/LocalAiGatewayConfiguration.cs:284

Agent review details

Security

Needs attention: Recovery can republish a stale endpoint before validating its live listener; no unrelated supply-chain changes were found.

Review metrics

Metric	Value	Why it matters
Production and test growth	Production +1,049/-116; tests +1,677/-45; docs +6	The growth implements a dedicated recovery transaction with substantial regression coverage, although the full rollback boundary remains insufficiently covered.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #1307
Summary: This PR is the active candidate for the reported destructive Local AI retry; the issue must remain open until a safe fix lands.

Members:

• canonical: Local AI: "Retry setup or download" destroys WSL sandbox, then "Install & setup" stays greyed out #1307 - Its expected behavior directly matches the non-destructive retry implementation.
• adjacent_distinct: fix(setup): allow Local AI setup on busy GPUs and current Gateway #1399 - The merged busy-GPU and Gateway CLI fix changes the installation contract but does not implement this recovery flow.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

1. Repair and prove the complete rollback (recommended)
   Verify listener ownership before restoring routing, cover same-port replacement cleanup, and demonstrate recovery without mandatory model loading.
2. Hold for recovery validation
   Keep this landing candidate open until the transaction and declared capacity-dependent proof can be evaluated together.

Technical review

Best possible solution:

Keep the in-product non-destructive repair flow, bind rollback to verified live listener ownership, coordinate process cleanup with the retained route, and preserve deferred model loading.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: main sends retry into general onboarding, and the proposed rollback paths expose explicit stale-listener and same-port triggers. No runtime reproduction was executed in this read-only review.

Is this the best way to solve the issue?

The dedicated recovery pipeline is the right boundary, but this implementation is not yet safe: listener authorization must precede provider restoration, cleanup must follow the actual retained route, and installation must retain main's deferred-load behavior.

Full review comments:

• [P1] Verify the original listener before restoring its provider — src/OpenClaw.SetupEngine/LocalAiGatewayConfiguration.cs:282-286
  If recovery moves from endpoint A to B and A exits or its port is reassigned, this branch restores A's provider without validating its live owner. The added probe in PreserveLocalAiRecoveryGatewayStep runs later in reverse rollback and only gates receipt restoration, so it cannot prevent Gateway traffic reaching a dead or foreign listener. This prior blocker remains: establish live managed-listener ownership before applying the restore batch, and preserve a safe route when that cannot be proven.
  Confidence: 0.98
• [P1] Preserve replacements that reuse the original endpoint — src/OpenClaw.SetupEngine/LocalAiSetupSteps.cs:823-830
  When the original process has stopped and recovery starts its replacement on the same configured port, the provider matches the current install, so ConfigureLocalAiGatewayStep never sets LocalAiRecoveryProviderTransition. A later failure therefore bypasses this guard and disposes the replacement while rollback retains the matching provider. The setup runtime uses the no-op Gateway lifecycle, so disposal does not withdraw that route. This prior blocker remains: coordinate cleanup from actual route/process ownership, including equal endpoints.
  Confidence: 0.97
• [P1] Keep model-loading diagnostics out of recovery installation — src/OpenClaw.SetupEngine/SetupPipeline.cs:80-82
  The newly integrated main accepts supported-but-busy GPUs and defers model loading, but this recovery pipeline still requires inference and GPU-load verification. With another application occupying VRAM, asset repair and router startup can succeed before VerifyLocalAiInferenceStep fails and forces rollback, leaving Retry unable to complete. Remove these mandatory diagnostics from recovery and retain them in the existing opt-in proof path, with a busy-GPU recovery regression test.
  Confidence: 0.95

Overall correctness: patch is incorrect
Overall confidence: 0.96

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against c71121cdcdc9.

Labels

Label justifications:

• P0: The linked existing-user failure describes retry destroying the working WSL environment and leaving Local AI recovery unusable.
• merge-risk: 🚨 compatibility: The new recovery installation path requires model loading despite main's supported busy-GPU installation contract.
• merge-risk: 🚨 auth-provider: Rollback can retain a provider pointing at a replacement runtime that cleanup subsequently stops.
• merge-risk: 🚨 security-boundary: Recovery republishes an old endpoint before establishing that its current listener is still authorized.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦪 silver shellfish.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: Authority-chain proof required: ConfigureLocalAiGatewayStep and runtime cleanup are exercised through fake Gateway commands, injected health results, and disposal counters. No supplied trace shows allowed recovery plus rejection of a foreign or reassigned listener before prompt delivery, including same-port rollback. Collaborator status exempts unrelated contributor proof; the separately declared GPU and WinUI gaps remain a maintainer choice. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

Security concerns:

• [high] Stale endpoint restoration can authorize a foreign listener — src/OpenClaw.SetupEngine/LocalAiGatewayConfiguration.cs:284
  The retained receipt supplies the original provider URL, but rollback applies it before any live ownership check. A different process occupying that port can receive subsequent Gateway prompts; the later health probe cannot undo that exposure.
  Confidence: 0.97

What I checked:

• Applicable repository policy: Read the complete root AGENTS.md and the applicable proof-validation skill. No ancestor-scoped policy was found under the changed source, test, or docs paths; .agents/maintainer-notes is absent. Applied the window ownership, recovery validation, and explicit proof-blocker guidance. (AGENTS.md:88, 3763c1954341)
• Central fix is still absent from main: Main still routes RetrySetup through ShowOnboarding. Its default setup pipeline includes gateway/distro cleanup and provisioning; the dedicated recovery pipeline belongs to this PR. (src/OpenClaw.Tray.WinUI/Presentation/LocalAiPageViewModel.cs:196, c71121cdcdc9)
• Provider restoration precedes listener verification: Rollback builds and applies the original provider at lines 282-294 without checking its live listener. The new probe runs later in PreserveLocalAiRecoveryGatewayStep, after provider rollback and runtime cleanup, and only controls receipt restoration. (src/OpenClaw.SetupEngine/LocalAiGatewayConfiguration.cs:284, 3763c1954341)
• Same-port replacement bypasses preservation: When the existing provider matches the replacement endpoint, matchesRecoveryInstall remains false and no recovery transition is recorded. StartLocalAiRuntimeStep then disposes the replacement during rollback; its runtime uses the default no-op Gateway lifecycle, so disposal does not withdraw the retained provider. (src/OpenClaw.SetupEngine/LocalAiSetupSteps.cs:823, 3763c1954341)
• Recovery still requires model loading: The introduced recovery pipeline includes mandatory inference and GPU-load verification before gateway repair. Current main instead places these steps in opt-in diagnostics and explicitly says they must not gate installation. (src/OpenClaw.SetupEngine/SetupPipeline.cs:80, 3763c1954341)
• Re-review continuity: GitHub's comparison confirms that the latest merge adds current-main CLI parsing and busy-GPU changes without repairing the two previously reported rollback paths. A local historical-blob comparison failed with a promisor HTTP 403; the read-only GitHub comparison supplied the missing patch context. The busy-GPU finding follows the newly integrated base contract. (3763c1954341)

Likely related people:

• shanselman: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• joelagnel: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (8 earlier review cycles)

• reviewed 2026-09-10T00:32:04.706Z sha bded03c :: blocked before merge. :: none
• reviewed 2026-09-10T01:12:28.167Z sha bded03c :: blocked before merge. :: none
• reviewed 2026-09-10T02:02:34.128Z sha bded03c :: blocked before merge. :: none
• reviewed 2026-09-10T19:21:57.776Z sha a008e42 :: needs real behavior proof before merge. :: [P1] Verify the original listener before republishing its endpoint | [P1] Keep the replacement listener alive when compensation retains its route
• reviewed 2026-09-10T22:15:50.029Z sha a008e42 :: needs real behavior proof before merge. :: [P1] Verify the original listener before republishing its endpoint | [P1] Keep the replacement listener alive when compensation retains its route
• reviewed 2026-09-10T23:02:36.807Z sha c9dee41 :: needs real behavior proof before merge. :: [P1] Verify the original listener before republishing its endpoint | [P1] Keep the replacement listener alive when compensation retains its route
• reviewed 2026-09-10T23:09:53.470Z sha c9dee41 :: needs real behavior proof before merge. :: [P1] Verify the original listener before republishing its endpoint | [P1] Keep the replacement listener alive when compensation retains its route
• reviewed 2026-09-11T17:57:37.901Z sha 776c7f4 :: needs real behavior proof before merge. :: [P1] Verify the original listener before restoring its provider | [P1] Preserve replacements that reuse the original endpoint

[shanselman]

Pushed two targeted rollback fixes for the recovery-rollback race conditions identified in review:

1. PreserveLocalAiRecoveryGatewayStep now probes the original (A) endpoint's health via \LlamaServerClient.ProbeManagedModelAsync\ before restoring its manifest receipt on rollback. A stale receipt alone can't prove the original process is still alive; if the probe fails, the replacement (B) receipt is preserved instead of pointing Gateway at a dead process.
2. StartLocalAiRuntimeStep.RollbackAsync now skips disposing the replacement runtime when \ctx.LocalAiRecoveryProviderTransition\ is true and \ctx.LocalAiRecoveryReceiptRollbackAllowed\ is false, i.e. when Gateway config rollback could not confirm it stopped routing to the replacement's endpoint. This avoids killing B's process while Gateway may still be pointed at it.

Added regression tests for both fixes and updated existing recovery-rollback tests to stub the new health probe. Full validation: \�uild.ps1, Shared.Tests (3991 passed), Tray.Tests (2908 passed), SetupEngine.Tests (1191 passed, 1 pre-existing skip).