Skip to content

fix(#1462): a stranded Compiling claim is re-driven by the sweep that already runs - #1593

Merged
rbuergi merged 1 commit into
mainfrom
fix/1462-stranded-compile-claim
Aug 14, 2026
Merged

rbuergi merged 1 commit into
mainfrom
fix/1462-stranded-compile-claim

Conversation

@rbuergi

@rbuergi rbuergi commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Compiling is a non-terminal state that nothing reconciles. The flip to it is durable; the terminal write is not guaranteed — a process death mid-compile leaves the row there for good.

DynamicTypePreWarmer's "don't clobber an in-flight compile someone else already started" guard then declines to touch it on every subsequent sweep:

if (def.CompilationStatus is CompilationStatus.Pending or CompilationStatus.Compiling)
    return node;   // ← forever, for a claim whose owner died

So the type never reaches a terminal state. It is neither Ok (usable) nor Error (classifiable by #1396's discriminator), it holds portal readiness, and it parks every instance hub for the full activation budget. The row that revealed this sat at Compiling for ten weeks — harmless only because it was an orphan the prewarmer never enumerates. Apply the same mechanism to a row that is in the bake set and it is the readiness-refusal shape #1391 opened with.

The fix

The guard now defers only to a claim that can still be in flight:

state verdict why
Pending honoured it is what this prewarmer itself writes to ask for a compile — re-driving it would fight our own request
Compiling, started within the per-type budget honoured it can still be in flight; clobbering restarts work that is about to finish
Compiling, started longer ago than the budget stranded no driver is still working on it — it either finished (and would have written a terminal state) or it died
Compiling, no start timestamp stranded a live compile always stamps one, so this is the shape a row left over from an older write carries — and honouring it forever is how this became permanent

The bound is the sweep's own per-type budget, not a new number.

🚨 What this is not

No timer, no poller, no background sweep for stale rows. The issue rules those out explicitly ("Do not add a timer that polls for stale Compiling rows and flips them — that is the watchdog-shaped band-aid AGENTS.md rules out"), and so does AGENTS.md.

The recovery rides the enumeration that already runs, and only ever reinterprets a claim that has provably expired. StrandedCompileClaimTest pins exactly that distinction — including the case that must NOT change (a compile 5 s old is still honoured), because a fix here that clobbered live compiles would be worse than the bug.

Verification

  • MeshWeaver.Hosting.Test — 176/176, 8 of them new.
  • -c Release -warnaserror — MeshWeaver.Hosting, MeshWeaver.Hosting.Test: 0 errors, 0 warnings.

No What's New entry: the user-visible effect (a portal that will not become ready) is a failure that this prevents rather than a change they can notice.

Fixes #1462

🤖 Generated with Claude Code

https://claude.ai/code/session_01KmBFaHTqhy7h742wkAaJKj

… already runs

`Compiling` is a non-terminal state and nothing reconciles it. The flip to
`Compiling` is DURABLE; the terminal write is not guaranteed — a process death
mid-compile leaves the row there for good.

DynamicTypePreWarmer's "don't clobber an in-flight compile someone else already
started" guard then declines to touch it on EVERY subsequent sweep, so the type
never reaches a terminal state. It is neither Ok (usable) nor Error
(classifiable by #1396's discriminator), it holds portal readiness, and it parks
every instance hub for the full activation budget. One row on public.mesh_nodes
sat at `Compiling` for ten weeks this way — harmless only because it was an
orphan the prewarmer never enumerates. Apply the same mechanism to a row that IS
in the bake set and it is the readiness-refusal shape #1391 opened with.

The guard now defers only to a claim that can still be in flight:
  * Pending — always honoured. It is the state this prewarmer itself writes to
    ASK for a compile; re-driving it would fight our own request.
  * Compiling — honoured only while LastCompileStartedAt is within the per-type
    budget, the same bound the sweep gives a type to settle. Past it no driver
    is still working on it: it either finished (and would have written a
    terminal state) or it died.
  * Compiling with NO start timestamp — stranded. A live compile always stamps
    one, so an unstamped Compiling is exactly the shape a row left over from an
    older write carries, and honouring it forever is how this became permanent.

🚨 What this is NOT: no timer, no poller, no background sweep for stale rows —
the issue rules those out explicitly and so does AGENTS.md. The recovery rides
the enumeration that ALREADY runs, and only ever reinterprets a claim that has
provably expired. StrandedCompileClaimTest pins that distinction, which is the
whole substance of the change.

Verified: MeshWeaver.Hosting.Test 176/176 (8 new); MeshWeaver.Hosting and
MeshWeaver.Hosting.Test build -c Release -warnaserror with 0 errors, 0 warnings.

Fixes #1462

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KmBFaHTqhy7h742wkAaJKj
Copilot AI lite review requested due to automatic review settings August 14, 2026 15:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses issue #1462 where a NodeType can remain stuck in the non-terminal Compiling state indefinitely (e.g., after a mid-compile process crash), causing subsequent prewarm sweeps to permanently defer and never re-drive the type to a terminal status.

Changes:

  • Refines DynamicTypePreWarmer’s “don’t clobber an in-flight compile” guard to only defer to compile claims that can still be live (fresh Compiling within the existing per-type budget, or Pending).
  • Introduces IsLiveCompileClaim(...) to centralize and test the stranded-claim interpretation.
  • Adds StrandedCompileClaimTest to pin the intended behavior for fresh vs. stale/unstamped Compiling claims.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/MeshWeaver.Hosting/DynamicTypePreWarmer.cs Updates the sweep’s guard to re-drive stranded Compiling claims via IsLiveCompileClaim, preventing permanent non-terminal status.
test/MeshWeaver.Hosting.Test/StrandedCompileClaimTest.cs Adds unit coverage for the new “live vs stranded compile claim” logic.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/MeshWeaver.Hosting.Test/StrandedCompileClaimTest.cs
@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

728 tests  ±0   728 ✅ ±0   9m 56s ⏱️ - 2m 4s
 10 suites ±0     0 💤 ±0 
 10 files   ±0     0 ❌ ±0 

Results for commit b89c9eb. ± Comparison against base commit 0ae56bb.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 933 tests  +8   1 923 ✅ +8   7m 11s ⏱️ -19s
   11 suites ±0      10 💤 ±0 
   11 files   ±0       0 ❌ ±0 

Results for commit b89c9eb. ± Comparison against base commit 0ae56bb.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

1 925 tests  ±0   1 733 ✅ ±0   6m 3s ⏱️ -12s
   11 suites ±0     192 💤 ±0 
   11 files   ±0       0 ❌ ±0 

Results for commit b89c9eb. ± Comparison against base commit 0ae56bb.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

   11 files  ±0     11 suites  ±0   6m 5s ⏱️ +6s
1 840 tests ±0  1 737 ✅ ±0  103 💤 ±0  0 ❌ ±0 
2 186 runs  ±0  2 083 ✅ ±0  103 💤 ±0  0 ❌ ±0 

Results for commit b89c9eb. ± Comparison against base commit 0ae56bb.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

1 464 tests  ±0   1 464 ✅ ±0   6m 53s ⏱️ +38s
   11 suites ±0       0 💤 ±0 
   11 files   ±0       0 ❌ ±0 

Results for commit b89c9eb. ± Comparison against base commit 0ae56bb.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

2 553 tests  ±0   2 552 ✅ ±0   8m 46s ⏱️ -4s
   12 suites ±0       1 💤 ±0 
   12 files   ±0       0 ❌ ±0 

Results for commit b89c9eb. ± Comparison against base commit 0ae56bb.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

    66 files  ±0      66 suites  ±0   44m 56s ⏱️ - 1m 55s
10 443 tests +8  10 137 ✅ +8  306 💤 ±0  0 ❌ ±0 
10 789 runs  +8  10 483 ✅ +8  306 💤 ±0  0 ❌ ±0 

Results for commit b89c9eb. ± Comparison against base commit 0ae56bb.

@rbuergi

rbuergi commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Review comment checked and not applied — using MeshWeaver.Mesh.Services; is required, not unused.

CompilationStatus lives in MeshWeaver.Mesh.Services (src/MeshWeaver.Mesh.Contract/Services/CompilationStatus.cs), not in MeshWeaver.Graph.Configuration where NodeTypeDefinition lives — the test names both. Removing the line and building with the same flags:

StrandedCompileClaimTest.cs(27,43): error CS0246: The type or namespace name 'CompilationStatus' could not be found
StrandedCompileClaimTest.cs(75,55): error CS0246: …
StrandedCompileClaimTest.cs(71,17): error CS0103: The name 'CompilationStatus' does not exist in the current context
StrandedCompileClaimTest.cs(72,17): error CS0103: …

Restored, and dotnet build test/MeshWeaver.Hosting.Test -c Release -warnaserror is back to 0 Error(s), 0 Warning(s) — which is also what the green CI on this PR already showed, since CS8019 would have failed the build under -warnaserror exactly as the comment describes.

Resolving the thread.

@rbuergi
rbuergi merged commit a2671c5 into main Aug 14, 2026
23 checks passed
@rbuergi
rbuergi deleted the fix/1462-stranded-compile-claim branch August 19, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A NodeType row can sit at compilationStatus='Compiling' indefinitely — nothing reconciles a non-terminal compile state

2 participants