Skip to content

fix(compiler): never alias a read to a same-MVID load context that can no longer load its build (#5555) - #6261

Merged
rbuergi merged 2 commits into
mainfrom
fix/oom-live-heap-driver
Oct 7, 2026
Merged

rbuergi merged 2 commits into
mainfrom
fix/oom-live-heap-driver

Conversation

@rbuergi

@rbuergi rbuergi commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What is growing

On memex.systemorph.com a NodeType compile/bind loop drives the heap, not the document-part seal (Plugins#2979 is deployed and did its part).

  • Store/Plugin NodeType node: v12083 (10-06 17:00Z) → v18986 (10-07 13:22Z). That is ~6,900 writes, in bursts of ~1,000–2,000 per hour (10-06 22–00Z, 10-07 08–10Z). Each cycle writes Ok → Ok → Pending. The dispatched adoption lands the same bytes (MVID 1d9f4d8e…, hash 1db01ff9e178) under a new v{version} store name.
  • Instance side (Loki, governed Logs action Ops/Actions/logs-oom-storeplugin-bind-20261007): 604× in 25 min on one pod: EnrichWithNodeType: the recorded build of Store/Plugin (… v18979 …) still does not load in this process … Failed to load assembly at /data/assembly-cache/Store_Plugin/v18979-c003e001-1db01ff9e178.dll — No file at '/data/assembly-cache/Store_Plugin/v14766-c003e001-1db01ff9e178.dll'. Every plugin root (AppleMaps, RemoteControl, Crm, …) logs Overlay self-heal … stuck … AGAIN after 13 self-heal(s).
  • Heap: the coordinator's crash reading for pod 65f6466887-5cx77 shows a flat 11.3–11.6 GiB with GC at 56% of wall time from 09:31 to 10:03Z, which is inside this loop window.
  • memex-cloud: Store/Plugin is at v15859 there, so it runs the same loop.

Root cause

CompilationCacheService.FindLiveContextOfSameBuild aliases a read to a live context with the same MVID. BuildMvid is read once from the file header and outlives the file. A context created over v14766-… that never loaded its assembly was therefore handed to a read of the present v18979-…, and its load fails on the evicted path. Each self-heal recompile re-adopts identical bytes under a new version and is aliased back to the same dead context. The loop never ends.

Fix

NodeAssemblyLoadContext.CanLoadItsBuild returns true when the assembly is loaded or its file still exists. It now gates the alias. In-memory reuse is unchanged.

Tests

  • ScanPinSupersessionTest.AReadIsNeverAliasedToASameBuildContextThatCanNoLongerLoadIt is red on the old resolver. It fails at NotBeSameAs, a deterministic negative control.
  • The control ASameBuildContextThatHasLoadedItsAssemblyIsStillReusedAfterItsFileIsGone keeps the loaded-generation reuse.
  • ScanPinSupersessionTest passed 11/11 and AssemblyLoadContextLeakTest passed 6/6.
  • MeshWeaver.Compiler.Pipeline and its test project build clean with -c Release -warnaserror.

Not established

  • I did not establish why the old context's assembly was unloaded or never loaded, nor which sweep deleted v14766.
  • No heap dump was taken. The heap attribution rests on the loop's timing against the GC-bound window.
  • client-b's OOMs (ci.9887) are not explained: its Store/Plugin is at v207, so this loop is not running there.

Deploy / recycle

After the image carrying this rolls, every pod starts with a fresh context cache. No recycle is needed for the fix to take effect. Name to watch: Store/Plugin node version should stop climbing.

Pairs-with: none — no public surface removed (one public property added on a sealed internal-use class).
Companion: Systemorph/MeshWeaver.Plugins — recycle-on-new-build keys on the bytes (MVID), not the version-bearing path.

Refs #5555 #6161 #6163 #6152

🤖 Generated with Claude Code

…n no longer load its build

FindLiveContextOfSameBuild matched contexts by MVID alone. The MVID is read once from the file
header and outlives the file, so a context over an evicted store name (v14766-…) that never loaded
its assembly was handed to a read of the present copy (v18979-…) and every load answered
"No file at" the deleted path. Each recompile re-adopted identical bytes under a new node version
and was aliased back to the same dead context: a permanent self-heal/recompile loop. On
memex.systemorph.com 2026-10-07 Store/Plugin instances failed 604 times in 25 minutes on one pod,
the type node was written ~6,900 times in 15 hours, and the replica sat at an 11+ GiB heap with GC
above half of wall time (#5555, #6161).

NodeAssemblyLoadContext.CanLoadItsBuild (assembly loaded, or its file present) now gates the alias.
ScanPinSupersessionTest.AReadIsNeverAliasedToASameBuildContextThatCanNoLongerLoadIt is red on the
old resolver; its control keeps the in-memory reuse.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new filesystem existence check retains a TOCTOU race with concurrent retention eviction.

1 open finding
What changed in this PR

Prevents compiler-cache aliasing to assembly contexts whose backing files were evicted.

Changes:

  • Adds a loadability guard to same-MVID reuse.
  • Adds regression and loaded-assembly control tests.
  • Documents the incident and cache invariant.
File Description
CompilationCacheService.cs Gates same-build context aliasing.
ScanPinSupersessionTest.cs Tests evicted and loaded contexts.
NodeTypeCompilation.md Documents the failure and fix.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/MeshWeaver.Compiler.Pipeline/CompilationCacheService.cs Outdated
… — an existence check keeps the deletion race

Review on #6261: File.Exists then return leaves a TOCTOU window against the retention sweep.
TryMakeResident loads the candidate's assembly (the durable guarantee) and skips it when the load
loses the race (file gone, context retired).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 16:36
@meshweaver-cloud
meshweaver-cloud Bot enabled auto-merge October 7, 2026 16:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused race fix is consistent with the load-context lifecycle and has appropriate regression coverage.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

1 844 tests  ±0   1 842 ✅ ±0   4m 23s ⏱️ -14s
    4 suites ±0       2 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit ea1ef08. ± Comparison against base commit 051dfd6.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

    5 files  ±0      5 suites  ±0   4m 23s ⏱️ -16s
1 230 tests ±0  1 230 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 231 runs  ±0  1 231 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit ea1ef08. ± Comparison against base commit 051dfd6.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

  3 files  ±0    3 suites  ±0   7m 26s ⏱️ +45s
571 tests ±0  380 ✅ ±0  191 💤 ±0  0 ❌ ±0 
575 runs  ±0  384 ✅ ±0  191 💤 ±0  0 ❌ ±0 

Results for commit ea1ef08. ± Comparison against base commit 051dfd6.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    2 files  ±0      2 suites  ±0   8m 17s ⏱️ -17s
1 119 tests ±0  1 119 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 120 runs  ±0  1 120 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit ea1ef08. ± Comparison against base commit 051dfd6.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

    4 files  ±0      4 suites  ±0   6m 52s ⏱️ +14s
2 616 tests ±0  2 616 ✅ +1  0 💤 ±0  0 ❌  - 1 
2 620 runs  ±0  2 620 ✅ +1  0 💤 ±0  0 ❌  - 1 

Results for commit ea1ef08. ± Comparison against base commit 051dfd6.

This pull request removes 1 and adds 1 tests. Note that renamed tests count towards both.
Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "9414c37b-4c03-44a6-a12c-79bae5d35878")
Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "50bddc66-c8af-4530-8d38-332982982374")

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

    4 files  ±0      4 suites  ±0   16m 28s ⏱️ -29s
3 606 tests +2  3 606 ✅ +2  0 💤 ±0  0 ❌ ±0 
3 609 runs  +2  3 609 ✅ +2  0 💤 ±0  0 ❌ ±0 

Results for commit ea1ef08. ± Comparison against base commit 051dfd6.

This pull request removes 39 and adds 17 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Graph.Test.ScanPinSupersessionTest ‑ AReadIsNeverAliasedToASameBuildContextThatCanNoLongerLoadIt
MeshWeaver.Graph.Test.ScanPinSupersessionTest ‑ ASameBuildContextThatHasLoadedItsAssemblyIsStillReusedAfterItsFileIsGone
MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest ‑ Classifies(shape: "a MIXED aggregate — one transient branch, one genu"···, exception: System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (source B is misconfigured)
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
, expected: False)
MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest ‑ Classifies(shape: "a nested aggregate with one genuine leaf", exception: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (boom)) (Failed to connect to 10.42.18.4:5432)
 ---> System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (boom)
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---

   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
, expected: False)
…

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Test Results

    22 files  ±0      22 suites  ±0   47m 51s ⏱️ -17s
10 986 tests +2  10 793 ✅ +3  193 💤 ±0  0 ❌  - 1 
10 999 runs  +2  10 806 ✅ +3  193 💤 ±0  0 ❌  - 1 

Results for commit ea1ef08. ± Comparison against base commit 051dfd6.

This pull request removes 40 and adds 18 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…
Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "50bddc66-c8af-4530-8d38-332982982374")
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Graph.Test.ScanPinSupersessionTest ‑ AReadIsNeverAliasedToASameBuildContextThatCanNoLongerLoadIt
MeshWeaver.Graph.Test.ScanPinSupersessionTest ‑ ASameBuildContextThatHasLoadedItsAssemblyIsStillReusedAfterItsFileIsGone
MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest ‑ Classifies(shape: "a MIXED aggregate — one transient branch, one genu"···, exception: System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (source B is misconfigured)
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
, expected: False)
…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants