Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 40 additions & 1 deletion src/MeshWeaver.Compiler.Pipeline/CompilationCacheService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,35 @@ public string? BuildMvid

private readonly Lazy<string?> _buildMvidFromFile;

/// <summary>
/// Makes this context's build RESIDENT — its assembly loaded, so the bytes are in memory and
/// no later deletion of its file can take them away — and says whether that succeeded.
/// <see cref="BuildMvid"/> alone cannot answer "can this context serve its build": it is read
/// once from the header and outlives the file (#5555). An existence check would not answer it
/// either: the retention sweep can delete the file between the check and the caller's load.
/// Loading IS the durable guarantee, and a load that loses that race — the file already gone,
/// or the context retired meanwhile — answers <c>false</c>.
/// </summary>
public bool TryMakeResident()
{
try
{
return LoadNodeAssembly() is not null;
}
catch (ObjectDisposedException)
{
// Retired between the caller's IsRetired check and this load — not a fault, the
// outcome of the race: this context can no longer be handed to a new caller.
return false;
}
catch (FileNotFoundException)
{
// The sweep deleted the file between LoadNodeAssembly's existence check and its load —
// the very race this method closes; the bytes are not resident, so no reuse.
return false;
}
}

/// <summary>
/// Pins the context for the duration of a SCAN of its loaded assembly (GetTypes + attribute
/// reflection + Activator). While any pin is held, <see cref="Dispose"/> waits before
Expand Down Expand Up @@ -1470,7 +1499,17 @@ private NodeAssemblyLoadContext ResolveLoadContextForPath(
if (!string.Equals(context.NodeName, nodeName, StringComparison.Ordinal)
|| context.IsRetired)
continue;
if (string.Equals(context.BuildMvid, mvid, StringComparison.OrdinalIgnoreCase))
// 🚨 The same MVID is the same build only while the context can still LOAD it
// (#5555). Re-adopting identical bytes lands them under a new store name
// (v{version}-…), the retention sweep deletes the old one, and a context over the old
// name that never loaded its assembly keeps reporting the MVID it read from the
// header. Aliasing a read of the present copy to it made every load answer "No file
// at" the deleted path — permanently, since each recompile's identical bytes were
// aliased back to the same dead context. So the candidate's bytes are made RESIDENT
// before it is returned; a candidate that loses the deletion race is skipped and the
// read resolves a context over its own, present path.
if (string.Equals(context.BuildMvid, mvid, StringComparison.OrdinalIgnoreCase)
&& context.TryMakeResident())
return context;
}
return null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2540,6 +2540,23 @@ safe:
alias and creates a fresh context, exactly as a first read of that path always did.
- **One lease per context.** `LeaseNodeContexts` leases each context once however many keys name it,
so releasing the hub's one lease performs the deferred unload.
- **A context that can no longer LOAD its build is never aliased (#5555).** The MVID is read once,
from the header, and outlives the file. Re-adopting identical bytes lands them under a new store
name (`v{node version}-{tag}-{hash}.dll`) and the retention sweep deletes the old name, so a context
over the old name that never loaded its assembly still reports the build's MVID while it can load
nothing. Aliasing a read of the present copy to it made every load answer *"No file at"* the deleted
path, and every recompile's identical bytes were aliased back to the same dead context — a
permanent loop. Measured on memex.systemorph.com 2026-10-07: `Store/Plugin` instances failed 604
times in 25 minutes on one pod (*"Failed to load assembly at …/v18979-c003e001-1db01ff9e178.dll —
No file at …/v14766-c003e001-1db01ff9e178.dll"*), the type node was written ~6,900 times in 15 hours
(v12083 → v18986), the recycle-on-new-build rule read each re-adoption as a new build (2,126
recycle jobs for that type alone), and the replica held an 11+ GiB heap with GC taking more than
half of wall time. The resolver now makes the candidate's bytes RESIDENT before aliasing to it
(`NodeAssemblyLoadContext.TryMakeResident` — load the assembly; a load that loses the deletion race
skips the candidate). An existence check would not do: the sweep can delete the file between the
check and the caller's load, so loading is the only durable guarantee. Pinned by
`ScanPinSupersessionTest.AReadIsNeverAliasedToASameBuildContextThatCanNoLongerLoadIt` (red on the
old resolver) and its control `ASameBuildContextThatHasLoadedItsAssemblyIsStillReusedAfterItsFileIsGone`.

No identity ⇒ no alias: an unreadable file, or an in-memory context, resolves a fresh context as
before. Pinned in core by `ScanPinSupersessionTest.AReadOfTheStoreCopyOfThePublishedBuildIsThatBuildsContext`,
Expand Down
62 changes: 62 additions & 0 deletions test/MeshWeaver.Compiler.Pipeline.Test/ScanPinSupersessionTest.cs
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,68 @@ public void AReadOfTheStoreCopyOfThePublishedBuildIsThatBuildsContext()
read.LoadNodeAssembly().Should().NotBeNull();
}

/// <summary>
/// 🚨 THE SAME MVID IS NOT THE SAME LOADABLE BUILD WHEN THE BYTES ARE GONE (#5555). The
/// assembly store names a build <c>v{node version}-{tag}-{hash}.dll</c>, and re-adopting
/// identical bytes lands them under a NEW version, so one generation gathers many store names
/// over time while the retention sweep deletes the old ones. A context created over an old name
/// that never loaded its assembly still reports that name's MVID (read once, from the header),
/// so the same-build alias handed it to a read of the NEW, present name — and every load then
/// answered "No file at" the deleted path. Measured on memex.systemorph.com 2026-10-07:
/// <c>Store/Plugin</c> instances failed 604 times in 25 minutes on one pod with "Failed to load
/// assembly at …/v18979-c003e001-1db01ff9e178.dll — No file at …/v14766-c003e001-1db01ff9e178.dll",
/// each failure re-driving a recompile whose identical bytes were aliased to the same dead context.
/// </summary>
[Fact]
public void AReadIsNeverAliasedToASameBuildContextThatCanNoLongerLoadIt()
{
var emitted = EmitAssembly("emit");
var evicted = CopyToStore(emitted, "v14766-tag-hash.dll");
var current = CopyToStore(emitted, "v18979-tag-hash.dll");

// An instance read the old store name and never loaded it; its identity was read from the
// header while the file existed.
var stale = _service.GetOrCreateLoadContextForPath(NodeName, evicted);
stale.BuildMvid.Should().NotBeNull("premise: the old context knows the build's identity");
stale.LoadedAssembly.Should().BeNull("premise: the old context never loaded its assembly");

// The retention sweep deletes the old name.
File.Delete(evicted);

var read = _service.GetOrCreateLoadContextForPath(NodeName, current);

read.Should().NotBeSameAs(stale,
"a context whose bytes are neither loaded nor on disk cannot serve the build, so a read "
+ "of a present copy must not be aliased to it");
read.LoadNodeAssembly().Should().NotBeNull(
$"the file at the requested path exists; the load failed with: {read.LastLoadFailure}");
}

/// <summary>
/// The control for the test above: when the old context HAS loaded its assembly, the bytes are
/// in memory and the alias is the correct reuse — dropping it would mint a second collectible
/// context over identical bytes (the duplicate <c>AReadOfTheStoreCopyOfThePublishedBuildIsThatBuildsContext</c>
/// forbids), even after the old file is gone.
/// </summary>
[Fact]
public void ASameBuildContextThatHasLoadedItsAssemblyIsStillReusedAfterItsFileIsGone()
{
var emitted = EmitAssembly("emit");
var evicted = CopyToStore(emitted, "v14766-tag-hash.dll");
var current = CopyToStore(emitted, "v18979-tag-hash.dll");

var live = _service.GetOrCreateLoadContextForPath(NodeName, evicted);
var assembly = live.LoadNodeAssembly();
assembly.Should().NotBeNull();
File.Delete(evicted);

var read = _service.GetOrCreateLoadContextForPath(NodeName, current);

read.Should().BeSameAs(live, "the loaded generation serves its build from memory");
read.LoadNodeAssembly().Should().BeSameAs(assembly);
GC.KeepAlive(assembly);
}

/// <summary>
/// The reuse must be a REUSE and nothing else: once a publish has superseded the build, a read of
/// its store copy must NOT be handed the retired context (that would resurrect a generation the
Expand Down
Loading