Build race: concurrent LayoutMeshModuleClosures restores clobber a shared project.assets.json #2127

Signature

Build solution (once) (run 32578666522, PR #2078, 2026-08-22T14:33) failed with:

/usr/share/dotnet/sdk/10.0.400/NuGet.targets(198,5): error : The file '/home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.AI/obj/project.assets.json' already exists. [/home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Social/MeshWeaver.Social.csproj]
memex/MeshModulesPublish.targets(279,5): error MSB4181: The "MSBuild" task returned false but did not log an error.  (x2)

The PR's diff was ONE markdown file (src/MeshWeaver.Documentation/Data/Architecture/PlatformAndContent.md), and main's run on the identical base tree (865dc17, run 32577873729) was green minutes earlier — this is a nondeterministic race in the build itself, not a change-induced break.

Mechanism

LayoutMeshModuleClosuresAfterBuild (memex/MeshModulesPublish.targets) fires AfterTargets="Build" on every host that imports it. In the failing run, three hosts finished Build within ~7 s of each other and each launched its own closure lane:

• 14:33:16.5 Memex.Portal.Monolith → Determining projects to restore... at 14:33:16.88
• 14:33:20.5 Memex.Portal.Distributed
• 14:33:23.8 test/Memex.Portal.Shared.Test (subset lane, Cosmos+Snowflake) → Determining projects to restore... at 14:33:23.88, and a third restore at 14:33:25.69

Line 279 of the targets file is the closure lane's explicit Restore of @(MeshModuleClosure) (Social, Hosting.Cosmos, Hosting.Snowflake) — deliberate, because post-flip modules are outside the host's restore graph. BuildInParallel="false" serialises the three modules within one lane, but nothing serialises the lanes across hosts: they are separate NuGet RestoreTask executions whose project-closure graphs overlap (both Social's and the storage backends' closures pull src/MeshWeaver.AI, Mesh.Contract, etc.). Two concurrent RestoreTasks decided the shared MeshWeaver.AI/obj/project.assets.json was stale and raced the write; the loser's atomic temp-file move found the destination already present → already exists → task returns false → MSB4181, twice (two lanes lost).

MSBuild's scheduler cannot dedupe these: the entry projects differ (Social vs Cosmos vs the .targets self-invocation with per-host MeshModulesAppDir), so they are distinct build requests even though their restore graphs overlap.

Why the lane's restore rewrites at all when the solution-level dotnet restore already ran: the lane restores with a different global-property set than the solution restore, so NuGet's dgspec hash differs and the assets files are considered stale — every parallel lane then rewrites the same shared obj/ files.

Impact

Rare (main is green 8/8 recently) but when it hits it fails Build solution (once) for an unrelated PR and costs a full CI cycle plus a misattribution hunt. Per AGENTS.md this class must be root-caused, not retried into submission.

Fix directions (root-cause, not band-aid)

• Serialise the closure lanes process-wide (e.g. hoist the closure restore+layout to a single scheduled point instead of per-host AfterTargets="Build"), or
• Make the lane's restore property set hash-identical to the solution restore so an up-to-date assets file is never rewritten, or
• Restore the closure modules once (entry-point superset) from one place and have the per-host lanes consume only.

Evidence: full log of run 32578666522, job 97044706356.