Repository navigation
Transient CompilationException compiling Store/Plugin NodeType on pod cold start #1213
Description
What is failing
The MeshNodeCompilationService failed to compile the Store/Plugin NodeType assembly on a single pod during what appears to be a cold-start compilation. The NodeType has 57 Code nodes across 4 source namespaces (its own Source/ and Test/, plus shared sources from Store/Coupon, Store/Order, and Store/BillingProfile).
Probable cause
Cause unclear — transient compile failure, confidence moderate. The Store/Plugin NodeType compiled successfully at 2026-08-11T10:59:53 (one release minted) and its compilationStatus is currently Ok. The failure logged at 11:08 on pod memex-portal-deployment-686bdb74cb-vj9xl was a distinct compile attempt that didn't recur — suggesting a transient issue rather than a code defect. Likely candidates:
- Assembly store not yet populated on that pod — the pod started up and tried to compile, but the
IAssemblyStoredidn't have the shared dependency DLLs ready in time, causing a compile-leg timeout or a missing-reference error during Roslyn emit. - Resource contention during warm-up — the pod was compiling multiple NodeTypes concurrently and the Store/Plugin type (57 files, cross-namespace) hit a stage budget (
SourceSnapshotTimeoutat 30s orRoslynCompileTimeoutat 5 min).
No Roslyn CS#### diagnostics are present in the log sample, so the specific cause is invisible from evidence alone.
Impact
Low. 1 occurrence, 1 pod, single timestamp. The failure did not cascade — a working release was already available and the pod recovered without downstream symptoms. End users were not affected.
Where to look
MeshWeaver.Graph.Configuration.MeshNodeCompilationService— theRunCompilemethod that logs this failure. Check whether theCompilationExceptionis thrown with inner Roslyn diagnostics that got lost in the log pipeline.Store/PluginNodeType definition — the compile activity atStore/Plugin/_Activity/compile-...may contain the full diagnostics if a failed compile was recorded as a Failed Release.CompilationCacheOptions— the stage timeouts (especiallySourceSnapshotTimeoutandRoslynCompileTimeouton cold start with 57 source files).
Evidence
| Fingerprint | c159c066f6670322 |
| Category | MeshWeaver.Graph.Configuration.MeshNodeCompilationService |
| Severity | Error |
| Exception | MeshWeaver.Graph.Configuration.CompilationException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-686bdb74cb-vj9xl |
| Occurrences | 1 |
| First seen | 2026-08-11 11:08:11Z |
| Last seen | 2026-08-11 11:08:11Z |
Recent log lines
2026-08-11 11:08:11Z memex-portal-deployment-686bdb74cb-vj9xl fail: MeshWeaver.Graph.Configuration.MeshNodeCompilationService[0]
Failed to compile assembly for node Store/Plugin. Executed source queries (8):
- namespace:Store/Plugin/Source scope:subtree nodeType:Code
- path:Store/Coupon/Source nodeType:Code
- namespace:Store/Coupon/Source scope:subtree nodeType:Code
- path:Store/Order/Source nodeType:Code
- namespace:Store/Order/Source scope:subtree nodeType:Code
- path:Store/BillingProfile/Source nodeType:Code
- namespace:Store/BillingProfile/Source scope:subtree nodeType:Code
- namespace:Store/Plugin/Test scope:subtree nodeType:Code
Matched Code nodes (57):
- Store/Plugin/Source/NamespacePicker
- Store/Plugin/Test/BuildOrderTests
- Store/Plugin/Test/InstallStatusTests
- Store/Plugin/Source/VideoEmbed
- Store/Plugin/Source/Entitlements
- Store/Plugin/Source/DependencyCheck
- Store/Plugin/Source/ModuleDependencies
- Store/Order/Source/OrderControlPlane
- Store/Plugin/Source/StoreTexts
- Store/Plugin/Source/CouponRedemption
- Store/Plugin/Test/DependencyCheckTests
- Store/Plugin/Test/PluginCoverTests
- Store/Order/Source/StripeWebhooks
- Store/Plugin/Source/InstallDialog
- Store/Plugin/Source/ModuleRequirement
- Store/Coupon/Source/CouponContent
- Store/Plugin/Source/MeshQueries
- Store/Plugin/Source/PluginGate
- Store/Order/Source/StripeGateway
- Store/BillingProfile/Source/BillingProfileContent
- Store/Plugin/Test/RedeemBoundedTests
- Store/Order/Source/OrderContent
- Store/Plugin/Test/ModuleDependenciesTests
- Store/Plugin/Test/PackageGraphTests
- Store/Plugin/Source/Localizer
- Store/Plugin/Source/InstallStatus
- Store/Plugin/Source/StandardPacks
- Store/Plugin/Source/OpenGraph
- Store/Plugin/Source/RepoProvision
…[truncated]
Opened automatically from Admin/_LogIncident/c159c066f6670322. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 12, 2026
Covered by PR #1226 (merged
939871331, 2026-08-11 22:08Z — about 11 h after this incident at 11:08Z).The single reporter of this line is
src/MeshWeaver.Graph/Configuration/MeshNodeCompilationService.cs:1062(.Catch<…, CompilationException>). #1226 introducesSourceSnapshot+SourceDiscoveryUnavailableException, so a compile whose source set could not be established now throws a different exception type that never reaches that catch — it is classifiedUnavailable, notCompileError.Honest limit on this call: the incident's own cause is not provable from the evidence. 8 cross-namespace queries into
Store/Coupon|Order|BillingProfileon a cold pod is exactly the starved-discovery shape of #1218, and the absence of any RoslynCS####code matchesFormatCompileFailure's no-diagnostic fallback — but 57 nodes did match, so discovery was at least partly answered, and a genuine Roslyn emit failure with truncated diagnostics would look the same. Either way the starved-discovery route to this log line is now closed.Closing as covered. If
Failed to compile assemblyrecurs forStore/Pluginon an image built after939871331, reopen — that would be the genuine-emit-failure branch and needs different work.systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 20 more since the last update (21 total).
memex-portal-deployment-686bdb74cb-vj9xl,memex-portal-deployment-585f6d7b98-z7ktk,memex-bake-1786623419-kwv2c,memex-portal-deployment-66b79847f8-zdbtz,memex-portal-deployment-68f5f9b9d8-bwgvv,memex-portal-deployment-d976fd57-2g46v,memex-portal-deployment-648c756f74-gzznt,memex-portal-deployment-66dcb75dc9-6249g,memex-portal-deployment-65f89777db-t5rhr,memex-portal-deployment-75476f6c55-48t6h,memex-portal-deployment-84779c8448-dhrv5,memex-portal-deployment-5d6f77b68c-vsmvd,memex-portal-deployment-5d6f77b68c-srtqg,memex-portal-deployment-596d7ff9bf-vtsp6,memex-portal-deployment-7c88f7d9cd-4z9gr,memex-portal-deployment-7c47bc9c85-xhspn,memex-portal-deployment-65f87d4594-5vsw8,memex-portal-deployment-f47f8cdd-f8fmqRecent log lines
rbuergi commentedon Sep 17, 2026
Closing again — the fix is on
main, the type compiles today, and the fault has been silent for a month1. The defect class was fixed by core PR #1226 — "fix(compile): never compile against a source set that could not be established (#1218)", merged
939871331on 2026-08-11T20:08:52Z, about 9 h after this issue's single occurrence (11:08Z). That is what the original close said, and it still holds: a compile leg whose source snapshot could not be established now fails the snapshot instead of emitting against a partial set.2.
Store/Plugincompiles. Measured onmemex.meshweaver.cloudtoday (read-only, MCP):Store/Pluginis at v13940, stamped 2026-09-16T00:14:43Z, and is not in the error set — the whole error census on that mesh is 7 NodeTypes, allHosting/*(#4314). All 17Store/*types are present and clean.3. The reopen is the #4577 wave, and there has been no occurrence in 31 days. Reopened
2026-09-17T07:51:52Zunder "this fault is happening again", with its ownLast seen:reading 2026-08-17 06:42:10Z. The issue itself documents 1 occurrence on 1 pod with no cascade; the reopen rule is being fixed in Systemorph/MeshWeaver.Plugins#2014.A transient that has not recurred in a month, against a merged fix and a type that compiles, is closed. A future reopen needs an occurrence newer than this close.