fix(chart): more than one portal replica on a pod-local /data is refused at render (#6052) by rbuergi · Pull Request #6197 · Systemorph/MeshWeaver

rbuergi · 2026-10-06T07:18:12Z

Refs #6052. This PR is the platform half of ask 1. It does not close the issue; see below for what is left.

Root cause (mechanism traced in code)

On the estate in #6052, each pod's FileSystemAssemblyStore (collection local) sat under its own /data. The NodeType records that point into it are shared in the database.

A compile writes the shared record pointing at pod-local bytes. The compile runs on the pod that hosts the NodeType's own hub (NodeTypeCompilationHelpers watcher → DispatchCompileTrigger). It uploads to that pod's store (NodeTypeCompilationHelpers.cs ~3312) and writes LatestAssemblyCollection = "local" into the shared record (~3907).
Another pod misses and cannot recover. On a miss, pod B's instance enrichment (NodeTypeEnrichmentHelpers.cs ~1512–1560) calls TriggerRecompileAndRetry. That flips the type to Pending through the owner hub, so the recompile runs on pod A again and the bytes land on pod A again. After MaxRecompileAttempts pod B shows the AssemblyUnavailable overlay — the page from the issue.
Nothing fetches bytes across pods. No path fetches another pod's bytes on a miss. ResolveAssembly asks only the local store, and PluginBundleClient runs only at install and reconcile.
The chart allowed the shape. persistence.data.claimName defaults to "", which renders emptyDir. The chart only warns in comments, and chart-gate's invariant 3 checks only the values files tracked in this repository, not a client's.

Our AKS instances avoid all of this: /data is a ReadWriteMany Azure Files claim (values.aks.yaml, and the client-b record in Systemorph/Memex has memex-data).

Fix

deploy/helm/templates/memex-portal/deployment.yaml now fails the render under three conditions together:

keda.enabled is on, or replicas.portal is greater than 1;
persistence.data.claimName is not set;
persistDataVolume is off. That setting is a single-node hostPath, so it counts as shared.

The message names the missing input and the issue. This is the chart's established refusal pattern (#3780, the fabrikam refusals).

Tests

check-chart-invariants.sh gets a new refusal control. Its fixture is the valid two-replica shape (values.two-replicas-no-keda.yaml) minus the data claim, so the pod-local /data is the only thing the render can refuse. The run reports "All 17 values combinations render a self-consistent deployment, and all 7 refusal controls hold."
Negative control (run): with the fail neutralised, the gate goes red: "only 6 of 7 refusal controls held".
check-values-are-read.sh and test-chart-drift-render.sh stay green.

What this does NOT do / not established

The estate's own values were not read. That /data was pod-local there is inferred from collection=local bytes being missing on the other pod. The estate's values live in its own repository.
An existing install in this shape will have its next helm upgrade refused until it sets persistence.data.claimName. That is intended: the shape cannot serve compiled NodeTypes across pods. Every tracked fleet record I could read (memex, memex-cloud, client-b) has a data claim.
Asks 2 and 3 of A pod compiles module NodeTypes locally and the shared record points at bytes only it holds — refuse local compile, re-fetch the shipped bundle, let the bundle identity follow the running image #6052 are left as they are.
- Ask 2: re-fetch shipped bundle bytes when a record's bytes are missing on a pod. There is no such path today.
- Ask 3: bundles.identity is static per helm release. bundles.identityFile is the existing dynamic alternative.
Not built: making Modules:RequirePrebuilt the default for client instances (ask 1, "refuse local compile"). That is a policy decision.

🤖 Generated with Claude Code

…sed at render (#6052) /data holds the NodeType assembly cache (FileSystemAssemblyStore, collection "local") and every compiled NodeType's SHARED record names a file in it. On per-pod emptyDir a type compiled on pod A leaves a record naming bytes pod B never holds; B's instances render "its compiled assembly could not be loaded on this node", and a recompile does not help because it runs where the NodeType's own hub lives. The client estate in #6052 ran two replicas this way (/health bake-report bytesmissing=93 on both pods). chart-gate invariant 3 asserts RWX only over the values files tracked here; a client's are not. So deployment.yaml now fails the render when keda.enabled or replicas.portal > 1 and neither persistence.data.claimName nor persistDataVolume gives a shared /data. - check-chart-invariants.sh: refusal control with a fixture that is the valid two-replica shape minus the data claim. Neutralising the fail reds the gate (6 of 7 refusals held). - values.yaml: the replicas comment names the refusal. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions · 2026-10-06T07:21:52Z

Test Results

22 files ± 0 22 suites ±0 45m 19s ⏱️ +33s
10 828 tests +30 10 635 ✅ +30 193 💤 ±0 0 ❌ ±0
10 841 runs +30 10 648 ✅ +30 193 💤 ±0 0 ❌ ±0

Results for commit 2bd8706. ± Comparison against base commit 3a0c2e9.

This pull request removes 40 and adds 46 tests. Note that renamed tests count towards both.


   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "a5b600fc-ee38-4686-85be-25587b9cbf37")
Memex.Portal.Shared.Test.InstanceRebootFaultedRestartTest ‑ ARestartThatCrashed_IsFaulted_RetriedByThePass_AndCompletesOnTheNextAttempt
Memex.Portal.Shared.Test.ModuleReloadFaultedBesideActivationTest ‑ OneModulesCrashedAdopt_BesideAnotherModulesLiveActivation_IsFaulted
Memex.Portal.Shared.Test.ModuleReloadFaultedTest ‑ ADecidedFailure_IsFailed_AndIsNeverRetried
Memex.Portal.Shared.Test.ModuleReloadFaultedTest ‑ ATransientFault_IsFaulted_TheNextPassRetriesIt_AndItSucceeds
Memex.Portal.Shared.Test.ModuleReloadRestartAttemptTest ‑ AFailedRestartAttempt_IsFaulted_AndTheRetryAsksAgain
Memex.Portal.Shared.Test.ModuleReloadRestartAttemptTest ‑ AnInstallThatCannotRestart_IsFailed_AndNeverRetried
Memex.Portal.Shared.Test.ModuleReloadRulesTest ‑ ACrashedItem_IsFaulted_ADecidedFailure_IsFailed
Memex.Portal.Shared.Test.ModuleReloadRulesTest ‑ AFaultedRequest_IsDue_OnThePassCadence_DoubledPerAttempt
Memex.Portal.Shared.Test.ModuleReloadRulesTest ‑ Rearm_StartsTheNextAttempt_FromRequested_KeepingTheLog
…