Broken NodeType BinaryClickerV2/BinaryToggle never bakes, failing nodetype_bake readiness and stalling portal rollouts

What is failing

The NodeType BinaryClickerV2/BinaryToggle has no working baked build — its compile never completes and no CLR type is registered for it in the process (GetDiagnostics reports the assembly as unresolved, "its content stays an untyped element"). The nodetype_bake health check consequently reports Unhealthy and refuses pod readiness, so the rollout of memex-portal-deployment stalls with the previous image still serving.

Probable cause

High confidence the concrete trigger is that BinaryClickerV2/BinaryToggle fails to bake: the sample log explicitly says it "failed with no working build to regress from (already broken, or the first bake of this instance)", and live diagnostics show its compile stuck in progress (>100s elapsed, unresolved type) rather than completing or surfacing an error. Two structural concerns the fix should address:

The compile appears to hang rather than fail fast — no compile error is surfaced anywhere, which is why the node sits in limbo.
A single broken content-level NodeType blocks readiness for the whole portal deployment — the readiness check has no scoping or bypass, so one bad node wedges every rollout.

Impact

Judged from the numbers: 6 occurrences on a single pod (memex-portal-deployment-7596dd76bb-cbbj8) over ~50 seconds — this is a stalled rollout, not a serving outage (the previous image keeps serving traffic). However, until the NodeType is fixed or removed, no portal rollout can complete, which blocks any deploy. Workaround exists: remove or fix BinaryClickerV2/BinaryToggle (or bypass the check), hence sev:H.

Where to look

The broken node: BinaryClickerV2/BinaryToggle with its sources BinaryClickerV2/Source/BinaryToggleLayout and BinaryClickerV2/Source/BinaryToggleData (all created 2026-07-01, NodeType at version 245 but never resolving a CLR type).
The bake/readiness path: the nodetype_bake health check in the portal deployment (category Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService) — specifically why a compile of this NodeType neither completes nor errors, and why a single broken NodeType fails deployment-wide readiness.

Evidence


Fingerprint	`5d2390f56f66444b`
Category	`Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService`
Severity	Error
Namespace	`memex`
Pods	`memex-portal-deployment-7596dd76bb-cbbj8`
Occurrences	6
First seen	2026-09-23 16:06:54Z
Last seen	2026-09-23 16:07:44Z
Routing	not determined — no configured route matches the category `Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-23 16:07:04Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0013ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'
2026-09-23 16:07:14Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0015ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'
2026-09-23 16:07:24Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0019ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'
2026-09-23 16:07:34Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0019ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'
2026-09-23 16:07:44Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0012ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'

_{Re-addressed by the current identity function: this incident inherited 0a24845deb486a56 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/5d2390f56f66444b. Recurrences are folded into this issue rather than opening new ones.}

Still occurring: 7 more since the last update (13 total).

Last seen: 2026-09-23 16:08:44Z
Pods: memex-portal-deployment-7596dd76bb-cbbj8

Recent log lines

2026-09-23 16:08:34Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0026ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'
2026-09-23 16:08:37Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0014ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'
2026-09-23 16:08:44Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0017ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'

_{Re-addressed by the current identity function: this incident inherited 0a24845deb486a56 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Diagnosis 2026-09-23 ~16:40–18:15Z: the blocking type is `ContainerRegistry/Repository`, not `BinaryClickerV2/BinaryToggle` — a leftover install of a package retired upstream

Read-only, through the control instance's own instruments (Ops/Status/memex, three Logs actions, public /health). No roll, recycle or record write.

1. The title names the wrong type

The health line parses as N regressed: <keys> (<addenda>) (NodeTypeBakeGate.RegressedDetail). The regression is ContainerRegistry/Repository. BinaryClickerV2/BinaryToggle is in the "failed with no working build to regress from" addendum, which is non-blocking by design (withoutBaseline, WasHealthy=false). BinaryToggle is already tracked as #3883 (CS1929; the owner act there is a grant on binaryclickerv2 or deleting the demo partition). It does not hold any roll.

2. Why `ContainerRegistry/Repository` regressed on `3.0.0-ci.9260`

The pod log line, landed by Ops/Actions/logs-memex-20260923-containerregistry-compile-3 as Ops/Logs/memex-1790179612499805715-dd76bb-cbbj8 (16:06:52Z, pod memex-portal-deployment-7596dd76bb-cbbj8):

DynamicTypePreWarmer: ContainerRegistry/Repository → CompileError (Untracked module content: … partition 'ContainerRegistry' tracks no source on this mesh (no configured sync source), and no prebuilt bundle for framework s9e58afe…/linux-x64 names this type. This mesh does not compile a module's content it does not track … (MeshWeaver#3583).)

The ContainerRegistry package was retired in MeshWeaver.Plugins on 2026-09-08 (8543acb96, "retire the blob-backed ContainerRegistry module"; WhatsNew/2026-09-08-the-portal-no-longer-serves-an-empty-container-registry.md says "Portals drop the retired module on their next roll").
They did not drop it. Plugins/ContainerRegistry (a Package record, installedAtUtc 2026-09-07T23:00Z, autoUpdate: true, 32 installed nodes including ContainerRegistry/Repository.json and its sources) is still on the control instance, and the self-updater still counts it: [RollSelect] memex: 77 plugin(s) required … 1 package(s) would recompile at boot: ContainerRegistry (Ops/Logs/memex-1790181535124697767-dcd48f-sz5st).
No bake will ever name the type again, and Bundles are baked for ONE framework identity per run, so a portal's GitSynced sources stay days behind main (or its types go unadopted) — options 1-4 undecided #3583 refuses to compile an untracked partition. So every new framework identity turns the type from Ok on the old identity into CompileError, which the gate counts as a regression.

3. The live state

Ops/Status/memex at 16:40Z showed runningImage 3.0.0-ci.9260, generations: 2, converged: false. The new pod …7596dd76bb-cbbj8 (core 9a61fed9, started 15:45:33Z) answers /health 503. Two 3.0.0-ci.9218 pods serve. At 18:14Z /api/version still answers 39f49494 (9218), and the served replicas booted at 05:24:51Z and 16:27:24Z. So the roll has been stalled for about 2.5 h. The startup probe (failureThreshold 1080 × 10 s = 3 h) will restart cbbj8 around 18:45Z. After that restart it should pass, because the shared record now reads Error and the type counts as previouslybroken (the 16:27Z replica's bake-report already prints previouslybroken in ContainerRegistry/…). That is the gate working as designed on a zombie package, not a gate defect.

4. Classification: (c) owner act, plus one platform contradiction filed into triage

Owner act (the fix for this incident): uninstall the retired ContainerRegistry package on the control instance, and on memex.meshweaver.cloud if it is installed there too. That means removing Plugins/ContainerRegistry and the ContainerRegistry partition content through the platform's package-removal flow. It is a destructive write on a production partition that I (an agent, and unable to read that partition under RLS) did not take. Verification: the next [RollSelect] line no longer lists ContainerRegistry under "would recompile at boot", and the next framework-changing roll converges without a nodetype_bake Unhealthy line.

Platform contradiction, filed into bug triage rather than fixed here: ReleaseAvailability still says a missing content bake "would recompile at boot" (a cost, #3651), but #3583 made the boot refuse that compile for an untracked partition. The only exception it knows about is Modules:RequirePrebuilt. So RollSelect predicts a boot compile that the compile watcher then refuses, and the readiness gate stalls the roll. Turning that case into a hold is a change to the release-gate policy (it would also freeze CD on a zombie package until someone uninstalls it), so I did not make it unilaterally. Retired packages whose installs persist is a second triage item.

Not established

Whether memex.meshweaver.cloud still carries Plugins/ContainerRegistry (not read).
That cbbj8 actually passes after its probe restart. That is a prediction from the code (WasHealthy), not a measurement yet.
The ContainerRegistry/Repository node itself: RLS hides it from this credential (get → Not found, search namespace:ContainerRegistry lists only _Policy). Everything above comes from the pod's own log and the install record.

Refs #3883 (BinaryToggle), #3583, #3651.

Still occurring: 369 more since the last update (382 total).

Last seen: 2026-09-23 17:10:35Z
Pods: memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-56fbdcd48f-sz5st

Recent log lines

2026-09-23 17:10:24Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0022ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'
2026-09-23 17:10:34Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.1039ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'
2026-09-23 17:10:35Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0021ms with message 'NodeType bake regressed on this image — refusing readiness so the rollout stalls with the previous image still serving. 1 NodeType(s) regressed on this image: ContainerRegistry/Repository (1 failed with no working build to regress from (already broken, or the first bake of this instance) — BinaryClickerV2/BinaryToggle)'

_{Re-addressed by the current identity function: this incident inherited 0a24845deb486a56 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

2026-09-24 ~02:00Z: `ContainerRegistry/Repository` no longer blocks the roll. The roll is still stalled, now on something else

Read-only, on the control instance (Ops/Status/memex v2962 sampled 01:57:32Z, Admin/_LogIncident/*, public /health).

1. The retired package stopped gating, the way the gate is designed to

The 9218 replica 56fbdcd48f-sz5st (booted 16:27:24Z) prints this bake plan: previouslybroken in ContainerRegistry/…, Crm/…. The shared record for ContainerRegistry/Repository now reads Error. So a new pod treats the type as already broken (NodeTypeBakeEntry.WasHealthy == false), files it under without baseline, and does not gate on it. This is what the 09-23 18:14Z comment predicted.
This issue's fingerprint (5d2390f56f66444b) was last seen at 2026-09-23 17:10:35Z. It has not been logged since.

2. What holds the roll now: the 9260 pod never gets past enumeration

Ops/Status/memex: runningImage 3.0.0-ci.9260, generations: 2, converged: false. Two 9218 replicas are serving (4dpcn, sz5st). 7596dd76bb-cbbj8 (9260, commit 9a61fed9) has 4 restarts and still answers /health 503.

Its readiness line now comes from a different fingerprint, 4fb23aa33d4c054a, whose status is Suppressed. Every one of its 10 retained samples (01:59:04Z → 02:00:44Z) is from cbbj8:

Health check nodetype_bake with status Unhealthy … with message 'NodeType bake in progress — enumerating dynamic NodeTypes'

That incident was suppressed as expected startup noise ("resolving as soon as the bake completes"). On cbbj8 the bake does not complete. The pod has been out of rotation since 15:45Z, across 4 restarts. Meanwhile its LIVE RECORD CENSUS on the serving replicas shows 93 NodeType records re-keyed to s9e58afe (the 9260 framework) since they booted. So the stuck pod is still stamping shared records while it never becomes Ready. The serving replicas then have no usable assembly for those types.

Not established: how long a single boot of cbbj8 stays in enumerating, and whether the enumeration is stuck or just very slow. Ops/Status gives restart counts, not restart times. Pinning it down needs a Logs action on cbbj8, which is a control-instance write I did not take. Next step (owner): { "requestedAction": "Logs", "query": "{namespace=\"memex\", pod=\"memex-portal-deployment-7596dd76bb-cbbj8\"} |~ \"DynamicTypePreWarmer|enumerat\"", "sinceMinutes": 240 }.

3. The retired package is still installed on BOTH portals. Owner act, unchanged

Plugins/ContainerRegistry on memex.systemorph.com: v4, installedAtUtc 2026-09-07T23:00:00Z, autoUpdate: true, 32 nodes.
memex.meshweaver.cloud has the same record: v5, installedAtUtc 2026-09-07T22:24:41Z.
The package was deleted from MeshWeaver.Plugins in 8543acb96, so no source lists it any more.

Owner act: on both portals, delete the ContainerRegistry partition's content (the ContainerRegistry/Repository NodeType and its Source/Test Code nodes). Then remove the orphaned record through Catalog → orphaned install records. Order matters. PackageInstaller.RemoveInstalledRecord removes only the record, and the content partition is "a separate lifecycle". So removing the record alone leaves the type in every bake sweep.

Deleting the definition node is also what the gate already handles live. A regression whose node is pruned is withdrawn by the pre-warmer's recovery watch (TypeNodeExists → RetireRegression) with no restart. That is ARetiredNodeTypeIsNotARegressionTest.

4. Should the platform file an installed-but-retired package under `Retired` by itself? Not done here. Reason:

I looked for a clean core change and did not make one:

The gate already has the right buckets (Retired/Removed do not gate) and the right live withdrawal (definition node pruned). It also self-heals one level up: after the first stalled boot, the record reads Error and later pods do not gate (§1). So the cost is one stalled roll per retired package, not a permanent freeze.
"No configured source lists this package" is not recorded anywhere durable. PackageUpdateReconciler walks what sources serve, not what is installed. InstanceAutoRegistrationService.ReassertInstalled sees an orphan only for records that need repair, and only logs it. CatalogLayoutAreas.Orphaned computes it per page render. Classifying types as Retired from that signal means minting a new durable fact from a registry listing. A listing that answers successfully but omits a live package would then silence a real regression. That is a gate-policy decision, not a bug fix.
The contradiction Broken NodeType BinaryClickerV2/BinaryToggle never bakes, failing nodetype_bake readiness and stalling portal rollouts #5544's 09-23 comment named (ReleaseAvailability predicts "would recompile at boot" while Bundles are baked for ONE framework identity per run, so a portal's GitSynced sources stay days behind main (or its types go unadopted) — options 1-4 undecided #3583 refuses that compile for an untracked partition) is the same policy question one step earlier. Refs Bundles are baked for ONE framework identity per run, so a portal's GitSynced sources stay days behind main (or its types go unadopted) — options 1-4 undecided #3583, A platform roll is gated on measured loadability on the target, never on floors; a missing bake is a boot compile, not a hold #3651.

Refs #3883 (the BinaryClickerV2/BinaryToggle addendum in this issue's title).

2026-09-24: ContainerRegistry cleanup on memex.meshweaver.cloud not performed. The content is invisible to the operating credential

Read-only, through the memex MCP (memex.meshweaver.cloud):

get @Plugins/ContainerRegistry: present. Package v5, installedAtUtc 2026-09-07T22:24:41Z, autoUpdate: true, installedNodeCount: 32. installedFiles includes ContainerRegistry/Repository.json plus Repository/Source/ImageRepository{Areas,Content}.cs and Repository/Test/ImageRepositoryTests{,Area}.cs.
get @ContainerRegistry: present. Store/Plugin v4 (the plugin landing page).
get @ContainerRegistry/Repository returned Not found. search namespace:ContainerRegistry scope:descendants returned only ContainerRegistry/_Policy (coverage containerregistry). So the NodeType and its Source/Test nodes are RLS-hidden from this credential, the same way they were on the control instance.

Why I stopped: the required order is to delete the partition content first and the install record second. The content cannot be read, so it cannot be snapshotted or diffed. Deleting the readable root ContainerRegistry would recursively remove descendants I cannot see, and that would be a blind delete. Removing only Plugins/ContainerRegistry would leave the type in every bake. So neither step was taken, and nothing was written.

Owner steps (both portals): under an identity that can read the ContainerRegistry partition:

Delete ContainerRegistry/Repository with its Source/Test children, and the rest of the partition content.
Remove the orphaned Plugins/ContainerRegistry record through Catalog → orphaned install records.

2026-09-24: what the 9260 pod was actually doing, and the root fix (#5643, merged as `8a70b0fb`)

I read the pod's own lines through Logs actions on memex.meshweaver.cloud. The fleet watch there also observes memex. Nothing was written on the control instance.

1. cbbj8 is not stuck in "enumerating". That label was just never updated

Ops/logs-memex-20260924-cbbj8-bake-stall covers the boot at 03:55Z on memex-portal-deployment-7596dd76bb-cbbj8. Entries are under Ops/Logs/memex-179022…-dd76bb-cbbj8.

03:55:46 to 03:55:53: 168 × Prebuilt assembly ADOPTED for … (framework s9e58a…), then ShippedPrebuiltBundles: 168 prebuilt assembly(ies) from 65 shipped bundle(s).
03:55:56: BuildProtocol: claim granted.
03:56:00: 107 of 232 dynamic NodeType(s) need building (… batch direct-compile, perTypeBudget=00:05:00).
03:56:01: batched source discovery did not establish the source sets — … falling back to the activation-driven sweep for ALL 107 pending type(s).
From 04:01 on: → TimedOut — 300.0 s for every type, one after another. That is binary-clicker-space, BinaryClicker, BinaryClickerV2, then Crm/, DeepSign, Edu/, Hosting/, LinkedIn/, RFPProcessor/, and it was still going through rsalzmann/ at 06:40Z.

107 × 300 s is about 9 h. That outlasts the 3 h startup probe, so the pod is killed and boots into the same sweep again. The gate's detail was set once, before the seeding, and held for the whole sweep. That is why readiness kept saying enumerating dynamic NodeTypes.

2. Root causes, both deterministic

(a) The activation path deadlocks against its own admission. DynamicTypePreWarmer.WarmOne waited for the shared record to show a usable build. On a gated pod, MeshPublicationGate holds that compile stamp until the bake passes (#3478). The bake cannot pass until this wait answers. #3478 had already given the recovery watch the process-local witness (LocalNodeTypeBuilds); the sweep's own per-type wait never got it.

(b) An unready replica re-keyed shared records through prebuilt ADOPTION. #3478 gated the two compile stamps. PrebuiltAssemblySeeder's adoption stamp writes the same CompiledFrameworkVersion and assembly coordinates, and it was never offered to the gate. That is where the "93 re-keyed" come from. At 06:36Z, after their own 06:28Z restart, the serving 9218 replicas' census already read 21 NodeType record(s) were RE-KEYED … AFTER it booted, all s9e58afe, in LinkedIn/Marketing/SocialMedia/Essentials/Edu/Feedback. This is the inverse case of #5353: that change stops a LEAVING process re-keying backwards; this one stops a not-yet-JOINED process re-keying forwards.

3. What #5643 changes

The adoption stamp now goes through the new MeshPublicationGate.Offer.
- Held while Provisional. The adoption still counts for the pod itself: the bytes are on the store and the definition is in NodeTypeAdoptionRegistry, so the sweep still classifies those types as baked.
- Released on admission.
- Discarded when Refused.
WarmOne also completes on LocalNodeTypeBuilds, and reads content with ContentAs instead of a cast.
NodeTypeBakeGateState.MarkProgress: readiness now names the step, the type and the bound. Example: building 17 of 107 pending NodeType(s) by activation (the batch was not available): waiting on Crm/Country, for up to 00:05:00. The one unbounded wait, following another replica's GO, now says so.
Tests on a real monolith mesh, with negative controls run against the unfixed source:
- AGatedSweepHearsItsOwnCompileTest: before the fix, TimedOut at exactly the 60 s budget. After the fix, Compiled in seconds, while the shared record provably does not carry the build.
- AnUnadmittedProcessDoesNotReKeyByAdoptionTest: before the fix, the stamp was written immediately while Provisional, and a Refused process still adopted. After the fix, provisional adoptions are held then released, and refused ones are discarded.

4. memex-cloud does the same thing, but the numbers hide it

Ops/logs-memexcloud-20260924-bake-phases covers 24 h of bake phase lines on every memex-cloud pod. On almost every boot the pattern is 2 of 371 … need building, then batched source discovery did not establish the source sets, then warm-up complete in 00:04:46 up to 00:13:41 — compiled=0. So memex-cloud also falls back to the activation path and also waits out the per-type budget. It only has 2 pending types, so it finishes in about 5 to 9 minutes instead of about 9 h. The line that comes right before the stall is the same on both portals: the batched-discovery fallback.

Not established

Why batched source discovery fails, on cbbj8 and on nearly every memex-cloud boot. The exception line at 03:56:01Z was not retrievable: Loki reads newest-first and my windows were cut. The pod logs many System.TimeoutExceptions. This is the next root to chase. With fix(bake): an unready replica neither re-keys shared NodeType records by adoption nor deadlocks its own activation sweep #5643 the fallback no longer deadlocks, but it still costs one activation per type.
On the activation path, a type whose grain lives on another silo, or that fails its compile locally, still times out as unevaluated. Only local successes are published locally.
That the new image actually converges on memex. That needs a build containing 8a70b0fb.

Owner actions (not taken)

After CD produces an image containing 8a70b0fb, roll memex to it. cbbj8's current image (9260) cannot finish its sweep, and restarting it only repeats the same boot.
To verify: the new pod's readiness should name steps instead of enumerating, and it should reach Ready. The serving replicas' /health LIVE RECORD CENSUS should print "NONE was re-keyed since boot" while the new pod is still unready.
The ContainerRegistry zombie package from the earlier comments is unchanged and still an owner act.

The batched-discovery half of this stall is fixed in #5655 (open), filed as #5654.

The same boots that fell back to per-type activation all show one pattern. Evidence is from memex-cloud on core bf5ac85526 (which includes #5615), read with Logs actions on memex.meshweaver.cloud:

On all 8 boots between 07:12Z and 08:06Z, nodeType:Code partitions:all settled at 1586 nodes. About 25 ms later each boot logged batched source discovery did not establish the source sets … for ALL N pending type(s), with N between 38 and 80. So the line still occurs on bf5ac85526, and fix(query): a live query re-reads once per burst of changes, not once per change #5615's query-stall fix does not touch it.
The exception behind it is not a timeout and not the 15 s fan-in bound. It is Npgsql.PostgresException: 42703: column n.created_by does not exist, logged 13 µs after the abandonment on …6848f8-gf5df (entry Ops/Logs/memex-cloud-1790235255089822272-6848f8-gf5df). The namespace:*/Source fetch is one UNION over every schema's code table, and at least one of those tables lacks the V57 columns. The database side is filed as Systemorph/MeshWeaver.Plugins#2350.
The many System.TimeoutException lines in the same boots are EnrichWithNodeType probe … faulted and … recompile did not settle. None of them falls between the first fetch settling and the abandonment. Whether the activation sweep that follows causes them was NOT established.

#5655 makes discovery degrade per type: when the global pass fails, each type is re-resolved from its own anchored queries, and only the types whose own queries fail go to activation. BinaryClickerV2/BinaryToggle's own compile failure (the subject of this issue) is not addressed by it.

Not established: whether memex (the control instance) hits the same 42703. The systemorph MCP is disconnected, so I read no memex log.

Triage 2026-09-25 ~15:30Z: fixed on `main`, not deployed, plus one owner act. Leaving open.

Classification. The title's subject is not a defect. BinaryClickerV2/BinaryToggle sits in the non-blocking "no working build to regress from" addendum, and its own compile failure is #3883. What stalled the roll were the two platform roots found on 09-24, and both are merged:

root	fix	merged	on `main`
a gated sweep waited on the shared record, which the publication gate holds (activation-path deadlock)	#5643 `8a70b0fba9`	2026-09-24 08:21Z	yes
batched discovery abandoned for ALL pending types when one partition cannot answer (`42703`)	#5655 `777e84819a`	2026-09-24 09:50Z	yes

Neither is deployed. /api/version, 4 of 4 probes per host at 15:22Z:

memex.systemorph.com → 39f49494eb (fix(compiler): bound compile CPU on a dedicated-thread IIoPool lane #5375 merge, 2026-09-22 21:13Z)
memex.meshweaver.cloud → 140c5fb7ae (fix(i18n): a refused node-bound field write has a localized sentence (#5600) #5639 merge, 2026-09-24 07:08Z)

Both predate 8a70b0fba9. The control instance is itself stuck: Ops/Status/memex (sampled 15:23Z) has two ci.9218 replicas serving, and ci.9332 pod 9b8b44c58-48r6d has 91 restarts, /health connection refused, converged: false. That is the DbVersionGate V57 < V58 wall (#4764). No image carrying these fixes can start there until the V58 migration runs.

Production actions still owed (none taken here):

Run the V58 migration on memex, then on memex-cloud (Self-update cannot cross a schema bump: the updater patches the portal image only, the migration is a helm-run Job, and the new pod crash-loops on DbVersionGate (measured memex-cloud 2026-09-19) #4764). After that a roll can land a build that contains 8a70b0fba9 and 777e84819a.
Uninstall the retired ContainerRegistry package on both portals, in the order given in the 09-24 06:41Z comment: partition content first, then the orphaned Plugins/ContainerRegistry record. The 9218 replicas' bake-report still prints previouslybroken in ContainerRegistry/… today.
Plugins#2350, the code table missing the V57 columns, is the database half of fix(bake): one partition that cannot answer costs the batched discovery its own types, not all of them #5655.

Closes on: a roll of memex onto a build that contains both fixes. The new pod's readiness must name its steps rather than enumerating, it must reach Ready, and the serving replicas' LIVE RECORD CENSUS must print no re-keying while it is unready.

Two maintainer decisions (2026-09-26), recorded

Both are rows in the policy register (PolicyNotProse.md, added by #5749).

1. `bake-gate-readiness-only` — the bake gate holds READINESS only and never fails the STARTUP probe

Root cause (#5746, TheBakeGateOnlyStallsARoll.md):

The nodetype_bake verdict made /health red.
The startup probe (10 s × 1080 on memex) killed every container at the three-hour mark.
A restarted pod of the previous image could not pass that probe either.
The control instance was down from 20:54Z to 04:07Z.

Rule: the startup probe proves only that the process booted. The gate's verdict is read by /ready alone. A refusal therefore stalls a roll: the pod stays alive and out of the Service, nothing is killed, and a restarted previous-image pod keeps serving.

Implementation: #5749.

ProbeEndpoints.RollGateTag marks a check as a roll gate. ServiceDefaults.TagRollGates tags nodetype_bake that way whatever the host registered.
/ready reads the roll gates.
/health still prints the gate but excludes it from its status.
The chart's progressDeadlineSeconds carries the bake's time.
Invariant 10b and RollGateReadinessOnlyTest guard all of this.

Startup budget: Systemorph/Memex#547 lowers memex's failureThreshold from 1080 to 60.

Re-arming the gate: only once an image carrying #5725 and #5749 has rolled. That is stage 2 of Memex#523 (memex) and Memex#521 (memex-cloud).

2. `silos-pool-autoscaler-max` — the `silos` node pool's autoscaler max goes up by one (4 → 5)

Why: memex's rollout pod sat Pending ~2 h because silos was at its autoscaler max while memex-cloud's KEDA scale-out holds up to 6 × 4 CPU.

How:

The change goes through the governed InfraDeploy action over Memex infra/estate.bicep (Systemorph/Memex#546): what-if first, then an approved deploy.
It is never done with az aks nodepool update.
No existing action kind covers autoscaler bounds; UpgradeCluster only changes versions.

Owed before the deploy: hosting-operator needs the agentPools write grant, which is the same Azure Kubernetes Service Contributor Role that UpgradeCluster already owes.

This is a record of the decisions. It does not close this issue.

Production verification 2026-09-26 16:38Z: resolved

The current memex roll is converged at 3/3 Ready, all on 3.0.0-ci.9408 / core 45e7f94e4ec36417ebcbda6125f7061c0a803f10.

Both root fixes are ancestors of that served commit:

gated-sweep local-build witness, fix(bake): an unready replica neither re-keys shared NodeType records by adoption nor deadlocks its own activation sweep #5643 8a70b0fba9;
per-type source-discovery fallback, fix(bake): one partition that cannot answer costs the batched discovery its own types, not all of them #5655 777e84819a.

Each of the three ci.9408 replicas reports nodetype_bake: Healthy and completes in 7–9 minutes. Their outcome census is complete: 234/234 enumerated types produced a verdict and 0 reached no verdict/timed out. The new pods are Ready; none is held at the old “enumerating dynamic NodeTypes” state. Each replica's LIVE RECORD CENSUS reports that every typed record targets its own c003e001 framework (0 records unable to be typed), so the former unready-pod re-keying regression is absent.

BinaryClickerV2/BinaryToggle remains explicitly categorized as an already-broken, non-gating type (tracked separately by #3883); it did not block this roll. This issue's stalled-roll root is fixed and verified in production.

fix(prewarm): an adoption the compile watcher makes on a provisional …

fix(compile): the boot batch compile asks the untracked-module gate f…

Navigation Menu

Repository navigation

Broken NodeType BinaryClickerV2/BinaryToggle never bakes, failing nodetype_bake readiness and stalling portal rollouts #5544

Evidence

Diagnosis 2026-09-23 ~16:40–18:15Z: the blocking type is `ContainerRegistry/Repository`, not `BinaryClickerV2/BinaryToggle` — a leftover install of a package retired upstream

1. The title names the wrong type

2. Why `ContainerRegistry/Repository` regressed on `3.0.0-ci.9260`

3. The live state

4. Classification: (c) owner act, plus one platform contradiction filed into triage

Not established

2026-09-24 ~02:00Z: `ContainerRegistry/Repository` no longer blocks the roll. The roll is still stalled, now on something else

1. The retired package stopped gating, the way the gate is designed to

2. What holds the roll now: the 9260 pod never gets past enumeration

3. The retired package is still installed on BOTH portals. Owner act, unchanged

4. Should the platform file an installed-but-retired package under `Retired` by itself? Not done here. Reason:

2026-09-24: ContainerRegistry cleanup on memex.meshweaver.cloud not performed. The content is invisible to the operating credential

2026-09-24: what the 9260 pod was actually doing, and the root fix (#5643, merged as `8a70b0fb`)

1. cbbj8 is not stuck in "enumerating". That label was just never updated

2. Root causes, both deterministic

3. What #5643 changes

4. memex-cloud does the same thing, but the numbers hide it

Not established

Owner actions (not taken)

Triage 2026-09-25 ~15:30Z: fixed on `main`, not deployed, plus one owner act. Leaving open.

Two maintainer decisions (2026-09-26), recorded

1. `bake-gate-readiness-only` — the bake gate holds READINESS only and never fails the STARTUP probe

2. `silos-pool-autoscaler-max` — the `silos` node pool's autoscaler max goes up by one (4 → 5)

Production verification 2026-09-26 16:38Z: resolved

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Broken NodeType BinaryClickerV2/BinaryToggle never bakes, failing nodetype_bake readiness and stalling portal rollouts #5544

Description

Evidence

Activity

systemorph-com commented on Sep 23, 2026

rbuergi commented on Sep 23, 2026

Diagnosis 2026-09-23 ~16:40–18:15Z: the blocking type is ContainerRegistry/Repository, not BinaryClickerV2/BinaryToggle — a leftover install of a package retired upstream

1. The title names the wrong type

2. Why ContainerRegistry/Repository regressed on 3.0.0-ci.9260

3. The live state

4. Classification: (c) owner act, plus one platform contradiction filed into triage

Not established

systemorph-com commented on Sep 23, 2026

rbuergi commented on Sep 24, 2026

2026-09-24 ~02:00Z: ContainerRegistry/Repository no longer blocks the roll. The roll is still stalled, now on something else

1. The retired package stopped gating, the way the gate is designed to

2. What holds the roll now: the 9260 pod never gets past enumeration

3. The retired package is still installed on BOTH portals. Owner act, unchanged

4. Should the platform file an installed-but-retired package under Retired by itself? Not done here. Reason:

rbuergi commented on Sep 24, 2026

2026-09-24: ContainerRegistry cleanup on memex.meshweaver.cloud not performed. The content is invisible to the operating credential

rbuergi commented on Sep 24, 2026

2026-09-24: what the 9260 pod was actually doing, and the root fix (#5643, merged as 8a70b0fb)

1. cbbj8 is not stuck in "enumerating". That label was just never updated

2. Root causes, both deterministic

3. What #5643 changes

4. memex-cloud does the same thing, but the numbers hide it

Not established

Owner actions (not taken)

rbuergi commented on Sep 24, 2026

rbuergi commented on Sep 25, 2026

Triage 2026-09-25 ~15:30Z: fixed on main, not deployed, plus one owner act. Leaving open.

rbuergi commented on Sep 26, 2026

Two maintainer decisions (2026-09-26), recorded

1. bake-gate-readiness-only — the bake gate holds READINESS only and never fails the STARTUP probe

2. silos-pool-autoscaler-max — the silos node pool's autoscaler max goes up by one (4 → 5)

rbuergi commented on Sep 26, 2026

Production verification 2026-09-26 16:38Z: resolved

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Diagnosis 2026-09-23 ~16:40–18:15Z: the blocking type is `ContainerRegistry/Repository`, not `BinaryClickerV2/BinaryToggle` — a leftover install of a package retired upstream

2. Why `ContainerRegistry/Repository` regressed on `3.0.0-ci.9260`

2026-09-24 ~02:00Z: `ContainerRegistry/Repository` no longer blocks the roll. The roll is still stalled, now on something else

4. Should the platform file an installed-but-retired package under `Retired` by itself? Not done here. Reason:

2026-09-24: what the 9260 pod was actually doing, and the root fix (#5643, merged as `8a70b0fb`)

Triage 2026-09-25 ~15:30Z: fixed on `main`, not deployed, plus one owner act. Leaving open.

1. `bake-gate-readiness-only` — the bake gate holds READINESS only and never fails the STARTUP probe

2. `silos-pool-autoscaler-max` — the `silos` node pool's autoscaler max goes up by one (4 → 5)