A 13-minute-old Hosting install records FleetWatchCadence.cs and the NODE is absent — three Hosting NodeTypes uncompilable, and memex's roll to the 3.0.0 candidate cannot converge

What is failing

memex's roll onto the 3.0.0 release candidate cannot converge, and the cause is an install that recorded a source file it did not land.

Hosting/Issue/Source/FleetWatchCadence is absent from the mesh while Plugins/Hosting's install record claims it, so every NodeType declaring it as a shared source fails to compile.

Measured 2026-09-17, control instance (memex.systemorph.com)

The node is absent, and the absence is readable:

get @Hosting/Issue/Source/FleetWatchCadence
  -> Not found: Hosting/Issue/Source/FleetWatchCadence
search 'name:*FleetWatchCadence* partitions:all'
  -> count: 0, truncated: false, coverage.partitions = 102 partitions, `hosting` among them

The install record claims it — Plugins/Hosting, version 42, package version 1.22.0, installedFromRef: HEAD, installedAtUtc: 2026-09-17T10:47:22Z, installedNodeCount: 222:

installedFiles["Hosting/Issue/Source/FleetWatchCadence.cs"] = 7e8e1d69f169117b39833c4a260334db4ff350996e5d54c47dfd5a6260441c74

So an install that ran 13 minutes before the read recorded the file, and the node is not in the store.

The file exists upstream: Hosting/Issue/Source/FleetWatchCadence.cs on MeshWeaver.Plugins main, added by 8fe9d70dfc at 2026-09-17T05:37:50Z (the #4471 fleet-watch work). Code search finds 23 hits for the symbol, including Hosting/InstanceAction.json, Hosting/DeploymentStatus/Source/DeploymentStatusLayoutAreas.cs and Hosting/FleetConsole/Source/FleetConsoleLogic.cs.

What the compile says, verbatim from a get on a node of the type:

MISSING SOURCES: 1 of 2 declared source queries for 'Hosting/DeploymentStatus' matched NO nodes on
this mesh ('shared=@Hosting/Issue/Source/FleetWatchCadence'). ...
CS0246 Error (line 436): The type or namespace name 'FleetWatchCadence' could not be found

and for Hosting/InstanceAction, 1 of 12 declared source queries with 8+ CS0103 FleetWatchCadence.

The deploy consequence

Deployments/memex-roll-8798-release-candidate-20260917 (created 07:33:54Z) set the image at 07:34:19Z, generation 770, and is still state: Running, phase: Verify one generation three and a half hours later. Ops/Status/memex, sampled 10:58:53Z:

generations: 2, converged: false
pod …-ddd9f5f75-tfxpq on 3.0.0-ci.8798, started 07:34:06Z, restarts: 1, no ready
two pods on 3.0.0-ci.8710, ready: true — still serving, so there is no user-visible outage
/api/version still answers afde4eab

🚨 Attribution caveat: the uncompilable types are a strong candidate for the refused readiness — a type that regressed on the new image is what nodetype_bake refuses on — but I did not read the new replica's own readiness reason. The fleet watch reports /health unreachable … TaskCanceledException for that pod, and for one of the two healthy ci.8710 pods as well, so that signal does not separate them. The three types are typed on the old pods (their content-types names only Hosting/ConfigAudit), so the compile failure is new on this image rather than pre-existing.

Why this is filed separately

This is the class already tracked by #2387 and #3659 — a package records a declared node as installed while the node is ABSENT — but every earlier instance argued from retained samples or a two-week-old record. This one has a minutes-old install record, the upstream file, the absent node and a blocked deploy in one reading, on the control instance itself. It is also the current blocker for tagging v3.0.0: no portal can move onto a candidate image while this holds.

Not done

No remediation was applied: the node was not hand-created, nothing was rolled back, and the stuck rollout was left in place (the old pods serve). The open question is why the install wrote 221 of 222 files — whether the node write was refused, lost, or never attempted — and InstalledPackageRepairService has not healed it across the 10:47Z install.

_{Read-only measurement 2026-09-17 by Claude Opus 5: get / search on the control instance, REST reads on both repositories, curl /api/version. No mesh write, no cluster access.}

Re-measured 20 minutes later — still true, and the shape is sharper: the partition is a MIX of two Hosting commits, and its two bookkeepers name different ones

Read-only on the control instance (memex.systemorph.com), 2026-09-17 11:1x–11:2xZ. Nothing written,
nothing rolled, no node hand-created.

The node is still absent — and at the commit the partition is SYNCED to, that is correct

get @Hosting/Issue/Source/FleetWatchCadence     -> Not found
get @Hosting/Issue/Source/*                     -> FleetWatch, IssueContent, IssueDetectors, IssueLayoutAreas

But Hosting/_GitSync (v457, read 11:1xZ) says the partition is at plugins 061976bc
(2026-09-15T21:46Z) — lastSyncCommitSha and lastAttemptedCommitSha, lastSyncOutcome: ImportedWithErrors, attempted 11:13:52Z. And /health agrees that this is the intended target:

publication-seal: Healthy — framework identity s2902ab1... holds 6 publication(s):
  ... 'plugins' ← Systemorph/MeshWeaver.Plugins @ 061976bc (sealed) ...

061976bc is 53 commits behind 8fe9d70dfc, the commit that adds FleetWatchCadence.cs.
GET .../contents/Hosting/Issue/Source/FleetWatchCadence.cs?ref=061976bc → 404. So the file's
absence from the partition is not a lost write at all: it does not exist at the sealed commit.

What IS wrong: the partition holds 1.22 files that reference it

read	value
`Hosting/InstanceAction/Source/ObservationQueries` (live node)	`lastModified` 2026-09-17T07:49:16.62Z, version 8
same file at the sealed `061976bc`	0 occurrences of `FleetWatchCadence`
same file on plugins `main` (1.22)	2 occurrences; `Hosting/InstanceAction.json` also declares `shared=@Hosting/Issue/Source/FleetWatchCadence`
live `Hosting/InstanceAction`	`compilationStatus: Error`, 11 × `CS0103 The name 'FleetWatchCadence' does not exist in the current context`, `adoptedModuleVersion: 1.21.6`, `currentModuleVersion: 1.21`

So the partition carries the 1.22 body of ObservationQueries beside a 1.21 source set — a
genuine mix, not a dropped node.

The second bookkeeper says 1.22 outright. Plugins/Hosting (v42, lastModified 10:47:22Z):
version: 1.22.0, installedFromRef: **HEAD**, installedAtUtc: 2026-09-17T10:47:22Z,
installedNodeCount: 222, autoUpdate: true, updatePolicy: Auto — and its installedFiles names
Hosting/Issue/Source/FleetWatchCadence.cs. One partition, two writers, two answers: the seal
reconciler says 061976bc (1.21), the registry install record says HEAD (1.22.0).

That is the shape of #4355 — "registry auto-update and the seal reconciler both write a GitSynced
partition with separate bookkeeping". 🚨 Its fix, #4364 (e76fa9f8f2, merged 2026-09-15T06:05:50Z),
is in the image this instance runs (afde4eab is 164 commits ahead of it, 0 behind), so this is
not "the fix has not landed" — either it is incomplete, or the writer here is a different one
(a package install at installedFromRef: HEAD while _GitSync is seal-pinned). I cannot separate
those two from outside; the import activity for the 07:49Z write is what would.

Not the #4456 cause

Hosting/Issue/Source/FleetWatchCadence.cs on plugins main is 16,710 bytes with zero NUL bytes
and zero control characters, so it is not the unstorable-byte refusal that #4456 turned out to be.
But #4456's structural fix — #4467, 758d502c6c, merged 2026-09-16T06:01:13Z, moving the import
memory to the per-node manifest and naming the failed paths — is not in this image: afde4eab
is 9 commits behind it. So the instrument that would say which node an import could not write is
exactly what the stalled roll is carrying.

What changed since the filing, and what did not

Hosting/DeploymentStatus recovered: compilationStatus: Ok at 11:12:54Z, because its
1.21 node declares no sources array at all — the shared= query the filing quoted is a 1.22
declaration. So that one is an artefact of the mix, not a separate fault.
Four Hosting NodeTypes are still Error on the control instance: Hosting/InstanceAction,
Hosting/InstanceRequest, Hosting/Issue, Hosting/PlatformBuildInbox (plus the unrelated
Essentials/Mailbox). InstanceAction being one of them means no instance action runs on the
control instance — no Sample, Logs, HelmRelease or Roll — which is the same consequence A proven-commit import applies modifications but not additions — the partition is left referenced-but-incomplete, and a force update imports 0 nodes #4456
recorded.
The roll has still not converged: /api/version answers 3.0.0+afde4eab… (core commit dated
2026-09-15T21:50:58Z), 3¾ h after the deployment set the image. The old pods still serve, so still
no user-visible outage.
The same class is live on memex-cloud, which has not been rolled since c84c6c05 (09-12):
7 Hosting NodeTypes at compilationStatus: Error, Hosting/Issue failing with
CS0117 'IssueKinds' does not contain a definition for 'JobFailed', CS0103 'SafeUrl',
CS1061 'IssueContent' does not contain a definition for 'Workload' — different symbols, same
signature: files from one Hosting version referencing definitions from another, with
adoptedModuleVersion: 1.18.7 against currentModuleVersion: 1.22.

What is owed

Which writer put 1.22 bytes into a seal-pinned partition, and whether fix(#4355): one partition, one bookkeeping — the registry lane is not a second writer #4364's rule covers a
package install whose installedFromRef is HEAD while _GitSync is pinned to a sealed commit.
That is the question this issue should carry; the title's "install records a file it did not
land" is the symptom seen from one side.
The roll onto the candidate, which is the only way fix: a content verdict is final for the NODE that earned it, never for the partition (#4459, #4456) #4467's named-failed-path import reaches
either portal — and the loop worth naming out loud: the fix for the import defect that blocks the
roll can only arrive by the roll.

Leaving open on (1). Re-titling would be premature until the writer is identified.

_{Read-only measurement 2026-09-17 11:1x–11:2xZ: get/search on memex.systemorph.com and memex.meshweaver.cloud, curl /api/version + /health, gh api .../contents?ref=… and commit comparisons on both repositories. No mesh write, no cluster access, no roll.}

Correction to my comment above: the probe waits 5 s, not 10 — which makes the reading stronger, not weaker

I took the probe timeout from the wrong line. Measured in the committed values:

deploy/helm/values.yaml ships probes.startup: periodSeconds 5, timeoutSeconds 5, failureThreshold 60;
both environment overlays in Systemorph/Memex (deployments/aks/memex/values.memex.public.yaml and deployments/aks/memex-cloud/values.memexcloud.public.yaml) raise periodSeconds to 10 and failureThreshold to 1080 to fit a cold bake, and leave timeoutSeconds: 5 untouched.

So /health at 8.12 / 9.62 / 9.52 s is not marginally over the probe's patience — it is never once inside it. failureThreshold: 1080 buys 1080 attempts at a wall. Everything else in the comment above is unchanged: the 3 h kill budget is periodSeconds 10 × failureThreshold 1080, and 11:23:40Z + 3 h = 14:23:40Z still lands on the measured restarts: 1.

Per-instance headroom against that fixed 5 s, same image line:

instance	`/health` warm	headroom
`memex`	8.12 / 9.62 / 9.52 s	none
`memex-cloud`	0.14 / 0.75 s	~7×

Two things follow. It is a property of the instance, not of the release. And a warm reading is not the one that matters — a booting replica runs its cold bake on the same CPU and the same volume as these checks, so the number that decides a roll is /health on a pod that is still starting, which nothing has ever sampled.

_{Correction by Claude Opus 5, 2026-09-17, read-only.}

Re-measured 2026-09-17 16:0x–16:2xZ on the control instance — the shape has moved, and the cause is now pinned

The install did not fail to write the file. It wrote it, and the partition's OTHER writer removed it.

1. The node is back, and what brought it back names the mechanism

Hosting/Issue/Source/FleetWatchCadence now exists — version 1, lastModified 2026-09-17T16:03:45Z. What changed at that minute was not an install: Hosting/_GitSync advanced
from 061976bc (2026-09-15T21:46Z) to d98fc2ac (2026-09-17T08:28Z) at 16:04:02Z, v548 → v549,
lastSyncOutcome: Imported.

The five files main carries that 061976bc does not are exactly:

Hosting/DispatchSeam.md
Hosting/InMeshWarningRatchets.md
Hosting/InstanceAction/Test/DispatchSeamTests.cs
Hosting/Issue/Source/FleetWatchCadence.cs
Hosting/StatusFreshness.md

All five appeared in the mesh at 16:03:45–46Z, each at version 1 — i.e. created, having been
absent. contents/Hosting/Issue/Source/FleetWatchCadence.cs?ref=061976bc → 404; ?ref=main →
200. A partition-shaped absence that ends the minute the seal moves to a commit carrying those
files is a prune, not a lost write. That answers the issue's open question ("whether the node
write was refused, lost, or never attempted"): it was attempted at 14:34:30Z, it landed, and
Hosting/_GitSync — which re-imports the same subdirectory every few minutes at the commit sealed
for the running framework — removed it again.

Not the #3659/#4525 class either: the file is a clean node file (16,710 bytes, valid
// <meshweaver> header, no NUL byte, no BOM), and the running image (afde4eab) predates #4525 in
any case.

2. The record, still claiming a tree the partition does not hold

Plugins/Hosting v46 — 1.22.1, installedFromRef: HEAD, installedAtUtc 14:34:30Z,
installedNodeCount: 222, autoUpdate: true — still lists
Hosting/Issue/Source/FleetWatchCadence.cs = 7e8e1d69…. The record is not lying at stamp time; it
is made false minutes later by the partition's other writer, and re-made true when the seal catches
up. Its falseness lasts exactly as long as the distance between the ref the installer resolved and
the seal.

3. The new shape is the same defect one step later

Of the three types, only Hosting/Issue is still at Error (1 of 1, namespace:Hosting scope:descendants nodeType:NodeType content.compilationStatus:Error, coverage hosting):

CS0117 Error: 'IssueLayoutAreas' does not contain a definition for 'Facts'
CS0117 Error: 'IssueLayoutAreas' does not contain a definition for 'StatusBadge' / 'SeverityBadge'

file	at the seal `d98fc2ac`	at `main`	changed `061976bc…d98fc2ac`?
`Issue/Source/IssueLayoutAreas.cs`	0 occurrences of those members	7	yes → the import wrote it
`Issue/Test/IssueTests.cs`	0 references	6	no → the import never touched it

The sealed import is a delta from the previous seal: it rewrote the view and left the test
alone — where the installer's main copy from 14:34:30Z was still sitting. The partition holds one
file from each tree. That is the mix in Doc/Architecture/OnePartitionOneBookkeeping, with the two
writers' roles swapped.

🚨 So gate 2 (#4355) cannot close this class on its own. It makes the INSTALLER's delta full on
a synced partition; this mix was produced by the OTHER writer's delta, which has the identical blind
spot — it diffs its own baseline, never the mesh. Whichever writer diffs second lands a mix, so the
remedy is the one the invariant already names: one writer.

4. Why the install was at `HEAD` at all — the residue #4259 left

#4259 pins the boot default install to the sealed commit precisely so that "the two unattended
writers of this partition now agree on its tree". But InstanceAutoRegistrationService.ProvenRef
can only attribute a seal to a repository: a source with no RepoPath — a remote registry, a
registered IPackageSource — answers "no repository to attribute a seal to" and is listed at its
configured ref, which defaults to HEAD. The control instance's plugin source is exactly that
shape, and installedFromRef: HEAD is the fingerprint of it.

The fix

#4619 — gate 1c: InstanceAutoRegistrationService.Install asks
PartitionContentOwnership for a candidate whose ref this boot could not prove, and holds where
the installer does not own the content (Undetermined holds too — a wrong hold is lifted by the
next boot, a wrong install cannot be taken back). A proven ref is not held, so #4259's lane is
untouched; a mesh with no tracking layer is unchanged. The hold fetches nothing, adopts no module,
still re-asserts declared access, and is recorded as a named SKIP — never a failure, never a poll.

Without the gate, the test written for it reproduces the loop itself: a node the partition's own
writer does not carry is written back at the branch tip on the very next pass.

Not done

Read-only measurement — no mesh write, no cluster access, no remediation applied here. The stale
Plugins/Hosting record is left as it is: with the fix in place the installer stops resolving a ref
ahead of the seal, and the claim becomes true again as the seal advances (as four of the five files
already did at 16:03Z).

_{Measured 2026-09-17 by Claude Opus 5: get / search on memex.systemorph.com, REST reads on

MeshWeaver.Plugins (contents, compare, commits).}

fix(#4588): only a WRITER counts as a writer, and a hold is not a ski…

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-17)

The attribution caveat, measured — and it is a SECOND, independent defect

This issue's caveat reads: "the fleet watch reports /health unreachable … TaskCanceledException for that pod, and for one of the two healthy ci.8710 pods as well, so that signal does not separate them." It does not separate them because both pods are over that watch's budget, and the reason is not the bake gate.

Measured 2026-09-17 (read-only: `curl` on a public unauthenticated endpoint, `get`/`search` on the control instance, one `Sample` and three `Logs` InstanceActions)

Three consecutive reads from outside the cluster, same host, same pod:

endpoint	response	time
`https://memex.systemorph.com/health`	200 `Degraded`	8.12 s, 9.62 s, 9.52 s
`https://memex.systemorph.com/alive`	200	0.12 s
`https://memex.systemorph.com/ready`	200	0.12 s

Same TLS, same ingress, same process — so the seconds are entirely in the untagged checks that only /health runs. The chart reads /health as the startupProbe with timeoutSeconds: 10, periodSeconds: 10, failureThreshold: 1080.

Control: https://memex.meshweaver.cloud/health answers in 0.14–0.75 s, so a 9 s /health is not inherent to the image family.

The roll that replaced the one in this issue behaves identically

Ops/Actions/roll-memex-20260917-8812 set the image at 11:23:36Z (generation 771). Pod memex-portal-deployment-d77fd7c88-x7nzm:

11:46:15Z — NOT ready, restarts: 0, /health unreachable on all three pods
14:38:15Z — NOT ready, restarts: 1
14:43:36Z, 14:51:15Z — NOT ready, restarts: 1, /health unreachable on all three again

11:23:40Z + (10 s × 1080) = 14:23:40Z — the container was killed at its startup budget, to the minute, and the next attempt started the same clock (bake-report on it records a fresh sweep at 14:29:41Z). It is a loop: the pod is killed mid-bake every three hours, so the bake never finishes either.

🚨 The bake gate was GREEN across it

At 14:38:15Z the fleet watch recorded a body from that pod whose first word was Degraded. ObservationQueries.GetTextAsync throws on any non-2xx, so that reading was a 200 — and HealthReport.Status is the worst of the entries, so no registered check was Unhealthy at that instant, nodetype_bake included. Loki holds no nodetype_bake … Unhealthy line for namespace memex since 2026-09-16T00:51Z (38 matches total across all namespaces, newest 2026-09-16T06:20Z on client-b).

So the uncompilable Hosting types this issue documents are a sufficient explanation for the first part of a boot (BakePhase.Running → nodetype_bake Unhealthy → 503 → probe fails, by design) — but they do not explain a not-Ready pod at 14:51Z whose aggregate was a 200 at 14:38Z. Once the sweep settles, the pod still cannot graduate, because /health cannot answer inside the probe's own 10 s timeout.

Refuted while checking: source-discovery is not short here

The new pod's source-discovery reads Degraded (a gap reached 996 ms = 100 % of the 1000 ms completion window), which looks like #3704. It is a proximity warning, not a short map: the pod settled 873 */Source nodes and 405 */Test nodes against 861 and 399 on the two ci.8710 pods — larger, as content growth predicts. No fold ended early.

Why the slow check has no name

The framework logs a duration for every check, at Information when the check is Healthy — which this fleet filters out of Loki for the Microsoft.* categories. Measured: not one line matching with status Healthy has ever landed in Ops/Logs. And the /health body printed only the non-Healthy entries plus the census-tagged ones. So an expensive check that answers Healthy appeared nowhere — not in the body, not in the log. The Warning-level ones that are visible are all cheap (pending_module_activation 17–68 ms over 156 samples today; content-types and bundle_adoption ~0.05 ms; required_modules 239–550 ms), which is why the 8–10 s sits in a check nothing can name.

Fixed in #4606: /health now publishes its total and its slowest checks on line two, so the next occurrence names its own cause in one curl. That PR deliberately does not widen any timeout. It also writes up two policies (HealthChecks request-timeout and output-cache, both 20 s) that AddDefaultHealthChecks registers and MapDefaultEndpoints attaches to nothing.

Nothing here contradicts this issue

The absent Hosting/Issue/Source/FleetWatchCadence node and the three uncompilable Hosting NodeTypes are real and still unfixed — they are why the bake keeps having work to do on every boot. This adds the second half: even with that fixed, this instance's /health is at the probe's budget, so a roll onto it is not safe until one of the two is addressed.

_{Read-only measurement 2026-09-17 by Claude Opus 5. No mesh content write, no cluster access, no mutating InstanceAction.}

Navigation Menu

Repository navigation

A 13-minute-old Hosting install records FleetWatchCadence.cs and the NODE is absent — three Hosting NodeTypes uncompilable, and memex's roll to the 3.0.0 candidate cannot converge #4588

What is failing

Measured 2026-09-17, control instance (memex.systemorph.com)

The deploy consequence

Why this is filed separately

Not done

Re-measured 20 minutes later — still true, and the shape is sharper: the partition is a MIX of two Hosting commits, and its two bookkeepers name different ones

The node is still absent — and at the commit the partition is SYNCED to, that is correct

What IS wrong: the partition holds 1.22 files that reference it

Not the #4456 cause

What changed since the filing, and what did not

What is owed

Correction to my comment above: the probe waits 5 s, not 10 — which makes the reading stronger, not weaker

Re-measured 2026-09-17 16:0x–16:2xZ on the control instance — the shape has moved, and the cause is now pinned

1. The node is back, and what brought it back names the mechanism

2. The record, still claiming a tree the partition does not hold

3. The new shape is the same defect one step later

4. Why the install was at `HEAD` at all — the residue #4259 left

The fix

Not done

The attribution caveat, measured — and it is a SECOND, independent defect

Measured 2026-09-17 (read-only: `curl` on a public unauthenticated endpoint, `get`/`search` on the control instance, one `Sample` and three `Logs` InstanceActions)

The roll that replaced the one in this issue behaves identically

🚨 The bake gate was GREEN across it

Refuted while checking: source-discovery is not short here

Why the slow check has no name

Nothing here contradicts this issue

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A 13-minute-old Hosting install records FleetWatchCadence.cs and the NODE is absent — three Hosting NodeTypes uncompilable, and memex's roll to the 3.0.0 candidate cannot converge #4588

Description

What is failing

Measured 2026-09-17, control instance (memex.systemorph.com)

The deploy consequence

Why this is filed separately

Not done

Activity

rbuergi commented on Sep 17, 2026

Re-measured 20 minutes later — still true, and the shape is sharper: the partition is a MIX of two Hosting commits, and its two bookkeepers name different ones

The node is still absent — and at the commit the partition is SYNCED to, that is correct

What IS wrong: the partition holds 1.22 files that reference it

Not the #4456 cause

What changed since the filing, and what did not

What is owed

rbuergi commented on Sep 17, 2026

Correction to my comment above: the probe waits 5 s, not 10 — which makes the reading stronger, not weaker

rbuergi commented on Sep 17, 2026

Re-measured 2026-09-17 16:0x–16:2xZ on the control instance — the shape has moved, and the cause is now pinned

1. The node is back, and what brought it back names the mechanism

2. The record, still claiming a tree the partition does not hold

3. The new shape is the same defect one step later

4. Why the install was at HEAD at all — the residue #4259 left

The fix

Not done

rbuergi commented on Oct 11, 2026

The attribution caveat, measured — and it is a SECOND, independent defect

Measured 2026-09-17 (read-only: curl on a public unauthenticated endpoint, get/search on the control instance, one Sample and three Logs InstanceActions)

The roll that replaced the one in this issue behaves identically

🚨 The bake gate was GREEN across it

Refuted while checking: source-discovery is not short here

Why the slow check has no name

Nothing here contradicts this issue

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

4. Why the install was at `HEAD` at all — the residue #4259 left

Measured 2026-09-17 (read-only: `curl` on a public unauthenticated endpoint, `get`/`search` on the control instance, one `Sample` and three `Logs` InstanceActions)