Cold boot: 78 prebuilt assemblies adopted, the sweep's store probe counts 5 ten seconds later — 197 compiles instead of ~20, and the upstream half of #3663

The disagreement, in one boot's own log

memex-portal-deployment-7d5d458cc4-cbztk boot 0, namespace memex (memex.systemorph.com), image 3.0.0-ci.8059. Both lines are from the same process, 10 seconds apart:

00:30:36.791  ShippedPrebuiltBundles: bundle Doc.zip: adopted 4/4 prebuilt assembly(ies); 0 were already current
00:31:08.780  ShippedPrebuiltBundles: 78 prebuilt assembly(ies) from 38 shipped bundle(s) under
              /data/prebuilt-bundles/sb43f9287dbd6922a7937bd24be103937 are backed by the assembly store
              — 78 adopted now, 0 already current and skipped WITHOUT activating their NodeType hubs
              — in 00:00:35.2855670

00:31:18.482  DynamicTypePreWarmer: 204 of 209 dynamic NodeType(s) need building (sequential, dependency
              order, batch direct-compile, perTypeBudget=00:05:00) — 5 already on the share.
              framework=sb43f928 total=209 baked=5 pending=204 frameworkstale=201 previouslybroken=3

78 assemblies adopted into the store; ten seconds later the sweep's store probe counts 5. Per-type, the same boot logged the adoption explicitly and the sweep compiled the type anyway:

00:30:35.461  Prebuilt assembly ADOPTED for Doc/Architecture/BusinessRules/Cession at version 25
              (framework sb43f9287dbd6922a7937bd24be103937) — no compile needed
00:31:44.915  Failed to compile assembly for node 'Doc/Architecture/BusinessRules/Cession'.

The control that makes it a disagreement rather than a number

Same portal, memex-portal-deployment-7fd85c4468-gwghs, 2026-09-08 07:34Z, same framework identity:

07:34:xx  ShippedPrebuiltBundles: 78 prebuilt assembly(ies) from 42 shipped bundle(s) … are backed by
          the assembly store — 1 adopted now, 77 already current and skipped … in 00:00:06.5
07:34:xx  DynamicTypePreWarmer: 20 of 209 dynamic NodeType(s) need building — 189 already on the share.
          framework=sb43f928 total=209 baked=189 pending=20 previouslybroken=3 dependencystale=17

So the probe does see assemblies that were already current in the store from a previous boot (77 of them, inside a total of 189). What it does not see is the set the seeding pass adopted in this same boot. That is the discriminator, and it is why this is a defect rather than a cold-store fact.

Full memex series (alreadyBaked from the warm-up summary, which agrees with baked= in the enumeration line):

boot (UTC)	seeding	`baked=`	compiled
09-07 22:41 `pcxnd`	cold	1	205
09-07 22:43 `n5ws6`	—	190	16
09-08 00:31 `cbztk`/0	78 adopted now, 0 already current	5	197
09-08 03:32 `cbztk`/1	—	40	166
09-08 03:35 `6fq52`	—	109	97
09-08 07:34 `gwghs`	1 adopted now, 77 already current	189	17
09-08 07:36 `lg59f`	—	189	17

Why this matters beyond the wasted minutes

This is the upstream half of #3663. The four Doc/** NodeTypes had their CI-baked bytes adopted at 00:30:35–36 with no compile needed; the sweep put them on the compile path anyway 43 seconds later, and only because they were on the compile path could a short source-discovery pass (#3663) turn them into four false regressions and hold the portal out of rotation for three hours. #3698 fixed what a short pass may CONCLUDE. This is what made the short pass reachable at all.

It also inflates the cold-boot cost by roughly an order of magnitude: 197 compiles instead of ~20.

Reading — candidates, NOT established

I have not measured the mechanism. Three shapes fit the readings, and they are distinguishable:

The enumeration's snapshot predates the seeding. The sweep takes ONE enumeration snapshot; if the store probe's view is captured before (or concurrently with) the 35-second seeding pass, the newly written entries are invisible. Argues against: the probe reported frameworkstale=201, so it was reading the store and finding entries — just not this framework's fresh ones.
Adoption and the probe key differently. Adoption logs "at version 25"; the probe classifies per type as baked / frameworkstale / previouslybroken / dependencystale. If the adopted entry's key (version, framework, source fingerprint) is not the key the probe looks up, the bytes are present and unreachable.
Adoption is not durable at the moment the probe runs — the write to the shared assembly store had not landed/flushed for 73 of 78 within 10 seconds. Argues against: adoption logged per type as completed, and this was Azure Files under a 35-second burst.

What is left unexplained by all three is the 5: not 0, not 78. Whatever the probe is keying on, five types satisfied it.

The measurement that would settle it

On one cold boot, for a single named type (Doc/DataMesh/SocialMedia/Profile is the cheapest — 2 source files):

log the exact store key ShippedPrebuiltBundles writes at adoption, and the exact key the sweep's store probe looks up, side by side;
log the timestamp at which the sweep's store view is materialised, relative to the seeding pass's completion;
name the 5 types the probe did count as baked on a boot that adopted 78 — what they have that the other 73 do not is the answer.

If the keys match and the view is materialised after seeding, the cause is durability (3). If the view predates seeding, it is (1) and the fix is ordering, not retry. If the keys differ, it is (2) and neither ordering nor waiting helps.

🚨 Not a fix: sleeping, retrying, or re-probing the store after the sweep starts. If the sweep cannot see what the same process wrote 10 seconds earlier, the answer is to find which of the three it is, not to look again later.

References

#3663 — the readiness stall this enabled (closed)
#3698 — the fix for what a short discovery pass may conclude
Doc/Architecture/SourceSetEstablishment — carries the same two log lines under "The prebuilt bytes were already there"
Doc/Architecture/InstallTimePrebuiltAdoption, Doc/Architecture/BakeIdentityMismatch — the adoption lane's existing pages

The other open half of #3663 is #3704 — why the discovery pass was short. The two are independent defects with one shared consequence: #3703 puts a type on the compile path that did not need to be there, #3704 hands that compile a short source set. Fixing either alone would have prevented #3663's stall; neither is fixed.

It is not a store-visibility problem — the probe's answer is computed and then preempted

This issue reads the disagreement as "the sweep's store probe counts 5", i.e. that the probe cannot
see what the seeding pass just adopted. Read against the code, the probe sees it fine. The result is
discarded one branch earlier, and the log line in the issue body already names the branch.

The two lines, and which field decides

00:31:18  … 5 already on the share.
          framework=sb43f928 total=209 baked=5 pending=204 frameworkstale=201 previouslybroken=3
07:34:xx  … 189 already on the share.
          framework=sb43f928 total=209 baked=189 pending=20 previouslybroken=3 dependencystale=17

The cold boot's 204 pending are frameworkstale=201. The control's 20 are dependencystale=17
— zero framework-stale. Same framework identity in both. So the question is not "did the store
answer", it is "why did 201 types reach the FrameworkStale verdict".

`ProbeOne` does ask the store, with the LIVE tag

NodeTypeBakeStatus.ProbeOne (src/MeshWeaver.Compiler.Pipeline/NodeTypeBakeStatus.cs:364) says so
itself:

Probe whenever there is a version to probe WITH — not only when the record already agrees we are
current. The store key carries the LIVE framework tag, so the lookup answers "are there bytes for
THIS framework?" independently of what the record claims, and a record naming another framework can
still be sitting on top of a usable build.

and ClassifyDetailed has the matching rule:

🚨 BYTES WIN OVER THE RECORD, in both directions — this is the whole premise. … A record can
name another framework while the share already holds our build … Reporting that FrameworkStale
would rebuild something already sitting on the volume, which is exactly the wasted work this probe
exists to avoid.

That rule is exactly what should have turned the 78 adoptions into Baked. It never runs.

What preempts it

The per-type dependency check sits above the bytes-win rule and returns early:

if (definition.CompiledDependencies is { } record
    && liveDependencyIdOf is not null
    && CompiledDependencies.FindMismatchAfterReevaluation(…) is { } mismatch)
    return frameworkMoved
        ? (BakeState.FrameworkStale, mismatch)
        : (BakeState.DependencyStale, mismatch);

// 🚨 BYTES WIN OVER THE RECORD … (unreachable for any record-stamped type on a framework roll)
if (storeHasBytes) return (BakeState.Baked, null);

and its own comment states the frequency, which is the whole point:

The reserved '!toolchain' entry is a hash over the FullMvidAssemblies closure's implementation
MVIDs, and FrameworkVersion folds those SAME MVIDs … A framework roll therefore moves
'!toolchain' BY CONSTRUCTION, and every platform ref-asm entry moves with it — so for any
record-stamped type the dependency check fires on every deploy … (this still preempts the
bytes-win rule below).

So on the first boot after a framework roll, every record-stamped type mismatches on !toolchain,
returns before storeHasBytes is read, and is recompiled — with correct live-framework bytes
sitting in the store, adopted ten seconds earlier. The control boot is 3 h later, after the records
have been re-stamped, so the dependency check passes and bytes-win applies: baked=189.

That accounts for every number in the table, including the middle rows (40, 109) — partial re-stamping
as successive boots compiled subsets.

What I think the fix is — and why I have not made it

The preemption is "conservative by design — a bytes-hit under the live framework can be exactly the
drifted build this state exists to replace". True for a module entry. But the code already
distinguishes the case where the framework alone explains the mismatch — that is what frameworkMoved
selects the NAME with. When the only mismatch is the reserved !toolchain entry and the
framework moved, the store's bytes are keyed by the live framework, so they already account for that
move: recompiling cannot produce anything different from what is already there.

So: let bytes win when the mismatch is !toolchain-only and the framework moved; keep the
preemption for any mismatch naming a real module. That is not a widened bound — it is the file's own
stated premise reaching the one branch that departs from it.

🚨 I am not implementing it in this pass. It changes what the bake gate accepts, which is a
correctness boundary this repository has re-litigated several times (#1976, #1707 slice 2,
#2952), and it wants a maintainer's eye plus a test staging a framework roll with adopted bytes.
Recorded here because the mechanism is now read off the source rather than inferred, and because the
issue's own framing ("the store probe counts 5") would send the next reader to the store.

Blast radius if the reading is right: every cold boot after a framework roll recompiles
essentially the whole mesh instead of adopting it — 197 compiles against ~20, and it is also the
upstream half of #3663.

Correcting my own proposal: the `!toolchain`-only narrowing would be a NO-OP

Above I proposed "let bytes win when the mismatch is !toolchain-only and the framework moved".
Reading CompiledDependencies.Compare — which I should have done before proposing it — that
condition essentially never holds.

Compare walks every entry in the record and returns the first mismatch:

foreach (var (name, stamped) in record)
{
    …
    else live = liveIdOf(name) ?? AbsentId;
    if (!string.Equals(stamped, live, StringComparison.Ordinal))
        return $"'{name}' built against {stamped}, live is {live}";
}

and the entries are the build's assembly surface — platform ref-asms included. NodeTypeBakeStatus
says so in the same breath as the !toolchain point I quoted: "A framework roll therefore moves
'!toolchain' BY CONSTRUCTION, and every platform ref-asm entry moves with it". So on a
framework roll the mismatch set is !toolchain plus every platform entry, and a rule keyed on
"only !toolchain mismatched" would fire for nobody. The diagnosis stands; that fix does not.

The better-founded rule, and the hazard that keeps it a maintainer call

The non-sequitur is narrower than I first wrote, and it is about what the record is evidence
for:

the store key is (typePath, version, LIVE framework tag), so bytes under it were compiled
against this framework — by another replica, or adopted from a bundle sealed for this
identity;
the record describes the build the node last recorded, which — when frameworkMoved — is a
different, superseded build.

So when the framework has moved, the record's mismatches are evidence about a build the store's
bytes are not. Preempting bytes-win with it concludes "those bytes are stale" from a comparison that
never looked at them. When frameworkMoved is false, record and store key describe the same
build, the mismatch is real evidence about those exact bytes, and the preemption is right — which is
the control boot's dependencystale=17, still working.

That gives: bytes win when frameworkMoved and the store holds live-framework bytes; the
dependency check keeps its preemption when the framework did not move.

🚨 The hazard, stated rather than argued away. Modules land independently of the framework, so
live-framework bytes could have been produced before a module update inside the same framework
identity — and under this rule they would be served instead of rebuilt. That is the exact
stale-serve the preemption is conservative about. Which risk is worse is a judgement about this
deployment's cadence, not something the code can settle, so it stays a maintainer call:

	cost
keep preempting (today)	every cold boot after a framework roll recompiles essentially the whole mesh — 197 compiles against ~20, measured; the upstream half of #3663
bytes win on a framework move	a type may serve bytes predating a module update made within the same framework identity, until something re-stamps it

If the second is chosen, ClassifyDetailed is pure by design ("no hub, no mesh, no I/O … every
state is unit-testable without a fixture"), so the scenario stages directly: an old-framework record
whose dependency record mismatches, storeHasBytes: true, framework moved ⇒ Baked; the same with
the framework unmoved ⇒ DependencyStale.

It is not the classifier — I tried changing it, and the change is wrong

The obvious reading is that NodeTypeBakeStatus's dependency check preempts the "bytes win over the
record" rule, so adopted bytes get rebuilt. I wrote that fix and it is wrong. Recording the
experiment so nobody repeats it.

What looked like the cause

if (mismatch is { })
    return frameworkMoved ? (FrameworkStale, mismatch) : (DependencyStale, mismatch);
if (storeHasBytes) return (Baked, null);          // ← unreachable when a record is present

Every real record carries CompiledDependencies (#1707 slice 2 stamps it on compile;
PrebuiltAssemblySeeder stamps the producer's on adoption), and the reserved !toolchain entry
hashes the framework closure's MVIDs — so it moves by construction on every deploy. The check
therefore always fires on a framework roll and always preempts. Only the test fixture
(Healthy(), which stamps no record) can reach the bytes-win rule at all.

I made the preemption yield when the framework explains the mismatch, and wrote a test that failed
before and passed after:

Expected value to be Baked … but found FrameworkStale

Why that is wrong

It breaks two of the three cases
ContentKeyReevaluationTest.ClassifyDetailed_TheContentKeyDecides_AndAnAbsentOneRebakes pins:

case	required	my change gave
regenerated key moved	`FrameworkStale` — "the bytes in the share are not what a compile would produce now"	`Baked` ❌
no regenerated key	`FrameworkStale` — "the metadata-only rule governs, exactly as before"	`Baked` ❌

The framework moving does not imply the type's own generated input held still. Without a
regenerated content key — and liveGeneratedInputDigest is documented as "null, the default, and
every caller today" — nothing can tell those apart, so rebuilding is the only safe answer. The rule
is conservative on purpose: one redundant rebuild, never a stale serve.

Which narrows this issue considerably

PrebuiltAssemblySeeder stamps, on adoption:

CompiledFrameworkVersion = NodeTypeCompilationHelpers.FrameworkVersion,   // the LIVE framework
CompiledDependencies     = dependencies,                                  // the producer's, validated above

A record carrying those classifies Baked by construction — no framework move, no dependency
mismatch. CompiledDependencyRecordTest.Classify_DependencyStale_BeatsTheBytesWinRule already pins
exactly that ("a matching record keeps the bytes-win premise intact" → Baked).

🚨 So a sweep that sees an adopted type as stale is reading a record that does not yet carry the
adoption stamp. Those writes are cross-hub stream.Updates to each NodeType's owning hub, and the
seeder's own comments say so twice — the source-stamp is requested rather than written precisely
because "this write is CROSS-HUB … the lambda diffs against the MIRROR's snapshot".

That fits the timeline in this issue exactly: adoption reports 78 at 00:31:08, the pre-warmer
reads at 00:31:18, and frameworkstale=201 means 201 records still named the previous
framework at read time.

Where I would look next

Not at NodeTypeBakeStatus. At what DynamicTypePreWarmer reads its NodeTypeDefinitions from, and
whether that read is ordered after the adoption's record writes have landed — or whether it can be.
If it cannot be ordered, then the report needs to consult the adoption registry (which knows what was
adopted in-process this boot) rather than only the records.

I did not establish that last part; I am naming it because it is the half the classifier cannot fix.

The seeder/probe ordering alone does not establish durable visibility. At core 5e8239bd1, the hosted service sequences seeding before probing, and SeedObserved waits for stream.Update. However, DataExtensions.ArmPatchAckWatcher deliberately calls ackOnce(true, null) when the 10-second flush bound expires, leaving persistence running (#3112). It logs FLUSH_OUTLIVED_BOUND. The test FlushThatOutlivesTheBound_AcksTrueOnceOnTheCommit_AndKeepsTheFlushRunning explicitly pins that behavior. Consequently an ADOPTED result can follow an acknowledged in-memory stamp while the storage-backed readiness query still reads the old record. The read-after-write comments on the hook and update path are stronger than this implementation.

This identifies a concrete possible ordering gap, not yet the historical cause. The discriminating evidence is FLUSH_OUTLIVED_BOUND for one of the adopted paths before this boot's probe, plus its eventual durable write. Do not relax the classifier or simply restore a flush-timeout NACK: the existing test documents why that would report an already committed write as not applied.

Settled: the premise is wrong, the defect underneath it is real — fix in #3928

The share was fine. Both counters were right. They were never the same population.

What each instrument counts, with denominators

	`ShippedPrebuiltBundles` → 78	`DynamicTypePreWarmer` → 5
unit	assemblies (bundle entries — two bundles may name one NodeType)	NodeTypes
denominator	entries across the 38 bundles this mesh holds a NodeType for	209 dynamic NodeTypes in one mesh-wide enumeration
source	bytes it wrote + the record patch it posted	a CQRS query snapshot of each type's record
claim	"these bytes are on the assembly store"	"this type's RECORD names a live-framework build and the store resolves bytes at the version that record names"

🚨 There is no "store probe" in the sense the title uses. The store is asked exactly ONE
question per type — TryGetAssemblyPath(typePath, definition.LastCompiledVersion) — and the key
comes out of the record. A reader whose record snapshot is behind asks the wrong key, misses,
and reports FrameworkStale whatever is on the volume. baked=5 and frameworkstale=201 are
statements about records; 78 is a statement about bytes. The bake report is not a census
of the share and cannot be one.

So the three candidate mechanisms in the body are all "no": not (1) an enumeration taken before
seeding — it is sequenced after (seeded.SelectMany(_ => ProbeDynamicTypes(…))); not (2) a key
mismatch in the store — the key is the record's, which is the point; not (3) durability — the
control boot proves the records did land.

And the unexplained 5 has an ordinary answer: it is over all 209 types, and the 78 adopted
contributed zero. Five other types were already current from before. That is a total miss, not
a lag pattern, which is what makes it a defect rather than a race you can wait out.

The defect

The seeding pass writes each adopted record through GetMeshNodeStream(path).Update(…) —
authoritative. The sweep then reads Query<MeshNode>(MeshWideQuery.OfType(…)).Take(1) — a CQRS
projection (Doc/Architecture/CqrsAndContentAccess: "a query's answer for one path can be minutes
old"). There is no ordering between those two sources, and nothing in ClassifyDetailed can
tell a stale projection from a genuinely stale record — they are the same bytes.

This confirms the direction of the fourth comment above ("look at what DynamicTypePreWarmer reads
its NodeTypeDefinitions from") and rules out the classifier, exactly as the third comment
concluded. NodeTypeBakeStatus.ClassifyDetailed is unchanged.

The fix (#3928) — no retry, no wait, no bound moved

NodeTypeAdoptionRegistry already existed to stop the first-build kickoff racing an in-flight
adoption. It now also carries a ledger of completed ones, and the sweep classifies each type from
the newer of the two facts. The ordering is a fact: an adoption stamps the version it read and its
own write bumps the node, so snapshotVersion <= observedVersion ⇒ the snapshot cannot contain
that write; > ⇒ it contains that write or something later (an owner refusal #2813, a recompile) and
the snapshot wins. That second branch is what keeps it from becoming a stale serve, and it is
pinned by a test that fails against an unconditional overlay.

And the reporting, because a silent correction would be the next version of this bug

Each line now names its own population, and NodeTypeBakeReport.ClassifiedFromLocalAdoption (shown
as fromlocaladoption=N) makes a boot whose enumeration was behind say so in the line that reports
the verdict. A second silent truncation found on the way — n.Content is NodeTypeDefinition dropping
any row that arrived as JsonElement out of total=/baked=/pending= with nothing to grep — is
fixed and counted too.

Work products

new Doc/Architecture/AdoptionAndTheSweepCountDifferentThings
instance 14 in Doc/Architecture/ControlsThatCannotFail — "two counters over different
populations, printed as if they were one measurement"
a What's New entry

🤖 Generated with Claude Code

https://claude.ai/code/session_01S25PJZbVeCbkWhCfMub9P7

docs: answer the "5", and say why the overlay moves definitions and n…

Re-measured 2026-09-11 06:30Z — OPEN, narrowly: the fix is merged AND deployed; the reading that would confirm it has no portal answer

The closing condition as this thread last states it

The body's three candidate mechanisms were all falsified by the 2026-09-10 comment ("the premise is wrong, the defect underneath it is real"): 78 counts assemblies, 5 counts NodeTypes, the store is asked exactly one question per type keyed off the record, and the real defect is that the seeder writes records through GetMeshNodeStream(path).Update(…) while the sweep reads a CQRS projection with no ordering between them. The fix is #3928 (NodeTypeAdoptionRegistry carries a ledger of completed adoptions; the sweep classifies from the newer of the two facts), and the confirming instrument it added is fromlocaladoption=N on the bake report line.

So what closes this is a cold boot after a framework roll reporting a sane pending= with fromlocaladoption accounting for the adopted set.

What I could measure

The fix is on main and in BOTH running portal images — by FILE/SYMBOL, never by ancestry (queue-merged commits fail is-ancestor):

symbol	`origin/main`	`6231c4da…` (memex.meshweaver.cloud)	`45306a33…` (memex.systemorph.com)
`NodeTypeAdoptionRegistry.StampSuperseding`	2	2	2
`NodeTypeAdoptionRegistry.OverlayOnto`	2	2	2
`NodeTypeBakeStatus` : `ClassifiedFromLocalAdoption`	4	4	4
`NodeTypeBakeStatus` : `fromlocaladoption`	1	1	1

Both portals confirmed running today's code first: /api/version → 3.0.0+6231c4da… (2026-09-11T03:32Z) and 3.0.0+45306a33… (2026-09-10T23:37Z).

What I could NOT measure, and why — stated rather than guessed

The bake report is a log line, and there is no portal surface for it. I looked for one:

/health on both portals prints only non-healthy entries; nodetype_bake is not among them on either portal right now, so it is healthy and prints nothing.
The nodetype_bake incident that does exist on the control instance (Admin/_LogIncident/0a24845deb486a56, occurrences 49 945, lastSeen 2026-09-11T01:00:46Z) carries only the transient 'NodeType bake in progress — enumerating dynamic NodeTypes' message. No counts, no baked=, no fromlocaladoption=.
No node in the mesh carries NodeTypeBakeReport.
Loki / kubectl is break-glass and this session is not authorised to use it.

A per-NodeType census through search would be a proxy, not this issue's measurement, and it is RLS-filtered besides — so it cannot state a denominator for the 209 the report counts.

Verdict — OPEN

Diagnosis settled, fix merged, fix running on both live portals. The confirming reading is pending and cannot be taken with the instruments a portal-only session has. Two ways to discharge it, in order of preference:

Surface the bake report. NodeTypeBakeReport — total=/baked=/pending=/frameworkstale=/fromlocaladoption= — is exactly the shape /health already publishes for bundle_adoption and content-types, both of which I could read as an unauthenticated client with the process's own denominator. Publishing it there turns this issue's verdict into a one-line curl, permanently. That is the durable fix for the instrument gap, not just for this issue.
Failing that: read BatchBake/DynamicTypePreWarmer at the next cold boot after a framework roll and post baked=, pending=, frameworkstale= and fromlocaladoption= here.

_{Read-only through the portal API and /health; no cluster access, no writes.}

The instrument now exists, and the gate has moved from "unreadable" to "needs a roll"

#4015 merged as ac7617466 (2026-09-11), landing BakeReportHealthCheck.cs and
SourceDiscoveryHealthCheck.cs under memex/aspire/Memex.Portal.ServiceDefaults/, plus
src/MeshWeaver.Compiler.Pipeline/NodeTypeBakeStatus.cs.

That closes the reason this issue could not be settled at all. The 2026-09-11 verification sweep
found the fix (#3928's StampSuperseding / OverlayOnto / ClassifiedFromLocalAdoption /
fromlocaladoption) already present and running in both portal images, but the confirming
reading was a boot log line with no portal surface — checked /health, the nodetype_bake
incident and the mesh — and Loki is break-glass. So the verdict existed and nothing could read it.

🚨 It is NOT yet readable — measured, not assumed

A merge is not a deployment. Verified by FILE/SYMBOL (${ref}:${path}, because queue-merged core
commits fail --is-ancestor):

ref	`BakeReportHealthCheck.cs`
`ac7617466` — core `main` today	PRESENT
`6231c4da4` — memex.meshweaver.cloud, running now (`/api/version`, HTTP 200)	ABSENT
`45306a33e` — memex.systemorph.com, running now (`/api/version`, HTTP 200)	ABSENT

Both live portals predate the change. This issue stays open until a roll carries ac7617466 or
later, and then it is answerable by curl with no cluster access.

What to do after the roll

Read /health on the rolled portal and record the bake report. Two cautions for whoever does it:

🚨 /health prints only NON-healthy entries. An absent entry is not a clean one. Establish
what a clean bake report looks like on that surface before reading silence as success — that
distinction is the whole point of publishing it.
🚨 Repeating the call samples different replicas (measured: 5/5 and 6/2 splits across two
replicas of one deployment). State how many reads you took and how they split, not just the last
one.
/health is public, unauthenticated and system-side, so unlike a search/get sweep it is
not RLS-filtered — which is exactly why it can answer a question the mesh could not.

Do NOT settle this from logs

Measured 2026-09-11: the Hosting Logs action's sinceMinutes is a REQUEST, not coverage, and
the run never reports the window it actually covered. A query over a requested 2,880 minutes
returned entryCount: 0 while the positive control — same shape, a string known to occur — showed
real coverage of 55 minutes. A zero there is not evidence, and a boot-time line is exactly the
short-window shape that cannot be recovered retrospectively.

🤖 Generated with Claude Code

The reading is in. Both portals run c84c6c055 (carries #4015's bake-report on /health) and booted on a NEW framework identity (sd608997) on 2026-09-12 — the cold-boot-after-a-roll shape this issue was waiting for. Six GET /health per portal (07:06:01–07:06:38Z, two replicas reached each): the control instance's seeding replica reads total=215 baked=214 pending=1 previouslybroken=1 with 59 adoption stamps held by this process (this issue's boot: baked=5 pending=204 with 78 adopted); its sibling 214/215 with 1 stamp; memex-cloud 295/353 (55 framework-stale = authored types with no bundle, 105 stamps) and 350/353. All four Healthy = ClassifiedFromLocalAdoption == 0. Conserved in Doc/Architecture/AdoptionAndTheSweepCountDifferentThings § Measured after the fix by #4150, which closes this on merge.

Cold boot: 78 prebuilt assemblies adopted, the sweep's store probe counts 5 ten seconds later — 197 compiles instead of ~20, and the upstream half of #3663 #3703

Description

The disagreement, in one boot's own log

The control that makes it a disagreement rather than a number

Why this matters beyond the wasted minutes

Reading — candidates, NOT established

The measurement that would settle it

References

Activity

rbuergi commented on Sep 8, 2026

rbuergi commented on Sep 9, 2026

It is not a store-visibility problem — the probe's answer is computed and then preempted

The two lines, and which field decides

ProbeOne does ask the store, with the LIVE tag

What preempts it

What I think the fix is — and why I have not made it

rbuergi commented on Sep 9, 2026

Correcting my own proposal: the !toolchain-only narrowing would be a NO-OP

The better-founded rule, and the hazard that keeps it a maintainer call

rbuergi commented on Sep 9, 2026

It is not the classifier — I tried changing it, and the change is wrong

What looked like the cause

Why that is wrong

Which narrows this issue considerably

Where I would look next

rbuergi commented on Sep 9, 2026

rbuergi commented on Sep 10, 2026

Settled: the premise is wrong, the defect underneath it is real — fix in #3928

What each instrument counts, with denominators

The defect

The fix (#3928) — no retry, no wait, no bound moved

And the reporting, because a silent correction would be the next version of this bug

Work products

rbuergi commented on Sep 11, 2026

Re-measured 2026-09-11 06:30Z — OPEN, narrowly: the fix is merged AND deployed; the reading that would confirm it has no portal answer

The closing condition as this thread last states it

What I could measure

What I could NOT measure, and why — stated rather than guessed

Verdict — OPEN

rbuergi commented on Sep 11, 2026

The instrument now exists, and the gate has moved from "unreadable" to "needs a roll"

🚨 It is NOT yet readable — measured, not assumed

What to do after the roll

Do NOT settle this from logs

rbuergi commented on Sep 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`ProbeOne` does ask the store, with the LIVE tag

Correcting my own proposal: the `!toolchain`-only narrowing would be a NO-OP