Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,63 @@ anything has been compiled. It is the only independent witness available at disc
is used in both directions — the same field, read for both polarities, which is what makes the two
classifications agree by construction rather than by coincidence.

`NodeTypeBatchBake.DiscoveryUnestablished` is the one predicate. When it answers true the whole
batch is abandoned with `SourceDiscoveryFailedException` and the pod falls back to the
activation-driven sweep, which re-resolves each type individually — slower, correct, and impossible
to mistake for a content verdict.
`NodeTypeBatchBake.DiscoveryUnestablished` is the one predicate. When it answers true for any type
after the GLOBAL pass, that pass is distrusted as a whole (a short read that empties one type may
have shortened others undetectably) — and the batch then re-resolves every pending type from its OWN
anchored queries, as described next. A type the predicate still refuses is left out of the batch and
warmed by activation, which resolves its sources itself — slower, correct, and impossible to mistake
for a content verdict.

## The unit of "I don't know" is the type, not the batch

The global pass is three mesh-wide fetches (`nodeType:Code partitions:all`, and the
`namespace:*/Source` / `namespace:*/Test` subtrees). On a partitioned Postgres mesh each is ONE
`UNION ALL` over every partition schema, so **one partition that cannot answer fails every fetch** —
and until this change that failed every pending type with it.

Measured on memex-cloud (core `bf5ac85526`, 2026-09-24): on every one of eight boots between 07:12Z
and 08:06Z, `nodeType:Code partitions:all` settled at 1586 nodes and ~25 ms later the pod logged
`batched source discovery did not establish the source sets — abandoning the batch and falling back
to the activation-driven sweep for ALL N pending type(s)`, with N between 38 and 80. That cost each
boot 5–14 minutes of per-type activations; on memex, combined with the activation wait deadlock
fixed in #5643, it held a pod unready for hours.

So when the global pass faults, times out, reaches its ceiling or is contradicted by a type's
record, `ResolveSources` falls back to `ResolvePerQuery`: every DISTINCT source query the pending
types expand to is run on its own — anchored, so pinned to the one partition it names — with a
concurrency of four, and each type is handed the union of its own queries' answers
(`AssemblePerQuery`). A type joins the batch only when every one of its queries answered and the
answer is established; a type left out is ABSENT from the map, and the sweep warms exactly that type
by activation. It is never handed an empty set, which would be #1216's fabricated verdict.

Two properties keep the fallback honest:

- **A floor is not an answer.** Every discovery query (global and per-type) refuses a snapshot whose
frame names `SilentProviders`. That field is set when a provider never answered, or answered over
reads it could not complete (`SnapshotIncomplete`, which is how a partitioned provider reports a
partition it dropped). Folding such a frame would hand the compiler a PARTIAL set.
- **One budget, not one per query.** The whole per-type pass shares ONE deadline of
`DiscoveryBudget`, the same budget the global pass has. Any query still unanswered at that
deadline fails as itself, so stalled queries cannot add up to several budgets before the sweep
starts.

The fault itself was read off the same boot (pod `…-6f9d6848f8-gf5df`): the line logged 13 µs
after the abandonment at 07:34:15.089Z is `Npgsql.PostgresException (0x80004005): 42703: column
n.created_by does not exist` — the `namespace:*/Source` fetch reads the `code` satellite table of
every schema, and at least one schema's `code` table lacks the authorship columns V57 adds. The
same `42703` recurs on that pod every 60 s from some other mesh-wide read. Which schema, and why V57
did not reach it, was NOT established.

What this does NOT do: it does not name or repair the partition that could not answer. The global
fetch's fault is logged at Warning with its exception (`BatchBake: the GLOBAL source-discovery pass
did not establish the source sets (<exception type>) …`), and the types withheld are named with the
query and fault that withheld them — the partition itself is a separate defect in the backend that
answered.

Pinned by `OnePartitionCostsOnlyItsOwnTypesTest`: a query provider that refuses every mesh-wide
`nodeType:Code` fetch and the anchored reads of ONE type leaves the healthy type in the batch with
its real source set and the broken one absent. With the fallback replaced by a rethrow, the same
test fails with the provider's fault — the pre-fix behaviour.

## What went wrong (issue #3663)

Expand Down
33 changes: 24 additions & 9 deletions src/MeshWeaver.Hosting/DynamicTypePreWarmer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1223,26 +1223,36 @@ IObservable<PreWarmOutcome> Sweep(
// #5544 — the readiness message names THIS type and the bound it may take,
// so a sweep that is slow reads as slow, on the right type, rather than as
// "enumerating" for hours.
//
// 🚨 A type ABSENT from batchSources had no source set the discovery could
// establish (its own query failed, or its record contradicts an empty answer).
// It is warmed by activation — ONE type leaves the batch, never all of them, and
// it is never compiled against an empty set (that would be #1216's fabricated
// verdict).
IReadOnlyList<MeshNode>? batchSet = null;
MeshNode? typeNode = null;
var inBatch = batchSources is not null
&& batchSources.TryGetValue(p, out batchSet)
&& nodes.TryGetValue(p, out typeNode);
progress?.Invoke(
$"building {pending.IndexOf(p) + 1} of {pending.Count} pending NodeType(s) "
+ (batchSources is not null
+ (inBatch
? "by direct batch compile"
: bytesMissing.Contains(p)
? "by a store-miss rebuild"
: "by activation (the batch was not available)")
: "by activation (the batch did not establish its sources)")
+ $": waiting on {p}, for up to {budget}");
var warm = batchSources is not null && nodes.TryGetValue(p, out var typeNode)
? NodeTypeBatchBake.BakeOne(
mesh, typeNode,
batchSources.TryGetValue(p, out var srcs) ? srcs : Array.Empty<MeshNode>(),
budget, logger)
var warm = inBatch
? NodeTypeBatchBake.BakeOne(mesh, typeNode!, batchSet!, budget, logger)
: missingBytes
? RebuildMissingBytes(workspace, accessService, p, budget, logger)
: WarmOne(workspace, accessService, p, budget, logger);

return warm
.DelaySubscription(
i == 0 || batchSources is not null ? TimeSpan.Zero : pacing)
// (A type the batch withheld is activated WITHOUT pacing too: batchSources is
// non-null only on a gated pod, which serves nobody while it bakes.)
// 🚨 A verdict against a node that NO LONGER EXISTS is not a verdict about
// the image. The definitions were enumerated once, at the start of the
// sweep; a repository sync can prune a retired type at any moment after
Expand Down Expand Up @@ -1288,8 +1298,13 @@ or PreWarmStatus.Retired
// does not know what the sources ARE, and a compile driven from a set you did not establish
// produces verdicts about code from evidence you do not have — on memex-cloud 2026-08-11
// that read as "169 of 237 types are content-broken" and, on an ungated pod, would have
// baked a fleet of empty assemblies with nothing refusing readiness. Whole-batch fallback is
// the only safe answer: the activation-driven sweep resolves each type's sources itself.
// baked a fleet of empty assemblies with nothing refusing readiness. The unit of that "I don't
// know" is the TYPE, not the batch: ResolveSources re-resolves every type from its own
// anchored queries when the global pass cannot be trusted, and leaves out of the map only the
// types it still could not establish — those, and only those, take the activation-driven
// sweep (which resolves their sources itself). Measured on memex-cloud 2026-09-24: one
// unanswerable global fetch sent ALL 38–80 pending types of every boot down that road. What
// still reaches the Catch below is a fault in discovery's own composition, not a query's.
IObservable<ImmutableDictionary<string, IReadOnlyList<MeshNode>>?> BatchSources() =>
Observable.Defer(() =>
{
Expand Down
Loading
Loading