Repository navigation
fix(bake): one partition that cannot answer costs the batched discovery its own types, not all of them - #5655
Conversation
…ry its own types, not all of them The batched source discovery resolved every pending NodeType from three mesh-wide fetches — each one UNION over every partition schema on Postgres — and surfaced any fault as a fault of the WHOLE batch. On memex-cloud (bf5ac85) every boot's `namespace:*/Source` fetch hit 42703 in one schema's code table, so all 38–80 pending types took the activation sweep (5–14 min per boot; hours on memex with #5643). When the global pass cannot be trusted (fault, timeout, ceiling, or a record that contradicts an empty set), ResolveSources now re-resolves each type from its OWN anchored queries and withholds only the types whose own queries fail; the sweep warms exactly those by activation and bakes the rest directly. Refs #5654, #5544, Systemorph/MeshWeaver.Plugins#2350 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical source-completeness and empty-set validation issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (7)
What changed in this PR
This PR isolates batched source-discovery failures to affected NodeTypes instead of abandoning the entire batch.
Changes:
- Adds per-query fallback discovery and per-type source assembly.
- Updates prewarming to activate types lacking established sources.
- Adds integration and truth-table tests.
- Documents type-level source-establishment semantics.
Unresolved review findings include incomplete-query handling, empty-source validation, cancellation and timeout behavior, and mutable collection usage.
| File | Summary |
|---|---|
test/MeshWeaver.Hosting.Test/OnePartitionCostsOnlyItsOwnTypesTest.cs |
Tests healthy and failed partition behavior. |
src/MeshWeaver.Hosting/NodeTypeBatchBake.cs |
Implements per-query fallback and source assembly. |
src/MeshWeaver.Hosting/DynamicTypePreWarmer.cs |
Activates types whose sources were not established. |
src/MeshWeaver.Documentation/Data/Architecture/SourceSetEstablishment.md |
Documents type-level discovery uncertainty. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| return queries | ||
| .Select(q => RunQuery(meshService, q, logger, discovery) | ||
| .Timeout(DiscoveryBudget) | ||
| .Select(nodes => new QueryAnswer(nodes, null)) | ||
| // Not a swallow: the fault becomes THIS query's recorded answer, and every type that | ||
| // reads from it is withheld from the batch and named, with this fault, below. | ||
| .Catch<QueryAnswer, Exception>(ex => Observable.Return(new QueryAnswer(null, ex))) |
There was a problem hiding this comment.
Fixed in 2e23aa7. RunQuery now records SilentProviders on every frame it folds, and it refuses the settled map as a FLOOR (SourceDiscoveryFailedException) when any are named. The fan-in folds SnapshotIncomplete into SilentProviders, so a provider that dropped a partition's rows is covered. This applies to both the global pass and the per-type pass. In the test, the broken type's own read now answers an Initial marked SnapshotIncomplete. With the check disabled, the test fails on 'the type whose own query fails is withheld'.
| if (failure is null | ||
| && DiscoveryUnestablished(matched.Count, pending.DeclaresSources, pending.KnownSourceCount)) | ||
| failure = "its own queries resolved an EMPTY set that its record contradicts"; |
There was a problem hiding this comment.
No change needed here. The predicate already returns true for that state: DiscoveryUnestablished(matchedCount: 0, declaresSources: true, knownSourceCount: null) evaluates as 0 == 0 && null != 0 (true) && (true || null > 0) = true. It is pinned by the existing EmptySourceSetNeedsAWitnessTest.AbsentSnapshot_LeavesThePreExistingRuleUntouched (Assert.True for exactly those arguments). AssemblePerQuery calls the same predicate, so it withholds such a type.
| .Merge(PerQueryConcurrency) | ||
| .ToList() |
There was a problem hiding this comment.
Fixed in 2e23aa7. The per-type pass now computes ONE deadline (now + DiscoveryBudget) when it is subscribed, and every query uses Timeout(deadline). A query still running or not yet started at that deadline fails as itself, so its types alone are withheld, and the pass ends within one budget. Per-query fault isolation is kept.
| private async Task UntilListed(string path, CancellationToken ct) | ||
| => await Observable.Interval(200.Milliseconds()).StartWith(0L) | ||
| .SelectMany(_ => MeshService | ||
| .Query<MeshNode>(MeshQueryRequest.FromQuery($"path:{path}").AsSystem()) | ||
| .Take(1)) | ||
| .Where(change => change.Items.Any(n => n.Path == path)) | ||
| .FirstAsync() | ||
| .Should().Within(TestTimeouts.Convergence) | ||
| .Emit($"discovery must be able to see {path}", ct); |
There was a problem hiding this comment.
Kept as is, on purpose. What gates discovery is whether the query index LISTS the node, because ResolveSources reads through queries and not through the node stream. A GetMeshNodeStream read would prove the store has it while the index could still trail, and the test would then measure an index lag instead of the fallback. The wait re-issues the query every 200 ms (Interval + SelectMany), so a stale negative is retried rather than trusted. This is the same readiness wait ABakeCompilesTheDefinitionItResolvedTest.UntilListed uses.
| var definitions = new Dictionary<string, NodeTypeDefinition?> | ||
| { | ||
| [GoodType] = new() { Configuration = "config => config" }, | ||
| [BrokenType] = new() { Configuration = "config => config" }, | ||
| }; |
There was a problem hiding this comment.
Fixed in 2e23aa7: both fixtures (definitions, and answers/perType in the pure test) are now ImmutableDictionary / ImmutableList.
…ype pass (review) - RunQuery refuses a frame naming SilentProviders (a provider that did not answer, or answered over a partition it dropped — SnapshotIncomplete), so neither pass folds a PARTIAL set; the test's broken type now answers incomplete, and a negative control (check disabled) fails the test. - The per-type pass shares ONE DiscoveryBudget deadline instead of one per query. - Established sets are ImmutableList; test fixtures immutable; log wording. Refs #5654 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…unded (MergeBoundedRatchetGuard) Its inners complete synchronously (.Catch → Return), exactly the shape that makes a bare Merge(n) recurse per queued inner. Refs #5654 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
What
NodeTypeBatchBake.ResolveSourcesno longer treats one unanswerable partition as a failure of every pending NodeType.ResolvePerQuery: every distinct source query the pending types expand to runs on its own (anchored, so pinned to the one partition it names;Merge(4)), andAssemblePerQueryhands each type the union of its own answers.DynamicTypePreWarmer's sweep now warms exactly that type by activation. It never compiles a type against an empty set.DiscoveryBudgetand the query fan-in's own stall terminal.Why (measured)
On memex-cloud, core
bf5ac85526, 2026-09-24. These areLogsactions on memex.meshweaver.cloud:Ops/logs-memexcloud-20260924-discovery-limit25000,…-batch-discoveryand…-gf5df-42703.nodeType:Code partitions:allsettled at 1586 nodes. About 25 ms later each boot loggedbatched source discovery did not establish the source sets — … for ALL N pending type(s), with N between 38 and 80.…6848f8-gf5dfcame 13 µs after the abandonment at 07:34:15.089Z:Npgsql.PostgresException: 42703: column n.created_by does not exist. Thenamespace:*/Sourcefetch UNIONs every schema'scodetable, and at least one of those tables lacks the V57 authorship columns (Systemorph/MeshWeaver.Plugins#2350).Test
OnePartitionCostsOnlyItsOwnTypesTesthas two cases.nodeType:Codefetch and the anchored reads of one type. The healthy type must keep its real source set, and the broken type must be absent. Negative control: I replaced the fallback with a rethrow, and the test then fails with the provider's42703fault, which is the pre-fix behaviour.AssemblePerQuerytruth table: a failed query and a contradicted empty set each withhold only their own type.Related bake/prewarmer suites, run locally in Release: 81/81 passed (
ABakeCompilesTheDefinitionItResolvedTest,EmptySourceSetNeedsAWitnessTest,SourceDiscoveryChunkTimingTest,AGatedSweepHearsItsOwnCompileTest,DeclaredSourcesMissingIsNotAnImageVerdictTest, …).MeshWeaver.HostingandMeshWeaver.Hosting.Testbuild with-c Release -warnaserror: 0 warnings, 0 errors.Not established / not done here
created_by, and why V57 missed it. That is tracked in Systemorph/MeshWeaver.Plugins#2350. Until it is repaired, the global pass will keep falling back on memex-cloud; with this change the fallback is per-type, not all-or-nothing.systemorphMCP is disconnected.Doc:
Doc/Architecture/SourceSetEstablishment, new section "The unit of 'I don't know' is the type, not the batch".After deploy: nothing needs recycling. This is boot-time pre-warm code, so it takes effect on the next pod start of an image that carries it.
Refs #5654, Refs #5544
Pairs-with: none — no public type or member removed (
ResolveSourcesoverloads unchanged; the new members are internal/private)🤖 Generated with Claude Code