Repository navigation
AdoptedSourceStamp logs 'the write did not converge' for stamps that converged — the commit mark moved inside the lambda (split off #1105) #5662
Description
Split off #1105. The log-incident watcher re-addressed legacy fingerprint b85ee940c57e9a79 onto incident Admin/_LogIncident/4e5a9252dc04fca8, so #1105 has been collecting a different defect since 2026-09-22. This issue is that defect.
The line
MeshWeaver.Graph.CompileWatcher at Error, from NodeTypeCompilationHelpers.InstallAdoptedSourceStampWatcher:
[AdoptedSourceStamp] <NodeType>: the adopted build's source stamp was committed for request <T> and the request is still standing with IsDirty=true — the write did not converge; the next release request will recompile this type
Recent samples on memex (memex.systemorph.com): Manufacturing/Customer (2026-09-23 22:55:02Z), Manufacturing/Product (23:27:04Z), Hosting/TriageStatus, Hosting/TriageItem, Hosting/PlatformBuildInbox, Hosting/InstanceRequest, and Manufacturing/AuditDossier twice at 2026-09-24 06:10:37Z. All are on pod 56fbdcd48f-sz5st, with some on -4dpcn and 5dd79dc746-*.
Root cause
The watcher advanced its "committed" high-water mark inside the write's lambda, before the stamp was applied or the write landed. It then reported any later emission of the same request as a write that did not converge. Three ordinary outcomes produce that Error when nothing is wrong:
- Pre-commit emissions. The lambda runs on the data source's serialized write path. Emissions reach the watcher through the shared reduced stream. An emission of a state committed before the stamp can still be delivered to the watcher after the lambda has moved the mark. That state still carries the request, and it is not evidence of anything.
- Deferred judgement (Gate: a NodeType activated before its shared sources landed adopts NOTHING and is PARKED red — 'MISSING SOURCES: 3 of 3 queries matched NO nodes', live fingerprint e3b0c442 (empty) #4208/A StaleAdopted live-source compile is dispatched DURING its package's install: source discovery under-matches, CS0246 on committed siblings, and CompileWatcher PARKS the transient failure #4280).
ApplyAdoptedSourceStampreturns the definition untouched, with the request standing, while the live source set is still short of the bundle's paths. The mark had already moved, so the next emission, which should fulfil the request, was reported as non-convergence. - A write that faults after its lambda ran. This is already reported at Warning. The mark still claimed a commit.
Fix: #5460. The mark moves in the write's onNext, and only when the committed definition no longer carries the request (AdoptedSourceStampLedger.RecordCommit). The owner's own-node stream is one shared reduced stream (Workspace.GetCachedStream), so every emission committed before the stamp reaches the watcher before the write's own echo completes UpdateOwn. Pre-commit emissions therefore see the mark unmoved and re-enqueue an idempotent no-op. A request that is standing again after a confirmed consuming write is still the Error.
Evidence
Production sequence, from a Logs action on memex (Ops/logs-memex-20260924-1105-auditdossier-stamp, 107 lines, not cut). Pod 56fbdcd48f-sz5st, Manufacturing/AuditDossier, 2026-09-24:
| time (Z) | line |
|---|---|
| 06:10:37.764 | Prebuilt assembly … DECLINED before writing (#2813) |
| 06:10:37.784 | … no build resolves on this process … the version-compatible bundle (1.3.2 over 1.3) is ADOPTED … stale-but-serving (#3583) (writes RequestedSourceStampAt) |
| 06:10:37.859 | [AdoptedSourceStamp] … the source moved past the adopted build (bundle 97e51e789a1b732c, live 36baa77ebf872081) … StaleAdopted (the judgement ran) |
| 06:10:37.889 | Error … committed for request 06:10:37 … did not converge |
| 06:10:37.912 | Error (same) |
| 06:10:37.917 | [AdoptedSourceStamp] … the source moved past … StaleAdopted (judged again) |
The two Errors land 30–53 ms after the judgement, inside the burst of writes an adoption makes. The pre-commit shape predicts exactly that.
Convergence observed on the other portal. Measured read-only on memex.meshweaver.cloud (see #5460): Store/Order logged this Error at 2026-09-22 22:54:24Z. 74 s later the node read requestedSourceStampAt absent and buildProvenance: AdoptedVerified.
Tests. AdoptedSourceStampCommitIsConfirmedTest covers the deferred-judgement shape with a negative control, which fails when RecordCommit counts any pass as a commit.
Not established
- The pre-commit shape has no deterministic in-mesh repro. It needs a delivery interleaving the test mesh cannot force. The claim rests on the timing above and on the stream topology, not on a failing test.
- I could not read the memex node state (the control-instance MCP is disconnected). So "converged afterwards" for
Manufacturing/*on memex is inferred, not measured. The.917second judgement means the request was standing again after.859. The likeliest cause is the seeder's cross-hub write being re-applied, which mints a newUtcNow, but I did not prove that. - Incidental, and not this defect:
sz5stre-runsDECLINED → no build resolves → ADOPTED stale → StaleAdoptedforManufacturing/AuditDossierevery few minutes (06:10:37, 06:13:25, 07:23:49, 07:54:06, 08:01:19 …; at 06:23:35 it loggedthe adoption is NOT judged yet). A new pod,745745db7c-b2gjn, adopts with the judgement deferred becauseManufacturing/Source/AreaReadsnever lands (A StaleAdopted live-source compile is dispatched DURING its package's install: source discovery under-matches, CS0246 on committed siblings, and CompileWatcher PARKS the transient failure #4280). Both belong to triage, not to this issue. - The "second writer mid-roll" hypothesis is not needed to explain these samples, and not refuted either.
Closing condition
After a roll to an image containing #5460, the Logs query {namespace="memex"} |~ "did not converge" returns no line from a pod on that image.
Activity
rbuergi commentedon Sep 24, 2026
#5460 merged. It is not live until memex (and memex-cloud) roll to an image that contains it; that roll is owner action. Closing condition unchanged: no
did not convergeline from a pod on that image.rbuergi commentedon Sep 28, 2026
Production closure verified 2026-09-28 19:25Z. Both current portal ReplicaSets (memex
85469d6f4b, memex-cloud59bb87c9d5) run ci.9554/core3e63924f, which contains #5460 merge commit9a120100. A Loki query over those exact pods from the roll (12:30Z) to 19:25Z processed 162,105 log lines and returned zerodid not convergelines. Positive control: the same image and window returned 28[AdoptedSourceStamp]judgements across Main and Cloud, so the affected watcher did execute. This meets the stated closing condition. It does not claim to fix the separate StaleAdopted/source-delivery behavior tracked under #3583.