Skip to content

AdoptedSourceStamp logs 'the write did not converge' for stamps that converged — the commit mark moved inside the lambda (split off #1105) #5662

Closed

Description

@rbuergi

Split off #1105. The log-incident watcher re-addressed legacy fingerprint b85ee940c57e9a79 onto incident Admin/_LogIncident/4e5a9252dc04fca8, so #1105 has been collecting a different defect since 2026-09-22. This issue is that defect.

The line

MeshWeaver.Graph.CompileWatcher at Error, from NodeTypeCompilationHelpers.InstallAdoptedSourceStampWatcher:

[AdoptedSourceStamp] <NodeType>: the adopted build's source stamp was committed for request <T> and the request is still standing with IsDirty=true — the write did not converge; the next release request will recompile this type

Recent samples on memex (memex.systemorph.com): Manufacturing/Customer (2026-09-23 22:55:02Z), Manufacturing/Product (23:27:04Z), Hosting/TriageStatus, Hosting/TriageItem, Hosting/PlatformBuildInbox, Hosting/InstanceRequest, and Manufacturing/AuditDossier twice at 2026-09-24 06:10:37Z. All are on pod 56fbdcd48f-sz5st, with some on -4dpcn and 5dd79dc746-*.

Root cause

The watcher advanced its "committed" high-water mark inside the write's lambda, before the stamp was applied or the write landed. It then reported any later emission of the same request as a write that did not converge. Three ordinary outcomes produce that Error when nothing is wrong:

  1. Pre-commit emissions. The lambda runs on the data source's serialized write path. Emissions reach the watcher through the shared reduced stream. An emission of a state committed before the stamp can still be delivered to the watcher after the lambda has moved the mark. That state still carries the request, and it is not evidence of anything.
  2. Deferred judgement (Gate: a NodeType activated before its shared sources landed adopts NOTHING and is PARKED red — 'MISSING SOURCES: 3 of 3 queries matched NO nodes', live fingerprint e3b0c442 (empty) #4208/A StaleAdopted live-source compile is dispatched DURING its package's install: source discovery under-matches, CS0246 on committed siblings, and CompileWatcher PARKS the transient failure #4280). ApplyAdoptedSourceStamp returns the definition untouched, with the request standing, while the live source set is still short of the bundle's paths. The mark had already moved, so the next emission, which should fulfil the request, was reported as non-convergence.
  3. A write that faults after its lambda ran. This is already reported at Warning. The mark still claimed a commit.

Fix: #5460. The mark moves in the write's onNext, and only when the committed definition no longer carries the request (AdoptedSourceStampLedger.RecordCommit). The owner's own-node stream is one shared reduced stream (Workspace.GetCachedStream), so every emission committed before the stamp reaches the watcher before the write's own echo completes UpdateOwn. Pre-commit emissions therefore see the mark unmoved and re-enqueue an idempotent no-op. A request that is standing again after a confirmed consuming write is still the Error.

Evidence

Production sequence, from a Logs action on memex (Ops/logs-memex-20260924-1105-auditdossier-stamp, 107 lines, not cut). Pod 56fbdcd48f-sz5st, Manufacturing/AuditDossier, 2026-09-24:

time (Z) line
06:10:37.764 Prebuilt assembly … DECLINED before writing (#2813)
06:10:37.784 … no build resolves on this process … the version-compatible bundle (1.3.2 over 1.3) is ADOPTED … stale-but-serving (#3583) (writes RequestedSourceStampAt)
06:10:37.859 [AdoptedSourceStamp] … the source moved past the adopted build (bundle 97e51e789a1b732c, live 36baa77ebf872081) … StaleAdopted (the judgement ran)
06:10:37.889 Error … committed for request 06:10:37 … did not converge
06:10:37.912 Error (same)
06:10:37.917 [AdoptedSourceStamp] … the source moved past … StaleAdopted (judged again)

The two Errors land 30–53 ms after the judgement, inside the burst of writes an adoption makes. The pre-commit shape predicts exactly that.

Convergence observed on the other portal. Measured read-only on memex.meshweaver.cloud (see #5460): Store/Order logged this Error at 2026-09-22 22:54:24Z. 74 s later the node read requestedSourceStampAt absent and buildProvenance: AdoptedVerified.

Tests. AdoptedSourceStampCommitIsConfirmedTest covers the deferred-judgement shape with a negative control, which fails when RecordCommit counts any pass as a commit.

Not established

  • The pre-commit shape has no deterministic in-mesh repro. It needs a delivery interleaving the test mesh cannot force. The claim rests on the timing above and on the stream topology, not on a failing test.
  • I could not read the memex node state (the control-instance MCP is disconnected). So "converged afterwards" for Manufacturing/* on memex is inferred, not measured. The .917 second judgement means the request was standing again after .859. The likeliest cause is the seeder's cross-hub write being re-applied, which mints a new UtcNow, but I did not prove that.
  • Incidental, and not this defect: sz5st re-runs DECLINED → no build resolves → ADOPTED stale → StaleAdopted for Manufacturing/AuditDossier every few minutes (06:10:37, 06:13:25, 07:23:49, 07:54:06, 08:01:19 …; at 06:23:35 it logged the adoption is NOT judged yet). A new pod, 745745db7c-b2gjn, adopts with the judgement deferred because Manufacturing/Source/AreaReads never lands (A StaleAdopted live-source compile is dispatched DURING its package's install: source discovery under-matches, CS0246 on committed siblings, and CompileWatcher PARKS the transient failure #4280). Both belong to triage, not to this issue.
  • The "second writer mid-roll" hypothesis is not needed to explain these samples, and not refuted either.

Closing condition

After a roll to an image containing #5460, the Logs query {namespace="memex"} |~ "did not converge" returns no line from a pod on that image.

Activity

  1. added
    bugSomething isn't working
    sev:MMedium - secondary path broken, or easy workaround
    area:meshCore subsystem: the mesh - nodes, partitions, addressing, activation
    on Sep 24, 2026
  2. rbuergi commented on Sep 24, 2026

    @rbuergi
    ContributorAuthor

    #5460 merged. It is not live until memex (and memex-cloud) roll to an image that contains it; that roll is owner action. Closing condition unchanged: no did not converge line from a pod on that image.

  3. rbuergi commented on Sep 28, 2026

    @rbuergi
    ContributorAuthor

    Production closure verified 2026-09-28 19:25Z. Both current portal ReplicaSets (memex 85469d6f4b, memex-cloud 59bb87c9d5) run ci.9554/core 3e63924f, which contains #5460 merge commit 9a120100. A Loki query over those exact pods from the roll (12:30Z) to 19:25Z processed 162,105 log lines and returned zero did not converge lines. Positive control: the same image and window returned 28 [AdoptedSourceStamp] judgements across Main and Cloud, so the affected watcher did execute. This meets the stated closing condition. It does not claim to fix the separate StaleAdopted/source-delivery behavior tracked under #3583.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:meshCore subsystem: the mesh - nodes, partitions, addressing, activationbugSomething isn't workingfeature:node-compilationFeature: node compilationsev:MMedium - secondary path broken, or easy workaround

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions