Repository navigation
CompileWatcher never advances latestReleasePath after a successful compile, and its re-cut path treats a 10s bounded wait as release-missing #5263
Description
What is failing
The CompileWatcher's release post-condition (MeshWeaver.Graph.CompileWatcher, [ReleasePostCondition]) consumes a release request and compiles the NodeType successfully, but afterwards latestReleasePath still names a release cut for an earlier build (e.g. Edu/Lesson lastCompiledVersion 1811 → 1815 while the pointer still names the 1811-era release). For three NodeTypes (Edu/Lesson, Edu/Module, Collaboration/Review) the pointer was never advanced, so new instances keep binding a stale release instead of the build the node now advertises — the node's record names a build no release points at.
Probable cause
Confidence: medium-high. Two coupled weaknesses in the release settle path, both visible in the log's own wording:
- The settle that advances
latestReleasePathraces (or loses to) the compile-completion path: the compile succeeded and consumed the request, but the pointer-update either never ran or ran against a stale read. - The recovery path — re-cutting the release from the bytes the compile produced (the mechanism added for issue fix(chat): AI-menu "New thread" opens the composer in MAIN, closing the side panel #781) — gives up after a 10-second bounded wait and treats "did not land within 00:00:10" as failure. The log itself states the bound only stops the process waiting, not the create: the release node may well exist, but nothing advanced the pointer to it. In 3 of 6 samples the re-cut reported exactly this timeout, leaving the stale pointer in place with no retry.
So the re-cut is conflating "the create is slow" with "the release is missing", and abandoning the wait leaves latestReleasePath permanently stale for that type.
Impact
Six occurrences over ~30 seconds on a single portal pod (memex-portal-deployment-5c89c7f5f9-sddb5), affecting three NodeTypes. Not an outage, but it is silently wrong routing: instances of Edu/Lesson, Edu/Module and Collaboration/Review bind the previous release (e.g. Edu/Module/Release/20260922113336-L9MQ81KA) until a release is cut for the newer build. Workaround exists and is cheap: create/advance the release for the named build manually — the samples even list the expected release paths (e.g. Edu/Lesson/Release/20260922123830-fL_1fq3J) to check before concluding the release is missing.
Where to look
MeshWeaver.Graph.CompileWatcher— the[ReleasePostCondition]log site and the release-settle logic that advanceslatestReleasePathafter a successful compile.- The re-cut path introduced for issue fix(chat): AI-menu "New thread" opens the composer in MAIN, closing the side panel #781 — the 10s bounded wait should distinguish "create still in flight" (keep waiting / re-check the pointer) from "create genuinely failed", and should verify the pointer after the bound instead of declaring failure.
Related but not a duplicate: this fingerprint supersedes Admin/_LogIncident/b85ee940c57e9a79 (filed as #1105), which covered the compile-failure parking at the same log site; this incident is a different defect — the release pointer, not the compile.
Evidence
| Fingerprint | 5bee95af1e40ce61 |
| Category | MeshWeaver.Graph.CompileWatcher |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-5c89c7f5f9-sddb5 |
| Occurrences | 6 |
| First seen | 2026-09-22 12:38:39Z |
| Last seen | 2026-09-22 12:39:07Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Graph.CompileWatcher. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 12:38:43Z memex-portal-deployment-5c89c7f5f9-sddb5 fail: MeshWeaver.Graph.CompileWatcher[0]
[ReleasePostCondition] Edu/Lesson: a release request (requestedReleaseAt=2026-09-22T10:28:25.1509071+00:00) was consumed and this compile succeeded, yet latestReleasePath still names 'Edu/Lesson/Release/20260922113337-Gpgzxj2_' — cut for an EARLIER build (lastCompiledVersion 1811 → 1815). The settle's own create: the create did not land within 00:00:10 — the bound stops this process WAITING, not the create, so the node may well exist and nothing here advanced the pointer to it: LOOK AT 'Edu/Lesson/Release/20260922123830-fL_1fq3J' before concluding the release is missing. Re-cutting the release from the bytes this compile produced (no recompile) — see issue #781.
2026-09-22 12:38:52Z memex-portal-deployment-5c89c7f5f9-sddb5 fail: MeshWeaver.Graph.CompileWatcher[0]
[ReleasePostCondition] Edu/Module: a release request (requestedReleaseAt=2026-09-22T10:28:25.1519686+00:00) was consumed and this compile succeeded, yet latestReleasePath still names 'Edu/Module/Release/20260922113336-L9MQ81KA' — cut for an EARLIER build (lastCompiledVersion 2268 → 2272) — AND the release could not be re-cut: the create did not land within 00:00:10 — the bound stops this process WAITING, not the create, so the node may well exist and nothing here advanced the pointer to it: LOOK AT 'Edu/Module/Release/20260922123839-Ap2GdT7o' before concluding the release is missing. The node advertises a build no release names; instances will keep binding 'Edu/Module/Release/20260922113336-L9MQ81KA' until a release is created for it.
2026-09-22 12:38:56Z memex-portal-deployment-5c89c7f5f9-sddb5 fail: MeshWeaver.Graph.CompileWatcher[0]
[ReleasePostCondition] Collaboration/Review: a release request (requestedReleaseAt=2026-09-20T22:30:37.6175436+00:00) was consumed and this compile succeeded, yet latestReleasePath still names 'Collaboration/Review/Release/20260922113311-vlSwOE4U' — cut for an EARLIER build (lastCompiledVersion 1074 → 1084). The settle's own create: the create did not land within 00:00:10 — the bound stops this process WAITING, not the create, so the node may well exist and nothing here advanced the pointer to it: LOOK AT 'Collaboration/Review/Release/20260922123843-bzfmylxh' before concluding the release is missing. Re-cutting the release from the bytes this compile produced (no recompile) — see issue #781.
2026-09-22 12:38:57Z memex-portal-deployment-5c89c7f5f9-sddb5 fail: MeshWeaver.Graph.CompileWatcher[0]
[ReleasePostCondition] Edu/Lesson: a release request (requestedReleaseAt=2026-09-22T10:28:25.1509071+00:00) was consumed and this compile succeeded, yet latestReleasePath still names 'Edu/Lesson/Release/20260922113337-Gpgzxj2_' — cut for an EARLIER build (lastCompiledVersion 1811 → 1815) — AND the release could not be re-cut: the create did not land within 00:00:10 — the bound stops this process WAITING, not the create, so the node may well exist and nothing here advanced the pointer to it: LOOK AT 'Edu/Lesson/Release/20260922123843-fL_1fq3J' before concluding the release is missing. The node advertises a build no release names; instances will keep binding 'Edu/Lesson/Release/20260922113337-Gpgzxj2_' until a release is created for it.
2026-09-22 12:39:07Z memex-portal-deployment-5c89c7f5f9-sddb5 fail: MeshWeaver.Graph.CompileWatcher[0]
[ReleasePostCondition] Collaboration/Review: a release request (requestedReleaseAt=2026-09-20T22:30:37.6175436+00:00) was consumed and this compile succeeded, yet latestReleasePath still names 'Collaboration/Review/Release/20260922113311-vlSwOE4U' — cut for an EARLIER build (lastCompiledVersion 1074 → 1084) — AND the release could not be re-cut: the create did not land within 00:00:10 — the bound stops this process WAITING, not the create, so the node may well exist and nothing here advanced the pointer to it: LOOK AT 'Collaboration/Review/Release/20260922123856-bzfmylxh' before concluding the release is missing. The node advertises a build no release names; instances will keep binding 'Collaboration/Review/Release/20260922113311-vlSwOE4U' until a release is created for it.
Re-addressed by the current identity function: this incident inherited b85ee940c57e9a79 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/5bee95af1e40ce61. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 22, 2026
Still occurring: 21 more since the last update (27 total).
memex-portal-deployment-5c89c7f5f9-sddb5,memex-portal-deployment-7f4f9b7689-d67t5Recent log lines
Re-addressed by the current identity function: this incident inherited
b85ee940c57e9a79(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 23, 2026
Duplicate of #5057. Every sample on this ticket is from an image that predates #5201, the fix for #5057.
What distinguishes the two images. #5201 (merged 2026-09-22 10:45Z,
072f8e230c) made two changes:Stamped on the node as unreleasedBuildPath=…(src/MeshWeaver.Compiler.Pipeline/ReleasePostCondition.cs, the[ReleasePostCondition] … could not be re-cutlog site).None of this ticket's lines carries that ending: not the six at 12:38–12:39Z (pod
memex-portal-deployment-5c89c7f5f9-sddb5), and not the 17:56Z update (pod…-7f4f9b7689-d67t5). #5057's thread already established that ReplicaSet7f4f9b7689was serving before #5201 merged. The mechanism these lines show is the one #5201 removed: the create did not land within 00:00:10, followed by a re-cut at a fresh id. That fresh id could neither adopt the late landing nor stay idempotent with the create still in flight.Post-#5201 samples. Lines that carry the
Stamped …ending have a different, distinct failure.HubDisposedBeforeResponseExceptionhits the release create because it is issued from the NodeType hub that the compile is recycling. That is #5358, which is being fixed on its own. The post-roll read that closes #5057 is written on that issue. Following up there.