Repository navigation
Release 3.0.0, then open 3.1.0 (tracking) #3355
Description
The plan (maintainer, 2026-09-05)
Release 3.0.0 first — only once every open issue is closed — then open 3.1.0. No rc labels, ever. A release is a promotion of a sealed continuous set (release.yml), never a rebuild. NuGet publication is retired (last packages on nuget.org: 3.0.0-rc13).
🎯 Release candidate — 3.0.0-ci.7917 is SUPERSEDED for the tag (20:00Z): the first sealed set cut from #3470's merge or later
A live user-facing regression sits in 7917, caught by the Plugins pin move (#1415, Portal hosts (shard 1)): core #3384 (3ef661981) made an undeclared catalog Format mean node-repo for both readers, so every undeclared package.json catalog renders an EMPTY page — two migrated PluginCatalog tests rendered the shell with zero cards; the same shard passes on the old pin. Shape 7 (behaviour behind an unchanged signature), the second instance today beside #3465. Status 22:1xZ (via meshweaver-58): the #3465 gate shipped as #3487; #3489 records four implementer-obliging shapes it still cannot see, each with a positive control — two cheap (an interface gaining a base interface; a protected abstract on a public abstract class), one disproportionate, one undetectable by construction (shape 7 wearing an interface). The interface-addition hole is narrowed and documented, not closed. Fix: #3470 — an undeclared format on a local checkout is detected from the layout (declared wins; <subdir>/<id>/package.json with no <x>/index.json Space root anywhere ⇒ package-json; everything else, URLs included ⇒ the shipped default), all three readers through one rule, #3384's guard kept. The tag is cut on the first set whose seal sub-jobs read success at a commit ≥ #3470's merge (#3470 MERGED 20:17Z, 85ab1dfbe). CD 7937 on that commit FAILED its seal at 21:25Z — the bake gate's Chess install timed out ([FAIL] Chess — install: TimeoutException, the #2543 family; the framework-identity check itself read MATCH) — so the candidate is the next slot run to seal (7938+, all ≥ #3470). 21:45Z: the rule tightens to ≥ #3480 as well — #3480 (in CI) turns the #2543 silence that killed 7937's own seal into a retryable NACK; a release set that still parks writers for 31 s on this arm is not the set to promote. So: the first sealed set at a commit ≥ #3480's merge. #3480 MERGED 21:58Z (8ebce31e8) — CD 7942 on that commit was CANCELLED at 22:20Z by #3482's merge (the one pending slot moves with every merge); 7943 (74eb540c5) holds it now and is itself replaceable while 7941 (b8accc2b2, in flight, no #3480) runs — so the candidate is whichever pending run is alive when 7941 completes — **and 7946 (ddb95e65f, SCHEDULE-triggered 22:25Z) is in flight NOW beside 7941: a schedule run bypasses the one-pending-slot rule (7940 did the same at 21:24Z), so it can seal the same commit 7945 is pending on. 8ebce31e8 and 85ab1dfbe are both ancestors of ddb95e65f (verified by meshweaver-58), so 7946 carries #3470 and #3480; 7946 FAILED 22:55Z before its seal — Module tests (MeshWeaver.AI): MeshPluginTest.FullCrudWorkflow_CreateGetUpdateDelete waited out 20 s for a node whose authoritative NotFound was in the log 0.1 s after the delete → #3498 (the re-ask arm of JsonSynchronizationStream logs a non-transient failure as resubscribe failed and drops it — #1029's other door; identical code in 7917/7926/7938; 1 in 6 runs of that job; local 24/24 because the race needs the change-feed event to beat the owner's teardown notice). Fix + deterministic test = PR #3501 (23:1xZ; negative control measured: red 30 s without the fix, green 239 ms with it; Data.Test 489/489, Documentation.Test 304/304). 7941 FAILED 22:56Z at its seal — [FAIL] Hosting — install: TimeoutException with writes parking on Hosting/DeploymentStatus, ModuleReportInbox, Issue → #3499 (a package ROOT hub disposed mid-install under a reconcile: 7937 Chess, 7941 Hosting, Plugins#1422 Chess; filed 23:02Z by another session; meshweaver-58 takes it). Interim Plugins pin move — Plugins#1429 (7917 → 7938) MERGED 01:1xZ (shard 2 red once on #1394's FIFO race — C,A,B, recorded on #1394 with the pin-unchanged evidence — then green on a recorded re-run at the same head: one-against-one, not clearance): the pin sat 90 of the 120-commit staleness bound behind core main at ~15 merges/h (meshweaver-bd's measurement, verified by 58), ~2 h from redding every open Plugins PR. 7938 is strictly ahead of 7917, carries #3470, predates #3475/#3480/#3498 — it buys hours, it is not the wave; Plugins pins the tag's set afterwards. Verified: pin checker, --check-tags against ACR, caller schema vs the lanes at aa40758 (the lanes changed: #3401, #3338). ACR (58, 00:5xZ): five manifests hand-locked writeEnabled:false + deleteEnabled:false (two are 7917's tester/portal, three Education's) — a 3.0.0 promotion RETAGS, and a retag is a write; moot for the tag's set (new digests, unlocked), keep them as purge protection. #3507 (merged, decccecee) is the automatic replacement (01:00Z, --delete-enabled false only) but CANNOT RUN: locking needs repositories/metadata/write (Container Registry Repository Writer), held by none of the registry's three service principals — the last production action on #3438 is that role grant; its report-only run reads 7 pinned manifests unprotected, incl. the 7926 portal/migration pair both deployments run. Tag rule now: the first sealed set ≥ BOTH #3501 (#3498) and #3505 (#3499) — #3501 MERGED 23:35Z as 99bafdc10 (queue; CI green, Copilot approval recommended); #3505 MERGED 23:59Z as 155b4fa17 (its correction push 5e2cb82 comment/doc only). 🚨🚨 LIVE OUTAGE, found 00:3xZ by the (working) sweep on memex.meshweaver.cloud — the COMMERCIAL portal: Store/Catalog, Store/Order, Store/Plugin, Store/Maintenance in compile Error since 22:39:32Z. Every missing symbol is #1413's Payments abstraction (IMessageHub.PaymentProvider, CommerceBaseUrl, PaymentDelivery, PaymentCheckoutEvent/Request, PaymentRecurrence, cref IPaymentProvider); the instance compiles on s2f22764 (rc9 line: no core #3434, no Payments module; module set pinned at boot). Its Store SOURCES synced to #1413 at 22:39Z ahead of its runtime. memex.systemorph.com has the SAME four in Error since 22:38:39Z — both portals within 70 s of Plugins main run 34061098155's Register the publication with memex (22:38:18Z), whose commit 8d4920c93 PREDATES #1413: the registration syncs sources at main's HEAD, not at the publication's commit. Paying customers cannot buy or install. Plugins#1430 filed (both portals + trigger); maintainer paged; NO mutation done. Remedies (maintainer's call): roll memex-cloud to a set with #3434 + Payments (= the tag's set), or restore_version the four types to pre-22:39Z + hold the Store sync until the roll. 🚨 Roll caveat per portal (meshweaver-64, measured on a 7926 roll attempt): memex.systemorph.com's NodeType bake gate REFUSES readiness on any new set until #3485's content gap is repaired (Feedback/Feedback/Source/FeedbackContent missing there; present on memex.meshweaver.cloud at v9) — that portal was reverted to ci.7693. So the roll remedy reaches the commercial portal now and the staff portal only after #3485; the Store outage on the staff portal needs restore_version + a sync hold meanwhile. 🚨 RULE UNTIL THE ROLL (02:5xZ), TWO CLAUSES (91): a Plugins change that needs a core seam is gated by (1) the LANE PIN — MW_PLATFORM_REF must carry the seam or CI is red by construction (7938/aa4075832 does NOT carry #3450's EmailSendAsCapability; git cat-file -e <pin>:<path> is the instrument, never a date) — and (2) the PORTALS' RUNNING framework (s2f22764) — or the next registration syncs main's HEAD into both instances and they cannot compile it (#1430). Different staleness questions, different owners; a repo can fail either alone. 🚨 The re-pin lands BEFORE the roll, so a held PR turns GREEN while the portals would still break — a green lane is not the un-draft signal; only #3355 recording both portals on the chosen set is. 58's ledger closed: #3494 merged 4c89dc6b6, #3413 closed, #3515 (#3472 follow-up) merges itself. First application: meshweaver-91's mail-module adoption of #3450's ObserveSendAsCapability/EmailSendAsCapability (cores ≥ 7962) is **Plugins#1431 (DRAFT, auto-merge off; release condition and both absence reasons in its body; 155/155 at 6577052, guard verified by mutation; core #3450 stays open — the two Markdown.Export sites need a re-probe design) and stays DRAFT until both portals run the chosen set; mail module itself verified 146/146 at 7962 (Plugins e8f315bf vs core 6577052), and the un-adopted seam is pre-existing behaviour (the 'not connected' lie on a timed-out Graph read, Plugins#1398), not a cut blocker. Durable half: a package source sync must not land sources the instance's module set cannot compile — a new blocker by the rule, and the cross-repo-halves shape (#3137/#3276 family) one layer up: source halves reach instances by SYNC, dependency halves only by ROLL. 🚨 #3511 (58, 00:2xZ): the pre-prod sweep AGENTS.md prescribes — search 'nodeType:NodeType compilationStatus:Error' — CANNOT match anything: QueryEvaluator.GetDirectPropertyValue reflects the selector against MeshNode, which has no CompilationStatus (it lives on NodeTypeDefinition inside Content) and there is no Content fallback; same evaluator in process for Postgres. A code read; the falsifying check is on the issue. Until resolved, an EMPTY sweep after the roll is NOT a clean mesh — the AGENTS.md instruction itself needs correcting. #3510 (00:1xZ, THIRD seal-killing mechanism, from CD 7950 = #3480 in, neither fix): the root of the package BEING INSTALLED (Hosting, 145 files) was disposed at 23:37:51Z under its own install — every write-through owner under it vanished (ADVANCE_WITHOUT_HANDOFF ×9), the four nodeops creates never replied, the install ran out its 10-min bound; all of #3499's and #3498's discriminators read ZERO. It is #3499's original sentence, which the count-based falsification does not reach (the discriminator is whether the recycled root is the one installing, not how many recycle); #3505 has no verdict to classify, #3506 is a different phase (intake at Started). Disposer not in the log; rebind recycle is the leading hypothesis (never measured). ROOT FOUND (58's agent, 00:5xZ, code-proven): the disposer is PackageInstaller.SettleRetypedRoot — the installer's own ordered recycle of a plugin root (dynamic type Store/Plugin), fired after adoption for EVERY plugin package install, 4/4 in 7950 with +2…15 s lag, succeeding every time (root back inside RootReadyTimeout), present 7–12× in every run INCLUDING sealed 7938. So the recycle is not the defect. The defect is a totality hole on the upsert lane: MeshExtensions.cs ~4740/4930 subscribes to the inner CreateNode/UpdateNode with onNext/onError only — an owner disposed under the request neither emits nor errors; no reply, no diagnostic (zero [CreateOrUpdate] lines in 7950), no WriteVerdictBound equivalent — same class as #3033/#3194. Fix NOT buildable tonight (58's agent, 01:1xZ, and rightly stopped): the UPDATE leg is already total by measurement (RegisterOwnerDisposingNack mints OwnerDisposing, the writer re-enqueues, caller gets success); the uncovered leg is the CREATE leg (UPSERT_READ absent → create, exactly 7950's trail) and it could not be driven — parking the partition root's primary stream does not put a create behind it (two fences, 36 s each). Producing it needs a park on the seam the inner CreateNode traverses, or extracting the inner-write subscription into a pure composition as ArmPatchAckWatcher was for the patch lane — real work, not tonight. PR #3512 (draft, 7dd16fad0) carries the attribution half: LogInformation at SettleRetypedRoot's recycle post + a regression pin + doc; it cannot help 7962 (a log line names recycles only in runs that start after it merges). #3510 stays a real defect with a known root, a named reproduction gap and NO release gate; the question is only whether 7962 seals without it. 01:0xZ: 7959 (8ef74ed66, #3501 in / #3505 out) FAILED its seal on #3510's shape — Hosting install timeout, all counters zero, ADVANCE_WITHOUT_HANDOFF path=Hosting/… ×7, 10 HubDisposedBeforeResponseException. Second occurrence (7950, 7959) in the THREE runs that started with #3480 in, against one seal (7955): #3510's seal-blocking rate is measured at roughly 2/3, both times on Hosting (145 files, the largest dynamic-root package). 7962 (6577052de, all five) is IN FLIGHT since ~01:02Z, seal ~01:55Z at best; merge hold lifted. If it seals → tag with #3510 open and known; if it fails on this shape → the release is gated by a defect whose fix is not buildable tonight (create-leg totality, no reproduction yet) — the maintainer's call, measured. Decision 01:0xZ (with 58; maintainer may overrule): the rule STAYS ≥ #3501 + #3505; #3510 joins it only if 7961 fails on its shape. Grounds: 7950 is n=1 on that shape and the recycle is in every run including sealed 7938, so the seal-blocking rate is unmeasured, and #3510's fix costs a merge that replaces the pending candidate. 58's #3509 (the #3472 adopt-time identity guard) landed at 00:14:57Z (merged_at; its merge commit 6577052de carries a 00:06:37Z committer date — the queue writes the commit when it builds the group, minutes before it lands) three seconds before its draft, so the pending candidate is 7962 (6577052de, created 00:18:33Z, carries #3470 #3480 #3501 #3505 #3509); 7959 (8ef74ed66, in flight since 23:48Z) ends first, then 7962 starts with the queue quiet; its seal (~00:50Z at best) or its failure shape decides. 11 of the last 12 completed CD runs were cancellations — one pending run starts per in-flight completion, which is why candidates were scarce, not why seals failed (7941/7946/7950 each ran to a distinct mechanism). A BLOCKER by the transitive rule; 7960 can meet it — candidate reads add ADVANCE_WITHOUT_HANDOFF under the installing package + HubDisposedBeforeResponseException. Both preconditions are on main: the tag's set is the first main-cd run to START at ≥ 155b4fa17 and seal with both sub-jobs success; read VERDICT_TIMEOUT, both re-enqueue arms and the bake wall clock vs 7938's 27m41s before the conclusion. #3499's root (meshweaver-58, measured over all four bake jobs with tracing): NOT a root disposed mid-install — [QUIESCE-TIMEOUT] peaks in the run that SEALED; the discriminator is VERDICT_TIMEOUT (7937: 5, Plugins#1422: 10, 7938: 0, 7941: 0). In 10/10 trails the patch had already committed and only the durability leg faulted on a closing lifetime scope, which ClassifyPatchException read as Unknown = terminal → the writer reported a lost write for a landed patch. #3505 changes classification only (HubDisposingException.IsDisposedContainer consulted first, over the exception graph); #3506 filed for #3261's intake window (6 of 9 trails, benign). 7941's failure carries two resubscribe failed lines and zero VERDICT_TIMEOUT — it is #3498's shape, so the two fixes cover two distinct mechanisms. Exposure on the candidate seal (58, 23:5xZ): #3505's reclassified fault happens once the hosted hub's scope is closed, i.e. past DisposeHostedHubs, so its verdict routes through the LATE re-enqueue arm by construction — the exact arm #3477 measured going silent (LATE_NACK_REENQUEUE … attempt=1, nothing after, 1/330). The candidate's failure mode is therefore SILENCE, not a red: read the bake job's wall clock against 7938's 27m41s and count LATE_NACK_REENQUEUE lines without a terminal; #3477's request-id link is the first post-tag follow-up, deliberately not folded into the release-blocking fix.; the first run ≥ 8ebce31e8 whose BOTH Plugins: bake + seal sub-jobs read success names the tag's set.
🎯 THE TAG'S SET, sealed 01:38:49Z — 3.0.0-ci.7962 (CD run 34069402974, core 6577052de; BOTH Plugins: bake + seal sub-jobs success; publication _complete, 36 bundles, both storage targets; identity sb6d0f798bf359b3aa990173cb39757a2; carries #3470 #3480 #3501 #3505 #3509): tester sha256:bfd5e5718c5796516e26728fa907ecd1fc99d649366fbaaa91c4c297ca5bdc76, portal sha256:fbfd161c019349113283bde89b2664e961efcd96b8ac6906632a76abe934c21f, migration sha256:fcf348228bab8cbfaa955e5b8776705fd1b6524be99688a3858e897815c2740e. Bake 11m43s (222 content-addressed cache hits, 2 compiles). Counters: VERDICT_TIMEOUT=3 (01:33:24Z on Hosting/Issue, Hosting/LogEntry, Hosting/DeploymentStatus _Activity/compile-state, bound 31 s, no terminal, NO re-enqueue marker — which is #3510's CREATE leg, not a gap in #3505: the update leg always re-enqueues (RegisterOwnerDisposingNack), and the paths are 7950's exact node family. So #3505 is exonerated and #3510 has SIX occurrences (7950, 7959, 7967, 7968, 7976 failed; 7962 passed) — of the TEN runs that started since #3480, five sealed (7955, 7962, 7964, 7969, 7974) and five failed, every failure this shape, Hosting every time: one started run in two; 7967 is the first run with #3512's line and the transcript now names the recycle itself (recycling root Hosting … at 03:44:19.082, 10.6 s before the quiesce cancel of the pending CreateNodeRequest), whose own text — 'work in flight beneath this root is answered by the teardown, not abandoned' — is the contract the create leg breaks; the first owner loss precedes the recycle by 5 s, so a child teardown is a second seam to park — what separates them is whether the un-terminated write is one the installer waits on; the install passed here), resubscribe-failed=6, both re-enqueue arms 0. ALSO SEALED 02:32:25Z — 3.0.0-ci.7964 (run 34072269844, core 93dd88941 = 6577052 + #3512, the #3510 attribution line): VERDICT_TIMEOUT=0, resubscribe-failed=3, arms 0; tester sha256:3f8ca46be368180bd11290562624240f2b5b15c9630a77c37a1832c1b26796d4, portal sha256:da16394eb37ebff7c14f68cd9edbc593268e71983fbfebc1d787002ad1063f3a, migration sha256:6063fff80b3cfc695de518c03571fc7fd2e5be672990ec4ae72a228426c6dae4. ALSO SEALED 04:30:18Z — 3.0.0-ci.7969 (schedule run 34080162460, core e64689e7f = 7964 + #3494 (message-trace collection in the shard lane) + #3515 (#3472 follow-up: the review findings + #3511's re-framing); VERDICT_TIMEOUT=0, resubscribe-failed=1, arms 0; bake 04:04:21–04:30:18): tester sha256:539fec3af4a4d93d4d61a5b14d36acd916a183ead58a6c74e284c4dc51727354, portal sha256:a25a694d87825df7d3632337dcff540b0046dfe8682fae1c7ab51d789d309978, migration sha256:3b6eec86b605248f2704d1c69fa10b9ba3c4706a7a8cd6dd5b07c9e18f60cbb4. ALSO SEALED 05:33:26Z — 3.0.0-ci.7974 (schedule run 34083637477, core c7fef7a2d = 7969 + four Dependabot bumps only: NSubstitute 6.2.0, ModelContextProtocol 2.2.0 + .Core, Microsoft.Reactive.Testing 7.0.0; ALL counters zero incl. resubscribe-failed; bake 05:06:24–05:33:26): tester sha256:b7119dd1d339d836401eb565e6612572e95d538aca0c57aa6a666a6369a6ea5f, portal sha256:aca3211151ebc5d65db6cab2690c93a1f61437c4ef9486bad79e6ffe7545e845, migration sha256:3a5aa1189b3743ea5907d3c1e28826d08371941660106b05c97ed2cb44d86b99. Started-run tally since #3480: 10 started, 5 sealed (7955, 7962, 7964, 7969, 7974), 5 failed (7950, 7959, 7967, 7968, 7976 — all #3510). By the rule the set is 7962; 7964 (+#3512) and 7969 (+#3494 +#3515) are the newer sealed sets — the maintainer picks; the CUT itself is the maintainer's decision. 7967 (4c89dc6b6, between them) failed on #3510. #3494 un-drafted at the seal as promised and MERGED as 4c89dc6b6 after its review fix (the hard-coded temp path). The schedule twins 7963/7965 on the same commits failed at 'CD delivered, or had a good reason not to' ('main is GREEN and has an INCOMPLETE image set, and this reconcile published nothing') — the hourly reconcile racing its own in-flight workflow_run twin; not a seal failure — filed as its own issue (reconcile checks for an in-flight run on the same head and no-ops green; the genuinely stuck case keeps its red). 7966 (schedule, 93dd889, after the twin finished) succeeded as exactly that no-op. Argument for 7964 over 7962 (58): it carries #3512, so a #3510 recurrence on the chosen set names its recycle in the transcript; 7962 cannot. NEWER SEALED FALLBACK, 00:27Z — 3.0.0-ci.7955 (schedule run 34066838312, core 1340e351c = #3500's merge; BOTH Plugins: bake + seal sub-jobs success; bake 23:58:01–00:27:40, 29m39s): tester sha256:fd2790e72e80356e69f1c3a2dee26d7da6a6f1fc24f3d00ee8c240396dcb80d8, portal sha256:3864c2b047cbc886606a839864b71afabc6d909b4e45ade2898cfe8c3a89dad7, migration sha256:2f6a4c6fbedd19b7b0354e67f9a96a399921c24f083a4f4bc9f8f06a798cb073. Carries #3470, #3480, #3500; NOT #3501/#3505 (99bafdc10 and 155b4fa17 are not ancestors). Counters: VERDICT_TIMEOUT=0, both re-enqueue arms 0, resubscribe-failed=1 with a green seal — #3498's line can appear without killing the seal. Supersedes 7938 as the fallback if the tag cannot wait for 7962. Sealed FALLBACK, 22:1xZ — 3.0.0-ci.7938 (CD run 34058471459, core aa4075832): Promote, Verify, Bake platform content and BOTH Plugins: bake + seal sub-jobs success on attempt 1, zero failures in the run; tester sha256:4025917c502d2af056351e36f084b62967d7ddd8c66c1e5a8371b69d4f57b48a, portal sha256:7f7a9a5fb7887507aa3c43f6bb2475bb5952165cee8a12291803db04c74a62b6, migration sha256:bc08c1d023bbf87d16108216ef760d2ca46ad88826afd939d06d3edd9b7594e8. Carries #3434, #3440, #3446, #3470; does NOT carry #3480 (the #2543 silence arm that killed 7937's own seal is unfixed in it). Cut on it only if the tag cannot wait for the next seal past #3480. (CD 7940 on 62039c996 FAILED at 21:5xZ on infrastructure — GitHub's OIDC token endpoint timed out during the migration image's Azure login, no promote — not the content; 7941+ retries.) Plugins lands on 7917 now after all: at 22:11Z the two fixtures were made to declare Format = "package-json" (2f1471a96, honest — they ARE package.json catalogs — but not the users' fix), so #1415 can go green today and Plugins main publishes again; all five satellites then sit on 7917 and re-align onto the tag's set at the tag. Verified by inspection that none of the four exercises the catalog node. 🚨 Until the tag re-alignment, a satellite that ADDS a PluginCatalog node over a package.json checkout must declare Format: package-json, or it inherits 7917's dark render silently.
The 7917 record, for the fleet that is on it:
| set | 3.0.0-ci.7917 — CD run 34048513298; Promote, Verify every image shipped, Bake platform content and BOTH Plugins: bake + seal sub-jobs success, attempt 1 (read with jobs?filter=all) |
| core | 1b5350d547473a5e2ca81e793e774cc962acfeb3 — carries #3434 (IPaymentProvider, Plugins#1413's contract), #3440 (every instance reports what it runs), #3446 (EA credential seam, #3433); NOT #3460 (unmerged) |
| tester | mw-plugin-test:3.0.0-ci.7917 = sha256:642f686fed743289ca1211b73bf8091c5d7deaaa3d4435746a4868b653e4fc8c |
| portal | memex-portal-ai:3.0.0-ci.7917 = sha256:31aab07d271aedc257700780417a060a6511eb8271b2e3982d47145a4fa5e9c9 |
| migration | memex-migration:3.0.0-ci.7917 = sha256:23e088df938f9a740dfe1f44a805885f94372f736839b440fd1a0717478a1b35 |
ci.7913 (6cddde50f) sealed clean too and carries the same three merges; 7917 was named for eleven more commits of staleness headroom. ci.7868 is superseded (predates all three). The requirement was stated as a COMMIT ("cut from 6cddde50f or later") and the set named only after its seal — the one-pending-slot rule cancelled 7910 → 7911 → 7912 → 7915 → 7916 → … → 7926 under valid commits today; #3452 (merged) writes the protocol into the CD contract and the release skill.
ONE satellite pin move, landing on 7917 (19:05Z). Plugins' MW_PLATFORM_SET was 3.0.0-ci.7859 (bbcb22f25), past the 120-commit staleness bound since ≈17:30Z (147 commits at 17:45Z; Plugins main published nothing until the move). State of the wave, driven by the release session on the drafts the issue-fixing session staged:
| repo | PR | state |
|---|---|---|
| Plugins | #1415 | MERGED 20:5xZ (main ef51fd50) — 14 sites + bd's one-file test-double hunk (core #3446 widened IEaGraphAuth; #3465) + the two catalog fixtures declaring package-json (2f1471a96, third session; honest, not the users' fix — #3470 is). Plugins main publishes again; its first run on 7917 is being read job by job; on green through the gates, Plugins#1413 (#1328) and #1416 un-draft |
| SocialMedia | #138 | MERGED — four-part commit (lane sha, platform-image + portal digest, webhook pair, tester digest) |
| Manufacturing | #56 | MERGED — same four parts |
| Crm | #56 | MERGED — four parts; its first gate red was the ACR token lacking memex-portal-ai pull (every sibling map had it): crm-ci-pull-scope-map extended, re-run green |
| Reinsurance | #170 | MERGED 19:4xZ — #169 (ci.7896) closed as superseded; all twelve sites, both pin gates ✓; its gate passed on 7917 without the #2543 shape this run |
| Education | — | out (#3438 purged its tester image) |
Every caller was schema-checked against its lane at its own sha (no missing required input, nothing undeclared); digests resolved from ACR by tag. #3454 (three satellites verify nothing about the digest they pin) and #3461 (two lanes publish one identity prefix) stay open as tag-time decisions.
#2543 — arm named (21:40Z): Reinsurance#169's gate showed twelve trails ending at PATCH_ECHO_SEEN (+2ms); #3464 (merged) added the flush stages and pool numbers; CD 7937's bake host (core 85ab1dfbe, carries #3464) then showed 13 stalls, every trail ending at PATCH_ECHO_SEEN, zero PATCH_FLUSH_SUBSCRIBED, pool pendingWork=0 — the starved-pool arm is refuted; the only code between those stages is the ack watcher's own onNext, where a synchronous throw (a closed lifetime scope's GetService, #3196's shape) ESCAPES the echo subscription and produces exactly that silence. #3480 (CI): the arm catches, NACKs with the classified error and records PATCH_FLUSH_FAULTED_SYNC building <exception>; the flush guards its service resolution. Not yet proof of the closed-scope cause — the next occurrence's trail names the exception.
#3477 (21:20Z) — a write-loss-class race, not attributed to the pin: Plugins main's first run on 7917 failed LateNackReenqueueTest 1/330 on shard 3 (late NACK fired, re-enqueue issued, nothing landed or answered in 45 s, no log after LATE_NACK_REENQUEUE); shard 3 was green ×4 on the old pin, and the test passes 28/28 locally across BOTH pins (linked cores verified; 12 of them at DOTNET_PROCESSOR_COUNT=4). Suspect, unconfirmed: #3321 step 3. Not a tag blocker at n=1; the issue names the instrumentation the next occurrence needs (link the re-enqueued request's trail). The un-draft signal waits on the run's gates, not on this shard. #1328 CLOSED 22:17Z — Plugins#1413 merged as ea9aa9934 (the Store binds IPaymentProvider; a payments-free mesh compiles all 17 Store/* types; with the module 63/63), which is why every satellite's registry-modules: … Stripe workaround is now revertible and why Plugins main's module versions moved 17 modules at once. #1396 resolves only PARTLY (4 of its 8 declines are MeshWeaver.Maps) — stays open. Held for the cut, by the shared-action rule: core #3494 (message-trace collection in the shard lane, #3413) is a deliberate draft — gh pr ready 3494 the moment the set seals; and #3450/#3473's Plugins half needs a pin bump, which moves only after the tag and only if the maintainer wants it. Gate blind spots filed while landing #3487 — NOT tag blockers, disposition agreed with meshweaver-58 22:4xZ: #3492 (a UTF-8 BOM hid 16 public types from the removal gate) and #3489 (four measured shapes the addition half cannot see). Both are pre-existing: the gate has been blind for as long as the BOMs have been there, so the cut changes nothing about the exposure, and each is cheaper to fix after than to fold into it. Maintainer may override here. SENT 22:16Z: run 34061098155 (8d4920c93) green through Validate, Compile every NodeType and the gate shard, zero red jobs — meshweaver-58 un-drafts #1413 and #1416. Run completed/success 22:38Z with publish-bake / Bake + publish and Register the publication with memex both success — Plugins main PUBLISHES again on 7917.
Plugins#1426 (22:07Z) — a false red on every intermediate merge-queue commit of Plugins main, fix Plugins#1427 MERGED 23:08Z (2c57a52c5, review's clean-tree condition included): gen-manifests.py derived each commit's module versions against the remote's LIVE tip, so a four-PR queue group read two commits that never touched Essentials as "lock 1.1.1, requires 1.1.3" and every gate downstream went red with them; the tip (6554274d) validates clean. Fix: a checkout that is an ancestor of the tip derives against itself, both witnesses (trunk lock + tags --merged), with a self-test whose control proves the fixture reproduces the red. Not a tag blocker (nothing published wrongly; the tip's run is the one that publishes), but it is the reason a "newest main run" watcher reads Plugins main as broken. Fleet: Education/Reinsurance/SocialMedia/Crm carry their own diverging copies with the same witness; Manufacturing's predates it — port-or-centralise is a triage call recorded on the issue.
🚨 NEW TAG BLOCKER 20:30Z — #3472 (portal loads NodeType assemblies compiled for a different framework identity). On memex.systemorph.com Crm/Offer and Crm/Opportunity are dead since 19:28Z: the portal (running rc9.ci.7647, identity s2f22764…) adopted a prebuilt bake published under sc273ee3… while its NodeTypes still self-report compilationStatus: Ok; every deal and offer page on the client portal is affected. Measured 20:35Z: Crm's post-merge publish-bake (run 34053641389, 19:1xZ) published under s556a0d43… — 7917's identity — so the adopted sc273ee3… assemblies are neither the running framework's nor today's publication's; CORRECTED 22:06Z (#3472 comment, measured from the bake logs): sc273ee3… is the identity of 7926, a sealed-green set (7859's bake never ran; 7917 = s556a0d43…, 7938 = s6765d418… — the identity moves on nearly every run), so a satellite pinned to set N publishes under an identity only an instance running set N reads. Blast radius measured 01:0xZ (58's agent, read-only, denominator-controlled): memex.systemorph.com holds 41 NodeTypes stamped with the refused identity sc273ee39f… (19:18–19:19Z, the two-image window), ALL reading Ok, 0 cellSurface; 85 carry s2f227642d… (the running identity, inferred). Crm/Offer and Crm/Opportunity healed only because someone OPENED them (instance-activation self-heal); nothing re-drives the other 39 (DecideStaleBuildAction answers Skip). After #3509 rolls, six load paths that used to load those bytes REFUSE them and the self-heal fires on first open. The release ROLL to a new identity recompiles all of them at boot — the roll is the cleanup. The four Store Error types cannot flip to Foreign (non-Ok passes through; rogue-identity ∧ non-Ok = 0). With the !toolchain mvids differing between the two pairs of types, the reading that fits is two memex replicas on two images (one s2f22764, one sc273ee3) each compiling and neither loading the other's output — #3395's ping-pong, which means memex was rolled today (Ops/Status still says rc9.ci.7647 from 09-03 — stale). First measurement for the owner: per-pod images on memex; if two are live, finish or revert the roll to ONE image, then evict the two prebuilts; #3440's inventory report answers the per-pod identity the moment memex is deployed with it. The tag waits for the root cause or a refusal guard on identity mismatch.
Cut the release (maintainer)
On a commit on main whose CD run shows Promote, Verify every image shipped and Plugins: bake + seal = success:
git tag -a v3.0.0 -m "MeshWeaver 3.0.0" <sha> && git push origin v3.0.0release.yml promotes the set to the clean 3.0.0, mirrors it to GHCR, records _releases/3.0.0, publishes the GitHub Release from Doc/ReleaseNotes/3_0_0, and opens release: PlatformVersion 3.0.0 → 3.1.0. Merging that PR is the cut to 3.1.0.
🚨 Ordering, accepted and documented: until the clean 3.0.0 exists, a 3.0.0-ci.<n> build sorts below the 3.0.0-rc* images already in ACR (SemVer compares ci < rc as text), so a Continuous install on an rc build stays put; the clean 3.0.0 outranks every rc and moves the fleet. Two 3.1.0-ci sets (7832, 7835) exist in ACR from the hours the props read 3.1.0; an install that took one will skip the clean 3.0.0.
Done (2026-09-05)
| MeshWeaver #3354 | the promotion lane release.yml, NuGet lanes retired, docs |
| MeshWeaver #3356 | release notes page |
| MeshWeaver #3358 | build-project --bind-to-image: modules carry the image's binding identity |
| MeshWeaver #3364 | the probe captures its output (no grep -q on a live pipe) |
| MeshWeaver #3366 | PlatformVersion 3.0.0, 3.0.0-first docs |
| MeshWeaver.Plugins #1366 | pin readers accept -ci.<n> |
| MeshWeaver.Plugins #1367 | AssemblyVersion follows the platform in hand |
| MeshWeaver.Plugins #1368, #1371 | pins onto sealed 3.1.0 sets; self-update fixtures relative to the installed version; lane refs aligned |
| MeshWeaver #3361 → #3362 | MeshWeaver.ContainerRegistry name collision, renamed to MeshWeaver.ContainerImages |
🚦 The 3.0.0 BLOCKER set (maintainer, 2026-09-06 midday)
Rule (maintainer): "these were generated by solving others from the blocking set ⇒ we must keep expanding the blocking set." The set is TRANSITIVELY CLOSED — any defect found while fixing a blocker is a blocker. That is today's history: #3395 came out of #3201's pre-deploy measurement, #3392 out of the approved cleanup, Plugins#1396 out of the Reinsurance pin move, #3401 out of a satellite gate on the current set. A blocker is what stops the tag producing a set memex/memex-cloud can ADOPT, plus everything that work uncovers. meshweaver-58 drives the set; the target is the whole fleet at zero (39 open across MeshWeaver, Plugins, Memex, SocialMedia, Manufacturing at 12:45Z).
-
Store/Payments — DECIDED (maintainer, 2026-09-06 ~15:00Z): land MeshWeaver.Plugins#1328 BEFORE the tag — the payments module becomes genuinely optional and the Store binds an abstraction. 3.0.0 does NOT ship a Store that hard-requires
MeshWeaver.Payments.Stripe; the centralStripedeclaration is OFF the table. Consequences: Reinsurance'sregistry-modules: … Stripe(fix(ai): truthful agent-selection errors + model-path factory resolution #167) and any other satellite's copy are revert-after workarounds, never doctrine; Plugins#1396 (Education/SocialMedia declining the same Store assemblies) resolves with fix: classify a wrapped HubDisposingException as teardown, not a failed round (#1300) #1328 rather than on its own. meshweaver-58's agent is on fix: classify a wrapped HubDisposingException as teardown, not a failed round (#1300) #1328's first PR. -
Portal pods refuse readiness and hold rollouts after SubscribeRequest to Admin/Build times out — even when the build root is already Ready #3404 — portal pods refuse readiness after the
Admin/Buildsubscribe times out and rollouts HOLD on the previous image. meshweaver-58 has it: the pre-warmer gets the second, DURABLE door (a GO for the pod's OWN framework fingerprint, never one written for another) — no timeout raise, no retries, fail-closed stays, the transport fault stays logged at full severity. -
NodeType compiles in ONE boot resolve DIFFERENT module sets 15s apart — a healthy type becomes failed, and that is the shape the readiness gate refuses forever #3395 — a healthy type recorded FAILED with no source change. MECHANISM FOUND (meshweaver-58, PR fix(#3395): two compiles in one process resolve one module set #3417
Refs #3395): not one boot — THREE memex-cloud replicas of one Deployment, each pinning its module generations at its OWN boot (ModuleGenerationPin, Module generations GC deletes lazily-loaded dependency DLLs out from under a running process; activation can point at a never-landed generation #2509); a landing wave at 12:18–12:27Z wrote new generations into the shared/data/modules/, so the pod booted at 12:51 pinsStripe@458afe55while the two booted at 11:33/11:41 pin@8f251f57(39 of 40 generations differ). They share one NodeType node; each compile stamps the module set IT resolved, the other replica correctly reads that stamp as stale for its environment and rebuilds (ping-pong), and where a replica's pinned set lacks a module the sources need it FAILS — healthy → failed. The single-process race is excluded. It worsens with replica count and with any landing wave during a roll — a release's exact conditions — and it manufactures the regression the readiness gate refuses. DECIDED (maintainer): force convergence on a landing wave — one module set per mesh at a time; the stamp stays one per NodeType and landing gains a coordination step. meshweaver-58's agent is on it. Until it lands: no arming on memex-cloud during the release. -
the readiness gate, per namespace — MEASURED 2026-09-06 (meshweaver-58,
printenvinside all 5 portal pods + the pods' own registration logs): memex is ARMED, enforcing and safe —GateReadiness=trueeffective,nodetype_bakeregistered, a pod held 3 m 12 s (compiled=78 alreadyBaked=122 compileErrors=4,previouslybroken=8) then went Ready; only a REGRESSION gates (MarkOutcomereturns early for!WasHealthyBeforeBake), so the "triage the previously-broken types or the roll stalls 3 h" precondition was FALSE. Rolling memex armed is not a risk item. memex-cloud is INVERTED: every readable artefact (ConfigMap, values, Deployment,helm get manifest) says armed, the pods runGateReadiness=falsefrom an inlineenv:set bykubectlon 2026-09-03T17:32Z during the sign-in-outage roll — a pod started after helm v23 still readsfalse. 🚨 Removing that inline entry ARMS memex-cloud; on memex the same command only clears a duplicate. Do NOT arm memex-cloud on the release (bigger sweep, 13 previously broken,BuildProtocol=true, Portal pods refuse readiness and hold rollouts after SubscribeRequest to Admin/Build times out — even when the build root is already Ready #3404 + NodeType compiles in ONE boot resolve DIFFERENT module sets 15s apart — a healthy type becomes failed, and that is the shape the readiness gate refuses forever #3395 open).PreWarm__BuildProtocoldefaults ON — blank is not off. Memex#171 closed on this; Memex#190 corrected the docs. -
Satellite gate 404s on a bundle its sealed publication names, while core CD republishes that identity #3401 — a satellite gate 404s on a bundle its own sealed index names while core CD republishes that identity (N+1 non-atomic read). A blocker by the rule; concretely it reds the fleet pin PRs that gate against the current ci set, so it lands BEFORE the Plugins/Reinsurance pins move. Being traced (comments 12:09/12:33Z; meshweaver-58 is asking meshweaver-64 and meshweaver-bd).
-
DeleteNode drain reports "did not drain within 30s" after removing 162 of 161 planned paths — and aborts the rest of an approved operation #3392 — the delete drain reports "did not drain within 30s" after removing 162 of 161 planned paths and aborts the rest of an approved operation. Found by the approved cleanup ⇒ in the set.
-
the roll — memex and memex-cloud off
3.0.0-rc9.ci.7693/UpdatePolicy: Noneonto the candidate (process, not an issue), then the rule-10003 re-scan that closes BlazorMonaco 3.5.0 bundles monaco-editor 0.42.0-dev with DOMPurify 3.2.7 — flagged by ZAP (2026 XSS advisories); override the bundled editor #3378. -
Flake: BakeEquivalenceTest — the mesh-driven bake fatals on Widget after its NodeType passed, and the assertion truncates the reason at 50 chars #3370 —
BakeEquivalenceTestred on ordinary PRs (Two teardown/disposal races red one Build-and-Test run: a mesh shutting down mid-bake, and a faulted change-feed entry whose release does not claim it #3333's shape): the bake read a NodeType record while the installer's release compile was still in flight and the store had evicted the version the record named. FIXED by fix(bake): the bake ships from a SETTLED record, not from the first one it sees (#3370) #3420 (merged ade8c45: the bake waits for a SETTLED record); Flake: BakeEquivalenceTest — the mesh-driven bake fatals on Widget after its NodeType passed, and the assertion truncates the reason at 50 chars #3370 CLOSED. -
The plugin-tester's bake refuses every package's content assets — no SyncContentFilesRequest handler on its package roots, so 15 packages log 'binaries are not being served' per run and the gate never exercises content #3424 (PR fix(#3424): the plugin tester's gate serves — and verifies — a package's content assets #3430) — package CONTENT assets were never published during a bake:
PackageInstaller.SyncPackageContentpostsSyncContentFilesRequestto the package root, whose handler exists only viaAddContentCollections()— on every portal per-node hub, but not inAddGraph()'s default chain nor theStore/PluginNodeType most roots declare — so roots answered "No handler found" and the publish was refused before the collection question was asked. Measured on 11 CI logs (Education 34036343718/34037514537: 60 "No handler found", zeropublished N/N; Reinsurance test-repos 40/80): silently true, not Education-specific. The gate now mounts content the portal way and reads every published asset back (N/M content asset(s) served); falsified. Readiness: a 3.0.0 module that ships content assets has not been verified as served until fix(#3424): the plugin tester's gate serves — and verifies — a package's content assets #3430 lands. -
An allow entry can outlive the merge it was written for — the gate detects it stale, but nothing stops the PR that adds it from reddening the fleet on merge #3422 — an allow entry can outlive the merge it was written for: fix(#3406): the navigation rail preserves the supplier's order #3414's
Railline inscripts/record-signatures.allowreddened every C#-touching PR and merge-queue build (fix(#3390): stamp the dispatched inputs at every Pending door and clear them on completion #3415, fix(#3392): the drain predicate admits a finished delete and refuses an unfinished one #3418, fix(bake): the bake ships from a SETTLED record, not from the first one it sees (#3370) #3420) from 13:38Z while main read green (gate reuse-skipped); ci(allow): drop the stale Rail record-signature allow entry (#3414 has merged) — unblocks the queue #3421 (MERGED) removed the line; An allow entry can outlive the merge it was written for — the gate detects it stale, but nothing stops the PR that adds it from reddening the fleet on merge #3422's fix (PR by meshweaver-58's agent, auto-merge armed) makes allow entries EXPIRE with their PR — the gate resolves the entry's PR live (expired/not-a-pull-request/unresolved, none of which allow); falsified on both gates against origin/main (record 3/10→10/10, type-move 3/9→9/9). Joined the set: found by Flake: BakeEquivalenceTest — the mesh-driven bake fatals on Widget after its NodeType passed, and the assertion truncates the reason at 50 chars #3370's work.
Still open but not (yet) in the set: #3390, #3399, #3338, #3353, #3321, #3201, #3176, #2543 (sets do seal; it delays them), #890, #3363, #3384 — each joins the moment blocker work touches it.
Before the tag
-
roll memex + memex-cloud to the candidate (or newer sealed) and set both
Admin/UpdatePolicyback toContinuous— both run3.0.0-rc9.ci.7693, an unsealed build, frozen since 09-03 -
the defect list: Portal managed heap leaks ~215 MB/h through 267 gen2 collections, with ZERO sync/bake events — the cause #2194 is not tracking #3321 (heap leak — step 1 feat(#3321): let a consumer ask whether a synchronization stream is alive #3380 and step 2 fix(#3321): step 2 — migrate stream.Hub dereferences onto TryGetHub #3386 merged, step 3 open), Rotate the plugin-registry token that sat in plaintext on both portals 2026-09-03 → 2026-09-11 (the shadow itself is removed, and the two tokens were EQUAL) #3201 (plaintext registry token — values half merged; the inline shadow stays deliberately until Portal pods refuse readiness and hold rollouts after SubscribeRequest to Admin/Build times out — even when the build root is already Ready #3404 and NodeType compiles in ONE boot resolve DIFFERENT module sets 15s apart — a healthy type becomes failed, and that is the shape the readiness gate refuses forever #3395 land, not cleared in the release window), Bake wedge: portal/nodeops stops draining mid-bake — one shared node-CRUD hub seen from eight places (queue latency bimodal: <=3.2s vs 33-49s) #2543 (bake wedge — hit again by CD #7845's seal on 09-05 AND by the cc7e31d run on 09-06 10:43Z; today's log rules out a compile and names the concurrent writers; diag(#2543): the patch commit path and nodeops' upsert record their arms on the request trail #3393 puts stages on both hops so the next occurrence names its wait), Core PR CI never restores the satellite tree: 47 package pins can be deleted with every check green #3349 (core PR CI never restores the satellite tree), CD heal ledger: the MODULE half is what has been stalling, not the image set (anchor refusal #3293; per-module identity #3306) #3176, CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890, Escalate the inbox's
signature: not-requiredwarning to an error, once the receiver answersverified#3338, Localization: viewer-locale chrome inside authored English content yields mixed-language pages (Run = 'Ausführen', quiz = 'Question 1 of 6'); one rule for in-content controls is needed #3203; MeshWeaver.Plugins NodeType bake single-silo guard: the lease works, but failover is a 10-minute timer where Orleans membership already knows #1355 (four mails per click), Activity tracking fails when PostgreSQL connection pool is temporarily exhausted #1310 (null authorship), NodeType bake: the image contributes ZERO baked assemblies — every pod boot recompiles all 280 types #1347 (exit 139 reads as a pass), ci: stop MeshWeaver.FutuRe.Test walking into a CoreCLR background-GC bug #1274 (release-wave emitter in-mesh), CompileWatcher faults with Postgres timeout during source re-establishment query #1316, Activity tracking fails because permission fold faults on Npgsql timeout #1313 -
QueuedWriteAdvancesOnHandoffTest: the successor's caller can miss its terminal when the first write's ack is late (1/330, Plugins shard 3) #3363 — queued-write handoff race (1/330 on a Plugins shard): no null-terminal path found by reading, 40/40 locally; MeshWeaver.Plugins#1391 (merged) records each caller's notification sequence so the next occurrence names it — waits for a sighting
-
CD: the hourly reconcile publishes a commit whose push-triggered publish is still in flight — two digests, one commit #3376 — the hourly reconcile published a commit whose push-run publish was in flight — fixed by main-cd: one commit yields one image set — a newer run defers to an older run still publishing it (#3376) #3379 (merged cc7e31d:
decidedefers to an older in-flight run) -
BlazorMonaco 3.5.0 bundles monaco-editor 0.42.0-dev with DOMPurify 3.2.7 — flagged by ZAP (2026 XSS advisories); override the bundled editor #3378 — DOMPurify 3.2.7 inside BlazorMonaco's Monaco bundle: FIX MERGED (MeshWeaver.Plugins#1393 — the portal builds its own Monaco with DOMPurify 3.4.14). Closes after rule 10003 reads PASS on memex, which needs the roll below (memex is on
UpdatePolicy: None) -
OWASP ZAP scan — every release, before the tag — docs(release): the OWASP ZAP scan is a release precondition; the notes page is a flat file (was served at _3_0_0) #3382 MERGED (
Doc/Architecture/SecurityScanning, the release skill, ReleaseProcess §4, a Security section on the 3.0.0 notes; ZAP pinned 2.17.0). Both 2026-09-06 runs readFAIL-NEW: 0. The non-10003 WARNs now carry MEASURED dispositions (follow-up PR): Backup File Disclosure and Proxy Disclosure are false positives (the icon route synthesises an icon for any name; TRACE/OPTIONS answer 405), Dangerous JS Functions and the timestamps leave with the old Monaco loader (Plugins#1393), CSP carried by design (MemexPortalComposition.cs, Add security response headers to the portal (CSP staged Report-Only; nosniff, Permissions-Policy, CORP enforced) #1988) withunsafe-evalto re-measure once the new Monaco rolls, COEP/CORP accepted. Override any of them on the page if you disagree. -
docs(release): the OWASP ZAP scan is a release precondition; the notes page is a flat file (was served at _3_0_0) #3382 also moved the notes page to the flat
Data/ReleaseNotes/3_0_0.md: as3_0_0/index.mdit was embedded as_3_0_0(MSBuild makes a digit-leading FOLDER an identifier) and served atDoc/ReleaseNotes/_3_0_0; release.yml reads the flat path -
Self-registry install silently stops mirroring its mounted checkouts — the repos land on the seed-once lane, and neither auto-update mechanism applies #3359 — self-registry install stops mirroring its mounted checkouts: FIXED by fix(catalog): a mounted checkout's packages reconcile on every boot instead of seeding once (#3359) #3385 (merged) (
ConfiguredPackageSource.LocalCheckout+ a reconciled lane for the operator's patterns over a local checkout; tests falsified). Its option 3 is now its own issue, A PluginCatalog node over a local node-repo checkout lists nothing — BuildSource never selects the node-repo source, and the content has no format field #3384 (a PluginCatalog node over a local node-repo checkout lists nothing) — nobody has it -
fleet mains: Crm and Manufacturing (compile-check / test-repos), Education (bake), Memex (
Deploy drift), Reinsurance (red since 09-05 03:50Z on an unchanged sha: its pins sit on3.0.0-rc9.ci.7574— a 09-02 08:52Z tester that predates Owner-side patch ack watcher has no completion arm: a stream that ends without emitting leaves a cross-hub write unacknowledged #3033/Gate seed race: one bundle's adoption write times out at the owner during the mass install (ADVANCE_WITHOUT_HANDOFF → LATE_NACK_TERMINAL on Radzen/Gallery) and the consumption postcondition reds a satellite PR #3112 — so the scheduled bakes fail the bake-identity gate against newer release-side inputs and the push gate hits the pre-Gate seed race: one bundle's adoption write times out at the owner during the mass install (ADVANCE_WITHOUT_HANDOFF → LATE_NACK_TERMINAL on Radzen/Gallery) and the consumption postcondition reds a satellite PR #3112 10 s NACK; measured on runs 33942938986/33975864339/34009974381/34027833694. Pins MOVED to ci.7868 in Reinsurance fix(ai): truthful agent-selection errors + model-path factory resolution #167 (merged6142af1a, mounting theStripepackage as the stated workaround). 🚨 Its firstpublish-bakeon the new pins failed on the Bake wedge: portal/nodeops stops draining mid-bake — one shared node-CRUD hub seen from eight places (queue latency bimodal: <=3.2s vs 33-49s) #2543 shape — 16 pathsOwnerUnreachable … no verdict within 31s, both the 20 s ack bound and the 31 s verdict bound,HANDLER_EXIT state=Processed → silence— on a core that has Owner-side patch ack watcher has no completion arm: a stream that ends without emitting leaves a cross-hub write unacknowledged #3033 + Gate seed race: one bundle's adoption write times out at the owner during the mass install (ADVANCE_WITHOUT_HANDOFF → LATE_NACK_TERMINAL on Radzen/Gallery) and the consumption postcondition reds a satellite PR #3112 but NOT diag(#2543): the patch commit path and nodeops' upsert record their arms on the request trail #3393's stages (7868 =1bc0256d8<30b8733f4). No before/after baseline: this lane reached its install phase for the FIRST time (earlier pushes died at test-repos or the identity gate), so it is newly observable, not demonstrably newly broken. Consequence for the release story: Reinsurance has no sealed publication under identitys8a52cb27…untilpublish-bakegets through, and itsregister-publicationwebhook wiring is still unexercised) -
Plugins pin: main pins
3.0.0-ci.7859(corebbcb22f25; measured 16:55Z — the earlier7835here was stale); re-align all three on the sealed set named under the candidate section (7910 today, the release set at the tag). 🚨 Trap measured on Reinsurance fix(ai): truthful agent-selection errors + model-path factory resolution #167: at lane shas past ci: core publishes ONE event and finishes — withdraw notify-dependents; the release wave is memex's, subscribers are the Deployment records #3185 theregister-publicationjob is RED (never skipped) withoutvars.PLATFORM_WEBHOOK_URL+secrets.PLATFORM_WEBHOOK_SECRET, andpublish-bakedoes not run on a PR — so a lane-sha move without both wired is green on the PR and red on main's first push. Check both before moving -
Deployment inventory — "track which instance has what deployed", before the release (maintainer, 2026-09-06). SHIPPED as three PRs: core Every instance reports what it runs — the fleet inventory files itself hourly #3440 — MERGED 17:10Z (
DeploymentReportService: every instance reports platform / commit / framework identity / update policy / every module's coordinate to the control instance'sHosting/Modulesinbox, once the default install settles and hourly; the chart renders the keys;Doc/Architecture/DeploymentInventory), MeshWeaver.Plugins The compile-progress overlay covers only Overview — every other area of an instance mid-compile serves a false "Area not found" #1411 (the Fleet area's per-instance table), Systemorph/Memex Coder agent still selectable in chat agent dropdown — should be surfaced via skill #200 — MERGED (keys on the deployment records'extraPortalConfig; the vault keyHosting-ModuleReportSecretis provisioned and in bothhelm-values-*halves); MeshWeaver.Plugins The compile-progress overlay covers only Overview — every other area of an instance mid-compile serves a false "Area not found" #1411 — MERGED 21:2xZ (the Fleet's per-instance table). What is LEFT is thehelm-release.yml → deployof memex + memexcloud, which rides the release roll. What is LEFT is operator provisioning, in order: merge Every instance reports what it runs — the fleet inventory files itself hourly #3440 → Coder agent still selectable in chat agent dropdown — should be surfaced via skill #200 →az keyvault secret set --vault-name Systemorph --name Hosting-ModuleReportSecret→ addsecrets.memex_portal.Hosting__ModuleReportSecrettohelm-values-memexandhelm-values-memexcloud→helm-release.yml → deployfor both releases (a ConfigMap change is a helm upgrade; self-update never re-renders it). Fleet reality: live = memex (control) + memex-cloud; client-i decommissioned; client-b planned and pre-armed
Decisions still open
-
🚨 Store/ hard-requires
MeshWeaver.Payments.Stripesince Plugins589a17a1— DECIDED: option (b), MeshWeaver.Plugins#1328 lands before the tag* (measured on Reinsurance fix(ai): truthful agent-selection errors + model-path factory resolution #167 run 34032702362:adopted 210 of 214,CS0234on the fallback compile). Option (a), a centralStripepackage requirement, is off the table; satellites that mountedStripeas a workaround revert after fix: classify a wrapped HubDisposingException as teardown, not a failed round (#1300) #1328. -
NuGet: retired. If
3.0.0packages should go to nuget.org after all, one PR adds a pack-and-push job to the promotion lane. -
the two
3.1.0-ciimage tags (7832, 7835): hide them in ACR (digests stay; Plugins pins by digest) so no Continuous install skips the clean 3.0.0 — one command, reversible. -
six preview/beta dependency pins in
Directory.Packages.props(Aspire.Hosting.Kubernetes, Azure.AI.*, Microsoft.Agents.AI, Microsoft.Extensions.AI.AzureAIInference) -
portal-next: the deployment runs
3.0.0-ci.2054while ACR has3.0.0-next.39— in 3.0.0, or explicitly out
🤖 Generated with Claude Code
Activity
[-]Release 3.1.0 — the first clean release of the 3.x line (tracking)[/-][+]Release 3.0.0, then open 3.1.0 (tracking)[/+]rbuergi commentedon Sep 7, 2026
Release-relevant state, 2026-09-07 15:1xZ
This body is stale in places — most of the blocker list above is closed. Rather than rewrite it,
here is what a reader needs to know today, measured rather than recalled.
Both portals are rolled and healthy
3.0.0-ci.7981(head525a1c489)3.0.0-ci.8009(head242637ff2)Both carry
91ef23179bygit merge-base --is-ancestor. The staff portal had been stuck on3.0.0-rc9.ci.7693since 09-03 because the bake gate refused readiness over an incompleteFeedbackinstall (#3485); that install was repaired at 10:34:02Z via
Store/Maintenance/RefreshModules(four missing nodes re-landed —
FeedbackContent,feedback-agent, the/feedbackSkill,Guide),and
Feedback/Feedbacknow compilesOk. #3485 is closed.portal-nextis 1/1 Running on both on3.0.0-next.43with Helm ownership metadata on Deploymentand Service, so Memex#172 is closed and no
kubectl deleteis outstanding. The Store packagefeed has recovered — all five git sources plus the registry read
fault None, zeroBad credentialsacross both replicas' full retained logs — so Memex#165 is closed.
What still gates a tag
bound, which reddened every open PR in that repo as a six-check red where five were
Every need this gate builds on must have succeeded. meshweaver-64 owns the move (Plugins#1463).Its remaining reds were three distinct real causes, not one: a contract change core fix(#3474): a create is an import too — a repo file's compile verdict never becomes a type's initial live state #3475
retired, the CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 Roslyn canary, and a shard-fold accounting refusal.
release's Artifacts) and fix(#3510): the upsert's CREATE leg answers instead of hanging forever #3603 (the upsert's CREATE leg answers instead of hanging). The re-pin
target is the first sealed set carrying both.
workflows: write, so a lane that cannot push has been failing silently. That is why the pindrifted, and it is a provisioning act.
Deliberately still open, and why
gets from
RegisterOwnerDisposingNackis still missing on the CREATE leg.lastCheckedAtoff the record sobookkeeping stops clobbering the policy is not done.
PATCH_ECHO_SEENstall trails,PATCH_FLUSH_FAULTED_SYNCzero.corr=id.Maintainer-only, outstanding
gh secret set --app dependabot(Reinsurance + CrmMW_REGISTRY_KEY, SocialMediaREGISTRY_PUBLISH_TOKEN, ManufacturingACR_USERNAME/ACR_PASSWORD/MW_REGISTRY_INSTANCE_KEY).🚨 Crm is listed as clean in Dependabot PRs are red fleet-wide: the Dependabot secret store is a partial mirror of Actions secrets #3399 and is not — its preflight is green while the lane dies on
compose-sealed-modules.sh: --registry-url needs --registry-key.PluginCatalog__RegistryTokenand rotate it; fixing supply didnot revoke the disclosed value, and
Chart Driftnames it.Container Registry Repository Writergrant and thelatest-tag policy.🚨 Memex#219 covers neither (all 13 open Memex issues searched, titles and bodies, 0 hits).
AgenticPrimeron the staff portal — install it, or repoint the prod probe. Note the probe samplesone path three times, so it cannot distinguish "asset missing" from "route down"; memex.systemorph.com: the Prod synthetic probe names AgenticPrimer, which is not installed there — decide whether to install it or repoint the probe #3578's
filed premise is falsified (the route serves and discriminates 200/404).
Measured during a fleet-wide sweep. 40 issues opened and 41 closed since 07:00Z; 40 core PRs
merged. The fleet count is flat because discovery kept pace with repair — that is the transitively
closed blocking set working as ruled, not stalled throughput.
rbuergi commentedon Sep 9, 2026
Release-gate state, measured 2026-09-09 ~08:0xZ
The plan on this issue is "Release 3.0.0 first — only once every open issue is closed", so the gate
is literally the fleet's open-issue count. Here it is, and — more usefully — what shape the
remainder is, because a flat count hides that most of it is not "unfixed defect".
44 open, excluding this tracking issue (core 29 + Plugins 10 + Education 2 + Reinsurance 1 +
SocialMedia 1 + Manufacturing 1; Crm is at zero).
It is not 44 pieces of engineering — it is roughly five kinds
A. Blocked on a person, not on code (4). No amount of work here moves them.
permission-workflows: write; the bump lane has never once moved apin. Org-admin grant.
node-repo-publish-bake.ymlis explicitly the maintainer's;af has it staged behind a decision.
B. One publication defect wearing four faces (4). #3732 (two producers of one module), #3768
(
FrameworkDeclined, 25/25 on the staff portal), Manufacturing#67 (gate cannot registerSkill/Agent), Reinsurance#178-adjacent.Admin/UpdatePolicy.heldReasonon memex states itdirectly — "the set is inconsistent and nothing rolls" — so this one cause is also holding
self-update. Fixing it should close several at once.
C. Live but already owned by someone (3). Education#298 (blocked on PR #299, which is red for a
startup failure I diagnosed there today —
runner.tempin a job-levelenv:), Education#300,core#3790.
D. Recorded findings rather than open defects (~10). #3703, #3704, #3773, #3782, #3780, #3488,
#3221, #3519-class… These are measurements someone wrote down so the next reader would not re-derive
them. Under a "close everything" gate they need a disposition — 3.1.0 milestone, or closed as
recorded — and that is a call for you, not something to fix.
E. Genuine multi-hour engineering (~20). #890 (Roslyn Emit poisoning), #2543 (node-CRUD
serialisation — core#3759 is drafted and its consequences are mapped on the PR), #3432 (2.7 GB of
sync/hubs), #3488 (no metrics at all), #1483, #1501, #1449, #3645, #3711 (narrowed to oneunmeasured question today), #3686 (two causes ruled out today).
What I would flag before the release
🚨 #3768's live consequence is user-visible right now:
memex.systemorph.comanswersDegraded,with 7 node types unable to type their content — including
Hosting/Deployment ×7, which is therecord a record-driven Reconcile reads. That one is worth fixing before a release regardless of how
the gate is counted.
🚨 The release-following fan-out is not currently delivering. The four subscribers
(
pluginRepos[].isRegistrySource: Plugins, Education, Reinsurance, Crm) took 111 / 130 / 72 / 31dispatches on 09-08 and 1 / 2 / 0 / 1 today, all before 00:16Z, while the platform sealed
s1218dc0…at 02:53–04:33Z. A promotion that cannot reach the satellites is worth knowing aboutbefore promoting.
(SocialMedia and Manufacturing are not subscribers — their zeros are the designed state, not a
symptom. I had that wrong earlier today and corrected it on #3768.)
rbuergi commentedon Sep 9, 2026
Gate state, 2026-09-09 ~09:40Z — 42 open (core 28 incl. this issue and one self-healing auto-file, Plugins 10, Education 2, Reinsurance/SocialMedia/Manufacturing 1 each, Crm 0)
Six PRs merged into core today; the count moved 29 → 28 in core because three of those PRs
deliberately leave their issue open, each with a stated closing condition rather than a link and a
hope.
Merged today
None— #3790 and #3542 closedWhat the remaining 28 actually need — five buckets
A · Blocked on a person, not on engineering (5). #3201 (needs #3663 settled + a rotation and a
maintainer call on a committed plaintext credential;
kubectl set envcannot take effect while bothportals are held at the bake gate), #3378 (needs a ZAP re-scan; the flagged file is gone from the
image — 0 routes, 404 — but "the rule no longer fires" is not something I can assert),
#3653 (a publish decision), #3773 / #3782 / #3780 (cluster/infra, and operations go through the
memex API by directive).
B · One publication defect wearing four faces (4). #3732 · #3768 · #3221 · #3461. Production
answered #3221's design question outright: memex's live
heldReasonnames the exact pair — "twodifferent builds of
MeshWeaver.Markdown.Collaboration… the loader keeps whichever it sees first… the set is inconsistent and nothing rolls". The implementation is #3732 and it is a sequencing
call, not a coding one.
C · Narrowed by measurement today; each needs ONE more datum (7).
drainsAwaitingSchedulerportal/nodeopsnever produced a routing verdict — the ledger's@is the recording hub, so this is #2543's family, not a routing miss!toolchainby constructionStoreD · Genuine engineering, unstarted (7). #890, #3353, #3438, #3477, #3488, #3583, #3711.
#3477 has a live repro harness in its own thread (6/9 under host contention) and gained a second
sighting today, in a different test and the other repository, which removes the pin suspicion its
control table could not settle. #3711 is narrowed to a single unmeasured link — one
render_areacall on memex-cloud tells you whether
dataContextreaches the client.E · Self-healing / transient (1). #3801 — main's HEAD came out of CD with an incomplete image
set; the #3176 reconciler is on attempt 1/3 and the issue closes itself on a heal.
The honest read
Buckets A and B are 9 of the 28 and none of them is mine to close. Bucket C is 7 issues that are
each one measurement from a decision. That leaves 7 in D as the actual remaining engineering, plus
whatever the satellites need.
🚨 Three of today's findings were corrections to standing conclusions, and each would have
produced wrong code: #3703 (my own proposed fix, falsified by reading
Compare— it would have beena no-op), #3432 (the proposed reclaim trades a silent correctness break for memory), #3674 (the
trail was read as "forwarded by every hub" when the third hub never answered at all). Worth
weighting when reading any "✅ established" in this fleet's threads — including mine.
rbuergi commentedon Sep 10, 2026
Reopened — the 2026-09-10 sweep was not a triage decision
This issue was closed
not_plannedat 2026-09-10T14:31:34Z by a bulk sweep that PATCHed every open issue in all 13Systemorphrepositories — 311 issues in twelve minutes, no comment on any of them — after a session was asked to "close all issues in all repos". Nothing on this thread was read, and no judgement was made about this issue.All 82 issues the sweep closed in the MeshWeaver fleet were reviewed afterwards. This one is unresolved: This is the release tracker. Its gate is "release 3.0.0 only once every open issue is closed" — the sweep satisfied that gate by fiat, and 3.0.0 has not been cut.
Sixteen of the 82 were left closed — their fix had actually landed, or the author had withdrawn them; those should have read
completed, notnot planned.rbuergi commentedon Sep 13, 2026
Gate state, 2026-09-13 ~07:1xZ — not tagged, stays open
v3.0.0tag exists:git ls-remote --tags origin 'v3*'ends atv3.0.0-rc9; GitHub Releases' newest non-draft is stillv2.0.2(2025-03-24). So the tracker's own gate has not been met and this issue cannot close.search/issues is:issue is:open, PRs excluded), measured after today's adjudication pass: core 51 (incl. this one), Plugins 42, Memex 17, Education 3, Crm 2, Reinsurance 1, SocialMedia 1, Manufacturing 1. Up from the 42 recorded on 09-09 — the fleet has been filing faster than closing since the 09-10 sweep was reversed.completed; The auto-arm lane reports 'armed' when GitHub refused the arming — a green verdict for work that did not happen #4118 closednot planned(premise falsified by the PR's own event timeline); the rest recorded with their remaining condition on each thread. Each closing comment carries its evidence, so the count above is a measured number, not a sweep.Corrected 2026-09-13 (
date -u09:1xZ): the Z stamps I wrote in this comment were derived from a local clock, notdate -u, and read about an hour ahead; fixed in place to the comment's owncreated_at. GitHub-sourced stamps were unaffected.rbuergi commentedon Sep 15, 2026
Per-item re-measurement, 2026-09-15: the headline has not started, and three tracked lines are still not done
Read-only, core
origin/mainb8da52e054. The 09-13 comment records the missing tag and the open counts; these five readings are not on the thread.The release lane has never run.
actions/workflows/release.yml→state: active,total_count: 0. So there is no promoted_releases/3.0.0marker, no GHCR mirror, no Release body — and no next-line bump PR, because that PR is opened only byrelease.yml'sopen-next-linejob (release.yml:339-431). Tags end atv3.0.0-rc13/rc9; the newest non-draft Release is stillv2.0.2(2025-03-24).The 3.1.0 line is not open.
Directory.Build.props:122→<PlatformVersion Condition="'$(PlatformVersion)' == ''">3.0.0</PlatformVersion>(there is noversion.json;Version = $(PlatformVersion)-ci.$(_CiBuildNumber)at:299-300). Corroborated by memex.systemorph.com running3.0.0-ci.8612.The roll item is half done, and the half that is missing is the one this tracker names. Both portals are off
rc9.ci.7693— memex on coreb814339, memex-cloud onc84c6c0— butAdmin/UpdatePolicyis stillNoneon both:lastCheckVerdict= "updates are disabled on this install (Admin/UpdatePolicy = None); the registry was not listed.",lastCheckedAt2026-09-15T07:22:50Z (systemorph) and 07:52:52Z (memex-cloud). "Roll both portals … and setUpdatePolicyto Continuous" is therefore not complete, and it appears twice in the body.The preview/beta pins are seven, not six.
Directory.Packages.props:68(Aspire.Hosting.Kubernetes 13.4.6-preview.1.26319.6),:334,:335,:336(Azure.AI.*),:339(Microsoft.Agents.AI.AzureAI 1.0.0-rc5),:340,:342.The "Plugins pin re-align" item was superseded, not executed. Plugins
ci.ymlno longer carries a set or digest pin at all —MW_IMAGE_DIGESTresolves at run time fromneeds.platform-ref.outputs.image-digest(:871,:3508,:3532,:3770,:3812), andMW_PLATFORM_REFis a freeze variable documented as "unset in the normal state" (ci.yml:32).grep -cE 'sha256:[0-9a-f]{16,}'over that file → 0. Worth rewording the box rather than leaving it to be chased.The gate is further from met than when it was written. Open issues fleet-wide today, PRs excluded: MeshWeaver 36 · Plugins 14 · Memex 15 · Education 3 · Crm 1 · Reinsurance 0 · SocialMedia 0 · Manufacturing 0 = 69 (42 on 09-09). And of the "before the tag" defect list, #3201 is open (reopened) — the one core blocker from that list still standing. Fleet mains are amber right now, not green: Memex
Deploy driftfailure, EducationEducation Content CIfailure, coreCombo verification (candidate × instance)failure (#3848).Two items remain (d) — unanswerable from any repo: hiding the two stray
3.1.0-citags (7832, 7835) in ACR, and theportal-next3.0.0-ci.2054vs3.0.0-next.39question. Instrument for both:az acr repository show-tags --name meshweaver --repository memex-portal-ai/az acr manifest list-metadata.Everything in the "Done (2026-09-05)" block and the whole BLOCKER set except the roll is confirmed closed. Staying open on: the tag→promote→open-3.1.0 headline (not started), UpdatePolicy → Continuous, the seven preview pins, and #3201.
rbuergi commentedon Sep 17, 2026
Release status, measured 2026-09-17 — 3.0.0 is NOT released; the gate has moved a long way today
The headline has still not started.
v3.0.0tagtagsend atv3.0.0-rc13/v3.0.0-rc9v2.0.2, 2025-03-24actions/workflows/release.yml/runstotal_count: 0— the promotion lane has never run, so there is no_releases/3.0.0marker, no GHCR mirror, no Release body, and noopen-next-linePRDirectory.Build.props:122<PlatformVersion …>3.0.0</PlatformVersion>— the 3.1.0 line is not openWhat the two live portals actually run (
/api/version, andAdmin/UpdatePolicyread on each):3.0.0+afde4eabe0740ff658f3dae8a3073c33a2b3ea02Continuous, pattern3.0.0-ci*— newly so since the 09-15 reading. Handed3.0.0-ci.8816toHosting/PlatformBuildsat 11:28:56Z,comboVerifications: [], verdict UNVERIFIED (#3848)3.0.0+c84c6c05503228860df03c4a8b596e497e6d218cNone— and HELD at3.0.0-ci.8339since 2026-09-11T08:57:47Z, because ~25 landed modules (Chat, Mcp, every AI provider, Blazor.*, Observability, …) have no build published for that framework identitySo the tracked item "roll both portals and set
UpdatePolicyto Continuous" is half done: the controlinstance is on Continuous; the commercial portal is neither rolled nor on Continuous, and its hold is a
module-publication problem, not a decision.
Remaining tracked items, re-measured:
Directory.Packages.props: 7 (Aspire.Hosting.Kubernetes 13.4.6-preview,Azure.AI.Agents.Persistent 1.2.0-beta.10, Azure.AI.Inference 1.0.0-beta.5, Azure.AI.OpenAI 2.9.0-beta.1,
Microsoft.Agents.AI.AzureAI 1.0.0-rc5 + .Persistent 1.4.0-preview, Microsoft.Extensions.AI.AzureAIInference
10.0.0-preview) — still (b), a decision per pin;
3.1.0-ciACR tags and theportal-next3.0.0-ci.2054vs3.0.0-next.39question remain(d) — unanswerable from any repo, one
az acrread each.The gate — "release only once every open issue is closed" — is closer than at any previous reading.
Open issues today, PRs excluded: MeshWeaver 97, Plugins 14, Memex 14, Education 3, Crm 1, Reinsurance 1,
SocialMedia 1, Manufacturing 0 = 131, and core's number is falling through the day (it was 164 after the
recurrence-bot wave this morning, #4577 — 53 of which were re-closed on measured evidence). A fleet-wide
build/deploy/CI triage pass is running right now on the maintainer's instruction, so this count is moving.
Staying open on: the tag → promote → open-3.1.0 headline (not started), memex-cloud's roll +
Continuous,the seven preview pins, and the two (d) items.
Read-only measurement, 2026-09-17 by Claude Opus 5 (1M context). Nothing was tagged, promoted or rolled.
rbuergi commentedon Sep 21, 2026
Triage 2026-09-21: the cut has NOT happened, and this issue is the goal's own tracking record
Blocked on: the cut, which is the maintainer's decision — and on this issue's own stated precondition, "Release 3.0.0 first — only once every open issue is closed".
Measured over REST today:
refs/tags/v3.0.0-rc2 … rc13exist; there is nov3.0.0, and the newest GitHub Release isv2.0.2(2025-03-24). So nothing in the promotion lane has fired, the tag rule recorded here ("the first sealed set ≥ both #3501 and #3505") is still a rule rather than a history, and the named candidate sets (3.0.0-ci.7962/7964/7969/7974) are still candidates.That makes this the tracking issue for the zero-issues goal, not a work item, and it should be the LAST thing closed rather than triaged toward closure: the release cannot be cut while it is open by its own rule, and it cannot be closed before the release is cut.
Two things worth recording so the state is not re-derived:
Before the tagandDecisions still openchecklists are the remaining substance, and several items are ordinary open issues in this backlog rather than release mechanics — so closing them is the same work as driving this issue toward the cut, not separate work.chore, no severity — correct: a tracking issue is not a defect. It should staychoreeven though what it tracks includes blockers, because letting a tracker carry a severity is how a release gate fills with rows that are not defects.rbuergi commentedon Sep 22, 2026
Post-roll verification sweep, 2026-09-22 ~12:05Z — what is live, what each fix's owner asked for, and which of those reads exist without a write
Read-only session. No portal write, no
InstanceAction, no recycle. Nothing is closed by this sweep — the one read that came back clean cannot tell a working fix from an absent condition, and it says so below.1. What is running, and how far the roll actually got
Both portals expose a per-replica census read-only, through
get @Ops/Status/<deployment>— theSampleInstanceAction is not needed for this question whilenotScrapedisfalse, and it isfalseon both right now (#4218's flip, measured).memex(memex.systemorph.com)memex-cloud(memex.meshweaver.cloud)3.0.0-ci.91623.0.0-ci.9172c55301f396ef60e1a8a965334a407e433d97c55392174c2ed5e3f1af76838616c960380eefaff72cgenerations: 1,converged: truegenerations: 1,converged: trueimageProvenancesampledAtSo the roll landed at ≈09:03Z on
memexand ≈11:34Z onmemex-cloud, and at 11:59Z both were converged on one generation./healthand/api/versionagree independently (publication-sealnamesmeshweaver-content ← Systemorph/MeshWeaver @ c55301f3and@ 92174c2e, bothsealed).🚨 One caveat on the roster: a direct
/healthat 12:02:10Z hit amemex-cloudreplica reporting booted at 2026-09-22T12:01:20Z — not in the 11:59:26Z roster. Its seal (92174c2e) and framework (sca5a128) match the two rostered ones, so it is build-identical, but the census is a snapshot and the deployment changed pods after it was taken.Ancestry re-verified locally rather than inherited —
git merge-base --is-ancestor <fix> <running commit>:memex-cloudmemex78bbf49e036b3fda2a487a39fb241277d8dcf5fac9a3ccd0cecc889c1e265d28773b342. 🚨 Instrument calibration — the 6 h is the watcher, not Loki, and that changes what is blocked
The house rule warns that a constant ingestion lag makes a fresh stop and a long-dead stop look identical. I measured the lag rather than assuming it, and it splits in two:
Admin/_LogIncidentis exactly 6h00m00s behind. Three independent incidents, each node written at T carrying its newest sample stamped T − 6h (±0.3 s):96c85f04cadba7f7(this issue's routing shape, #5009)afc6241b330977b0(process_progress)833c5f2e6337f4d3(Polly, #4528)Loki itself is ~2–4 minutes behind. Two
Hosting/LogEntrynodes underOps/Logs, comparingcontent.timestampto the node'screatedDate:memex-1790071806302558326-…— line at 10:10:06.302Z, written 10:11:57.836Z → 1m 51smemex-1790071678607290710-…— line at 10:07:58.607Z, written 10:12:05.662Z → 4m 07sConsequences, both of them load-bearing:
Admin/_LogIncidentreading available today stops at ≈06:04Z — that is 3 h before thememexroll and 5.5 h before thememex-cloudroll. No incident-based statement about a post-roll pod is possible on either portal until ≈15:03Z / ≈17:34Z. Any "no recurrence since the roll" read inside that window is vacuous by construction, and so is every occurrence count the recurrence bot has posted today.LogsHosting/InstanceActionreads Loki from the control instance's own pod, and Loki has the post-roll window now. Every (b) row below is one write away, not six hours away.I also searched the incident store directly for the post-fix wordings —
per-channel queue/ordered channels queued(#5151) and aShuttingDownpod-hub verdict (#5174). Neither appears anywhere. Every back-pressure row still carries the pre-fixdeepest per-destination queuespelling, consistent with (1). Denominator:coverage.partitions: ["admin"]for the incident reads and["ops"]for the status reads — theadminandopspartitions as readable by me.3. The table
Class: (a) doable read-only · (b) needs a
Logs/Sample/deleteInstanceAction — a WRITE, the maintainer's act · (c) impossible as written.[ROUTE] Routing back-pressureline readingdeepest per-channel queue 0withordered channels queued≈ in-flight. A line still spellingdeepest per-destination queueis a pre-fix replica, not a result.…-7b4776b9f-8khld,deepest per-destination queue 62oncache/T2W4ynBASEOZcVKyp3_tLg— a pre-fix pod on a pre-fix image, so by the owner's own rule it says nothing either way.info:…surfacing ShuttingDown DeliveryFailure; afail:/Failedline on a post-roll pod is the fix not holdingInformationand the red-log watcher collectsfail:/crit:only. The owner states this on the thread: "the next reading of 'is this still happening' cannot come from this incident's occurrence count any more." NeedsLogs. Also: this issue is wrongly CLOSED — see §5.TimedOut; a type in a confirmed-absent partition reportsRemovedmemexreplicas: "sweep completed … 230 of 230 enumerated type(s) reported an outcome; of those, 0 reached NO verdict (timed out, or waiting on something that did)". 🚨 The owner records that #5163 is inert onUWDeepfield(its store exists, so the store witness truthfully confirms nothing absent). So this read cannot separate the fix fired from the condition never arose — a negative control that passes. One soft signal I could not resolve:memexreports 1 type "withdrawn by their own repository (retired or removed)" while bothmemex-cloudreplicas report 0; the public/healthbody deliberately does not name the type, so Retired and #5163'sRemovedare not separable from outside.delete @Admin/Partition/UWDeepfieldas a platform admin →[StrandedPartition] finishing the teardown…+Dropped partition…, thenget→ Not found, then the next replica'sbake-reportdrops that partitioncc889c1e26is not an ancestor of either running build, and the read is adelete.4b361ba397c4432b, 3 occurrences, last seen 2026-09-19T05:35:41Z — a base rate of ~1 per 6 days. A ~3 h post-roll window has no statistical power at that rate, and the instrument cannot see past 06:04Z regardless. Closing this on silence is precisely the reading the brief forbids.Logswithin minutes forauthenticated=False→redirecting … /login?returnUrl=%2Fauthorize%3F…(a local target) →Issued OAuth authorization code …→Issued OAuth access token …. The third line could not exist before #5160.Logsaction within minutes.5d28773b34is not in either build, so even the new attribution lines are not live.4. Closed by this sweep: nothing
No row reaches the bar. The one (a) read that came back clean (#5073 d1) is disqualified by its own author's inertness finding; every other row is either not live, needs a write, or is unobservable from incidents by construction.
5. 🚨 #2299 has been closed by a keyword misfire for the second time in three hours — reopening
Timeline from
repos/Systemorph/MeshWeaver/issues/2299/timeline:#5190 is
docs(triage): the line's wording dates a sample against a merged fix— a documentation change that closed asev:Hrelease blocker whose owner had, 57 minutes earlier, written on the thread that it "stays open for the re-activation bounce". It is the same trap the house memory already names (a closing keyword parses per number, not per sentence), tripped twice against the same issue by two different PRs.Reopening #2299 and noting it there. It is a
sev:Hand it is unverified — it belongs in this tracker's blocking set.6. What the release cut is actually waiting on — and who has to do it
Three acts, all the maintainer's, none available to an agent:
LogsHosting/InstanceActionper portal, taken now. Loki is ~2–4 min behind, so the post-roll window is already there. This settles RoutingGrain dispatch slots saturate with legs head-of-line blocked behind one shared-cache destination while the second pod starves on load #5009 (query: "Routing back-pressure"— read the field name:per-channel= post-fix,per-destination= a pre-fix replica) and RoutingGrain turns a stale pod-hub route into a sender-visible DeliveryFailure instead of re-resolving it #2299 (query: "Directed delivery to pod hub"—info:/ShuttingDown= the fix holding,fail:/Failed= not) in a single pass each. Without it, both wait until ≈17:34Z for the incident watcher to catch up — and RoutingGrain turns a stale pod-hub route into a sender-visible DeliveryFailure instead of re-resolving it #2299 will never be answerable from incidents, because its post-fix line isInformation.memex-cloud, then aLogsaction within minutes — the only thing that can produce MCP OAuth state is per-process — on a two-replica deployment the token exchange reaches the wrong replica about half the time #5074 defect 1's positive read.cc889c1e26, thendelete @Admin/Partition/UWDeepfield— for A new pod waits forever on a partition deleted long ago — the readiness gate holds on a NotFound it treats as transient #5073 defect 2. Not live; nothing to verify yet.7. What I could NOT establish
sev:Hrows, none that discriminates.memex-cloudis converged right now. It was at 11:59:26Z; a replica booted at 12:01:20Z is outside that snapshot. Build-identical by seal and framework, but the roster is stale by ~3 minutes.memexbake-report's "1 withdrawn (retired or removed)" is. The public body does not name the type by design (bake-report names no type: the one past-RLS census counts a permanently-broken NodeType and drops its path one call before publication #4258), so fix(mesh,hosting): a recycle and a bake must ask a survivor, not the thing that is dying (#5099, #5073) #5163'sRemovedand an ordinary retirement are indistinguishable from outside.UWDeepfieldstill appears inmemex'sframeworkstalelist. The/healthbody truncates the partition list at 12 with "(+7 more partition(s))", andUWDeepfieldsorts after the last one printed. Not readable from the public body.Instruments:
git merge-base --is-ancestor;GET /api/versionandGET /healthon both hosts;get @Ops/Status/{memex,memex-cloud};get/searchunderAdmin/_LogIncidentandOps/Logson the control instance (coverage.partitions: ["admin"]/["ops"], RLS-filtered as me);repos/Systemorph/MeshWeaver/issues/2299/timeline. No write of any kind.rbuergi commentedon Oct 9, 2026
Closing as superseded. This tracked the 3.0.0-ci/rc era cut. Policy
platform-semver-versioning(#6208,Doc/Architecture/PlatformVersioning) replaces it: continuous builds become3.<minor>.<run>, and the 3.1 line opens with the minter flip owed in that doc's section 4. A deliberate release cut is promoted byrelease.ymland gated byGovernance/Standards/release.cut(policyrelease-blocker-gate: sev:B and sev:H at zero), so it does not need a tracking issue.rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-09-22)
What blocks the cut today — measured against the POLICY bar, not the body's
The bar is policy
release-blocker-gate(AGENTS.md → "Work by PRIORITY"):sev:Bandsev:Hboth zero across the seven gated repositories;sev:M/sev:Lnever gate. The precondition this thread has been carrying — "only once every open issue is closed" — is the body's original wording and is superseded by that policy; measuring against it would hold the cut onsev:Mbacklog that the policy says is a priority conversation, not a gate. The gate is enforced, not remembered:Governance/Standards/release.cut.json(MeshWeaver.Pluginsorigin/main8db241c2f2) carries 14NoOpenIssuesgates over MeshWeaver, MeshWeaver.Plugins, .Crm, .SocialMedia, .Reinsurance, .Manufacturing, .Education (Memex is not gated), plus twoChecksgates (core@main, Plugins@main), aManualnotes gate andSignatures count=2.Snapshot ~06:20Z today, REST only (
repos/{o}/{r}/issues?labels=…&state=open&per_page=100, paginated, PRs filtered on.pull_request == null), every label's existence confirmed with the SINGLE-label endpointrepos/{o}/{r}/labels/sev:{B,H}(14 of 14 answered 200 — an absent label would fold its gate to a false green):sev:Bsev:Hstate=opencontrol on the five answers 1/1/1/0/1, so the zeros are real small populations, not a truncated read)So 12 of 14 issue gates are green and 2 are red: 26 open
sev:H— 18 core, 8 Plugins. Read per issue from each one's timeline (issues/{n}/timeline, cross-referenced merged PRs) and its last two comments:LogsInstanceAction, a write), #3432 (needs an in-process ClrMD census, break-glass)Four of the (iii) rows are a human's act, not a code change, and no PR moves them: core #3848 (a naming decision on the duplicate
memexinstallation, then two secrets in the Actions store), Plugins #2125 (a GitHub Appactionspermission grant), Plugins #1109 (a Stripe webhook registration plus one test purchase — that is one of the items this brief says to leave alone, so it is only counted here), core #4684 (deployment-record state on memex-cloud and client-b).The
Checksgate is red as well. On Plugins head8db241c2f2thePlugin Catalog CIrun 35688806005 failed (portal-hosts network-133 legs 3/8 and 5/8, Module bundle batch 5/8) andPortal image rebuildfailed at token mint — which is #2125 again; the same two workflows are red on the previous three main heads. Core@main is green on94689968ef(run 35693857440).Newest sealed set: CD #9141 (
35688584567, head6b3fda2a4833) — the bake job'sPush the sealed publication to the fleet registryandRegister the publication with memexsteps bothsuccess, promote phases A–D andVerify every image shippedsuccess. #9143 concludedsuccesswith every build/promote/seal jobskipped(the decide-not-to-publish branch — a green that sealed nothing), #9144/#9145 cancelled, #9146 (94689968ef) in progress at the time of reading. 🚨 A run's conclusion is not a seal — read the seal step.So the honest content of this tracker is: the cut is blocked on 26
sev:H(6 of them one roll-and-read from closed, 4 waiting on a person), on a redchecks:Plugins@main, and on the maintainer's decision. Nothing here is this tracker's own work; it stays open until the cut, as the previous comment says, and it stayschore.Not established: the tag of #9141 (by run-number convention
3.0.0-ci.9141, not read from a log); whether anysev:Hwas downgraded rather than fixed; whether any plugin is held at a seal older than #9141; the Manual notes gate and the two signatures. The gate rechecks every 5 minutes, so this is a snapshot.