Repository navigation
A freshly written hub serves every new subscriber a Full behind its own outbound chain — constant-tuple frame loss, and a copied Code cell's Overview never renders (Education mesh-gate red on wave #7955) #3520
Description
What is measured
Education's disposable-mesh gates are red on every run since the pin moved to wave #7955 (portal sha256:47dc1750…, core 1340e351c), with the compile-verdict chain (#3474/#3475) gone and the module assets landed. What is left is one shape, on every course with an embedded Code cell — all four e2e-mesh shards and e2e-install fail the same "install as a fresh learner and assert the copies exist" step (Education run 34077755860, head 62d0f3a8d, on Education #274).
The portal log of that run's e2e-install (30 min, one course, one learner) carries 202 [SYNC_STREAM] Frame loss detected warnings. By owner:
| owner (the hub the mirror asks for a fresh snapshot) | warnings |
|---|---|
e2e-admin/ThinkInStreams/09-RxVersusIx/Exercise/OperatorForge/Source/OperatorForge (the learner's copy of the exercise's Code cell) |
109 |
e2e-admin/Packages |
53 |
e2e-admin/ThinkInStreams/09-RxVersusIx/Exercise/OperatorForge (the exercise node) |
25 |
ThinkInStreams, its lessons, the learner's copies of them |
15 |
The tuples are constant per owner across dozens of DISTINCT stream ids. On the cell:
[SYNC_STREAM] Frame loss detected for <StreamId>: incoming Patch v38 chains onto v37 but the last applied frame is v26 — a frame was lost in transport; requesting fresh snapshot from e2e-admin/ThinkInStreams/09-RxVersusIx/Exercise/OperatorForge/Source/OperatorForge
46 times, 46 different StreamIds, ~4 per minute from 03:04:21 to 03:28 (the run's end), and then incoming Patch v42 chains onto v41 but the last applied frame is v38 14 times. Elsewhere the gap is almost always exactly three: v9→v12 (on v11) ×25, v10→v13 (on v12) ×23, v18→v22 (on v21) ×31, v26→v30 (on v29) ×6, v38→v42 (on v41) ×14.
Random transport loss does not produce identical numbers for 46 subscribers in a row over 24 minutes. Since the mirror adopts the owner's Version on apply (SynchronizationStream.UpdateStream, "Adopt the OWNER's Version") and the chain is per forwarding subscription (JsonSynchronizationStream.ToDataChanged, lastSentVersion), both numbers are the owner's clock for that one subscription: the owner emitted a Full stamped v26 to a brand-new subscriber and, on the same subscription, eleven frames (v27–v37) that never arrived, then v38. The owner hands each new subscriber a Full that is behind its own outbound chain, deterministically, long after it stopped changing.
Timeline on the cell (all from the same log):
03:03:58 Node created at e2e-admin/…/Source/OperatorForge by e2e-admin (the install's copy)
03:04:12 Compile watcher: install#1439 on e2e-admin/…/Source/OperatorForge (a Code node under Source/ is NodeType source)
03:04:12 [ROUTE] Directed delivery to pod hub 'portal/sCK_…' was refused: its OWNER RELEASED the address …
03:04:21 first "v38 chains onto v37 but the last applied frame is v26"
03:04 … 03:28 the same tuple, ~4/min, distinct StreamIds; the numbers never move again
03:10:46 Layout area 'Overview' on e2e-admin/…/Source/OperatorForge was torn down having never rendered — the subscriber only ever saw the "awaiting first data" placeholder
03:11:52 / 03:17:24 / 03:21:48 the same, one per probe round
So the hub took ~12 writes in its first 20 s (copy, compile-watcher stamp, …) and was static from then on — and every stream it served for the next 24 minutes started at v26.
Neighbouring lines, for completeness: 0 × Resync gave up, 0 × re-asserting the current snapshot, 0 × Patch before base Full, 0 × Stale JSON snapshot, 103 × Dropping DataChangedEvent … the target stream is gone (disposed), 33 × [ROUTE] … OWNER RELEASED the address. No client console error; the Playwright screenshot shows the exercise page with its brief and the cell's node card header, and no editor and no Run button where the workbench belongs.
What it costs
Layout area 'Overview' … was torn down having never rendered is the owner-side host's disposal diagnostic with rendered == 0 (LayoutAreaHost, the Warning arm) — the render pipeline on the cell's own hub never emitted a frame, in four consecutive probe rounds. MeshNodeLayoutAreas.Overview is GetMeshNodeStream() ⨯ GetEffectivePermissions(hubPath) ⨯ ObservePartitionRoot(...); the third leg StartWith(null)s, so it is one of the first two. A Debug run (MeshWeaver.Layout, MeshWeaver.Data.SubscribeHandler, MeshWeaver.Data.ClientSubscriptionEviction, MeshWeaver.Mesh.Security at Debug) is in flight on Education #274 (run 34080328179); I will attach which leg stalls, and add the MeshWeaver.Data.Serialization loggers on the next push for the per-frame owner-side trace (is the stale Full built stale, or built fresh and then overtaken?).
Control
- Production (
memex.meshweaver.cloud) renders the ORIGINAL cell'sOverviewinstantly (get …/Source/OperatorForge/area/Overview: the splitter with the Code Files nav and the nestedLayoutAreaControlon the same address). What fails is a freshly created copy whose hub is written to while subscribers arrive. - The warning is not new; the non-convergence is. Loss detection (
79bc1a6bd, 08-10) is in the green 09-04 pin (03642028e), and Education's older red runs on thebbcb22f25pin show 112–216 of the same warnings. Between the green pin and #7955 the sync stream gained23e32e28f(Six of the last ten CI failures were unrelated single integration tests — a flake population taxing every second PR #1384: fault instead of asking forever),2346063e5,804b9beca(Resync give-up (#1384) fires on a routine bulk-install frame loss: ~1-in-3 flake on main, reproducible in MeshWeaver.PluginTester.Test #3058),f95de0621(Race: Layout.Test EditorTest.TestEditorWithDelayed — five UpdatePointer round-trips, the delayed editor's stream emits NOTHING in 30 s (shard 4) #2701) and67e71f53d(Portal managed heap leaks ~215 MB/h through 267 gen2 collections, with ZERO sync/bake events — the cause #2194 is not tracking #3321 step 3). Neither pin's log is a clean control for the cell, because the green run's diagnostics artifact is gone; what is certain is that the same content rendered on 09-04 and does not on #7955. - The terminal
StreamNotConvergingExceptionneedsMaxUnansweredResyncs(3) acknowledged-and-unanswered re-asks on one stream ("counted HERE and nowhere else"). Here every subscriber is a fresh stream and every re-ask IS answered (0 × gave up, 14 streams show a second gap after applying the re-asked v38), so the fault cannot fire by construction — the mirrors converge after one or two round trips. The wedge is upstream of them, on the owner.
🚨 Not a retry/budget problem: the Blazor side already re-subscribes and the cycle repeats; widening anything would only hide the stale Full.
Reproduction
Education #274's CI: pin wave #7955, install ThinkInStreams as a fresh learner (e2e-install), open the copied exercise 09-RxVersusIx/Exercise/OperatorForge, wait for the embedded Source/OperatorForge cell — .md-code-cell / the Run button never appears; the portal log shows the constant-tuple frame loss above.
Ownership checked with meshweaver-bd (core's open issues hold nothing on subscribe-time stale snapshots; #3498 and #3404 are the re-ask and readiness arms, both closed). Filed from Education by meshweaver-education-34.
Activity
rbuergi commentedon Sep 7, 2026
Negative result, recorded as promised: the frame loss is NOT what blanks the cell
The Debug run (34080328179, Layout / SubscribeHandler / Mesh.Security at Debug) plus the Playwright trace's DOM snapshot settle the render question the other way round:
Overview), a host is built, and it renders — the client DOM under "Your workbench" holds the node card, then the generic property grid of the content (Code/Language: csharp/Is Executable: true/Last Executed At… /Activity Parent Path: {viewer}), thenComments. The singletorn down having never renderedper round is a host disposed 10 ms after creation and replaced by a second one (a client re-subscribe), i.e. the Warning arm's "client navigated away" case — not a wedge.MeshWeaver.Graph.Views—GraphViewsModuleAttribute: .AddNodeHubContribution(CodeNodeType.NodeType, c => c.AddCodeViews())— which left the platform image in refactor(views): the node-surface view layer leaves the platform for a module #2678 (db040f4b1, merged 2026-08-29 19:17Z). Education's "green 09-04 pin" (03642028e) is a core commit from 08-29 14:03Z, five hours BEFORE that merge; wave #7955 is the first Education pin whose image has no Code views, and the disposable mesh never loads the module (its nine[ModuleLoad]lines are identical in every run: Grpc, Blazor.Views, Hosting.Instance from appsettings; AI, Blazor.EntityViews, Blazor.Graph, Maps, Markdown.Collaboration, Payments.Stripe from the store). Production hasDefaultViews(pre-installed,module: MeshWeaver.Graph.Views) and renders the shell.module-artifacts: module-bundle-MeshWeaver.{AI,Markdown.Collaboration,Maps,Payments.Stripe}in Plugins'publish-bake, so a pre-installed package's module is sealed out by construction — filed against Plugins).What remains of this issue is exactly the sync-stream signature and nothing else: every stream a freshly created host serves loses the frames that follow its Full — deterministically, the same tuple per page load — and heals by re-ask (0 ×
Resync gave up; the two-gap streams show the re-asked Full applied). The cost is one or two extra round trips per subscription and 100–200 Warnings per run, not a wedge. Severity down accordingly; theMeshWeaver.Data.Serializationper-frame trace is running on Education #274 (run 34082816482) and I will attach whether the lost burst is emitted before the subscriber's transport attaches (built fresh, overtaken) or the Full is built stale, so the elimination is on record.rbuergi commentedon Sep 7, 2026
Verified and closing — both halves accounted for, independently re-measured 2026-09-07 07:2xZ
I treated the "it's fixed" triage as a hypothesis and measured each half myself.
Half 1 — the sync-stream signature: FIXED ON
mainMerged as #3530 (
merged=true,merged_at 2026-09-07T05:29:03Z, merge commitb75f50107). Read back onorigin/main(
67bd3d064):src/MeshWeaver.Data/Serialization/SynchronizationStream.cs~L470-492 — the value-dedup arm nowdoes
current = current with { Version = value.Version }before returning, with the reasoninginline: the owner chained the next frame onto the skipped one, so leaving the mirror's clock
behind turned every no-op into a proven "frame loss" and the mirror re-asked for a snapshot it
already held.
test/MeshWeaver.Data.Test/SameValuePatchKeepsTheChainTest.cs— present onorigin/main(
git cat-file -eexits 0), 101 lines, reproducing the wire shape (Full v2, Patch v3 basedOn v2[], Patch v4 basedOn v3) and failing on the old code with a secondSubscribeRequest.That matches this issue's own narrowing exactly. The commit's measurement — 164 of 164
Frame loss detectedwarnings in one Education run preceded by precisely that skip, and not onea real loss — is the same denominator-carrying shape the issue asked for, and it lands on the arm
the issue identified ("the owner hands each new subscriber a Full that is behind its own outbound
chain"). Cost was a round trip per page load plus 100-200 Warnings per run; no wedge.
Half 2 — "a copied Code cell's Overview never renders": NOT this issue, and it has two homes
The negative result already recorded here on 2026-09-07T04:36Z stands and I did not re-derive it:
the Overview did render — as the generic property grid — because the Code-specific views are
contributed by
MeshWeaver.Graph.Views, which left the platform image in#2678 (
db040f4b1, 2026-08-29 19:17Z), andthe disposable mesh never loads that module. That half is tracked, twice and correctly:
MeshWeaver.Graph.Views, so every Code cell renders the property gridGraph.Viewsbuilt against a newer core and every render throwsNeither is the sync-stream defect and neither is fixable in this repository's
SynchronizationStream, so keeping this issue open would not track them — it would only make themharder to find.
Disposition
Closing. Half 1 is on
mainwith a regression test; half 2 is homed atMeshWeaver.Plugins#1438 and MeshWeaver#3538. No re-scope needed — the title's second clause is
answered by the relocation, not by a rename.
Measured 2026-09-07 07:2x-07:3xZ from a worktree at
origin/main67bd3d064; artefacts:a0713614e, PR #3530 /b75f50107,src/MeshWeaver.Data/Serialization/SynchronizationStream.cs:470-492,test/MeshWeaver.Data.Test/SameValuePatchKeepsTheChainTest.cs.