Skip to content

A freshly written hub serves every new subscriber a Full behind its own outbound chain — constant-tuple frame loss, and a copied Code cell's Overview never renders (Education mesh-gate red on wave #7955) #3520

Closed

Description

@rbuergi

What is measured

Education's disposable-mesh gates are red on every run since the pin moved to wave #7955 (portal sha256:47dc1750…, core 1340e351c), with the compile-verdict chain (#3474/#3475) gone and the module assets landed. What is left is one shape, on every course with an embedded Code cell — all four e2e-mesh shards and e2e-install fail the same "install as a fresh learner and assert the copies exist" step (Education run 34077755860, head 62d0f3a8d, on Education #274).

The portal log of that run's e2e-install (30 min, one course, one learner) carries 202 [SYNC_STREAM] Frame loss detected warnings. By owner:

owner (the hub the mirror asks for a fresh snapshot) warnings
e2e-admin/ThinkInStreams/09-RxVersusIx/Exercise/OperatorForge/Source/OperatorForge (the learner's copy of the exercise's Code cell) 109
e2e-admin/Packages 53
e2e-admin/ThinkInStreams/09-RxVersusIx/Exercise/OperatorForge (the exercise node) 25
ThinkInStreams, its lessons, the learner's copies of them 15

The tuples are constant per owner across dozens of DISTINCT stream ids. On the cell:

[SYNC_STREAM] Frame loss detected for <StreamId>: incoming Patch v38 chains onto v37 but the last applied frame is v26 — a frame was lost in transport; requesting fresh snapshot from e2e-admin/ThinkInStreams/09-RxVersusIx/Exercise/OperatorForge/Source/OperatorForge

46 times, 46 different StreamIds, ~4 per minute from 03:04:21 to 03:28 (the run's end), and then incoming Patch v42 chains onto v41 but the last applied frame is v38 14 times. Elsewhere the gap is almost always exactly three: v9→v12 (on v11) ×25, v10→v13 (on v12) ×23, v18→v22 (on v21) ×31, v26→v30 (on v29) ×6, v38→v42 (on v41) ×14.

Random transport loss does not produce identical numbers for 46 subscribers in a row over 24 minutes. Since the mirror adopts the owner's Version on apply (SynchronizationStream.UpdateStream, "Adopt the OWNER's Version") and the chain is per forwarding subscription (JsonSynchronizationStream.ToDataChanged, lastSentVersion), both numbers are the owner's clock for that one subscription: the owner emitted a Full stamped v26 to a brand-new subscriber and, on the same subscription, eleven frames (v27–v37) that never arrived, then v38. The owner hands each new subscriber a Full that is behind its own outbound chain, deterministically, long after it stopped changing.

Timeline on the cell (all from the same log):

03:03:58  Node created at e2e-admin/…/Source/OperatorForge by e2e-admin         (the install's copy)
03:04:12  Compile watcher: install#1439 on e2e-admin/…/Source/OperatorForge      (a Code node under Source/ is NodeType source)
03:04:12  [ROUTE] Directed delivery to pod hub 'portal/sCK_…' was refused: its OWNER RELEASED the address …
03:04:21  first "v38 chains onto v37 but the last applied frame is v26"
03:04 … 03:28  the same tuple, ~4/min, distinct StreamIds; the numbers never move again
03:10:46  Layout area 'Overview' on e2e-admin/…/Source/OperatorForge was torn down having never rendered — the subscriber only ever saw the "awaiting first data" placeholder
03:11:52 / 03:17:24 / 03:21:48  the same, one per probe round

So the hub took ~12 writes in its first 20 s (copy, compile-watcher stamp, …) and was static from then on — and every stream it served for the next 24 minutes started at v26.

Neighbouring lines, for completeness: 0 × Resync gave up, 0 × re-asserting the current snapshot, 0 × Patch before base Full, 0 × Stale JSON snapshot, 103 × Dropping DataChangedEvent … the target stream is gone (disposed), 33 × [ROUTE] … OWNER RELEASED the address. No client console error; the Playwright screenshot shows the exercise page with its brief and the cell's node card header, and no editor and no Run button where the workbench belongs.

What it costs

Layout area 'Overview' … was torn down having never rendered is the owner-side host's disposal diagnostic with rendered == 0 (LayoutAreaHost, the Warning arm) — the render pipeline on the cell's own hub never emitted a frame, in four consecutive probe rounds. MeshNodeLayoutAreas.Overview is GetMeshNodeStream() ⨯ GetEffectivePermissions(hubPath) ⨯ ObservePartitionRoot(...); the third leg StartWith(null)s, so it is one of the first two. A Debug run (MeshWeaver.Layout, MeshWeaver.Data.SubscribeHandler, MeshWeaver.Data.ClientSubscriptionEviction, MeshWeaver.Mesh.Security at Debug) is in flight on Education #274 (run 34080328179); I will attach which leg stalls, and add the MeshWeaver.Data.Serialization loggers on the next push for the per-frame owner-side trace (is the stale Full built stale, or built fresh and then overtaken?).

Control

🚨 Not a retry/budget problem: the Blazor side already re-subscribes and the cycle repeats; widening anything would only hide the stale Full.

Reproduction

Education #274's CI: pin wave #7955, install ThinkInStreams as a fresh learner (e2e-install), open the copied exercise 09-RxVersusIx/Exercise/OperatorForge, wait for the embedded Source/OperatorForge cell — .md-code-cell / the Run button never appears; the portal log shows the constant-tuple frame loss above.

Ownership checked with meshweaver-bd (core's open issues hold nothing on subscribe-time stale snapshots; #3498 and #3404 are the re-ask and readiness arms, both closed). Filed from Education by meshweaver-education-34.

Activity

  1. rbuergi commented on Sep 7, 2026

    @rbuergi
    ContributorAuthor

    Negative result, recorded as promised: the frame loss is NOT what blanks the cell

    The Debug run (34080328179, Layout / SubscribeHandler / Mesh.Security at Debug) plus the Playwright trace's DOM snapshot settle the render question the other way round:

    • On the cell hub every page load subscribes to the EMPTY area (resolved to Overview), a host is built, and it renders — the client DOM under "Your workbench" holds the node card, then the generic property grid of the content (Code / Language: csharp / Is Executable: true / Last Executed At … / Activity Parent Path: {viewer}), then Comments. The single torn down having never rendered per round is a host disposed 10 ms after creation and replaced by a second one (a client re-subscribe), i.e. the Warning arm's "client navigated away" case — not a wedge.
    • So the Overview is the plain MeshNode details view. The Code-specific Overview (the "Code Files" splitter, the cell, the Run button) is contributed by MeshWeaver.Graph.Views — GraphViewsModuleAttribute: .AddNodeHubContribution(CodeNodeType.NodeType, c => c.AddCodeViews()) — which left the platform image in refactor(views): the node-surface view layer leaves the platform for a module #2678 (db040f4b1, merged 2026-08-29 19:17Z). Education's "green 09-04 pin" (03642028e) is a core commit from 08-29 14:03Z, five hours BEFORE that merge; wave #7955 is the first Education pin whose image has no Code views, and the disposable mesh never loads the module (its nine [ModuleLoad] lines are identical in every run: Grpc, Blazor.Views, Hosting.Instance from appsettings; AI, Blazor.EntityViews, Blazor.Graph, Maps, Markdown.Collaboration, Payments.Stripe from the store). Production has DefaultViews (pre-installed, module: MeshWeaver.Graph.Views) and renders the shell.
    • Fix on the Education side: land DefaultViews' bundle the way the harness already lands EntityViews/GraphViews (the sealed module set on the registry is the hard-coded module-artifacts: module-bundle-MeshWeaver.{AI,Markdown.Collaboration,Maps,Payments.Stripe} in Plugins' publish-bake, so a pre-installed package's module is sealed out by construction — filed against Plugins).

    What remains of this issue is exactly the sync-stream signature and nothing else: every stream a freshly created host serves loses the frames that follow its Full — deterministically, the same tuple per page load — and heals by re-ask (0 × Resync gave up; the two-gap streams show the re-asked Full applied). The cost is one or two extra round trips per subscription and 100–200 Warnings per run, not a wedge. Severity down accordingly; the MeshWeaver.Data.Serialization per-frame trace is running on Education #274 (run 34082816482) and I will attach whether the lost burst is emitted before the subscriber's transport attaches (built fresh, overtaken) or the Full is built stale, so the elimination is on record.

  2. rbuergi commented on Sep 7, 2026

    @rbuergi
    ContributorAuthor

    Verified and closing — both halves accounted for, independently re-measured 2026-09-07 07:2xZ

    I treated the "it's fixed" triage as a hypothesis and measured each half myself.

    Half 1 — the sync-stream signature: FIXED ON main

    $ git merge-base --is-ancestor a0713614e origin/main && echo YES
    YES
    $ git log -1 --format='%H %ci %s' a0713614e
    a0713614ec011cceb69a0a1e5efda927bd3c2f30  2026-09-07 07:12:02 +0200
    fix(sync): a no-op patch is not a lost frame — the mirror adopts its version (#3520)
    

    Merged as #3530 (merged=true,
    merged_at 2026-09-07T05:29:03Z, merge commit b75f50107). Read back on origin/main
    (67bd3d064):

    • src/MeshWeaver.Data/Serialization/SynchronizationStream.cs ~L470-492 — the value-dedup arm now
      does current = current with { Version = value.Version } before returning, with the reasoning
      inline: the owner chained the next frame onto the skipped one, so leaving the mirror's clock
      behind turned every no-op into a proven "frame loss" and the mirror re-asked for a snapshot it
      already held.
    • test/MeshWeaver.Data.Test/SameValuePatchKeepsTheChainTest.cs — present on origin/main
      (git cat-file -e exits 0), 101 lines, reproducing the wire shape (Full v2, Patch v3 basedOn v2
      [], Patch v4 basedOn v3) and failing on the old code with a second SubscribeRequest.

    That matches this issue's own narrowing exactly. The commit's measurement — 164 of 164
    Frame loss detected warnings in one Education run preceded by precisely that skip, and not one
    a real loss — is the same denominator-carrying shape the issue asked for, and it lands on the arm
    the issue identified ("the owner hands each new subscriber a Full that is behind its own outbound
    chain"). Cost was a round trip per page load plus 100-200 Warnings per run; no wedge.

    Half 2 — "a copied Code cell's Overview never renders": NOT this issue, and it has two homes

    The negative result already recorded here on 2026-09-07T04:36Z stands and I did not re-derive it:
    the Overview did render — as the generic property grid — because the Code-specific views are
    contributed by MeshWeaver.Graph.Views, which left the platform image in
    #2678 (db040f4b1, 2026-08-29 19:17Z), and
    the disposable mesh never loads that module. That half is tracked, twice and correctly:

    where it bites tracked as state
    a fresh / disposable mesh never seals MeshWeaver.Graph.Views, so every Code cell renders the property grid MeshWeaver.Plugins#1438 — publish-bake seals a hard-coded four-module set open (2026-09-07T04:36:53Z)
    a live portal ADOPTS a Graph.Views built against a newer core and every render throws MeshWeaver#3538 open (2026-09-07T06:12:48Z)

    Neither is the sync-stream defect and neither is fixable in this repository's
    SynchronizationStream, so keeping this issue open would not track them — it would only make them
    harder to find.

    Disposition

    Closing. Half 1 is on main with a regression test; half 2 is homed at
    MeshWeaver.Plugins#1438 and MeshWeaver#3538. No re-scope needed — the title's second clause is
    answered by the relocation, not by a rename.

    Measured 2026-09-07 07:2x-07:3xZ from a worktree at origin/main 67bd3d064; artefacts:
    a0713614e, PR #3530 / b75f50107,
    src/MeshWeaver.Data/Serialization/SynchronizationStream.cs:470-492,
    test/MeshWeaver.Data.Test/SameValuePatchKeepsTheChainTest.cs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions