Skip to content

A course root that ships its own NodeTypes stops serving its layout areas after a recycle — the Store paywall never paints, while a NodeType-free root recycled seconds earlier serves in 4.9s #1349

Closed

Description

@rbuergi

Symptom

A course root that ships its own NodeTypes does not serve its Store/Plugin layout areas — the
Subscribe paywall and the cover's CoverCta — to a viewer, for minutes, while a NodeType-free
root recycled in the same loop seconds earlier serves in 4.9 s. The portal is healthy throughout:
Running=true OOMKilled=false Dead=false RestartCount=0, and unrelated courses pass their full
journey on it during the dark window.

This is not a wedge, not OOM, not a dead container, and — since MeshWeaver#1199 — not the old
"activation parks behind an in-flight compile" behaviour either.

The discriminator is exact

Across three CI runs of the Systemorph/education disposable-mesh suite, the split is clean and has
no exception:

ships own NodeTypes courses /{course}/Subscribe
yes AgenticOffice (4 types), AgenticPrimer (2), AgenticPrimerDe (2) never paints
no SwissSolvencyTest, DataModeling, AdvancedBusinessRules, RiskTransfer, ThinkInStreams, ReinsurancePractice paints in 5–10 s

Not language (AgenticPrimer is English, AgenticPrimerDe German), not course kind (AgenticOffice is an
exercise course, the primers are content courses), not size (AgenticOffice 91 files vs
SwissSolvencyTest 71), not root count (AgenticOffice declares 4 install roots; DataModeling declares
22 and is fine).

The controlled observation

Run 31637704027, shard 3. Both
course roots are recycled by the same bootstrap loop, seconds apart:

+482s  20:37:13   AgenticOffice imported (content renders)     ← admin reads a GATED page: root serving
+550s  20:38:21   SwissSolvencyTest imported (content renders)
+581s  20:38:53   course roots recycled                        ← both, same loop
       20:38:54 → 20:48:04   /AgenticOffice/Subscribe paints NONE of its three states,
                             2 × 275 s (10 polls each, 27.6 s per poll)
       20:48:04 → 20:51:05   SwissSolvencyTest: cover CTA 4.9 s, install 35.9 s,
                             copies 19.8 s, exercises 16.3 s — all green

So the same portal, in the same minute, served one recycled root instantly and the other not at all.
Note the AgenticOffice root was serving before the recycle (the admin read a gated page at +482 s).

Independently, on Systemorph/education#144's run, /AgenticPrimerDe/Subscribe did not come up
through 180 s of explicit retries — so this is not a race that retrying wins.

Where I think it lives

MeshWeaver.Plugins' AddPluginGating already carries a mitigation for a neighbouring failure, with
a first-hand incident note (Store/Plugin/Source/PluginLayoutAreas.cs:108-178):

the subtree query is a HUB ROUND-TRIP whose response routes back into the very hub that is still
activating … on a big one (a whole course — hundreds of nodes) the grain saturates and the caller's
SubscribeRequest hits its 60 s timeout, so ACTIVATION ITSELF FAULTS. Orleans then retries and
faults again, forever — the node is permanently unreachable and the page sits on "Subscribing to …".
Recycles and a pod restart do NOT help.
— memex 2026-07-29, AgenticEngineering

The plugins-side answer was DelaySubscription(ReconcileStartDelay) = 5 s. A NodeType-shipping
root's activation additionally drags in enrichment for its own types, and MeshWeaver#1199's
InFlightOverlayGrace is also 5 s (NodeTypeEnrichmentHelpers.cs:71). Two 5 s timers landing on
the same turn of the same activation is at least worth looking at directly.

The consequence is what makes it a core concern rather than a Store one: if the root's activation
does not complete, PluginGate.SeedGating never runs, the cover's Public/Anonymous grants are
never (re)seeded, and an unentitled viewer is denied — which is precisely the observed shape
(the entitled admin could read before the recycle; only fresh learners are dark afterwards). Earlier
runs show the matching AccessControlPipeline: … lacks Read permission storms (48 and 70 lines) on
exactly these courses.

I am flagging that last paragraph as the best-supported hypothesis, not an established mechanism —
the portal's own log for the dark window sits between the captured head/tail excerpts and is only in
the uploaded artifact, so I have not read the activation lines themselves.

What would settle it in one run

Systemorph/education#143 un-quarantines the assertion that measures this (it had been fixme'd
since 2026-08-07 on a cause that MeshWeaver#1199 has since falsified). With education#139's
timestamped diagnostics now in place, three lines discriminate outright:

  • EnrichWithNodeType: compile for '…' still in flight after 5s ⇒ the overlay engaged, compile
    latency is not the story;
  • has no compile in flight and did not settle within 60s, or Store/Plugin … is PARKED ⇒ the
    Phase-A / park path;
  • [ACTIVATE] Grain AgenticOffice: activation faulted + TimeoutException … SubscribeRequest ⇒ the
    activation-fault mechanism above.

Why it matters beyond CI

This is the learner-visible failure "an entitled learner has no way to install the course, and an
unentitled one cannot even reach the paywall to buy it" — on the courses that are most valuable,
since shipping NodeTypes is what makes a course interactive. It reproduces on a clean disposable mesh
with no load, so it is not an artefact of CI resource pressure.

Refs Systemorph/MeshWeaver.Plugins#362 (the Store-side triage, which cleared the Store logic),
MeshWeaver#1199, MeshWeaver#1172 / #1284 (a different, load-driven wedge — explicitly not this).

Activity

  1. rbuergi commented on Aug 12, 2026

    @rbuergi
    ContributorAuthor

    Correction: the discriminator is necessary but NOT sufficient, and it is intermittent

    The table in the description says NodeType-shipping courses' paywall "never paints". That is
    overstated
    — a third run has since landed and AgenticPrimer bootstrapped cleanly twice. Correcting
    it here rather than leaving a claim that would send someone hunting for a deterministic reproducer
    that does not exist.

    The full per-run record, by the one signature that identifies this defect
    (/{course}/Subscribe paints none of its three states → redeeming the … coupon never succeeded, or
    its downstream form NodeType '…/DiceGame' never came live + an AccessControlPipeline denial storm):

    course own types 31601088052 31637704027 31638147896
    AgenticOffice 4 FAIL FAIL FAIL
    AgenticPrimerDe 2 FAIL FAIL FAIL
    AgenticPrimer 2 FAIL pass (8.4 m, 0 denials) pass (8.3 m, 0 denials)
    SwissSolvencyTest 0 pass (CTA 8.8 s) pass (CTA 4.9 s) pass (CTA 4.9 s)
    DataModeling, AdvancedBusinessRules, RiskTransfer, ReinsurancePractice, ThinkInStreams 0 — pass pass

    So the accurate claims are:

    1. Necessary, not sufficient. Every occurrence of this signature is on a course that ships its own
      NodeTypes. No NodeType-free course has ever produced it.
    2. Intermittent for the 2-type courses, consistent for AgenticOffice (3/3) and AgenticPrimerDe (3/3).
      AgenticPrimer — AgenticPrimerDe's structural twin, same two types, English instead of German —
      failed once and passed twice. So this is load- or timing-sensitive, not a hard property of the
      course.
    3. Retrying does not cure it. education#144 retried /AgenticPrimerDe/Subscribe for a full 180 s
      and never got the surface. So the intermittency is in whether the root comes up serving at all,
      not in a short window the caller can wait out.

    One near-miss to rule out explicitly

    ThinkInStreams (0 own types) also fails beforeAll at 0 ms in shards 4 of both runs, which looks
    identical in a test-list summary. It is not this defect: its errors are
    apiRequestContext.post: Timeout 30000ms exceeded and page.goto: Timeout 60000ms exceeded — the
    dev-login/transport timeouts of the separate portal-pressure problem (MeshWeaver#1172 / #1284) — never
    redeeming the … coupon never succeeded. Anyone triaging from the ✘ list alone will conflate the two;
    the error string is what separates them.

    What this changes about the hypothesis

    It argues against anything that would be deterministic for a given course shape (a static type-count
    threshold, a fixed activation budget always being exceeded) and toward a race — most plausibly
    between the root's activation/enrichment work and the gating reconcile, where AgenticOffice's four
    types simply lose it every time and the primers' two lose it sometimes. The ReconcileStartDelay
    (5 s) / InFlightOverlayGrace (5 s) coincidence noted in the description fits a race better than it
    fits a threshold, so I would start there.

    Everything else in the description stands: the recycle-adjacent timeline, the same-loop
    SwissSolvencyTest control, and Running=true OOMKilled=false RestartCount=0 on the portal throughout.

  2. rbuergi commented on Aug 13, 2026

    @rbuergi
    ContributorAuthor

    The portal log for the dark window, finally read — and it rules out the whole compile/overlay family

    I downloaded mesh-diagnostics/portal.log from run 31637704027's shard-3 artifact (6.9 MB, not the ~100 MB one — shard 3 is where AgenticOffice runs). It covers 20:28:22 → 20:51:11, i.e. the entire dark window.

    1. Every line the description offered as a discriminator has ZERO occurrences

    line the issue said would settle it occurrences in 5,379 log lines
    EnrichWithNodeType: compile for … still in flight after 5s 0
    did not settle within 60s 0
    … is PARKED 0
    [ACTIVATE] Grain … activation faulted 0
    compilation-in-progress / compilation-error overlay 0
    EnrichWithNodeType (any) 0

    So the flagged-as-unverified activation-fault explanation is falsified, and so is the overlay one. This is not a NodeType-compilation problem.

    2. The ReconcileStartDelay / InFlightOverlayGrace coincidence is not a mechanism, and cannot be

    Independently of the log, the two 5 s windows cannot interact. InFlightOverlayGrace gates on IsCompileInFlight(typeNode) for the NodeType named by the instance's nodeType field — for a course root that is Store/Plugin, and nothing else (NodeTypeEnrichmentHelpers.cs:724-726, :653-685). The only things that can put Store/Plugin into Pending/Compiling are: the first-build kickoff (needs CompilationStatus is null, NodeTypeCompilationHelpers.cs:315-326), a RequestedReleaseAt stamp on the Store/Plugin node (:777), its own stale-Ok self-heal / TriggerRecompileAndRetry (NodeTypeEnrichmentHelpers.cs:409-441, :1167-1178), and RecompileClosure.AffectedTypes, which marks a type affected only when its own node or its own expanded Source/Test query paths changed (RecompileClosure.cs:41-72). A course's own NodeTypes compiling touches none of them. Please don't spend another run on those two constants.

    3. The paywall DOES come up — it renders the priced branch, coupon row and all

    20:38:57.xxx Created node at e2e-get-agenticoffice/_Billing/Profile by system-security

    _Billing/Profile is written by PluginLayoutAreas.CreateBillingNode, which is only reached past the coupon row on the priced branch of BuildSubscribe. So on the very first poll after the recycle the Subscribe area rendered, for the fresh learner, in the correct state. "The paywall never paints" is the wrong frame.

    The whole learner bootstrap runs cleanly and fast, exactly as SwissSolvencyTest's does 9½ minutes later:

    20:38:54  e2e-get-agenticoffice                                  (partition root)
    20:38:54  e2e-get-agenticoffice/_Access/e2e-get-agenticoffice_Access
    20:38:55  …/_Memex/Selection, …/_Thread/ThreadComposer, …/_Memex/AiSettings
    20:38:56  …/_Notification/9jjS2BKAh0afQgeoAPosHg
    20:38:57  …/_Billing/Profile                    ← the paywall's priced branch rendered
    20:39:23  …/_UserActivity/AgenticOffice         ← the learner is ON the course page
    

    4. What never happens is the WRITE the redemption needs

    In the whole 23-minute log there is not one of:

    • AgenticOffice/_Entitlements/{learner}
    • AgenticOffice/_Access/{learner}_Access
    • Admin/Purchases/*-AgenticOffice-*

    The control course has all three, ~3 s after its learner appears:

    20:49:47  Created node at SwissSolvencyTest/_Entitlements/e2e-get-swisssolvencytest
    20:49:47  Created node at SwissSolvencyTest/_Access/e2e-get-swisssolvencytest_Access
    20:49:47  Created node at Admin/Purchases/20260812-204947197-SwissSolvencyTest-e2e-get-swisssolvencytest
    

    So the failure is the coupon redemption's grant, not the surface that offers it. redeeming the e2e coupon … never succeeded turns out to be literally true and literally the whole story: the learner pressed Redeem ten times over nine minutes and nothing was ever written.

    It is silent server-side by construction — the click's error goes to the panel, never to a log:

    CouponRedemption.RedeemBounded(…)
        .SelectMany(_ => InstallTail(…))
        .Subscribe(s => host.UpdateData(CouponStatusId, s),
                   exception => host.UpdateData(CouponStatusId, exception.Message));

    5. …and the one thing that IS broken sits on exactly the namespace that write targets

    PluginGate.Enroll writes into {course}/_Access. That satellite hub is the only thing in the log with a genuine fault, and it is AgenticOffice-only:

    20:34:07  Created node at AgenticOffice/_Access/Public_Access by system-security
    20:34:17  GetMeshNode('AgenticOffice/_Access/Public_Access') timed out after 10.0s (budget 10s)
              — the owning per-node hub never answered the GetDataRequest. This is NOT 'node not found'.
    20:34:17  Notifier: failed for AgenticOffice/_Access/Public_Access
    20:34:17  Created node at AgenticOffice/_Access/Anonymous_Access by system-security   ← twice, a retry
    20:34:17  Created node at AgenticOffice/_Access/Anonymous_Access by system-security
    20:34:18  Orleans WorkGroup: long-running task … Activation: …/messagehub/AgenticOffice/_Access/Public_Access
    20:34:19  Orleans WorkGroup: long-running task … Activation: …/messagehub/AgenticOffice/_Access/Anonymous_Access
    

    SwissSolvencyTest's two grants land at 20:37:44 with none of it — no timeout, no notifier failure, no duplicate, no work-item warning.

    So the race is between the {course}/_Access satellite hub answering its GetDataRequest inside its 10 s budget, and everything else the course's own NodeType compiles are driving through the same silo at recycle time. More shipped types ⇒ more competing per-node activations ⇒ the _Access hub misses the budget ⇒ the gating seed is incomplete and the redemption's grant write has nowhere to land. That is a load-widened window, which is exactly the intermittency profile the correction comment described (AgenticOffice 4 types 3/3, AgenticPrimerDe 2 types 3/3, AgenticPrimer 2 types 1/3, 0 types never) — and unlike a 5 s constant, it explains why retrying does not win: the retry re-enters the same saturated hub.

    6. Two things that look like the bug and are not

    • Frame loss is background noise here, not the discriminator. The dark window's only AgenticOffice traffic is [SYNC_STREAM] Frame loss detected … requesting fresh snapshot — one fresh stream per 27.6 s poll. But the PASSING course loses more: SwissSolvencyTest averages 7.8 gaps/stream (up to v132) against AgenticOffice's 3.1 (v38–v44). Real defect, separate issue.
    • The AccessControlPipeline … lacks Read permission storms (104 lines) are all e2e-admin, and all before the recycle (20:29:34 → 20:37:58). They are not the learner and not in the dark window.

    What would settle the remaining step in one run

    The redemption failure is invisible because the panel swallows it. Two cheap additions:

    1. Log the redemption failure server-side in CouponRedemption's subscribe (it currently only calls host.UpdateData), so "the learner pressed Redeem and it failed with X" is greppable.
    2. Capture the #store-plugin-coupon panel text in the e2e helper when redeemCouponOnSubscribe gives up — the message the buyer sees is already the diagnosis, and the harness throws it away.

    Refs: falsification of the compile/overlay family is from the run's own portal log, quoted above; the code-level null result in §2 is from NodeTypeEnrichmentHelpers.cs, NodeTypeCompilationHelpers.cs and RecompileClosure.cs at a59c1879.

  3. rbuergi commented on Aug 13, 2026

    @rbuergi
    ContributorAuthor

    Tested against #1362's mechanism: it does not match — #1349 is not a duplicate

    Suggested pairing: #1362's HandleGetDataRequest subscribes with no completion arm, so a stream disposed by a recycle completes silently, the terminal signal is discarded, and the caller's callback sits registered to the wall-clock bound. Shape: HANDLER_EXIT state=Processed → [SYNC_STREAM] … disposed ~30 ms later → silence.

    I re-grepped the shard-3 portal log (no new run). The signature is absent, and what is there is incompatible with it.

    1. None of the markers are near any timeout

    marker occurrences where
    ShuttingDown 0 —
    HandleGetDataRequest 0 —
    stream is disposed 13 20:30:49 and 20:51:06 only
    is shutting down 25 Doc/*, mesh/*, and the AgenticOffice recycle at 20:38:43

    Every GetMeshNode timeout is at 20:34:17 – 20:34:59. The nearest disposed line is 3½ minutes earlier, the nearest shutdown 3¾ minutes later.

    2. The timeouts precede the recycle by four minutes, on freshly-created hubs

    #1362 needs an already-disposed stream. These hubs had just been born:

    20:34:07.046  Node created at AgenticOffice/_Access/Public_Access by system-security
    20:34:17.117  GetMeshNode('AgenticOffice/_Access/Public_Access') timed out after 10.0s
    

    Created, then timed out exactly 10.0 s later. Nothing had been disposed — the AgenticOffice recycle is at 20:38:43.

    3. The runtime was frozen for 9.23 s of the 10.0 s budget

    The line between those two, which I had not looked for before:

    20:34:16.517  Orleans.Runtime.Watchdog: .NET Runtime Platform stalled for 00:00:09.6946577.
                  Total GC Pause duration during that period: 00:00:09.2307230.
                  We are now using a total of 4702MB memory.
                  Collection counts per generation: 0: 2044, 1: 864, 2: 18
    20:34:16.915  crit: [ROUTE] 64 route dispatches are in flight on this silo and not terminating.
    

    92% of the budget was stop-the-world GC. Two of the seven timeouts even report more elapsed than their own budget (after 11.3s (budget 10s), after 10.2s (budget 10s)) — the timer itself fired late, which only happens when the process is not running.

    4. The hub answered late; it did not complete silently

    20:34:18.695  Orleans WorkItemGroup: long-running task in
                  Activation: …/messagehub/AgenticOffice/_Access/Public_Access
    

    That is 1.6 s after the timeout fired. A discarded completion leaves no activation work at all. Your own discriminator — "a saturated hub would eventually answer; a discarded completion never will" — resolves here in favour of saturation.

    5. Five stalls, monotonically rising heap, and the control course hits it too

    20:34:16   stalled 9.69s   GC 9.23s   4702MB
    20:34:34   stalled 7.85s   GC 7.45s   4880MB
    20:37:05   stalled 11.44s  GC 11.16s  5264MB
    20:40:27   stalled 9.78s   GC 9.04s   5641MB
    20:51:02   stalled 11.18s  GC 10.95s  6289MB
    

    resources.csv from the same artifact: the container goes 0.85% → 80% of its 7.752 GiB limit by 20:38 and stays there. And SwissSolvencyTest/06-AggregationAndMvm/_Access/Public_Access times out too, at 20:37:56, co-timed with the 11.4 s stall — so this is not an AgenticOffice property. AgenticOffice simply presents more _Access nodes to seed (7 timeouts vs 1) because it has more children.

    A correction to my own earlier comment

    I wrote that "the retry re-enters the same saturated hub." That phrasing is wrong, and resources.csv is what corrects it: through the dark window (20:39 → 20:48) the portal is idle — 10–15% CPU, memory plateaued at ~78–80%. The redemption is not failing against a busy hub.

    The accurate statement is that the damage was done earlier and is not self-repairing: the gating seed failed on seven of AgenticOffice's _Access nodes at 20:34 during the GC storm (AccessGrantNotifier: failed for AgenticOffice/_Access/Public_Access, plus a duplicated Anonymous_Access create), and after the 20:38:43 recycle the redemption's grant write has nowhere to land — silently, on an idle box, ten times over.

    Verdict

    Shipped

    Systemorph/MeshWeaver.Plugins#453 — the first of the two diagnostics. Every acquisition arm ended in exception => host.UpdateData(CouponStatusId, exception.Message), so all ten redemption failures went into a Blazor data stream and were dropped on navigation. They are now logged (stage, package, viewer, reason) with the panel text kept byte-identical, because StageOf classifies that string and decorating it would read as a stage.

    Still open, in Systemorph/education: have the e2e helper capture the #store-plugin-coupon panel text when redeemCouponOnSubscribe gives up — the message the learner is looking at is already the diagnosis, and the harness throws it away.

  4. rbuergi commented on Aug 13, 2026

    @rbuergi
    ContributorAuthor

    This now reproduces with the memory confound REMOVED — on a portal at 57–61% RSS with zero swap

    Chasing Systemorph/education#77 I re-ran the education disposable-mesh suite against an image carrying #1345 (the Workspace.GetStream per-call hub leak). That fixed the memory failure #77 was about — and #1349 survived it, unchanged, which is the useful part: it now reproduces on a demonstrably healthy portal.

    The discriminator holds exactly, and it is the only thing left failing

    Two runs, same education commit 13288911, two different post-#1345 images (4bcf5e1, dd8e528):

    shard courses outcome portal peak RSS swap used
    1 AdvancedBusinessRules, ThinkInStreams ✅ 5.11 GiB (65.9%) 116 MB
    2 AgenticEngineering, RiskTransfer, AgenticPrimerDe ❌ 4.72 GiB (60.9%) 0 MB
    3 AgenticOffice, SwissSolvencyTest ❌ 4.43 GiB (57.2%) 0 MB
    4 DataModeling, ReinsurancePractice ✅ 6.155 GiB (79.4%) 1286 MB

    The two failing shards are exactly your "ships own NodeTypes" row — AgenticPrimerDe and AgenticOffice — and nothing else. SwissSolvencyTest shares shard 3's mesh and passes.

    They fail at the LOWEST memory of any shard, with zero swap and ~2 GB available, while the shard that PASSES is the one at 79% with 1.3 GB of swap in use. So "the root is slow because the box is thrashing" is now excluded by the data, not just by argument.

    The health signature on the failing shards' full portal.log is clean too — 1 GC stall (3.2 s, heap 2743 MB), 1 long grain turn, 0 [ROUTE] … dispatches are in flight. Pre-#1345 the same shard showed 7 stalls escalating to 12.8 s at 6366 MB, 375 long turns and 7 route reports. The portal is genuinely idle while the root refuses to serve.

    The failures, and the recycle trigger you describe

    shard 2:  Error: NodeType 'AgenticPrimerDe/DiceGame' never came live:
              /AgenticPrimerDe/Wuerfelspiel did not render its type's own surface within 300s.
    shard 3:  Error: installing 'Agentische Buchhaltung' never made its exercise pages render
    

    and the bootstrap timeline puts it immediately after the recycle, as #1349 predicts:

    [bootstrap +101s] course roots recycled
    [bootstrap +124s] AgenticPrimerDe installed (public lesson renders)
       … 300 s of nothing on /AgenticPrimerDe/Wuerfelspiel
    

    Note the install work all completes inside ~2 minutes and the box is idle for the remaining 5 — so the 300 s budget is not the constraint and raising it would only wait longer.

    Why this matters for scheduling

    This is now the only thing keeping Systemorph/education's disposable-mesh gate red, and therefore the only thing blocking its four open PRs (#138, #140, #143, #144), none of which touch these courses. Tracked education-side as Systemorph/education#146.

    Runs: 31652113987 · 31652894389

  5. rbuergi commented on Aug 13, 2026

    @rbuergi
    ContributorAuthor

    Root cause: the two failing courses are the only two that declare language: de-CH, and the harness only speaks English

    My GC-storm conclusion is superseded. Not partial — superseded. It described something real and co-present, and it was not the cause. Details in §4; the answer first.

    The discriminator, exhaustively

    Every course in Systemorph/education, by its index.json → content.language:

    course language result
    AgenticOffice de-CH FAIL 3/3
    AgenticPrimerDe de-CH FAIL 3/3
    AgenticPrimer en pass 2/3
    SwissSolvencyTest en pass
    AdvancedBusinessRules, AgenticEngineering, DataModeling, ReinsurancePractice, RiskTransfer, ThinkInStreams (unset → en) pass

    The two de-CH courses are exactly the two that fail, and no en course has ever produced this signature. "Ships its own NodeTypes" was a confound — AgenticPrimer ships the same two types as AgenticPrimerDe, is English, and passes. That also settles the "something else separates the twins" question: it is the language tag, nothing structural.

    The mechanism, both halves verified

    1. A de-CH course renders its paywall in German for every viewer. StoreTexts.ResolveLocale takes the plugin's declared language first, and matches on the primary subtag:

    public static string ResolveLocale(string? pluginLanguage, string? viewerLocale) =>
        Primary(pluginLanguage) ?? Primary(viewerLocale) ?? "en";
    // Primary("de-CH") → "de";  Supported = { "en", "de" };  For("de") → German

    2. The harness only matches English. e2e/helpers.ts → redeemCouponOnSubscribe:

    const subscribed = page.getByText(STORE.subscribed).first();            // "You're subscribed."
    const redeemBtn  = page.getByRole('button', { name: STORE.redeemCoupon }).first(); // "Redeem coupon"
    const getFree    = page.getByRole('button', { name: /^Get — free$/ }).first();
    await subscribed.or(redeemBtn).or(getFree).first().waitFor({ timeout: 20_000 }).catch(() => {});
    …
    } else {
      return false; // surface not up yet — retry later
    }

    The German paywall's button reads Gutschein einlösen (StoreTexts.German.RedeemCouponAction), and the entitled marker reads Du bist eingeschrieben. Neither locator matches, so the helper falls into return false — "surface not up yet" — and redeemAsLearner retries 10× / 6 s before throwing the error we have all been reading:

    redeeming the e2e coupon for 'AgenticOffice' never succeeded — is the course's Subscribe paywall up, and does the bootstrap-seeded coupon cover this course?

    The paywall was up. Nobody ever pressed the button.

    Every observation fits, including the ones that killed the other models

    • No server error anywhere in the 9-minute dark window — nothing was ever submitted.
    • Zero _Entitlements, zero _Access, zero Admin/Purchases for the learner — same reason.
    • e2e-get-agenticoffice/_Billing/Profile created at 20:38:57 — I flagged this before as proof the paywall rendered. It is now positive evidence: that node is written by CreateBillingNode, reachable only on the priced branch past the coupon row. The German paywall painted completely, coupon box, billing editor and all.
    • Idle box, 10–15% CPU through the dark window.
    • Retrying never wins — the label does not change between retries.
    • Deterministic 3/3 for both de-CH courses.
    • Survives core fix: share a workspace's local reduced streams instead of minting a hub per call (#1324) #1345 and fails at the lowest-memory shards — memory was never the discriminator, in either direction.
    • Immediately after course roots recycled — coincidence: that is simply where the learner journey starts in the bootstrap loop.

    4. On my GC-storm model — superseded

    I claimed a GC storm caused a gating-seed failure on seven _Access nodes and that the broken state outlived the pressure. The A/B kills it: the storm is gone (1 stall / 3.21 s, long grain turns 375 → 1) and the failure is unchanged. The GetMeshNode … timed out after 10.0s warnings and the 9.23 s pause were real and co-present, and causally irrelevant to this failure — they are the separate load problem #1345 fixed, and I mistook co-occurrence for mechanism. The "outlives the pressure" half was right for the wrong reason: nothing was ever broken and then unrepaired; nothing was ever attempted.

    What survives, and stands on its own evidence:

    • The compile/overlay/activation-fault family is falsified — zero still in flight, did not settle, PARKED, activation faulted, EnrichWithNodeType in 5,379 lines. Absence of markers, independent of load.
    • The ReconcileStartDelay / InFlightOverlayGrace coincidence cannot be a mechanism — pure code reasoning.
    • ThreadAgentIntegrationTest blocks shard 4: a per-node hub's GetDataRequest handler runs and never answers #1362 is still not this, and importantly its refutation does not lean on the GC point: the markers are absent, the timeouts precede the recycle by four minutes on freshly-created hubs, and the hub answered late (a WorkItemGroup 1.6 s past the timeout). Only argument 3 of the five was the GC pause; drop it and the null result holds.

    A falsifiable prediction

    Re-running the gate on the current image will still fail on shards 2 and 3, and my #453 diagnostic will emit nothing — no [Acquisition] … FAILED line at all — because no acquisition is ever attempted. If a Warning does appear, this analysis is wrong and the redemption is genuinely failing server-side. I would rather have that checked than assumed; it costs one run and it is a clean two-way test.

    Where the fix goes

    Systemorph/education, e2e/helpers.ts (tracker #146) — the harness must resolve its expected labels from the course's declared language instead of pinning English. The same blindness affects STORE.subscribed, /^Get — free$/, and any other pinned string a de-CH course renders, so a per-course text table keyed off index.json's language is the durable shape, not a one-off string swap.

    A product question worth a separate decision, not a fix here: ResolveLocale makes the plugin's declared language win over the viewer's, so a de-CH course renders German to an English-speaking learner. That is defensible — a German course is German content — but it is the reason a language-blind client breaks, and it is worth stating explicitly somewhere the next harness author will read.

    This is not a core defect. Unless the language-precedence decision above is revisited, #1349 should close as not a platform bug, with the work moving to education #146.

  6. rbuergi commented on Aug 13, 2026

    @rbuergi
    ContributorAuthor

    Confirmed by CI: a harness-only change turns the red green. #1349 is not a platform bug.

    education#147 changes only e2e/ — no platform code, no node content. Result on the required 4-shard matrix:

    shard courses before (main 31652113987) after (#147)
    1/4 DataModeling, ThinkInStreams pass pass
    2/4 AgenticEngineering, RiskTransfer, AgenticPrimerDe fail fail — different test (below)
    3/4 AgenticOffice, SwissSolvencyTest fail pass
    4/4 AdvancedBusinessRules, ReinsurancePractice, AgenticPrimer pass pass

    Shard 3 — AgenticOffice — is green. The step that threw redeeming the e2e coupon for 'AgenticOffice' never succeeded in all three prior runs now reads:

    success  Install shard 3/4's courses as fresh learners and assert the copies exist
    skipped  Mesh diagnostics on failure
    skipped  Run actions/upload-artifact@v4
    

    Shard 2's failure MOVED, and the move is the confirmation

    failing test error
    before bootstrap.setup.ts › the Edu plugin and every course are installed NodeType 'AgenticPrimerDe/DiceGame' never came live: /AgenticPrimerDe/Wuerfelspiel did not render its type's own surface within 300s
    after 15-install-lands.spec.ts › Install lands — AgenticEngineering › the copies are REAL nodes the copy of declared root 'AgenticEngineering/Exercises' must exist at /e2e-get-agenticengineering/AgenticEngineering/Exercises

    The before line is this issue's own downstream form of the signature — and it is the same root cause wearing a different mask: the bootstrap's entitleViaCoupon silently failed on AgenticPrimerDe (German labels), so the admin was never entitled, so a gated page of that course could not render, which the harness reports as "the NodeType never came live". Fix the language and it goes away.

    The remaining failure was masked, not caused. bootstrap is a project dependency of the chromium project, so its failure skipped the entire spec suite — shard 2 had never reached 15-install-lands at all. With bootstrap passing, it does, and a genuinely separate defect surfaces.

    The residual is a real product defect, and it is NOT this issue

    From the run's portal log, the AgenticEngineering learner got everything except the content:

    01:21:30  AgenticEngineering/_Entitlements/e2e-get-agenticengineering
    01:21:30  AgenticEngineering/_Access/e2e-get-agenticengineering_Access
    01:21:30  Admin/Purchases/20260813-012130137-AgenticEngineering-e2e-get-agenticengineering
    01:21:30  e2e-get-agenticengineering/AgenticEngineering            ← the home root
    01:21:38  e2e-get-agenticengineering/_Install/Systemorph-education ← the manifest, written LAST
    

    …and no nodes under e2e-get-agenticengineering/AgenticEngineering/Exercises. The manifest is written last, so the install reported success with that declared root empty. AgenticEngineering declares 11 install roots. That is precisely the 2026-07-27 "entitled but empty" shape 15-install-lands was written to catch — server-side, in the install engine, and untouched by a harness change. It needs its own issue; it should not be folded back into #1349.

    The two-way test — honest outcome: consistent, but NOT discriminating

    The post-#453 failing shard's portal log carries zero [Acquisition] lines, which is what the model predicts. But I am not claiming it as proof, for two reasons:

    • In this run the redemptions succeeded, so no acquisition failure could be logged under either model.
    • The pre-change comparison run (31652113987, 23:46Z) predates Space header: tighten the description→body gap #453 (merged 00:35Z), so it could not have carried the warning either way.

    The discriminating evidence remains what it was: the pre-fix portal log with zero errors of any kind in the dark window, plus the fact that a change touching only e2e/ moves shard 3 from red to green.

    Worth noting for the record: this run also has zero Platform stalled, zero timed out after, and zero never came live. The whole GC/timeout family I originally built a model on is simply gone after core #1345 — and the failure it "explained" went away for an unrelated reason. That model was rightly superseded.

    Recommendation

    Close #1349 as not a platform bug, resolved by education#147 (tracker: Systemorph/education#146). The ResolveLocale precedence question — plugin language beating viewer language — is escalated separately and is deliberately untouched here; the harness is wrong under either answer.

  7. rbuergi commented on Aug 13, 2026

    @rbuergi
    ContributorAuthor

    Closing as not a platform bug. Resolved by education#147 — a change touching only files under e2e/.

    The discriminator was language, not NodeTypes

    course content.language result
    AgenticOffice de-CH FAIL 3/3
    AgenticPrimerDe de-CH FAIL 3/3
    AgenticPrimer en pass
    SwissSolvencyTest en pass
    the other six (unset → en) pass

    Exhaustive and exact. AgenticPrimer ships the same two NodeTypes as AgenticPrimerDe, is English, and passes — so "ships its own NodeTypes", the premise of this issue, was a confound throughout.

    Mechanism, verified in code, both halves:

    1. StoreTexts.ResolveLocale takes the plugin's language first and matches on the primary subtag — Primary("de-CH") → "de", which is supported, so the ?? short-circuits and the viewer's own locale is never consulted. A de-CH course renders German to every viewer.
    2. e2e/helpers.ts → redeemCouponOnSubscribe pinned English: 'Redeem coupon', "You're subscribed.", /^Get — free$/. The German button is Gutschein einlösen; the entitled marker is Du bist eingeschrieben. Nothing matched → return false // surface not up yet — retry later → 10 retries × 6 s → the timeout everyone had been reading as "the paywall never came up".

    The paywall was up the whole time. Nobody ever pressed the button. The _Billing/Profile node written at 20:38:57 is reachable only past the coupon row on the priced branch — the German paywall had painted completely.

    Verification

    Shard 3 (AgenticOffice) was red 3/3 across three prior runs and is now green, with Mesh diagnostics on failure skipped. No platform code moved.

    Shard 2's failure moved rather than clearing, and the move is itself confirming: bootstrap is a project dependency, so its failure had been skipping the whole spec suite — shard 2 had never reached 15-install-lands. What it now hits is a genuine, separate, server-side install defect (manifest written, zero nodes copied), filed as #1371. Masked, not caused.

    Superseded hypotheses, recorded so nobody re-walks them

    • Compilation / overlay / activation faults — falsified: zero markers across 5,379 lines of the dark window.
    • The ReconcileStartDelay (5 s) / InFlightOverlayGrace (5 s) coincidence — cannot be a mechanism: the grace gates only on the type named by the instance's nodeType (Store/Plugin), and nothing a course's own NodeTypes do can put Store/Plugin into Pending/Compiling.
    • ThreadAgentIntegrationTest blocks shard 4: a per-node hub's GetDataRequest handler runs and never answers #1362 (a handled request that never answers) — refuted on five independent grounds, including that the timeouts precede the recycle by four minutes on freshly-created hubs, and the hub answered 1.6 s late rather than never.
    • The GC-stall model (5 stalls, heap 4702 → 6289 MB, 92% of a 10 s budget in stop-the-world) — real and co-present, but causally irrelevant: the whole family is gone since fix: share a workspace's local reduced streams instead of minting a hub per call (#1324) #1345 and this failure persisted unchanged. Co-occurrence mistaken for mechanism.

    One product question left open deliberately

    ResolveLocale letting the plugin's language beat the viewer's is defensible ("a German course is German content") and so is the opposite. It is why a language-blind client broke, but it is not obviously wrong. ResolveLocale was left untouched; only the harness was fixed, which is wrong under either answer. Escalated to the repo owner rather than decided here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions