Skip to content

🚨 3.0.0-ci.8071: a CreateOrUpdateNodeRequest for an adopted NodeType's own path is Forwarded by every hub and handled by none β€” Reinsurance's Ifrs17/AocConfiguration render gate times out (ok on 8055)Β #3674

Closed

Description

@rbuergi

What

MeshWeaver.Reinsurance's release-wave gate (run 34182440102, 2026-09-08 03:07–03:40Z, meshweaver-framework-released version 3.0.0-ci.8071 = core 13d3249 Γ— plugins 25900d7) fails ONE type:

RED Ifrs17/AocConfiguration: compile=Ok render=FAILED tests=ok
    default area did not materialise within 120s
GATE FAILED β€” render: Ifrs17/AocConfiguration β€” 1 new failure(s)

Control: the same repository, same type, on 3.0.0-ci.8055 (core 7bfe2d9; PR #179's gate, run 34180740287): ok Ifrs17/AocConfiguration: compile=Ok render=ok tests=ok. Nothing in Reinsurance changed between the two gates but the platform.

The trail β€” a write nobody handles

The render waits on the type's release request; that write never gets a verdict:

03:33:16 [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Ifrs17/AocConfiguration seq=397 bound=5000ms β€” the owner never acknowledged this write
03:33:42 [UpdateRemote] VERDICT_TIMEOUT hub=cache/ug9x8HhPXEOl-R2MMjoxPg target=Ifrs17/AocConfiguration bound=31s corr=wDBGIzxJ1k-L1rAgZIpeTg
03:33:42 [RequestNodeTypeRelease] Trigger write failed … MeshNode OwnerUnreachable at 'Ifrs17/AocConfiguration': the owner returned no verdict within 31s
03:33:44 [STALE-CALLBACK] Ifrs17/AocConfiguration: qW9ZKtZgBECUoJmr0gX0eA=CreateOrUpdateNodeRequest@portal/nodeops-ug9x8HhPXEOl-R2MMjoxPg (32368ms)

and the delivery's own fate says where it died:

CreateOrUpdateNodeRequest: AWAITING β†’ POSTED target=portal/nodeops-… @Ifrs17/AocConfiguration
 β†’ RECEIVED@Ifrs17/AocConfiguration β†’ ENQUEUED
 β†’ RECEIVED@mesh/ug9x8Hh… β†’ ENQUEUED
 β†’ ROUTED onTarget=False state=Forwarded @Ifrs17/AocConfiguration
 β†’ RECEIVED@portal/nodeops-… β†’ ENQUEUED
 β†’ ROUTED onTarget=False state=Forwarded @mesh/ug9x8Hh…
 β‡’ the delivery reached a hub but no handler

The request for the NodeType's own path is forwarded by the node hub, by the mesh hub and by the nodeops hub β€” each says "not mine" β€” and no handler ever runs. Earlier in the same run (03:28:21) the receiving hub also logged Received '$type':'AocConfiguration' which is NOT registered in this (receiving) hub β€” it can only be read as an untyped JsonElement, and at 03:31:27 Prebuilt assembly ADOPTED for Ifrs17/AocConfiguration at version 1 β€” no compile needed.

Where to look

Core a57a6b52b..13d324992 is dominated by the module-adoption work β€” #3648 (declared floor advisory, load measured), #3649 (a generation that cannot load falls back to the previous one), #3650 (eager adoption, fallback re-examination, pending restart), #3651 (roll gate on measured loadability), #3662 (the module load context resolves through the platform before its own directory) β€” i.e. which assembly a prebuilt-adopted NodeType's types bind to. A type adopted "at version 1 β€” no compile needed" whose $type the receiving hub does not know, followed by a write to that node that every hub forwards, is the shape of an owner hub that came up with the wrong assembly identity for its own type.

This is the one thing holding Reinsurance's publication (and so its Hosting/Publication registration with upstreams: [plugins, crm]) on the newest platform; the directed wave, the crm grant and the resolve-by-version fix (Reinsurance#179) all passed in the same run.

Activity

  1. rbuergi commented on Sep 8, 2026

    @rbuergi
    ContributorAuthor

    The owner is gone, not misrouted. Same gate log, 10 s before the write in the report:

    03:33:06.243 [QUIESCE-START] Ifrs17: 1 pending callbacks at dispose entry: Qn1N3OfiYky0NlBFvRt3Kw=CreateOrUpdateNodeRequest@portal/nodeops-ug9x8HhPXEOl-R2MMjoxPg(12009ms)
    03:33:08.246 [QUIESCE-TIMEOUT] Ifrs17: 1 callback(s) still pending after 2s β€” forcibly cancelling
    03:33:08.253 [PluginGating] reconcile failed for Ifrs17 … HubDisposedBeforeResponseException: Hub Ifrs17 was disposed before the response arrived (request type CreateOrUpdateNodeRequest …)
    03:33:16.640 [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Ifrs17/AocConfiguration seq=397 β€” the owner never acknowledged this write
    03:33:42.662 [UpdateRemote] VERDICT_TIMEOUT target=Ifrs17/AocConfiguration corr=wDBGIzxJ1k-L1rAgZIpeTg
    

    The Ifrs17 PARTITION hub β€” the owner of Ifrs17/AocConfiguration β€” was disposed by the gate's plugin-gating reconcile (the recycle) while a CreateOrUpdateNodeRequest to it was in flight; the pending caller got HubDisposedBeforeResponseException (the reconcile) and the NEXT writes to the partition (the release trigger for AocConfiguration) found no owner and were forwarded by every hub, handled by none β€” the fate trail in the report. The same shape repeats for SST at 03:37:18 (QUIESCE-START SST: 1 pending callbacks … QUIESCE-TIMEOUT … forcibly cancelling).

    So this is the open owner-side half of #3510 ("the CREATE leg has no disposal-NACK under SettleRetypedRoot's recycle; owner-side disposal NACK still open") surfacing in a satellite gate, and what changed between 8055 and 8071 is most likely WHEN the recycle fires relative to the gate's writes (the #3648–#3651/#3662 adoption work moves module load/adoption into the boot the gate observes). Two halves: (1) a hub being disposed must NACK β€” not cancel β€” every pending and every subsequently-routed request for the paths it owned, so the caller gets a terminal instead of a 31 s timeout; (2) the recycle must not run while the gate's own writes to that partition are in flight.

  2. rbuergi commented on Sep 8, 2026

    @rbuergi
    ContributorAuthor

    The same race now reds core's own CD seal. CD run 34190841613 (core 4fedf45 Γ— plugins 7c640de, 2026-09-08 06:24Z), job Plugins: bake + seal … Bake + publish NodeType assemblies:

    06:24:08.936 [QUIESCE-START] Chess: 1 pending callbacks at dispose entry: LIQeRW4kD0O3UIS2-JmXGw=CreateOrUpdateNodeRequest@portal/nodeops-hEx9wO…
    06:24:10.939 [QUIESCE-TIMEOUT] Chess: 1 callback(s) still pending after 2s β€” forcibly cancelling
    06:24:10.939 [PluginGating] reconcile failed for Chess … HubDisposedBeforeResponseException: Hub Chess was disposed before the response arrived
    06:24:16.702 [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Chess/_Policy seq=640
    06:24:42.815 [UpdateRemote] VERDICT_TIMEOUT target=Chess/_Policy … MeshNode OwnerUnreachable at 'Chess/_Policy'
    [FAIL] Chess … idempotence: re-install failed: Install of 'Chess/_Policy' failed: Inner UpdateNode faulted: MeshNode Unknown at 'Chess/_Policy': TimeoutException
    GATE FAILED β€” idempotence: Chess β€” 1 new failure(s)
    

    Identical shape to Ifrs17/AocConfiguration above: the package root hub is disposed by the plugin-gating reconcile with a CreateOrUpdateNodeRequest in flight, the pending caller is force-cancelled, and the next write to a node of that partition finds no owner. Same core (4fedf45 vs 67cbbe0 differ only by #3678, which does not touch hubs) sealed fine an hour earlier on plugins 2e1e430 β€” so this is a race, not a deterministic break, and it decides whether a set seals. Until #3510's owner-side half lands, every seal and every satellite gate is a coin toss on this race.

  3. rbuergi commented on Sep 8, 2026

    @rbuergi
    ContributorAuthor

    Narrowed further β€” the block is a persistence READ that never completes, and it holds the mesh's ONE node-CRUD execution hub for the inner-create bound.

    The 51 s turn on portal/nodeops-ug9x8Hh… (SLOW_DISPATCH | CreateNodeRequest | Elapsed: 51514ms, 03:34:03) is Ifrs17/ValuationApproach/_Activity/compile-state; the same shape at 10.5 s and 9.4 s, and CreateOrUpdateNodeResponse dispatches of 26 s and 109 s on the same hub. The fate trail of one of them (LOohf5BJ…=CreateNodeRequest, path Ifrs17/YieldCurve/_Activity/compile-state):

    POSTED β†’ RECEIVED@nodeops β†’ ENQUEUED(+0ms) β†’ ROUTED onTarget=True(+5406ms) β†’ HANDLER_ENTER(+5406ms)
    β†’ CREATE_CHAIN_SUBSCRIBED path=Ifrs17/YieldCurve/_Activity/compile-state(+5406ms)
    β‡’ a handler was entered and no reply, completion or fault has been recorded since   [pool threads=18 pendingWork=0 completed=783500]
    

    CREATE_CHAIN_SUBSCRIBED is the stage HandleCreateNodeRequest records right after subscribing persistence.Read(node.Path); nothing follows, and the IO pool is IDLE (pendingWork=0) β€” so the read's observable never emitted and no IO was outstanding for it. The caller side confirms the bound: 03:33:42 [CreateOrUpdate] the inner CreateNode for Ifrs17/ValuationApproach/_Activity/compile-state produced no response within 00:00:36. Because the execution hub advances one turn at a time and the turn is the request until it is answered, every other write in the mesh queues behind that stalled read for those 36–51 s β€” which is exactly when the root recycle's QUIESCE-TIMEOUT cancels the reconcile's pending write (03:33:06) and the release trigger for AocConfiguration times out at 31 s (03:33:42). 51 distinct CreateOrUpdateNodeRequest@nodeops callbacks went stale in this run; their last stage is uniformly ENQUEUED@nodeops with no HANDLER_ENTER.

    So the chain is: a partition-scoped persistence read for Ifrs17/*/_Activity/compile-state never completes (during/after the Ifrs17 root recycle) β†’ the single node-CRUD turn waits its full bound β†’ every write in the mesh stalls β†’ the recycle cancels what is pending β†’ the next writes to the partition find no owner. The read is the thing to find; the code/activities satellite data source for a partition whose root hub is being recycled is where I would look first (a promise-cached data source that latches during disposal would produce exactly an observable that neither emits nor errors β€” the #1369 shape).

  4. rbuergi commented on Sep 9, 2026

    @rbuergi
    ContributorAuthor

    Still unverified β€” and the reason nobody has seen it again is that this repo is pinned to the CONTROL

    Checked whether the regression still reproduces. It has not been exercised since it was filed.

    MeshWeaver.Reinsurance's recent green runs are on 3.0.0-ci.8055 β€” measured from the Required CI inputs job of run 34294279477 (2026-09-09 00:16Z, repository_dispatch, success):

    MW_PLATFORM_SET: 3.0.0-ci.8055
    

    That is this issue's own control β€” the version where Ifrs17/AocConfiguration renders ok. The type is present in that run's gate log (Ifrs17/AocConfiguration [15 source(s), 7 dependency record entr(ies)]), so it was genuinely processed and genuinely passed. On 8055. Which we already knew.

    🚨 So the green runs are not evidence the 8071 defect is fixed. They are the control re-run. The failing side (8071 = core 13d324992) has not been retried, and Reinsurance's pin has not moved past 8055, so nothing here would surface it either way.

    What would settle it: a Reinsurance gate on a platform β‰₯ 8071 β€” either a pin move, or a release wave that actually reaches this repo. Note the second is currently not happening: no satellite has received a repository_dispatch since 2026-09-09T02:00Z, while the platform sealed s1218dc0… at 02:53–04:33Z (measured across all six repos; recorded on MeshWeaver#3768).

    One methodological note, because it nearly caught me: the gate log also contains the string 3.0.0-ci.7845, which is not the platform under test β€” it is a minMeshVersion inside a package's PluginContent. The platform version has to be read from MW_PLATFORM_SET in the preflight job, not grepped out of the gate log.

  5. rbuergi commented on Sep 9, 2026

    @rbuergi
    ContributorAuthor

    Re-reading the ledger: portal/nodeops did NOT forward it β€” it never produced a verdict at all

    This issue reads the trail as "forwarded by the node hub, by the mesh hub and by the nodeops hub β€”
    each says 'not mine'"
    . The @ suffix in a request-fate entry is the hub that recorded the
    stage
    , not the delivery's target:

    // src/MeshWeaver.Messaging.Hub/RequestFateLedger.cs
    public void Add(string stage, Address hub)
    {
        …
        var entry = $"{stage}@{hub}(+{elapsedMs}ms)";

    and the routing verdict is written by the hub doing the routing:

    // src/MeshWeaver.Messaging.Hub/MessageService.cs:1536
    fate?.Add($"ROUTED onTarget={isOnTarget} state={delivery.State}", Address);

    So regrouping the quoted trail by recording hub:

    hub stages recorded complete?
    Ifrs17/AocConfiguration RECEIVED β†’ ENQUEUED β†’ ROUTED onTarget=False Forwarded βœ… and correct β€” the target is portal/nodeops, not this node
    mesh/ug9x8Hh… RECEIVED β†’ ENQUEUED β†’ ROUTED onTarget=False Forwarded βœ… and correct β€” the router forwards
    portal/nodeops-ug9x8Hh… RECEIVED β†’ ENQUEUED β†’ nothing ❌ no ROUTED, no HANDLER_ENTER

    Two hubs did exactly what they should. The third enqueued the delivery and never took the turn.
    Truncation does not explain the gap either: the ledger keeps a sliding tail so "the terminal
    evidence the verdict reads can never be the part that gets dropped"
    , and a ROUTED@portal/nodeops
    would be newer than the ROUTED@mesh that IS shown.

    So this is not "handled by none" in the routing sense. It is a pump that did not turn, on
    portal/nodeops β€” and every other line in the report is downstream of exactly that:

    ADVANCE_WITHOUT_HANDOFF path=Ifrs17/AocConfiguration seq=397 bound=5000ms β€” the owner never acknowledged
    VERDICT_TIMEOUT hub=cache/… target=Ifrs17/AocConfiguration bound=31s
    [STALE-CALLBACK] …=CreateOrUpdateNodeRequest@portal/nodeops-…(32368ms)
    

    Which makes this the #2543 / #3593 family, on the hub #2543 names

    The candidate list in "Where to look" β€” the module-adoption work, #3662's load-context change,
    "an owner hub that came up with the wrong assembly identity for its own type" β€” would explain a
    handler that refuses or a type that will not bind. It does not explain a delivery that is
    enqueued and never dequeued, because nothing about assembly identity is consulted before the turn
    starts. Worth setting aside unless the pump reading is falsified.

    (The Received '$type':'AocConfiguration' which is NOT registered in this (receiving) hub line at
    03:28:21 is real but separate β€” that is the untyped-content family, five minutes earlier, and it
    would make a view render empty rather than strand a write.)

    What is now available that was not when this was filed

    #3798 merged today. The pump verdict no longer asserts its mechanism β€” it measures it, and prints
    which of three it is:

    • drainsInFlight > 0 β†’ a drain body is running and blocked before the dequeue;
    • drainsAwaitingScheduler > 0 β†’ the turn scheduler accepted work and has not run it;
    • both zero β†’ the latch leaked (a defect in MessageService, now unreachable).

    portal/nodeops is a hosted hub, so it runs on TaskScheduler.Default β€” which makes
    drainsAwaitingScheduler > 0 read as pool starvation or a local-LIFO stall, not a dead
    activation scheduler. That distinction is in Doc/Architecture/DisposalStallVerdicts.

    🚨 The verdict fires from the disposal stall detector, so it will not print for a hub that is
    merely stalled while alive. If this recurs before that gap is closed, the reading is still available
    directly: MessageService.GetQueueSnapshot() carries DrainsInFlight and DrainsAwaitingScheduler,
    and GetPendingRequestDiagnostics() prints them.

    Status

    Unchanged as to reproduction β€” this repository is still pinned to the control (3.0.0-ci.8055), so
    nothing here has re-run the failing side. What changes is where to look when it does: at the
    nodeops pump, alongside #2543, rather than at assembly identity.

  6. rbuergi commented on Sep 9, 2026

    @rbuergi
    ContributorAuthor

    The regression framing does not hold β€” both commits carry the same race

    This issue localises the failure by contrast: 3.0.0-ci.8055 (core 7bfe2d907) ok,
    3.0.0-ci.8071 (core 13d324992) red, "nothing in Reinsurance changed between the two gates
    but the platform"
    . Measured on main:

    commit date carries #3603 (MemoryCache.GetOrCreate update-queue publication)?
    7bfe2d907 β€” the control 2026-09-07 20:22Z yes
    13d324992 β€” the failure 2026-09-08 01:37Z yes
    git log -S"GetOrCreate" -- src/MeshWeaver.Hosting/MeshNodeStreamCache.cs
      88201f021  2026-09-07T14:16:45Z  Merge pull request #3603 …
    

    Both are downstream of it. So the two gates ran the same code in the respect that matters, and a
    green control is exactly what a race produces most of the time β€” it cannot localise anything. Any
    bisect between those two commits is looking for a difference that is not there.

    What the trail actually points at

    ADVANCE_WITHOUT_HANDOFF … the owner never acknowledged this write, then VERDICT_TIMEOUT β€” a write
    that is neither applied nor refused. #3818 (merged 2026-09-09 12:08Z) names that mechanism:

    Each overlapping cold factory can return its own candidate and replace an entry another caller has
    just published. […] replacement either leaves two live queues or disposes a queue another caller
    already holds. Disposing the subscription alone gives that caller no completion or error.

    "No completion or error" is precisely a write with no verdict. And #3603 β€” which introduced the
    GetOrCreate publication β€” merged 2026-09-07 14:16Z, six hours before the control gate and
    eleven before the failing one, which fits a race that had just become reachable and fired on its
    second outing rather than its first.

    🚨 What would confirm it, and what would not

    MeshWeaver.Reinsurance pins core, so its gate cannot see #3818 until MW_PLATFORM_REF moves past
    it. When it does:

    • A single green gate confirms nothing. The control here was a single green gate on the racy
      code. That is the trap this issue already walked into once, and I walked into its cousin on
      Reinsurance#178 today β€” I expected a specific commit to be the transition there and three runs had
      already gone green on the pre-fix binary.
    • What would confirm it is the mechanism: a recurrence after the pin move, carrying the same
      ADVANCE_WITHOUT_HANDOFF trail, would falsify it outright.

    Until the pin moves, the honest state is: the symptom matches a race that both of this issue's
    commits carry, and that race is now fixed on core main.

  7. rbuergi commented on Sep 9, 2026

    @rbuergi
    ContributorAuthor

    The pin that makes this testable is moving β€” MeshWeaver.Reinsurance#184

    This issue cannot see its own candidate fix, and the reason is structural: MeshWeaver.Reinsurance was pinned at 7bfe2d907 / 3.0.0-ci.8055 β€” this issue's own green control. So neither #3802 nor #3818 has ever been in front of the gate that reports the failure.

    MeshWeaver.Reinsurance#184 moves it to 3.0.0-ci.8195 / core 174f5ab77, the first sealed set carrying both:

    #3802 09:50Z node-CRUD response continuations leave the responding hub's action block
    #3818 11:55Z an update-queue cold factory could dispose a queue another caller already held β€” no completion, no error

    Sealed verified rather than assumed (CD run 34353419069, jobs?filter=all: Promote, Verify, Bake platform content and both Plugins: bake + seal sub-jobs success), and exercised β€” MeshWeaver.Plugins moved onto the same set today and its full CI is green on it.

    🚨 What a green gate afterwards will and will not mean

    Repeating the caution already established in this thread, because the bump makes it immediately relevant: a single green gate confirms nothing. The control that made this issue look like a regression was itself a single green gate on the racy code. What the move buys is the falsifying direction β€” a recurrence on this pin, carrying the same ADVANCE_WITHOUT_HANDOFF β†’ VERDICT_TIMEOUT trail, would rule the hypothesis out outright.

    So the honest state after #184 lands is unchanged in kind: the symptom matches a race both of this issue's commits carried, and that race is now fixed on the pin the gate runs. Only a recurrence, or a long clean run, moves it.

  8. rbuergi commented on Sep 10, 2026

    @rbuergi
    ContributorAuthor

    Re-measured 2026-09-10 β€” and one instance of this shape is now FIXED and pinned

    The falsifying observation is still unavailable, for a NEW reason

    The pin move landed (Reinsurance#184, merged 09-09 15:48Z) and the gate now resolves well past this issue's control β€” 3.0.0-ci.8261 / core 47d94201e, read from the preflight's own resolve-platform verdict, not grepped out of the gate log.

    🚨 But the gate is not reaching the render step at all. Every Reinsurance Plugins CI run today ends in test-repos / Gate shard 1/1 at:

    ##[error]upstream 'plugins' has no SEALED publication at https://memex.meshweaver.cloud
    for framework identity s6c3644c091f6675035c9cfaf59453d21.
    

    (run 34467892111, 10:47Z; same on 34454816677, 34454314821, 34454162282 β€” four runs, one cause.) Ifrs17/AocConfiguration has therefore still never been rendered on a platform β‰₯ 8071. The blocker moved from "the pin is the control" to "the upstream publication is not sealed for this identity" β€” different cause, same consequence for this issue.

    What I could reproduce deterministically

    Not the routing claim β€” that was already falsified above (the @ names the RECORDING hub). What reproduces is the shape: the caller of a sanctioned lifecycle request receives no terminal at all.

    HandleCreateOrUpdateNodeRequest returns Processed() at once and owes its response from chains that outlive the turn. Three of those chains subscribed with two of Rx's three terminations, so a source that completes empty ran neither arm β€” no reply, no fault, no log line, and a trail reading HANDLER_ENTER … β†’ HANDLER_EXIT state=Processed while the caller waits out its whole budget.

    The reachable one is the no-op probe, i.e. a RE-INSTALL. Identical content takes the no-op branch, which asks hub.GetEffectivePermissions(path, requestedBy) whether the caller could have written anyway β€” and that probe terminating without ever emitting is documented, not hypothetical: it is half of what HubPermissionExtensions classifies as Undetermined, "because it FAULTED, or because it terminated without ever emitting (#2742)".

    🚨 That CD 7950's [FAIL] Chess … idempotence: re-install failed took this branch is a FIT, not a measurement β€” a re-install is a no-op upsert, but a leg that logs nothing leaves nothing to name it with. The narrow claim is: the branch was silent on an outcome its own source is documented to produce.

    Control, both directions (Release, full rebuild each way), on #3915:

    • fix reverted, test kept β†’ RED in 36 s: Expected the observable to emit a value within 36s … but it did not. The observable emitted nothing at all.
    • fix re-applied β†’ GREEN in 0.7 s.

    Silence is now handed to the ordinary write path (exactly as the FAULT arm already was) so the owner stays the single authority on allow/deny; the other two legs refuse and say the outcome is unknown. No bound moved, nothing is retried, nothing swallowed.

    🚨 This does NOT close the issue β€” two things are untouched

    1. The nodeops pump. The 09-09 reading stands: portal/nodeops recorded RECEIVED β†’ ENQUEUED and never took the turn. Nothing about handler selection or assembly identity is consulted before a turn starts, so that is a separate defect (Bake wedge: portal/nodeops stops draining mid-bake β€” one shared node-CRUD hub seen from eight places (queue latency bimodal: <=3.2s vs 33-49s)Β #2543 / Hub disposal wedges before Quiescing: the action block never dequeues the buffered ShutdownRequest(Quiescing)Β #3593) β€” and fix(#3593): a failed drain schedule must not latch the pump, and the verdict must measure its mechanismΒ #3798's verdict still only prints for a hub stalled while disposing, never one merely stalled while alive.
    2. Whether the Reinsurance gate reproduces, which needs the plugins publication sealed for the current framework identity first. The thread's standing caution holds: a single green gate afterwards confirms nothing; only a recurrence (falsifying) or a long clean run moves it.

    Work product committed to Doc/Architecture/WriteVerdictTotality β€” including a correction: that page's per-leg table judged "covered" by "does it survive the owner being disposed under it?", and a leg can answer yes to that and still answer nobody on an empty completion.

  9. rbuergi commented on Sep 10, 2026

    @rbuergi
    ContributorAuthor

    Addendum: the control run reproduces THIS ISSUE'S OWN ARTEFACT, not merely its symptom

    Worth recording separately, because it settles the reading this thread has been arguing about for two days. With the fix backed out, this repository's test host emits the same line the report quotes β€” including the [STALE-CALLBACK] … CreateOrUpdateNodeRequest@portal/nodeops-… signature and the same interleaved Forwarded entries:

    [STALE-CALLBACK] client/…: 1 callback(s) pending > 30000ms:
      …=CreateOrUpdateNodeRequest@portal/nodeops-…(35000ms)
      AWAITING β†’ POSTED target=portal/nodeops-… β†’ RECEIVED@client/… β†’ ENQUEUED@client/…
      β†’ RECEIVED@mesh/… β†’ ENQUEUED@mesh/… β†’ ROUTED onTarget=False state=Forwarded@client/…
      β†’ RECEIVED@portal/nodeops-… β†’ ENQUEUED@portal/nodeops-…
      β†’ ROUTED onTarget=False state=Forwarded@mesh/…
      β†’ ROUTED onTarget=True state=Submitted@portal/nodeops-…
      β†’ HANDLER_ENTER@portal/nodeops-…
      β†’ UPSERT_READ existing β†’ no-op probe@portal/nodeops-…
      β†’ HANDLER_EXIT state=Processed@portal/nodeops-…
      β‡’ a handler was entered and no reply, completion or fault has been recorded since
    

    Compare with the report's β‡’ the delivery reached a hub but no handler. It is the same family of trail, and the two Forwarded lines are correct in both β€” the 09-09 regrouping was right. What names the defect is the part the report's capture did not carry: onTarget=True β†’ HANDLER_ENTER β†’ the leg's own stage β†’ HANDLER_EXIT state=Processed β†’ nothing.

    🚨 So the two candidate causes are distinguishable by exactly one stage, and it is worth stating the test explicitly for whoever gets the next capture:

    trail on portal/nodeops cause
    ENQUEUED and no HANDLER_ENTER the pump did not turn β€” #2543 / #3593, still open
    HANDLER_ENTER β†’ a leg's stage β†’ HANDLER_EXIT state=Processed β†’ nothing a detached-reply leg answered nobody β€” fixed by #3915

    #3674's own capture is truncated before either discriminator, which is why both readings survived it. The next one need not be: every leg now records a stage, so a trail that stops at a stage names the leg.

  10. rbuergi commented on Sep 10, 2026

    @rbuergi
    ContributorAuthor

    Reopened β€” the 2026-09-10 sweep was not a triage decision

    This issue was closed not_planned at 2026-09-10T14:30:54Z by a bulk sweep that PATCHed every open issue in all 13 Systemorph repositories β€” 311 issues in twelve minutes, no comment on any of them β€” after a session was asked to "close all issues in all repos". Nothing on this thread was read, and no judgement was made about this issue.

    All 82 issues the sweep closed in the MeshWeaver fleet were reviewed afterwards. This one is unresolved: One instance is fixed and pinned; the general shape and the falsifying observation are still open.

    Sixteen of the 82 were left closed β€” their fix had actually landed, or the author had withdrawn them; those should have read completed, not not planned.

  11. rbuergi commented on Sep 11, 2026

    @rbuergi
    ContributorAuthor

    Re-measured 2026-09-11 06:35Z β€” the falsifying observation is finally available, and it is green. Closing, with the residue named.

    The closing condition as this thread states it

    From the 09-10 comment, the two things that were untouched:

    1. The nodeops pump. … that is a separate defect (Bake wedge: portal/nodeops stops draining mid-bake β€” one shared node-CRUD hub seen from eight places (queue latency bimodal: <=3.2s vs 33-49s)Β #2543 / Hub disposal wedges before Quiescing: the action block never dequeues the buffered ShutdownRequest(Quiescing)Β #3593).
    2. Whether the Reinsurance gate reproduces, which needs the plugins publication sealed for the current framework identity first. … a single green gate afterwards confirms nothing; only a recurrence (falsifying) or a long clean run moves it.

    Item 2 is this issue's own symptom. It is now measurable, and measured.

    The gate reaches the render step, and Ifrs17/AocConfiguration renders

    The 09-10 blocker (upstream 'plugins' has no SEALED publication … for framework identity s6c3644c0…) is cleared. Two independent gates, two different platform sets, both far past this issue's 8071:

    run platform (read from the preflight's own resolve-platform verdict, not grepped from the gate log) verdict line
    34557281969 3.0.0-ci.8319 β€” core b65c1957d 03:29:23Z ok Ifrs17/AocConfiguration: compile=Ok render=ok tests=ok
    34567369483 3.0.0-ci.8323 β€” core 6231c4da4 06:02:08Z ok Ifrs17/AocConfiguration: compile=Ok render=ok tests=ok

    And it was genuinely processed, not skipped β€” from 34567369483's shard log:

    05:54:38  ok  Ifrs17/AocConfiguration [15 source(s), 7 dependency record entr(ies)]
    05:59:44  Prebuilt assembly ADOPTED for Ifrs17/AocConfiguration at version 0
              (framework s090ff34bfea40729de2c6b37f00a6fcb, module version cf802b1798cbd9c3) β€” no compile needed
    05:59:51  [ENRICH-DIAG] node=Ifrs17/AocConfiguration/GateProbe … def.Status=Ok
    05:59:52  ok  Ifrs17/AocConfiguration [compile:Passed render:Passed tests:Passed]
    06:02:08  ok  Ifrs17/AocConfiguration: compile=Ok render=ok tests=ok
              Tests host: Ifrs17/AocConfiguration/GateProbe β€” the probe instance the gate created for this check
    

    Not a single green gate β€” a clean run. test-repos / Gate shard 1/1 is success on 11 of the 11 completed Reinsurance Plugins CI runs that carry it today, 03:08Z β†’ 05:48Z (three further runs carry no gate job at all). Today's red runs are red in publish-bake / Bake + publish NodeType assemblies to portal β€” spot-checked on four of them (34565251320, 34564240111, 34563886244, 34562587738): the failing job is publish-bake in every case, never the gate.

    No recurrence of the trail. Neither ADVANCE_WITHOUT_HANDOFF path=Ifrs17/AocConfiguration nor VERDICT_TIMEOUT … Ifrs17/AocConfiguration appears in either gate log.

    The fix that covers the reproducible half is in the code those gates ran

    #3915 by FILE/SYMBOL (not ancestry β€” queue-merged commits fail is-ancestor):

    ref src/MeshWeaver.Mesh.Contract/DetachedReplyOutcome.cs MeshExtensions.cs references
    origin/main present 4
    b65c1957d (gate platform 8319) present 4
    6231c4da… (gate platform 8323, and memex.meshweaver.cloud) present 4
    45306a33… (memex.systemorph.com) present 4

    That is the leg whose control ran both ways on #3915: fix reverted β†’ RED in 36 s with "the observable emitted nothing at all"; fix re-applied β†’ GREEN in 0.7 s.

    Verdict β€” CLOSING, and here is exactly what is NOT being closed with it

    This issue's subject is the Ifrs17/AocConfiguration render gate. On platforms 248 and 252 sets past the one it failed on, with the type actually exercised, it passes, eleven runs in a row, with no trace of its own trail. The regression framing was already falsified on this thread (both 7bfe2d907 and 13d324992 carry the same race), and the reachable mechanism was fixed and controlled both directions.

    🚨 The residue stays open where this thread already put it, and it is not a formality:

    If the AocConfiguration timeout returns on a platform β‰₯ 8319, reopen this with the fate trail β€” that is the falsifying direction and it would settle it outright.

    Measured via the GitHub REST API only (no GraphQL), every log fetched and size-checked before grepping; platform versions read from resolve-platform, never grepped out of a gate log.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions