Incident filer normaliser misses per-episode identity in RoutingGrain back-pressure lines, flooding the tracker with duplicate Critical tickets

What is failing

The log-watch incident filer's message normaliser does not mask the per-episode identity that the RoutingGrain back-pressure line embeds inside its message text — the bracketed [activation#episode started timestamp] group (e.g. 5bdfaaf2#4 started 2026-09-22T12:43:54.1965294Z). One recurring back-pressure condition therefore mints a distinct Critical LogIncident (and ticket) per crossing instead of folding into one incident with a growing occurrence count. There are currently dozens of near-identical RoutingGrain: [ROUTE] Routing back-pressure incidents in Admin/_LogIncident spanning 2026-09-19 to 2026-09-22, differing only in those three values.

Probable cause — medium confidence

The masking rule for this log site has no pattern for the [hex-activation#episode started ISO-timestamp] prefix, so the fingerprint is computed over per-episode identity. This is the hypothesis recorded in feedback rbuergi/Feedback/route-backpressure-fingerprint-splits-20260921-2236; the normaliser rule itself has not been read, so it may have a pattern that fails to match rather than none at all. Supporting evidence: core issue 5018's consolidation comment records seven separate GitHub issues (5017, 5019–5022, 5060, 5018) carrying the same fingerprint 54ecfdaea23fd110 for the same incident window, and ~87 bot-filed tickets for this one log site over 2026-09-19–21, 48 hand-folded into core #5009 and 24 into core #5018.

Impact

This incident is a single occurrence on one pod and its sample line reads as absorbed load (64 dispatches at the threshold-64 crossing, oldest leg 84 ms, both pool gauges 0 — young legs, no leak). The defect is the flood: real slot-leak signals get buried among dozens of near-identical Criticals, and humans spend effort hand-folding duplicates (core #5118's own stated worry). Threshold raises and severity demotions were refused in core because they treat the symptom.

Where to look

The normaliser / masking rules of the log-watch incident filer in MeshWeaver.Plugins — add a case masking the bracketed [activation#episode started timestamp] group so recurrences fold onto one incident whose occurrence count becomes the signal.
The log call site (for reference, not to change): src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs, method ReportSaturation, which composes those three values into the message template — they are deliberately load-bearing for single-line diagnosis and should stay in the log text.

Related: core #5118 (alert flood verdict), #5018 (load verdict, carries the seven-issues-one-fingerprint measurement), #5009 (head-of-line verdict, fixed in core PR 5151, pending verification). This ticket is the remaining half of #5118: fix the grouping, not the log line.

Evidence


Fingerprint	`76dea3f1ce951aa2`
Category	`MeshWeaver.Hosting.Orleans.RoutingGrain`
Severity	Critical
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-5c89c7f5f9-sddb5`
Occurrences	1
First seen	2026-09-22 12:43:54Z
Last seen	2026-09-22 12:43:54Z
Routing	not determined — no configured route matches the category `MeshWeaver.Hosting.Orleans.RoutingGrain`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-22 12:43:54Z memex-portal-deployment-5c89c7f5f9-sddb5 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [5bdfaaf2#4 started 2026-09-22T12:43:54.1965294Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 8 over 3 stream destination(s), deepest per-channel queue 12, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 84 ms — stream-routed → portal/egXML8v7Y8GjHurnb8KTfloONWmGqcGbVAtCvfjeDWk (delivery K3ryrDUWuEexmuB1GJ7t9g). Latest dispatch target portal/egXML8v7Y8GjHurnb8KTfloONWmGqcGbVAtCvfjeDWk — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. �…[truncated]

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/76dea3f1ce951aa2. Recurrences are folded into this issue rather than opening new ones.}

Reopened — closed by the Fixes keyword in Systemorph/MeshWeaver.Plugins#2286 on merge, but a sev:H closes on production verification, not on a merge.

What was merged (b922f070fc, 2026-09-22T21:46Z): the diagnosis in this issue's body was not the defect. The episode stamp was already masked (LogLineParser.EpisodeStamp, Plugins#2167). What split the fingerprint was the oldest-leg label's delivery id (base64url, unmasked) and a single-segment destination (dispatch → Essentials). Two positional masks fix both; over the 146 back-pressure sample lines quoted in the open core issues (43 filed fingerprints) the identity now yields 3. Controls both ways, 400/400 in MeshWeaver.Observability.Test; doc Observability/ThePortalMasksWhatItHashes §3.

Close condition: after a portal roll carrying b922f070fc (the portal re-normalizes from raw samples, so no watcher roll is needed), a new back-pressure crossing must fold onto an existing incident (occurrence count rises, no new Admin/_LogIncident node, no new issue). A newly filed back-pressure issue on a post-roll pod is the fix not holding — its sample line is then the next splitter to read.

Not established: nothing measured in production; the 43 → 3 is over the samples the issues quote, not over the live incident store.

Measured against Plugins `main`: the fix folds every sampled line. The filings after the roll come from a host that does not have it yet

Measurement. I compiled LogLineParser.Normalize from Plugins origin/main (src/MeshWeaver.Observability.Contract/LogLineParser.cs, which contains b922f070) and ran it over every [ROUTE] Routing back-pressure line quoted in the 71 flood issues. That is 159 lines. Cut at the constant tail (— the address…, because the issue bodies truncate the lines at different lengths), they normalise to exactly 3 texts: one per leg kind (dispatch →, stream-routed →), plus the older wording of the line (stream destinations queued …). The episode stamps include ids that carry hex letters (f14edef0, 6f268927, 440d2a6b, c2b56e92, …). They all become [{hex}#{n} started {time}]: MixedHexId takes an id that has both digits and letters, EpisodeStamp takes whatever is still in the [<id># slot, and both produce the same placeholder. So the claim in #5415/#5421/#5422 that the stamp's hex letters are still unmasked is not true on main. Those three are folded here.

Why new back-pressure issues were still filed after the memex-cloud roll (#5427/#5428, #5429 and #5430, all on 39f49494 pods, three distinct fingerprints): the incident filer does not run on memex-cloud. Admin/_LogIncident/30c74d837b5174bc does not resolve on memex.meshweaver.cloud. The control instance memex.systemorph.com (/api/version → 3da3423c) has Plugins sealed at 954e7795 (read from Ops/Status/memex, publication-seal), and b922f070 is not an ancestor of 954e7795 (checked with merge-base --is-ancestor, exit 1). memex-cloud's seal does carry it (b922f070, then e9d9283a), but that is not the host that hashes. All three post-roll lines are among the 159 above, so under main each of them would have folded onto an existing identity.

So the close condition above needs one correction. The roll that has to carry b922f070 is the control instance's Plugins seal, not memex-cloud's portal roll. After that: a back-pressure crossing from either namespace should add to an existing incident and file nothing new. Plugins#2290 (open) goes further and folds the log site onto its existing issue (#5118).

Not established: I did not read the live incident store (the control instance was answering 503), so I have not confirmed which process computes the hash. The claim that the filer runs on the control instance rests on the two readings above and on the brief I was given.

Consolidated bot-filed duplicates (per-episode identity split of the RoutingGrain back-pressure line, closed not_planned): #5473 #5481 #5536. The site-level fold that stops such splits from minting tickets is Systemorph/MeshWeaver.Plugins#2290.

The split did not stop after the roll. The masking was correct; the cut sample was not. Fix: Systemorph/MeshWeaver.Plugins#2325

The fix was already live when the flood continued. The control instance's image is 3.0.0-ci.9218 (per Ops/Status/memex, 16:34Z). main-cd run 9218 built it from plugins e9d9283a (its gate log reads plugins HEAD (main) = e9d9283abe…), and b922f070 is an ancestor of that commit. The image therefore hashes with #2286's masking. Even so, namespace:Admin/_LogIncident name:*back-pressure* returned 60+ (truncated) incidents modified today, 40+ of them created after the 05:24Z boot. The listing names still read [{n}f6{n}e2#…, but that is the watcher's stored text. The node id is the portal's.

The splitter is the watcher's sample cut. The portal re-derives the identity from LogSample.Line. The watcher cuts every sample at MaxSampleLength = 2000 and appends …[truncated], and this line is about 3,400 characters. The cut falls at a fixed raw offset. The fields in front of it (the oldest leg's destination and the latest dispatch target) have different lengths on each crossing, so the masked text ends at a different word of the constant tail each time (AT THE CROSSIN…, AT THE CR…, visible in 78316ad8e1d82122's own samples). Each distinct cut position becomes its own fingerprint. The 04:43Z measurement above cut each line "at the constant tail" by hand before normalizing, which removed exactly this variable.

Repro (Plugins#2325, ATruncatedSampleKeysOnWhatItKeptTest). Four dispatch crossings carry the values from 78316ad8e1d82122's samples and are cut at the watcher's own default:

before the fix they give 3 identities, and one of them, ec4f187fd869d0be, is a live node id on the control instance;
after the fix they give 1.

The controls pass: the clear line still splits, two short lines that differ at the end still split, a cut inside the bound is not counted as evidence (this case goes red with the exclusion removed), and a cut through a stack trace still counts. MeshWeaver.Observability.Test passes 411/411 in Release.

Fix. The identity reads the first 1024 characters of the masked detail. A cut sample counts only when the cut falls at least 128 characters past that bound. Messages shorter than the bound hash exactly as before.

Close condition (corrected). memex rolls onto an image whose plugins sha contains the #2325 merge. After that, a back-pressure crossing adds to an existing incident and no new Admin/_LogIncident node appears for it. No watcher roll is needed, because Contract ships in the portal image.

Not established. I have not seen the fold hold in production. MaskSubjects is a known edge I did not measure: a subject that appears only past the cut could mask differently in a whole line than in a cut one.

Systemorph/MeshWeaver.Plugins#2325 is merged (cfd0576631, 2026-09-23 20:09Z). The head 5a57d945 was green on every required context, including Tests-area ratchet over the gate log = success, with zero failed check-runs. Copilot's review found that a cut through the first application frame could still split the identity. That is fixed in the same PR with its own case, ACutThroughTheFirstApplicationFrame_IsNotEvidence, which goes red with the frame check removed. Observability.Test passes 412/412.

Still open, because sev:H closes on production verification: memex has to roll onto a portal image whose plugins sha contains cfd0576631. After that roll, a back-pressure crossing has to fold onto an existing Admin/_LogIncident node and must not mint a new one. The filer runs there, so this is the roll that matters; no watcher roll is needed. If a new back-pressure incident appears on a post-roll pod, the fix does not hold, and its sample line is the next splitter to read.

Consolidated bot-filed duplicates of this defect (tickets that themselves describe the per-episode fingerprint split, closed not_planned): #5580 #5590 — both minted 2026-09-23 18:27Z/19:50Z, i.e. before Plugins#2325 (cfd0576631, merged 20:09Z) could be live. The 12 same-window crossings folded onto #5118 are further instances of this split.

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

What is running.

memex-cloud: /api/version returns 9a61fed964. Ops/Status/memex-cloud (06:42:00Z) shows 4/4 replicas on 3.0.0-ci.9263. CD run 9263 built core 9a61fed964 with Plugins 41bcfe968f.
memex: /api/version returns 39f49494eb. Ops/Status/memex shows the serving pods on 3.0.0-ci.9218, which is core 39f49494eb with Plugins e9d9283abe. A third pod, on ci.9260, is unready, so the roll is converged: false.

This issue's fix: Plugins#2325 (cfd0576631). It is not an ancestor of either running build (git merge-base --is-ancestor, exit 1).

Where it is. 3.0.0-ci.9287 (CD run 35956222018) is core bf5ac85526 with Plugins c4795a8d5b, and it contains this fix. The registry has sealed it. memex-cloud's self-updater logged [SelfUpdate] handing 3.0.0-ci.9287 to the control lane (https://memex.systemorph…) at 06:47Z. So the roll now waits on the control instance, which is an owner step and was not done here.

Still open. It closes on a post-roll reading once a ci.9287-or-later image serves the namespace this fingerprint fires in.

Post-roll note: the fix lives in the filer on memex, and memex has not rolled

Image. memex-cloud (memex.meshweaver.cloud) serves 3.0.0-ci.9291. /api/version returns core 140c5fb7ae, and Ops/Status/memex-cloud (sampled 14:49Z) shows all 4 replicas on that tag, converged: true. The Plugins half is b3601fecbe: the portal-ai build job of CD run 35970157348 (job 107542528900) logs HEAD is now at b3601fe. The first pods of this ReplicaSet (memex-portal-deployment-589ff8f895) started at 08:41:05Z and 08:42:17Z. More pods of the same ReplicaSet, on the same image, joined later (gqsfn at 14:06Z, vw5jj at 14:10Z, and others). The control instance memex.systemorph.com still serves core 39f49494eb.

Window and method. I ran Logs InstanceActions on memex.meshweaver.cloud between 14:43Z and 14:54Z, with sinceMinutes 370/375, filtered to this ReplicaSet with | pod=~"memex-portal-deployment-589ff8f895-.+". That is ≥ 6 h of this image only, from 08:41Z to about 14:50Z. Coverage control: Ops/logs-memexcloud-20260924-close6h-c-bootcoverage (warm-up complete) lands 9 lines from 9 pods of this ReplicaSet. The earliest are ghgjt, vn49b and tp564 at about 08:51Z, so the window reaches back to the first boots.

Fix: Plugins#2325 (cfd0576631). It is in memex-cloud's image (b3601fe). The incident filer that mints the fingerprints runs on the control instance (memex.systemorph.com), and Admin/_LogIncident is not on memex-cloud. The control instance serves core 39f49494eb with Plugins e9d9283abe, and cfd0576631 is not an ancestor of that (exit 1). So the fix is not live where it acts.

For reference, memex-cloud's back-pressure lines on the fixed image are still arriving: 118 lines, 59 episodes, and 4 of them Critical (Ops/logs-memexcloud-20260924-close6h-f-backpressure, …-f-routingcrit; details on #5389). That is live input for the filer once it rolls. Stays open until memex rolls onto a portal image containing cfd0576631.

Triage re-read 2026-09-25 ~15:30Z. Running builds (/api/version): memex.systemorph.com = 39f49494eb (ci.9218), memex.meshweaver.cloud = 140c5fb7ae (ci.9291). Ancestry was checked with git merge-base --is-ancestor against origin/main. Both portals are still held at the V58 migration: the ci.9332 pods crash-loop on DbVersionGate db_version=57 < expected 58. Nothing will roll until a governed Reconcile of Deployments/memex runs the migration Job.

Status: fixed in Plugins#2325 (cfd0576631) and waiting for the control-instance roll. The filer runs on memex, which still serves core 39f49494eb with Plugins e9d9283abe, and that build does not contain the fix. The close condition is unchanged: after memex rolls, a back-pressure crossing folds onto an existing Admin/_LogIncident and mints no new incident. No code action.

Post-roll verification, memex.systemorph.com, 2026-09-26 ~05:55Z. memex serves 3.0.0-ci.9332 = core 4028fd6468 (/api/version), Plugins half 1470fbf3 (CD run 9332 portal-ai job HEAD is now at 1470fbf; same ref as the /health publication-seal line). The DB is at V58: memex-migration-63 logged Database migration completed. Version: 58 at 2026-09-25 17:58:03Z. The roll is not converged: Ops/Status/memex at 05:48Z reads 1/3 ready; only 5dbb7df97b-dktrd (booted 04:06:39Z) serves. So every post-roll window below is about 1 h 50 min on one serving replica. All reads are Logs InstanceActions under Ops/Actions/logs-memex-20260926-postroll-*.

Still open. The post-roll reading REFUTES the fix, so this needs another look.

Plugins#2325 (cfd0576631) is an ancestor of 1470fbf3 (compare → ahead), so the filer carrying it is live on the control instance.
After the roll, ingest resumed (see Plugins#2152). It is still minting a new Critical incident per back-pressure crossing:
- Admin/_LogIncident/bab5b3e412f85ea5 was created 2026-09-26 04:29:49Z (ci.9332). Its normalizedMessage begins [ROUTE] Routing back-pressure [{n}f0{n}c#{n} started {time}]. Only the digit runs and the timestamp are masked. The hex letters of the activation id (7f03153c → {n}f0{n}c) survive, so every activation fingerprints differently. Its sample is a 2026-09-24 13:42Z line from the backlog, so the fingerprint was computed at ingest time on the new build.
- fbd27b4e342e917c and 981bb5aa56136a69 (both [{n}f1d1f4#{n} …], touched 05:18–05:19Z) and 0beb991b4f0f4645 ([b5{n}b9{n}#{n} …], 04:18Z) have the same shape.
Not established: whether this normalisation runs in the portal's ingest (LogIncidentIngestService, now on 1470fbf3) or in the out-of-cluster memex-log-watcher, whose image nothing rolls (#4326). The bab5b3e4 node carries reporterFingerprint: e4a97855ab595beb and siteFold: 8b02af1dbffbfd6d, so the watcher does contribute a fingerprint. If Four AI tests move to the AI suite; Content.Test stops referencing the engine (#2276) #2325 changed the portal side and the watcher computes this one, the fix has not reached the component that decides. If Four AI tests move to the AI suite; Content.Test stops referencing the engine (#2276) #2325 changed this component, then the masking rule does not cover mixed hex.

Next step (engineering): read where normalizedMessage is produced for an ingested report, and check whether a hex activation id containing letters is masked there.

Re-read of the 06:04Z "refuted" verdict: #2325 holds. One real residual split, fixed in Systemorph/MeshWeaver.Plugins#2391.

The {n}f0{n}c text is not what the portal hashes. An incident's normalizedMessage and its display name come from the REPORTER. That is LogLineParser.Normalize compiled into the separately shipped mw-log-watcher, which predates MixedHexId. The node id is the portal's. I fed the stored LogSample.Line of Admin/_LogIncident/bab5b3e412f85ea5 to LogIncidentIdentityResolution.Resolve at Plugins origin/main. It returns exactly bab5b3e412f85ea5, and its own normalization reads [{hex}#{n} started {time}].

What the four post-roll nodes are (control instance, read 06:10Z):

node	leg	occurrences	reading
`fbd27b4e342e917c`	stream-routed	415, many activations and pods	folding; last write 05:19Z
`981bb5aa56136a69`	dispatch	162, many activations	folding; last write 05:18Z
`0beb991b4f0f4645`	`NACK` label	3 (09-24)	a separate branch, by the same rule that keeps dispatch and stream-routed apart
`bab5b3e412f85ea5`	dispatch	1 (sample from 09-24 13:42Z)	the one real split

All four were created between 04:08Z and 04:29Z. That is the re-address at the new pod's boot, not fresh crossings: every one carries samples from before the roll. No back-pressure node newer than these exists.

The split: bab5b3e4… is a dispatch crossing whose only difference is Latest dispatch target rsalzmann. SpacedSubject masks a subject only when it starts upper-case or with a digit, which is the guard that keeps prose out. With the target capitalised, the same sample resolves to 981bb5aa56136a69. Plugins#2391 masks that slot by position (dispatch target <token> —). It has a regression test over both nodes' stored samples, verbatim, and the test is red without the rule, where it answers the live bab5b3e412f85ea5.

Production action (not taken; display only): roll mw-log-watcher. Its stale normalization is what the incident names show. It does not affect identity, but it is what made this look refuted.

Close condition, unchanged: once memex serves Plugins#2391, a back-pressure crossing folds onto an existing Admin/_LogIncident node, including one with a lower-case target.

Not established: whether lower-case tokens split other log sites. Only this line was measured.

Formatting error in this note; the following comment contains the complete verified production evidence.

Post-close lower-case-target sample, 2026-10-04 UTC: the untruncated two-hour memex-cloud Logs action recorded real ROUTE back-pressure starts at 05:52:56Z and 06:07:57Z. Their Latest dispatch target values were cache/yP1KZtdzR06mQc25TvM7eg and cache/gX3hcUIXUU6UZtkIaymgyQ. Each crossed 64 in flight with oldest leg 18 ms and 5 ms, then cleared. This meets the real lower-case-target event prerequisite. It does not prove end-to-end incident folding: the action returned 39 lines and no separate incident-ingest entry; the protected Admin incident record is not readable here, and NotFound is not evidence of absence. No new incident identity or fold is verified; no reopen or closure.

Fresh lowercase-target evidence, 2026-10-04. The completed memex-cloud Logs action Ops/Actions/logs-watch-20261004-0847-cloud-5271 covered 07:48:00–08:48:00Z and read/landed 61 lines, 0 failed, untruncated. Real back-pressure starts included cache/yP1KZtdzR06mQc25TvM7eg at 07:56:13Z (oldest leg 160 ms; episode cleared after 495 ms), cache/cb1MaiwXDEGx81Wz6ME5vA at 08:28:49Z (oldest leg 3,628 ms; cleared after 6,827 ms), cache/wN0sQtF_1UG6C9pBQeGpWw at 08:28:56Z (oldest leg 5,684 ms; cleared after 580 ms), and cache/mV9RSCXEYkKAqFbAM_W4Kw at 08:41:15Z (oldest leg 11 ms; cleared after 31 ms). The action also captured a NotFound for Admin/_LogIncident/fbd27b4e342e917c from the prior protected-record probe; that is not evidence about whether the incident exists or folded. This supplies real lowercase-target crossings but no positive event-to-existing-incident fold witness. Missing ingestion evidence is not a clean result; issue status unchanged.

Post-close fold diagnostic, 2026-10-04 UTC. The memex-cloud Logs action completed with 54/54 lines landed, 0 failures, untruncated (09:48:23–10:48:23Z). Three current cloud replicas emitted LogIncidentControlPlane idle messages at 10:35:31Z, 10:36:32Z, and 10:37:49Z: no LogWatch:DefaultRepository and no LogWatch:Routes are configured. Real lower-case-target back-pressure evidence was already recorded; this run contains no positive event-to-incident fold witness. No incident identity was invented and no protected Admin record was read or changed. Operator input is required: please provide the intended LogWatch default repository and route mapping so fold behavior can be verified. Issue status remains unchanged.

Fresh lowercase-target recheck, 2026-10-04 UTC: the memex-cloud Logs action covered 12:47:42–13:47:42Z and landed 18/18 lines with 0 failures, untruncated. It captured 64-dispatch crossings for cache/iBl425cfAkety9SC2KltNg at 13:02:16Z (oldest leg 1,605 ms; cleared after 41 ms) and portal/reads-iBl425cfAkety9SC2KltNg at 13:35:29Z (oldest leg 4 ms; cleared after 21 ms). This is real lowercase-target back-pressure, but there is no positive event-to-existing-incident fold witness in the sample. The prior cloud diagnostic found no LogWatch default repository or routes; the intended mapping still needs owner input. No protected Admin incident record was read or changed, no identity was invented, and issue status remains unchanged.

Production verification after a real lower-case-target event (memex, 2026-10-05 19:24–19:26Z): RoutingGrain emitted four Critical [ROUTE] samples for target cache/L1PlUIhTo0WIzSJqWuyx1Q. The distinct episode stamps (1f2573d8#1, 1a5c1b19#1, 22c699de#1, 009d2f51#1) appear in one Admin/_LogIncident/02ad6e292c7bbfb4 record with occurrences=4, variants=1; its normalizedMessage masks them as {hex}#{n} started {time}. This is a positive fold witness for that event group.

The underlying back-pressure was still Critical (64 dispatches in flight; oldest leg 137–165s against its 100s bound). This verifies folding only, not that routing recovered or that every distinct channel identity should share one incident. The issue remains closed.

Second production sample after a separate real lower-case-target event (memex, 2026-10-05 19:55–20:00Z): seven distinct episode stamps for cache/L1PlUIhTo0WIzSJqWuyx1Q share the same channel hash u0IKGcrOlk2Tdf286A3Vog and were recorded in one Admin/_LogIncident/5f6421c4b3852c28 node (occurrences=7, variants=1). The normalized message masks the episode id and timestamp. This is a second positive fold witness for the per-episode identity fix.

The route remained Critical: 64 dispatches in flight, deepest queue 61–62, and oldest legs 116–168s against a 100s bound. This verifies folding only; it does not clear the routing backlog or establish behavior for distinct channel identities. Leaving #5271 closed.

Fresh lower-case-target recheck from memex-cloud Logs, 2026-10-08 18:47–19:47Z: a real [ROUTE] back-pressure event at 18:58:18Z reported target portal/Aixc09CZk7NNhedCOGbi7T4jSOEPpZ0R-8zPTtXmd7I (lower-case portal/), episode 7dc0fba6#2, oldest leg 57 ms; later episodes #3 and #4 on that activation targeted the same lower-case portal path, and their clear lines followed. The sample classifies these as Warning/load, not a stuck Critical. I checked for a corresponding newly modified RoutingGrain LogIncident after 18:00Z but found no positive fold witness in the readable result; I am not treating absence as a clean pass or changing issue status.

Fresh memex-cloud Logs action (2026-10-10 08:18:29–09:18:29Z) landed 275/275 lines, zero failures, untruncated. On a 3.1.10429 pod it contains real back-pressure episodes 7dca13e3#2–#4 targeting the lowercase path cache/E0PMa3KAWUK0AfCbbWzukw at 09:03:32–09:03:35Z (64 dispatches; oldest 180 ms–1.01 s, with later clear lines). This meets the lower-case-target event prerequisite. I found no corresponding newer RoutingGrain back-pressure incident record in the latest readable incident results (latest matching record is Oct 6), so there is no fresh positive fold witness; I am not treating that absence as a pass or failure and left the issue closed.

Fresh memex-cloud Logs action for 2026-10-10 09:21:47–10:21:47Z landed 262/262 lines with zero failures, untruncated. It records seven real crossings on activation 79b53fc6; episode #3 at 09:37:54Z targeted lowercase portal/reads-x6qVqrFvlEKersrFOhkYXw (64 dispatches; oldest leg 0 ms, queued leg 5 ms, deepest channel queue 33) and cleared after 14 ms. This satisfies the lowercase-target event prerequisite. The latest readable targeted Admin/_LogIncident results still show no newer matching back-pressure incident than Oct 6, so this sample supplies no positive event-to-existing-incident fold witness. I am not treating that absence as a pass or a failure; issue status remains unchanged.

Fresh post-comment production sample (2026-10-10, completed Logs actions covering 10:19–11:19Z): memex and memex-cloud reads landed 119/119 and 115/115 lines respectively, with zero failed writes and no truncation. Real lower-case cache-target crossings include memex episode febe8c60#4 at 10:42:33Z (cache/i6XIVnDyhkGUH6o6tjlhOg) and memex-cloud episodes 9c06e502#1 at 10:42:23Z (cache/5lv_7zRG2kabm2rvmYCcsQ) and #3 at 10:43:46Z (cache/pkgTCpXzWki0T6W5p7hsiA); all reported 64 in-flight, with oldest legs under one second, and subsequent clear lines followed. The latest readable targeted Admin/_LogIncident search (200 results, truncated) has no matching incident modified after 2026-10-06, so this provides no positive event-to-existing-incident fold witness. I am not reading that absence as a pass or failure; the issue remains closed.

Fresh post-10442 Logs window 2026-10-10 11:23–12:23Z: memex and memex-cloud actions completed untruncated with 0 failed writes (179/179 and 176/176 lines). A real memex lower-case target back-pressure event occurred at 12:02:34Z: activation c9d27bd4 episode #1, stream-routed to cache/yGzNtcGVY0-NpSZdjz9WEA, 64 in-flight, oldest leg 30 ms; episodes #1–#4 have subsequent clear lines (1.953 s, 935 ms, 98 ms, 52 ms). The latest readable targeted Admin/_LogIncident RoutingGrain back-pressure search still has its newest matching record at 2026-10-06 and is truncated at 200; this is not a positive fold witness, and the truncated result is not proof of absence. Leaving issue status unchanged.

Fresh read-only log evidence from memex at 2026-10-10 12:52:38Z: real lowercase-target back-pressure episodes 542dd704#2 and #3. Episode #2 latest dispatch target was cache/z6m67w561kOM6VYVAR0_YQ with 64 dispatches in flight and 64 queued across three stream destinations; it cleared after 3 ms. Episode #3 again targeted that lowercase cache path; its oldest queued leg was on cache/yGzNtcGVY0-NpSZdjz9WEA (depth 2, 12 queued across three destinations), and it cleared after 41 ms. Recording the event only; no cause or fix is inferred.

New qualifying event: memex-cloud at 2026-10-10T14:55:10.799Z, pod memex-portal-deployment-868574bc48-qjtg4, activation 6c846f93#4. The real [ROUTE] line names lowercase target cache/ZNkpnzQTO0K6D3yr__JNBQ; it reports 64 dispatches in flight, 36 queued over one stream destination, deepest per-channel queue 1, subscribing 0 and waiting-for-pool-slot 0; it cleared after 11 ms. This satisfies the lower-case-target event prerequisite. The Admin/_LogIncident search for back-pressure is bounded (200 results, truncated); newest returned matching record is modified 2026-10-06T02:00:56Z, so this snapshot does not establish a positive fold witness. I am not treating missing evidence as a clean result or claiming the normalizer condition passed.

New qualifying real lower-case-target event, 2026-10-10 18:18:21.604Z: memex-cloud pod memex-portal-deployment-5499f76b76-sz46x, activation bef2718b episode #1, stream-routed target cache/QiYAcLAw1E6iwZGLqtVxsg; 64 dispatches in flight, 39 queued across one stream destination, deepest per-channel queue 1, pool gauges 0. The matching clear line followed 201 ms later. This meets the event prerequisite but does not establish incident-fold behavior. The targeted Admin/_LogIncident search remains truncated at 200 results and its newest matching record is still modified 2026-10-06, so I cannot verify whether this event folded or split; no closure claim.

Navigation Menu

Repository navigation

Incident filer normaliser misses per-episode identity in RoutingGrain back-pressure lines, flooding the tracker with duplicate Critical tickets #5271

Evidence

Measured against Plugins `main`: the fix folds every sampled line. The filings after the roll come from a host that does not have it yet

The split did not stop after the roll. The masking was correct; the cut sample was not. Fix: Systemorph/MeshWeaver.Plugins#2325

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

Post-roll note: the fix lives in the filer on memex, and memex has not rolled

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Incident filer normaliser misses per-episode identity in RoutingGrain back-pressure lines, flooding the tracker with duplicate Critical tickets #5271

Description

Evidence

Activity

rbuergi commented on Sep 22, 2026

rbuergi commented on Sep 23, 2026

Measured against Plugins main: the fix folds every sampled line. The filings after the roll come from a host that does not have it yet

rbuergi commented on Sep 23, 2026

rbuergi commented on Sep 23, 2026

The split did not stop after the roll. The masking was correct; the cut sample was not. Fix: Systemorph/MeshWeaver.Plugins#2325

rbuergi commented on Sep 23, 2026

rbuergi commented on Sep 23, 2026

rbuergi commented on Sep 24, 2026

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

rbuergi commented on Sep 24, 2026

Post-roll note: the fix lives in the filer on memex, and memex has not rolled

rbuergi commented on Sep 25, 2026

rbuergi commented on Sep 26, 2026

rbuergi commented on Sep 26, 2026

3 remaining items

rbuergi commented on Oct 4, 2026

rbuergi commented on Oct 4, 2026

rbuergi commented on Oct 4, 2026

rbuergi commented on Oct 4, 2026

rbuergi commented on Oct 4, 2026

rbuergi commented on Oct 5, 2026

rbuergi commented on Oct 5, 2026

rbuergi commented on Oct 8, 2026

rbuergi commented on Oct 10, 2026

rbuergi commented on Oct 10, 2026

rbuergi commented on Oct 10, 2026

rbuergi commented on Oct 10, 2026

rbuergi commented on Oct 10, 2026

rbuergi commented on Oct 10, 2026

rbuergi commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Measured against Plugins `main`: the fix folds every sampled line. The filings after the roll come from a host that does not have it yet