Route dispatch saturation on portal/reads channels: burst drains in seconds but still mints a per-stamp Critical incident — normaliser masking rule still missing #5593

2026-09-23 19:59:36Z memex-portal-deployment-56fbdcd48f-4dpcn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [506f05fc#6 started 2026-09-23T19:59:36.1976685Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 1 over 1 stream destination(s), deepest per-channel queue 52, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 2084 ms — dispatch → Manufacturing/MaintenancePlanner/_Activity/compile-20260923195923724072e90cfb4b64c5094997a149d70bdc2 (delivery pjQ-m0urskOCrsavliqGMQ). Latest dispatch target portal/reads-rqEHlOPNSkCV1G34WXuQ9A — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs …[truncated]
2026-09-23 19:59:37Z memex-portal-deployment-56fbdcd48f-sz5st crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [e9f417e5#3 started 2026-09-23T19:59:37.3353671Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 2 over 2 stream destination(s), deepest per-channel queue 62, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 1133 ms — stream-routed → portal/reads-jll3sSZfx061jLrYC3JaUA (delivery LA5W2M1fYUizY2XnmaP_lQ). Latest dispatch target portal/reads-jll3sSZfx061jLrYC3JaUA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE …[truncated]
2026-09-23 19:59:43Z memex-portal-deployment-56fbdcd48f-4dpcn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [506f05fc#7 started 2026-09-23T19:59:43.3475683Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 1 over 1 stream destination(s), deepest per-channel queue 60, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 1731 ms — dispatch → Manufacturing/Customer/_Activity/compile-20260923195924035cccea18a26b744fca413153050f288d8 (delivery nn5YSGdyyUG8DvCRAihNqA). Latest dispatch target portal/reads-rqEHlOPNSkCV1G34WXuQ9A — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND l…[truncated]
2026-09-23 19:59:50Z memex-portal-deployment-56fbdcd48f-4dpcn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [506f05fc#8 started 2026-09-23T19:59:50.7379258Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 1 over 1 stream destination(s), deepest per-channel queue 62, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 27 ms — dispatch → Hosting/Issue/_Activity/compile-20260923195938239fd7402304963494290814a9fdf7a3ad4 (delivery zCfR02EJ6kG4kBw9orN_dQ). Latest dispatch target portal/reads-rqEHlOPNSkCV1G34WXuQ9A — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say …[truncated]
2026-09-23 19:59:53Z memex-portal-deployment-56fbdcd48f-4dpcn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [506f05fc#9 started 2026-09-23T19:59:53.4119775Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 2 over 2 stream destination(s), deepest per-channel queue 58, routing pool subscribing 0, waiting for a pool slot 1; oldest leg in flight 2701 ms — dispatch → Hosting/Issue/_Activity/compile-20260923195938239fd7402304963494290814a9fdf7a3ad4 (delivery zCfR02EJ6kG4kBw9orN_dQ). Latest dispatch target portal/reads-rqEHlOPNSkCV1G34WXuQ9A — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to sa…[truncated]