docs: the 3.03 h memex churn was the startup probe killing gate-refused containers (#5704) by rbuergi · Pull Request #5746 · Systemorph/MeshWeaver

rbuergi · 2026-09-26T06:29:32Z

What

A documentation-only change. It records the root cause of the memex pod churn behind #5704, which also covers #5709, #5730 and #5705.

From 09-25 ~18:00Z to the break-glass at 04:05Z, every portal container on both images died 3.03 h after it booted, with no crash dump. That is the memex startup probe running out its budget: /health, periodSeconds 10 × failureThreshold 1080 = 10 800 s. While nodetype_bake refused (#5544), the probe never recorded a single success, so the kubelet killed each container and restarted it in place. The next attempt reached the same verdict, and the cycle repeated.

The mechanism also defeats the gate's own safety net. A serving ci.9218 pod that SIGABRTed (#4654) had to pass the startup probe again and could not.

Changes

TheBakeGateOnlyStallsARoll.md gets a new section covering:
- the readings: the probe spec from the record, the boot series, the heartbeat ceiling at tick ~1085, nodetype_bake Unhealthy every 10 s, and the 05:52:27Z kubelet Startup probe failed: 503 on the same pod;
- how to recognise this kind of churn;
- why the pods that looked "deleted and recreated" after 04:05Z were just the ordinary roll (plus 1 h 47 m of Insufficient cpu Pending);
- what is not established: the missing shutdown line.
OperatingFromThePortal.md lists the Sample action's new last-termination fields from Systemorph/MeshWeaver.Plugins#2397.

The code fix for the gate itself is #5725 (merged). The running memex image (4028fd6468) predates it, so the gate stays off until an image carrying it rolls.

Pairs-with: none — documentation only; no public surface changes.

🤖 Generated with Claude Code

…ed containers (#5704) TheBakeGateOnlyStallsARoll gains the outside view of the #5544 refusal: every memex container on both images died 3.03 h after boot because the startupProbe (/health, 10 s x 1080 = 10800 s) never saw one success while nodetype_bake refused, so the kubelet killed and restarted it in place, for ever. The readings (probe spec, boot series, heartbeat ceiling, nodetype_bake Unhealthy every 10 s, the 05:52 kubelet event), how to recognise it, why the "deleted and recreated" pods after 04:05Z are the ordinary roll, and what is NOT established (the missing shutdown line). OperatingFromThePortal lists the Sample's new last-termination fields (Plugins#2397). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Review effort: Lite
Findings: 1

Open (1)

Avoid claiming timeout sequence from earlier related incidents · New

What changed in this PR

Documents the startup-probe-induced container churn and new per-replica termination diagnostics.

Changes:

Adds incident measurements and diagnostic guidance.
Documents new Sample termination fields.

File	Description
src/MeshWeaver.Documentation/Data/Architecture/TheBakeGateOnlyStallsARoll.md	Updated as part of this pull request.
src/MeshWeaver.Documentation/Data/Architecture/OperatingFromThePortal.md	Updated as part of this pull request.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-26T06:45:23Z

+it booted**. There was no crash dump for these deaths. The Orleans timeouts that followed each one
+were filed as their own incidents ([#5704](https://github.com/Systemorph/MeshWeaver/issues/5704),
+folding #5709, #5730, #5705).


Agreed, fixed in 3efde72. #5704 was opened from 2026-09-24 samples, so the sentence claimed a sequence the cited evidence cannot support. It now:

calls Memory stream queue dequeues time out when target silo is unresponsive; pulling agent retries without recovering #5704 and the issues folded into it related;

cites the only in-window evidence: the Dequeue timeouts at 2026-09-25 18:57:39–41Z against silo S10.244.3.247 (cbms9), after its 18:55:12Z SIGABRT;

says plainly that the in-window samples tie the timeouts to a crash, and that the probe kills producing the same lines is inferred, not observed.

The table below that paragraph (the ggf88 boot series, the heartbeat ceiling, the nodetype_bake Unhealthy lines and the kubelet event) is all from the 09-25/26 window itself, so it is unchanged.

github-actions · 2026-09-26T06:33:24Z