Repository navigation
fix(chart): the startup budget the chart DESCRIBES is now the one it ships (#2787, #2274) - #2812
Conversation
…ships (#2787, #2274) Found while measuring #2787's "close it at the source" direction against the live cluster. deploy/helm/templates/memex-portal/deployment.yaml asserted the portal's startup-probe budget TWICE: :281 "Paired setting and full derivation: values.yaml (probes.startup) — 10 × 180 = 30 min." :455 "the startupProbe budget above … which managed envs set to 10 × 180 = 30 MINUTES" values.yaml ships probes.startup 5 × 60 = 300s, and memex-cloud was measured live on 2026-08-30 running exactly that (kubectl: failureThreshold=60, periodSeconds=5). That is not a cosmetic drift. PreWarm__GateReadiness — the pod-side backstop for the very rollout window #2787 is about, and recorded as "false" in #2274's measurement — carries three numbered prerequisites, and #2 is a startup budget covering a full COLD bake (~570s: ~2.4s per NodeType, sequential, ~240 types, derived by the chart itself from production Loki). An operator arming the gate reads prerequisite 2, is told by the next comment that the budget is already 30 minutes, and arms it on a 5-minute probe — so Kubernetes kills the pod mid-bake, every time, forever, which is verbatim what the same comment warns about. Prose asserting a guarantee that does not exist, in the one file where being wrong costs a rollout. - Both comments now describe what SHIPS, and say arming the gate means raising the budget in the same change. - PreWarmGateReadinessGuard makes the coupling arithmetic instead of prose: * UNCONDITIONAL: no line in the chart may state an "N × M" startup budget that is not the one values.yaml ships, unless the line marks itself as a suggested paired setting ("raise…", "suggested", "→"). This half fails TODAY on the drifted comment — a guard that only fired once the gate was armed would have caught nothing on the day the prose went wrong, which is exactly why it survived. * CONDITIONAL: GateReadiness=true requires budget ≥ 570s bake + 300s plain boot, PreWarm__DynamicTypes=true, maxUnavailable: 0, and a startupProbe on /health — the chart's own list, now enforced. It reads probes.startup out of values.yaml rather than restating it, and fails loudly if the `probes:` block moves (a guard that cannot find its subject passes having checked nothing). MUTATION-PROVED, both halves, not merely observed green: - restoring the false "10 x 180" prose → FAILS: "states a startup budget of 10 × 180 = 1800s … but values.yaml ships 5 × 60 = 300s" - arming GateReadiness on the shipped budget → FAILS: "armed, but probes.startup is 5 × 60 = 300s and a gated boot needs at least 870s" Restored and re-verified 2/2 green after each probe. 🚦 Recorded on #2787 as a NEGATIVE result too: the rollout config itself is correct (maxUnavailable:0 + maxSurge:1 + PDB, readiness deliberately on /alive to avoid the documented probe death-spiral, startup holding readiness on /health). The nginx 503 dependent repos see during promotion is NOT explained by those, so that mechanism stays open on #2787. Touches no workflow file, so ShardWeightHeadroomGuard's timeout parse is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR corrects misleading Helm-chart commentary about the portal startupProbe budget and adds a build-time guard to prevent future drift between what the chart says and what it ships, especially around PreWarm__GateReadiness rollouts.
Changes:
- Updates
memex-portalDeployment template comments to describe the shipped startupProbe budget and its coupling tovalues.yaml. - Adds
PreWarmGateReadinessGuardtest to enforce (a) “stated budget equals shipped budget” and (b) prerequisites whenPreWarm__GateReadinessis armed. - Adds a “What’s New” entry documenting the defect and the new guard.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| test/MeshWeaver.Documentation.Test/PreWarmGateReadinessGuard.cs | New guard test enforcing probe-budget/prose coupling and gate prerequisites. |
| deploy/helm/templates/memex-portal/deployment.yaml | Fixes misleading budget commentary to align with shipped values. |
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-30-the-chart-can-no-longer-describe-a-budget-it-does-not-have.md | Documents the change and rationale for operators. |
Suppressed comments (1)
test/MeshWeaver.Documentation.Test/PreWarmGateReadinessGuard.cs:151
- StartupBudget()’s probes-block regex isn’t indentation-scoped:
(?:\n(?:[ \t].*)?)+can consume non-indented top-level lines (e.g.grpc:) because the indented part is optional, so the guard can accidentally read periodSeconds/failureThreshold from outsideprobes.startupif those keys appear later in the file. Tighten the match to a realprobes.startupblock and extract keys from that block only.
private static (int Period, int Threshold) StartupBudget(string values)
{
var block = Regex.Match(values, @"^probes:\s*$(?<body>(?:\n(?:[ \t].*)?)+)",
RegexOptions.Multiline);
Assert.True(block.Success,
$"{Values} no longer has a top-level 'probes:' block — this guard reads the budget from "
+ "it, and a guard that cannot find its subject passes having checked nothing.");
var body = block.Groups["body"].Value;
var period = Regex.Match(body, @"periodSeconds:\s*(\d+)");
var threshold = Regex.Match(body, @"failureThreshold:\s*(\d+)");
Assert.True(period.Success && threshold.Success,
$"{Values} probes.startup no longer declares both periodSeconds and failureThreshold.");
return (int.Parse(period.Groups[1].Value, CultureInfo.InvariantCulture),
int.Parse(threshold.Groups[1].Value, CultureInfo.InvariantCulture));
}
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| --- | ||
| Name: The chart can no longer describe a startup budget it does not have | ||
| Category: Fix | ||
| Description: The deployment template said, twice, that the portal's startup-probe budget was 30 minutes. It ships 5. Anyone arming the NodeType bake gate on that sentence would have had every pod killed mid-bake — the exact failure the same comment warns about. The prose now states what ships, and a guard keeps it that way. |
| # periodSeconds × failureThreshold is the ceiling. What this chart SHIPS, and what | ||
| # every environment in the fleet runs today, is the ungated default — right for a plain | ||
| # boot and far too small for a cold bake, so turning the gate on without raising it in | ||
| # the SAME change kills the pod mid-bake, every time, for ever. The paired setting and |
| using System; | ||
| using System.Globalization; | ||
| using System.IO; | ||
| using System.Linq; | ||
| using System.Text.RegularExpressions; |
Test Results (shard 3)444 tests 444 ✅ 4m 49s ⏱️ Results for commit 99c2c01. |
Test Results (shard 4) 10 files 10 suites 5m 19s ⏱️ Results for commit 99c2c01. |
Test Results (shard 5)2 184 tests 2 180 ✅ 6m 35s ⏱️ Results for commit 99c2c01. |
Test Results (shard 1)988 tests 988 ✅ 6m 11s ⏱️ Results for commit 99c2c01. |
Test Results (shard 0)1 180 tests 987 ✅ 5m 28s ⏱️ Results for commit 99c2c01. |
Test Results (shard 2)624 tests 624 ✅ 6m 5s ⏱️ Results for commit 99c2c01. |
Test Results 42 files 42 suites 34m 30s ⏱️ Results for commit 99c2c01. |
Found while measuring #2787's "close it at the source" direction against the live cluster.
The defect
deploy/helm/templates/memex-portal/deployment.yamlasserted the portal's startup-probe budget twice:values.yamlshipsprobes.startup5 × 60 = 300 s, and memex-cloud was measured live on 2026-08-30 running exactly that (failureThreshold=60, periodSeconds=5).That is not cosmetic.
PreWarm__GateReadiness— the pod-side backstop for the very rollout window #2787 is about, recorded as"false"in #2274's measurement — has three numbered prerequisites, and #2 is a startup budget covering a full cold bake (~570 s: ~2.4 s per NodeType, sequential, ~240 types, derived by the chart itself from production Loki). So an operator arming the gate reads prerequisite 2, is told by the next comment that the budget is already 30 minutes, and arms it on a five-minute probe — at which point Kubernetes "kills the pod mid-bake, every time, forever", which is verbatim what the same comment warns about.Prose asserting a guarantee that does not exist, in the one file where being wrong costs a rollout.
The fix
Both comments now describe what ships. More usefully, the coupling stops being a paragraph:
PreWarmGateReadinessGuardN × Mstartup budget that is not the onevalues.yamlships, unless the line marks itself as a suggested paired setting (raise…,suggested,→). This half fails today, on the drifted comment. A guard that only woke up once someone armed the gate would have caught nothing on the day the prose went wrong — which is exactly why it survived this long.GateReadiness: "true"requires the budget ≥ 570 s bake + 300 s plain boot,PreWarm__DynamicTypes: "true",maxUnavailable: 0, and a startupProbe on/health(the only endpoint that reads the gate). The chart's own numbered list, now arithmetic.It reads
probes.startupout ofvalues.yamlrather than restating it, and fails loudly if theprobes:block ever moves — a guard that cannot find its subject passes having checked nothing.Mutation-proved, both halves
10 x 180proseGateReadinesson the shipped budgetRestored and re-verified 2/2 green after each.
Negative result, recorded on #2787
The rollout configuration itself is correct and I nearly "fixed" it before the chart's own comment stopped me:
maxUnavailable: 0+maxSurge: 1+ a PDB, readiness deliberately on/alive(pointing it back at/healthre-creates the documented probe death-spiral), and the startup probe holding readiness on/healthuntil the mesh is up. The nginx 503 dependent repos see during promotion is not explained by any of those, so that mechanism stays open on #2787.Touches no workflow file, so #2800's
ShardWeightHeadroomGuardtimeout parse is unaffected.🤖 Generated with Claude Code