Skip to content

fix(chart): the startup budget the chart DESCRIBES is now the one it ships (#2787, #2274) - #2812

Merged
rbuergi merged 1 commit into
mainfrom
fix/prewarm-gate-startup-budget-guard
Aug 30, 2026
Merged

rbuergi merged 1 commit into
mainfrom
fix/prewarm-gate-startup-budget-guard

Conversation

@rbuergi

@rbuergi rbuergi commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Found while measuring #2787's "close it at the source" direction against the live cluster.

The defect

deploy/helm/templates/memex-portal/deployment.yaml asserted the portal's startup-probe budget twice:

:281 — "Paired setting and full derivation: values.yaml (probes.startup) — 10 × 180 = 30 min."
:455 — "the startupProbe budget above … which managed envs set to 10 × 180 = 30 MINUTES"

values.yaml ships probes.startup 5 × 60 = 300 s, and memex-cloud was measured live on 2026-08-30 running exactly that (failureThreshold=60, periodSeconds=5).

That is not cosmetic. PreWarm__GateReadiness — the pod-side backstop for the very rollout window #2787 is about, recorded as "false" in #2274's measurement — has three numbered prerequisites, and #2 is a startup budget covering a full cold bake (~570 s: ~2.4 s per NodeType, sequential, ~240 types, derived by the chart itself from production Loki). So an operator arming the gate reads prerequisite 2, is told by the next comment that the budget is already 30 minutes, and arms it on a five-minute probe — at which point Kubernetes "kills the pod mid-bake, every time, forever", which is verbatim what the same comment warns about.

Prose asserting a guarantee that does not exist, in the one file where being wrong costs a rollout.

The fix

Both comments now describe what ships. More usefully, the coupling stops being a paragraph:

PreWarmGateReadinessGuard

  • Unconditional — no line in the chart may state an N × M startup budget that is not the one values.yaml ships, unless the line marks itself as a suggested paired setting (raise…, suggested, →). This half fails today, on the drifted comment. A guard that only woke up once someone armed the gate would have caught nothing on the day the prose went wrong — which is exactly why it survived this long.
  • Conditional — GateReadiness: "true" requires the budget ≥ 570 s bake + 300 s plain boot, PreWarm__DynamicTypes: "true", maxUnavailable: 0, and a startupProbe on /health (the only endpoint that reads the gate). The chart's own numbered list, now arithmetic.

It reads probes.startup out of values.yaml rather than restating it, and fails loudly if the probes: block ever moves — a guard that cannot find its subject passes having checked nothing.

Mutation-proved, both halves

probe result
restore the false 10 x 180 prose FAILS — "states a startup budget of 10 × 180 = 1800s … but values.yaml ships 5 × 60 = 300s"
arm GateReadiness on the shipped budget FAILS — "armed, but probes.startup is 5 × 60 = 300s and a gated boot needs at least 870s"

Restored and re-verified 2/2 green after each.

Negative result, recorded on #2787

The rollout configuration itself is correct and I nearly "fixed" it before the chart's own comment stopped me: maxUnavailable: 0 + maxSurge: 1 + a PDB, readiness deliberately on /alive (pointing it back at /health re-creates the documented probe death-spiral), and the startup probe holding readiness on /health until the mesh is up. The nginx 503 dependent repos see during promotion is not explained by any of those, so that mechanism stays open on #2787.

Touches no workflow file, so #2800's ShardWeightHeadroomGuard timeout parse is unaffected.

🤖 Generated with Claude Code

…ships (#2787, #2274)

Found while measuring #2787's "close it at the source" direction against the
live cluster.

deploy/helm/templates/memex-portal/deployment.yaml asserted the portal's
startup-probe budget TWICE:

  :281  "Paired setting and full derivation: values.yaml (probes.startup)
         — 10 × 180 = 30 min."
  :455  "the startupProbe budget above … which managed envs set to
         10 × 180 = 30 MINUTES"

values.yaml ships probes.startup 5 × 60 = 300s, and memex-cloud was measured
live on 2026-08-30 running exactly that (kubectl: failureThreshold=60,
periodSeconds=5).

That is not a cosmetic drift. PreWarm__GateReadiness — the pod-side backstop
for the very rollout window #2787 is about, and recorded as "false" in
#2274's measurement — carries three numbered prerequisites, and #2 is a
startup budget covering a full COLD bake (~570s: ~2.4s per NodeType,
sequential, ~240 types, derived by the chart itself from production Loki).
An operator arming the gate reads prerequisite 2, is told by the next comment
that the budget is already 30 minutes, and arms it on a 5-minute probe — so
Kubernetes kills the pod mid-bake, every time, forever, which is verbatim
what the same comment warns about. Prose asserting a guarantee that does not
exist, in the one file where being wrong costs a rollout.

- Both comments now describe what SHIPS, and say arming the gate means
  raising the budget in the same change.
- PreWarmGateReadinessGuard makes the coupling arithmetic instead of prose:
    * UNCONDITIONAL: no line in the chart may state an "N × M" startup budget
      that is not the one values.yaml ships, unless the line marks itself as a
      suggested paired setting ("raise…", "suggested", "→"). This half fails
      TODAY on the drifted comment — a guard that only fired once the gate was
      armed would have caught nothing on the day the prose went wrong, which is
      exactly why it survived.
    * CONDITIONAL: GateReadiness=true requires budget ≥ 570s bake + 300s plain
      boot, PreWarm__DynamicTypes=true, maxUnavailable: 0, and a startupProbe
      on /health — the chart's own list, now enforced.
  It reads probes.startup out of values.yaml rather than restating it, and
  fails loudly if the `probes:` block moves (a guard that cannot find its
  subject passes having checked nothing).

MUTATION-PROVED, both halves, not merely observed green:
  - restoring the false "10 x 180" prose → FAILS: "states a startup budget of
    10 × 180 = 1800s … but values.yaml ships 5 × 60 = 300s"
  - arming GateReadiness on the shipped budget → FAILS: "armed, but
    probes.startup is 5 × 60 = 300s and a gated boot needs at least 870s"
  Restored and re-verified 2/2 green after each probe.

🚦 Recorded on #2787 as a NEGATIVE result too: the rollout config itself is
correct (maxUnavailable:0 + maxSurge:1 + PDB, readiness deliberately on
/alive to avoid the documented probe death-spiral, startup holding readiness
on /health). The nginx 503 dependent repos see during promotion is NOT
explained by those, so that mechanism stays open on #2787.

Touches no workflow file, so ShardWeightHeadroomGuard's timeout parse is
unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 30, 2026 17:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR corrects misleading Helm-chart commentary about the portal startupProbe budget and adds a build-time guard to prevent future drift between what the chart says and what it ships, especially around PreWarm__GateReadiness rollouts.

Changes:

  • Updates memex-portal Deployment template comments to describe the shipped startupProbe budget and its coupling to values.yaml.
  • Adds PreWarmGateReadinessGuard test to enforce (a) “stated budget equals shipped budget” and (b) prerequisites when PreWarm__GateReadiness is armed.
  • Adds a “What’s New” entry documenting the defect and the new guard.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
test/MeshWeaver.Documentation.Test/PreWarmGateReadinessGuard.cs New guard test enforcing probe-budget/prose coupling and gate prerequisites.
deploy/helm/templates/memex-portal/deployment.yaml Fixes misleading budget commentary to align with shipped values.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-30-the-chart-can-no-longer-describe-a-budget-it-does-not-have.md Documents the change and rationale for operators.
Suppressed comments (1)

test/MeshWeaver.Documentation.Test/PreWarmGateReadinessGuard.cs:151

  • StartupBudget()’s probes-block regex isn’t indentation-scoped: (?:\n(?:[ \t].*)?)+ can consume non-indented top-level lines (e.g. grpc:) because the indented part is optional, so the guard can accidentally read periodSeconds/failureThreshold from outside probes.startup if those keys appear later in the file. Tighten the match to a real probes.startup block and extract keys from that block only.
    private static (int Period, int Threshold) StartupBudget(string values)
    {
        var block = Regex.Match(values, @"^probes:\s*$(?<body>(?:\n(?:[ \t].*)?)+)",
            RegexOptions.Multiline);
        Assert.True(block.Success,
            $"{Values} no longer has a top-level 'probes:' block — this guard reads the budget from "
            + "it, and a guard that cannot find its subject passes having checked nothing.");
        var body = block.Groups["body"].Value;
        var period = Regex.Match(body, @"periodSeconds:\s*(\d+)");
        var threshold = Regex.Match(body, @"failureThreshold:\s*(\d+)");
        Assert.True(period.Success && threshold.Success,
            $"{Values} probes.startup no longer declares both periodSeconds and failureThreshold.");
        return (int.Parse(period.Groups[1].Value, CultureInfo.InvariantCulture),
                int.Parse(threshold.Groups[1].Value, CultureInfo.InvariantCulture));
    }

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

---
Name: The chart can no longer describe a startup budget it does not have
Category: Fix
Description: The deployment template said, twice, that the portal's startup-probe budget was 30 minutes. It ships 5. Anyone arming the NodeType bake gate on that sentence would have had every pod killed mid-bake — the exact failure the same comment warns about. The prose now states what ships, and a guard keeps it that way.
# periodSeconds × failureThreshold is the ceiling. What this chart SHIPS, and what
# every environment in the fleet runs today, is the ungated default — right for a plain
# boot and far too small for a cold bake, so turning the gate on without raising it in
# the SAME change kills the pod mid-bake, every time, for ever. The paired setting and
Comment on lines +1 to +5
using System;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

444 tests   444 ✅  4m 49s ⏱️
  6 suites    0 💤
  6 files      0 ❌

Results for commit 99c2c01.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

   10 files     10 suites   5m 19s ⏱️
3 084 tests 3 084 ✅ 0 💤 0 ❌
3 754 runs  3 754 ✅ 0 💤 0 ❌

Results for commit 99c2c01.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

2 184 tests   2 180 ✅  6m 35s ⏱️
   10 suites      4 💤
   10 files        0 ❌

Results for commit 99c2c01.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

988 tests   988 ✅  6m 11s ⏱️
  5 suites    0 💤
  5 files      0 ❌

Results for commit 99c2c01.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

1 180 tests   987 ✅  5m 28s ⏱️
    5 suites  193 💤
    5 files      0 ❌

Results for commit 99c2c01.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

624 tests   624 ✅  6m 5s ⏱️
  6 suites    0 💤
  6 files      0 ❌

Results for commit 99c2c01.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

   42 files     42 suites   34m 30s ⏱️
8 504 tests 8 307 ✅ 197 💤 0 ❌
9 174 runs  8 977 ✅ 197 💤 0 ❌

Results for commit 99c2c01.

@rbuergi
rbuergi added this pull request to the merge queue Aug 30, 2026
Merged via the queue into main with commit 70e887d Aug 30, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants