Repository navigation
feat(operator): hosting-aks-upgrade — the cluster half of the governed UpgradeCluster action - #5682
Merged
Merged
Conversation
…d UpgradeCluster action The AKS Kubernetes upgrade becomes an operator command the aks-ops lane runs from a signed, mesh-approved plan (policy `cluster-upgrade-governed`), never an ad-hoc `az aks upgrade`: - `facts` (read-only): cluster version, offered upgrades + the region's version graph, pools, nodes, every portal Deployment's TEMPLATE image against the image its pods SERVE, /health, PDBs, the API server's deprecated-API metric (unreadable = UNREAD, never "none") and the scanned manifests' apiVersions — one base64 JSON fact for the plugin's pure plan. - `gate [--strict]`: a stuck roll refuses the pre-flight by name (memex 2026-09-25: template 3.0.0-ci.9260 over serving 3.0.0-ci.9218); between pools it waits inside the job budget and fails on what Kubernetes gave up on. - `control-plane` / `node-pool` / `verify`: idempotent, resumable hops (already there or already past → nothing issued; the runner's own pool detached; a budget pause is aks_upgrade=incomplete for the control plane to continue). The ORDER (one approval, control-plane minors chained, each pool once to the final version, portals' pool last) is the Hosting plugin's ClusterUpgradePlan; this script decides none of it. 19 behaviour cases against stubs; the RBAC map learns `pdb`; the plan half lands in MeshWeaver.Plugins, so the command is declared not-yet-wired until then. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Test Results (shard 0) 1 files 1 suites 2m 43s ⏱️ Results for commit 33995cb. |
Contributor
Test Results (shard 3)443 tests 443 ✅ 1m 1s ⏱️ Results for commit 33995cb. |
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Unresolved critical and moderate findings remain in hosting-aks-upgrade.
Review effort: Lite
Findings: None
What changed in this PR
Adds the operator-side governed AKS UpgradeCluster action with safety gates, resumability, verification, tests, fixtures, RBAC coverage, and documentation.
Changes:
- Adds
hosting-aks-upgradefacts, gates, upgrade, and verification commands. - Adds command stubs, AKS fixtures, and behavioral tests.
- Updates policy, workflow tracking, deployment guidance, and RBAC coverage.
Review found one critical and seven moderate unresolved issues in the upgrade command.
| File | Description |
|---|---|
src/MeshWeaver.Documentation/Data/Architecture/PolicyNotProse.md |
Registers the proposed upgrade policy. |
src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md |
Documents governed AKS upgrades. |
deploy/aks/operator/test/stubs/aks-upgrade/kubectl |
Kubernetes CLI test stub. |
deploy/aks/operator/test/stubs/aks-upgrade/curl |
Health probe test stub. |
deploy/aks/operator/test/stubs/aks-upgrade/az |
Azure CLI test stub. |
deploy/aks/operator/test/run-tests.sh |
Upgrade behavior tests. |
deploy/aks/operator/test/fixtures/aks-upgrade/versions.json |
Version graph fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/upgrades.json |
Upgrade offers fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/show.json |
Cluster state fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/show.after.json |
Post-upgrade cluster state fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/runner.json |
Runner pool fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/pools.json |
Node-pool fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/pools.after.json |
Post-upgrade node-pool fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/pods-memex.json |
Portal pod fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/pdbs.json |
Pod disruption budget fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/nodes.json |
Node fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/metrics.txt |
Deprecated API metrics fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/manifests/pdb.yaml |
Manifest API fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/manifests/cron.yml |
Manifest API fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/ingress-memex.json |
Ingress fixture. |
deploy/aks/operator/test/fixtures/aks-upgrade/deployments.json |
Deployment fixture. |
deploy/aks/operator/test/check-rbac-coverage.sh |
Adds PDB RBAC coverage. |
deploy/aks/operator/bin/hosting-aks-upgrade |
Implements AKS upgrade operations and safety gates. |
.github/workflows/hosting-operator.yml |
Tracks the operator command and test stubs. |
.claude/skills/deployment/SKILL.md |
Documents the upgrade action. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Test Results (shard 1)1 652 tests 1 652 ✅ 3m 16s ⏱️ Results for commit 33995cb. |
Contributor
Test Results (shard 4) 3 files 3 suites 6m 16s ⏱️ Results for commit 33995cb. |
Contributor
Test Results (shard 2)749 tests 557 ✅ 7m 8s ⏱️ Results for commit 33995cb. |
Contributor
Test Results (shard 5) 5 files 5 suites 13m 29s ⏱️ Results for commit 33995cb. |
Contributor
Test Results 17 files 17 suites 33m 56s ⏱️ Results for commit 33995cb. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The operator half of the governed
UpgradeClusterInstanceAction (policycluster-upgrade-governed, registered here asproposeduntil the plan half merges):deploy/aks/operator/bin/hosting-aks-upgrade, the only command that upgrades the AKS cluster's Kubernetes version, run by theaks-opslane from a signed, mesh-approved bundle — neveraz aks upgradeby hand.factsaks_facts=<base64 JSON>line: cluster version/state/tier/support plan, offered upgrades (previews out) + the region's version graph (for chaining minors and the LTS note), pools, nodes, every portal Deployment's template image vs the images its pods serve,/healthper host, PDBs, the API server'sapiserver_requested_deprecated_apismetric (an unreadable metric is carried as UNREAD, never "none"), and the apiVersion/kind of every scanned manifest (templated ones skipped)gate [--strict]--strict(the pre-flight) refuses a stuck roll by name — both images — having issued nothing; between pools it waits inside the job budget, pauses there, and FAILS on what Kubernetes gave up on (ProgressDeadlineExceeded, a pod not Ready past its own startup budget,/healthfailing with every pod Ready > 10 min)control-plane --versionnode-pool --pool --versionverify --versionaks_upgrade=completeonly when the control plane, every pool and every node are at the version and the gate is healthyResumability: a wait reaching
--budget-seconds(measured from the job's FIRST step) printsaks_upgrade=incomplete, exits 0, and every later step of that run is skipped — the control plane dispatches a continuation of the same plan.🚨 Why the stuck-roll refusal: measured 2026-09-25 — memex's Deployment template names
3.0.0-ci.9260(pods of that image deadlock at bake; fixed by #5643) while its serving pods are3.0.0-ci.9218, alive only because nothing evicts them. A node drain would recreate them from the template and take the control instance down.The order (one approval; control-plane minors chained one at a time; every pool upgraded ONCE to the final version; the pool hosting the portals last; a gate before each pool) is decided by the Hosting plugin's pure
ClusterUpgradePlan— this script decides none of it.Also
check-rbac-coverage.shlearns thepdbalias (the ClusterRole already grantspolicy poddisruptionbudgetsget/list).hosting-operator.yml: floor 31 → 32, the three new stubs in the tracked list, andhosting-aks-upgradedeclared not-yet-wired until the Plugins plan step merges (it moves up into the plan list in the change after).Verification
deploy/aks/operator/test/run-tests.sh: 592 passed, 0 failed locally (19 newhosting-aks-upgradecases; every waiting call carries a 5 s budget so a regression fails in seconds, never spins the suite for 35 min).shellcheck -S warningclean on the script and stubs;check-rbac-coverage.sh: 44 calls, 0 missing, 0 unknown;check-stderr-discarded.sh: 0 undeclared.dotnet build src/MeshWeaver.Documentation -c Release -warnaserror: 0 warnings, 0 errors.What this does NOT do
UpgradeClusterlane choice + read step (Systemorph/Memexaks-ops.yml) follow.az aks get-versionsin swedencentral (the parser follows the documentedvalues[].patchVersionsshape) and whetherhosting-operatoralready holds the ARM rights the reads/writes need ("Azure Kubernetes Service Contributor Role" on the cluster) — the command refuses a missing grant BY NAME rather than reading it as "nothing to upgrade".Pairs-with: none — no public .NET surface touched (bash + docs only).
🤖 Generated with Claude Code