Skip to content

fix(publish-bake): a python with venv + pip, so the self-hosted lane can publish - #4583

Merged
rbuergi merged 1 commit into
mainfrom
fix/publish-bake-venv-on-arc
Sep 17, 2026
Merged

rbuergi merged 1 commit into
mainfrom
fix/publish-bake-venv-on-arc

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

The outage

publish-bake-bundles.sh builds a venv and pip-installs its pinned Azure SDKs into it — on purpose, so a satellite whose ref is newer than its pin cannot run a script whose dependency nobody installed (see the script's own header).

The ARC runner image ships python3 3.12 without ensurepip/venv. So from the moment every private repo's jobs moved to aks-silos* this morning, the publish step died with:

The virtual environment was not created successfully because ensurepip is not
available. ... apt install python3.12-venv

No satellite could publish. Evidence — the publish job failed while every other job in the same run was green:

run repo job
35199593887 MeshWeaver.Crm publish-bake / Bake + publish NodeType assemblies to portal storage
35205340142 MeshWeaver.Manufacturing same

publish-bake is skipped whenever a run has nothing to bake, which is why the migration's own proofs did not catch this: it only fires on a run that actually publishes.

The fix

actions/setup-python brings its own interpreter with pip on any runner — hosted or ours — so the script's venv works everywhere. On a hosted runner the tool cache already carries it, so it costs nothing there. Not sudo apt-get install python3-venv: that needs root and the archive, and would be the only thing standing between the fleet and a failed publication.

The durable fix is baking python3-venv into the runner image (Systemorph/Memex deployments/aks/ci-runners/runner-image/Dockerfile); the step's comment says to delete it when that lands. The script asserts its own pins afterwards (sdk-check), so a wrong interpreter is red here rather than a later "unreadable".

Checks

check-workflow-yaml-keys, check-workflow-timeouts, check-reusable-workflow-runners and check-workflow-shell all pass --self-test and clean over the edited tree.

🤖 Generated with Claude Code

…can publish

publish-bake-bundles.sh builds a venv and installs its pinned Azure SDKs into
it — deliberately, so a satellite whose ref is newer than its pin cannot end up
running a script whose dependency nobody installed. The ARC runner image ships
python3 3.12 WITHOUT ensurepip/venv, so from the moment every private repo moved
to the self-hosted sets (2026-09-17) the publish step died with

  The virtual environment was not created successfully because ensurepip is not
  available. ... apt install python3.12-venv

and no satellite could publish: MeshWeaver.Crm run 35199593887 and
MeshWeaver.Manufacturing run 35205340142 failed there with every other job in
the run green — a delivery outage that only shows up on a run that actually has
something to bake, which is why the migration's own proofs did not catch it.

`actions/setup-python` brings an interpreter WITH pip on any runner, so the
script's venv works hosted or self-hosted; on a hosted runner the tool cache
already carries it. The durable fix is to bake python3-venv into the runner
image, and the comment says to delete this step when that lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 17, 2026 10:16
@rbuergi
rbuergi enabled auto-merge September 17, 2026 10:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The workflow change addresses the self-hosted runner dependency issue without unresolved blocking concerns.

Pull request overview

Updates the reusable publish-bake workflow to provision Python 3.12 with venv and pip on self-hosted runners before publishing bundles.

Changes:

  • Adds actions/setup-python@v7.
  • Runs setup only when publication is required.
  • Preserves pinned Azure SDK validation.
File summaries
File Description
.github/workflows/node-repo-publish-bake.yml Provisions Python for the publish script.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

264 tests  ±0   264 ✅ ±0   2m 38s ⏱️ -7s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 8809d5b. ± Comparison against base commit f993503.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

425 tests  ±0   425 ✅ ±0   49s ⏱️ -12s
  2 suites ±0     0 💤 ±0 
  2 files   ±0     0 ❌ ±0 

Results for commit 8809d5b. ± Comparison against base commit f993503.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

533 tests  ±0   533 ✅ ±0   58s ⏱️ -7s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 8809d5b. ± Comparison against base commit f993503.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

    3 files  ±0      3 suites  ±0   5m 1s ⏱️ -51s
2 285 tests ±0  2 285 ✅ ±0  0 💤 ±0  0 ❌ ±0 
2 286 runs  ±0  2 286 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 8809d5b. ± Comparison against base commit f993503.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 491 tests  ±0   1 489 ✅ ±0   7m 34s ⏱️ -7s
    4 suites ±0       2 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit 8809d5b. ± Comparison against base commit f993503.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    6 files  ±0      6 suites  ±0   11m 21s ⏱️ -12s
3 172 tests ±0  2 980 ✅ ±0  192 💤 ±0  0 ❌ ±0 
3 176 runs  ±0  2 984 ✅ ±0  192 💤 ±0  0 ❌ ±0 

Results for commit 8809d5b. ± Comparison against base commit f993503.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

   17 files  ±0     17 suites  ±0   28m 22s ⏱️ - 1m 38s
8 170 tests ±0  7 976 ✅ ±0  194 💤 ±0  0 ❌ ±0 
8 175 runs  ±0  7 981 ✅ ±0  194 💤 ±0  0 ❌ ±0 

Results for commit 8809d5b. ± Comparison against base commit f993503.

@rbuergi
rbuergi added this pull request to the merge queue Sep 17, 2026
Merged via the queue into main with commit a86b48c Sep 17, 2026
37 checks passed
@rbuergi
rbuergi deleted the fix/publish-bake-venv-on-arc branch October 10, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants