Skip to content

ci: a wake lane's "upstream not sealed" is a WAIT, and the poll never retries it - #3747

Merged
meshweaver-cloud[bot] merged 1 commit into
mainfrom
fix/wake-lane-not-yet-is-a-wait
Sep 8, 2026
Merged

meshweaver-cloud[bot] merged 1 commit into
mainfrom
fix/wake-lane-not-yet-is-a-wait

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes the shared half of MeshWeaver.Manufacturing#67 and MeshWeaver.Crm#66 — the nightly release poll is red in both repos and the shared lane's own diagnostic points the reader at a mechanism that does not exist on that trigger.

What was measured

repo trigger job that failed why
Manufacturing schedule (run 34184029469) test-repos / Gate shard 1/1 framework-identity: MISMATCH — the bake published under 's4e84d301…' but '/portal' resolves 's556a0d43…' — tester followed :latest, portal stayed a literal
Manufacturing schedule memex build plugin upstream 'plugins' has no SEALED publication … for identity s4e84d301… → exit 8
Crm schedule (run 34185317544) publish-bake same MISMATCH, s4e84d301… vs sf456af88…
Crm meshweaver-framework-released (run 34245799929) publish-bake Upstreams not ready — the lane resolved both by version (identity MATCH), plugins was simply not sealed yet

Crm's poll has been red 8 of the last 10 nights; Manufacturing's on both nights it has run.

1. The promise the lane could not keep

node-repo-gate.yml refused with "The upstream's own publish-bake for this identity re-enables this run." node-repo-publish-bake.yml's step summary for the same condition said the opposite:

Nothing to do — if this repo has the schedule trigger. The next scheduled run re-reads the released image and rebuilds once the upstream has published. Nothing wakes this repo otherwise: the dependent-dispatch lane was removed deliberately (no cross-repo write credential)…

Both halves are wrong. The dispatch lane exists and works — measured on Crm 2026-09-08: the availability gate failed at 15:36Z and the meshweaver-upstream-published wake for the same set succeeded at 16:01Z. And the poll does not retry the identity it failed on: it re-resolves a moving tag, so tomorrow's run asks about whatever has been promoted since.

Both lanes now carry one statement under one title (Upstreams not ready), with a table naming how each trigger learned the identity and which of them makes an unsealed upstream this repository's fault:

the run was it learned the identity from an unsealed upstream means
push / pull_request the repo's PIN the PIN is wrong
repository_dispatch the wake's own payload the wave is mid-flight — a WAIT
schedule a moving tag it resolved itself nobody promised the upstream published for it — a WAIT

🚨 It still fails RED on every trigger. No exit-zero arm, no continue-on-error:, no if: asking about an input — the event changes what is said, never whether the gate runs. Verified by executing the function: exit 1, one ::error title=Upstreams not ready, one step summary.

The rejected alternative is written into the doc so it is not re-proposed: having the poll walk back to the newest identity that has a complete publication reads like a narrowing but is a fallback — an upstream that stops publishing would leave every dependent's poll green forever, the same trapdoor as falling back to the pin.

2. The recipe that produced the mixed pair

CiContentBake.md's "four parts a repo has ALL of them or it follows nothing" told a satellite to "on a release trigger resolve the digest MW_TEST_IMAGE currently points at" — and stopped there. That text predates MeshWeaver#3022, which made the portal a second required image. A caller following it faithfully moves one half of the wave and leaves platform-image-digest: a literal → two CD waves in one bake → the MISMATCH above.

Part 3 now requires both images from one tag, with the per-trigger entitlement table (:main for the poll, never :latest — promote phase B applies latest to mw-plugin-test only, so the portal has no such tag), and the wait-vs-failure rule. MeshWeaver.Reinsurance#179 is the landed shape the two satellite PRs adopt.

3. Also

The gate's Azure seed probe now splits absent from could-not-determine, the way check-release-availability.sh already does — an unreadable share is a hold naming the share, not a missing upstream publication. exists=unknown used to be reported as "the upstream has not published".

Denominator — 6 of 6 satellites read, 3 carry the defect

repo platform-image-digest on the release path verdict
MeshWeaver.Manufacturing literal, tester followed the release ❌ → #72 (poll red 2/2 nights, all 10 upstream wakes red)
MeshWeaver.Crm literal, bake followed the release ❌ → #70 (poll red 8/10 nights)
MeshWeaver.SocialMedia literal, bake followed the release ❌ → #156 (poll red 7/8 nights) — found by this sweep, filed as SocialMedia#155
MeshWeaver.Reinsurance needs.preflight.outputs.portal-image-digest ✅ already fixed by its #179
MeshWeaver.Education needs.preflight.outputs.{portal,bake-portal}-image-digest ✅
MeshWeaver.Plugins expressions on gate + publish-bake; the two module-pack literals pass a matched tester+portal pair ✅ (and it is the root upstream — no upstream-seed)

Each ❌ was confirmed from the failing job's log, not inferred from the text: the same framework-identity: MISMATCH on all three, all three resolving tester s4e84d301… against their own older portal pin.

That three repositories reached the identical defect independently is the reason the doc change matters more than the three workflow fixes: they were each following what this page told them to do.

Pairs-with: none — no public type or member is removed; this diff is two workflows and one doc page.

Verification

  • check-workflow-shell.py, check-workflow-yaml-keys.py, check-workflow-timeouts.py — all clean.
  • Both lanes parse as YAML; every run: block passes bash -n with expressions neutralised.
  • The new upstream_not_ready function executed standalone: exit 1, correct annotation, correct summary.

… retries it

Manufacturing#67 / Crm#66 — the nightly release poll is red in both repos, and
the shared lane's own diagnostic sends the reader to a mechanism that does not
exist on that trigger.

Two halves, measured on run 34184029469 (Manufacturing, schedule) and
34185317544 / 34245799929 (Crm, schedule + framework-released):

1. THE PROMISE. node-repo-gate.yml said "The upstream's own publish-bake for this
   identity re-enables this run" and node-repo-publish-bake.yml's step summary
   said the opposite — "the next scheduled run ... rebuilds once the upstream has
   published", plus "the dependent-dispatch lane was removed deliberately". The
   second is stale (the dispatch lane exists and works: Crm red 15:36Z, its
   meshweaver-upstream-published wake green 16:01Z) and the first is false for a
   POLL, which re-resolves a moving tag and therefore asks about a DIFFERENT
   identity next time. Both lanes now say one true thing, under one title
   ("Upstreams not ready"), with a table naming how each trigger learned the
   identity and which of them makes an unsealed upstream this repo's fault.

   It still fails RED on every trigger. No exit-zero arm, no continue-on-error,
   no if: about an input — the event changes what is SAID, never whether the
   gate runs.

2. THE RECIPE. CiContentBake.md's "four parts" told a satellite to resolve the
   TESTER on a release trigger and stopped there — written before MeshWeaver#3022
   made the portal a second required image. A caller following it moves one half
   of the wave and leaves platform-image-digest a literal, which the lanes refuse
   as "the tester image and the platform image do NOT resolve one framework
   identity". Part 3 now requires BOTH images from ONE tag, with the per-trigger
   entitlement table (:main for the poll, never :latest — promote applies latest
   to mw-plugin-test only) and the wait-vs-failure rule.

Also splits ABSENT from COULD-NOT-DETERMINE in the gate's Azure seed probe, the
way check-release-availability.sh already does: an unreadable share is a hold
naming the share, not a missing upstream publication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 8, 2026 18:02
@meshweaver-cloud
meshweaver-cloud Bot enabled auto-merge September 8, 2026 18:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are narrowly scoped to CI/docs diagnostics and consistency, with no confirmed functional regressions found in the updated logic.

Pull request overview

Updates the shared node-repo CI guidance and reusable workflows to correctly describe (and consistently report) the “upstream not sealed for this identity” condition as a WAIT—especially for scheduled release polls that do not retry the same identity—and to document/require resolving both wave images (tester + portal) together.

Changes:

  • Rewrites the node-repo-publish-bake.yml “Upstreams not ready” step summary + annotation to reflect trigger-specific semantics (pin wrong vs wave in flight vs schedule wait) and the actual wake mechanism.
  • Adds a shared upstream_not_ready() diagnostic helper in node-repo-gate.yml and uses it for upstream-seed 404/“not sealed” failures; splits Azure probe “absent” vs “could-not-determine”.
  • Updates CiContentBake.md to require FOLLOW_RELEASE to resolve both tester+portal from one tag, documents which triggers may resolve which refs, and codifies the “WAIT but still RED” rule.
File summaries
File Description
src/MeshWeaver.Documentation/Data/Architecture/CiContentBake.md Updates the CI content bake contract to resolve tester+portal from the same wave and documents trigger entitlement + “upstream not sealed” semantics.
.github/workflows/node-repo-publish-bake.yml Aligns the “Upstreams not ready” diagnostics with measured behavior and clarifies what actually re-enables waiting runs.
.github/workflows/node-repo-gate.yml Introduces a unified upstream-not-ready diagnostic helper and improves Azure seed probe error classification.
Review details

Suppressed comments (1)

.github/workflows/node-repo-gate.yml:698

  • This error annotation appends This run's event was ..., but in a reusable workflow that value will be workflow_call and doesn't help diagnose the caller trigger (push/schedule/dispatch). Removing it avoids confusion and shortens the annotation.
            echo "::error title=Upstreams not ready — did not gate::upstream '$src' has no SEALED publication at $where for framework identity $IDENTITY. What re-enables it is the upstream's own publish-bake dispatching meshweaver-upstream-published for this identity — NOT the schedule poll, which re-resolves a moving tag and therefore asks about a different identity next time. This run's event was ${GITHUB_EVENT_NAME:-unknown}."
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +687 to +688
echo "This run's event was \`${GITHUB_EVENT_NAME:-unknown}\`."
echo ""
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

496 tests  ±0   496 ✅ ±0   35s ⏱️ -4s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit c6161c0. ± Comparison against base commit e103981.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

1 079 tests  ±0   1 079 ✅ ±0   1m 29s ⏱️ -12s
    2 suites ±0       0 💤 ±0 
    2 files   ±0       0 ❌ ±0 

Results for commit c6161c0. ± Comparison against base commit e103981.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

254 tests  ±0   254 ✅ ±0   2m 7s ⏱️ -23s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit c6161c0. ± Comparison against base commit e103981.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 669 tests  ±0   1 475 ✅ ±0   2m 7s ⏱️ ±0s
    4 suites ±0     194 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit c6161c0. ± Comparison against base commit e103981.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    5 files  ±0      5 suites  ±0   3m 0s ⏱️ -2s
1 764 tests ±0  1 764 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 765 runs  ±0  1 765 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit c6161c0. ± Comparison against base commit e103981.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 244 tests  ±0   1 244 ✅ ±0   4m 18s ⏱️ -5s
    3 suites ±0       0 💤 ±0 
    3 files   ±0       0 ❌ ±0 

Results for commit c6161c0. ± Comparison against base commit e103981.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results

   16 files  ±0     16 suites  ±0   13m 38s ⏱️ -47s
6 506 tests ±0  6 312 ✅ ±0  194 💤 ±0  0 ❌ ±0 
6 507 runs  ±0  6 313 ✅ ±0  194 💤 ±0  0 ❌ ±0 

Results for commit c6161c0. ± Comparison against base commit e103981.

@meshweaver-cloud
meshweaver-cloud Bot added this pull request to the merge queue Sep 8, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 8, 2026
@meshweaver-cloud
meshweaver-cloud Bot added this pull request to the merge queue Sep 8, 2026
@meshweaver-cloud

Copy link
Copy Markdown
Contributor

🔀 Merge-queue steward: re-queued alone (bisecting) — bisecting: your group of 3 pull requests failed on an uncatalogued assertion while this PR's own run was green; re-queued ALONE — if the solo build fails too, the PR stays out (group: #3746, #3739).
Group build: https://github.com/Systemorph/MeshWeaver/actions/runs/34262161269

  • MeshWeaver.Data.Test.ReadDuringDisposalWindowTest.ReadThatFaultsOnTeardown_IsNackedShuttingDown_NotAnsweredAsAbsent — MeshWeaver.Reactive.Assertions.AssertionException : Expected "Hub data-owner/1 is shutting down (RunLevel=Quiescing, activation #017DC7A2) — cannot process GetDataRequest; the address may reactivate (recycle / restart). Rejecting now." to

enqueued

Merged via the queue into main with commit 5241ef4 Sep 8, 2026
34 checks passed
@rbuergi
rbuergi deleted the fix/wake-lane-not-yet-is-a-wait branch September 13, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants