Rehearse the release weekly from the release pipeline itself

[Alexandre Zollinger Chohfi (azchohfi)]

Why

ES/1ES policy changes keep surfacing mid-release — the most expensive possible moment. The release stalls, someone edits the pipeline under time pressure, and we ship late or ship twice.

This makes release.yml rehearse a release every Monday from main, so a policy or credential failure shows up days before anyone cuts rel/v*.

How

There is no separate dry-run pipeline. The real release pipeline runs weekly with only the final publishing actions omitted — same 1ES Official template, same stages, same jobs, same tasks, same service connections. A lookalike pipeline could only ever approximate those.

The mode is derived from the branch, never from a parameter. This is the load-bearing detail: Azure DevOps compiles both scheduled and branch-triggered runs with the default values of parameters:. A dryRun parameter therefore cannot distinguish a Monday rehearsal from a real release — defaulting it false makes the weekly run publish for real; defaulting it true makes every real release silently ship nothing.

Every gate is a positive test for rel/v*, so an unexpected branch fails closed. Grep refs/heads/rel/v.

What the rehearsal skips

Action	Gate
ESRP code signing	DoEsrp ANDed with the branch
GitHub release	compile-time ${{ if }}
Symbol publication	stage omitted
npm publish	stage omitted
nuget.org push	stage omitted
WinGet fork sync + --submit	WINGET_SUBMIT env, 'false' unless rel/v*
MS Learn push + PR	MSLEARN_PUBLISH env, 'false' unless rel/v*

Everything else runs for real: the full stable build with telemetry unstubbed, all tests, packaging, release-notes generation, the real Release_GitHub job up to the upload, WinGet manifest generation (installers downloaded and hashed, stopping at --submit), and MS Learn porting/validation/commit (stopping at the push).

Review findings addressed

A full multi-dimension review ran on this branch. The headline catch came from the different-model cross-check:

Critical — main moved under this branch. #795 reverted the NuGet-publish split, putting a Release_NuGet + 1ES.PublishNuget@1 stage back into release.yml gated only on DoEsrp. Since DoEsrp defaults to true and scheduled runs compile with defaults, merging main would have made the weekly rehearsal push to nuget.org. Now branch-gated like every other publishing stage.

Also fixed:

• Credentialed steps restricted to main with a positive condition, rather than "any branch that isn't rel/v*". Production tokens were reachable from any branch someone could queue.
• WingetPkgsFork validated via env instead of ADO macro expansion — it was interpolated unquoted into an inline PowerShell body, where ; is a statement separator.
• WINGET_SUBMIT / MSLEARN_PUBLISH defined explicitly as 'false' in a rehearsal. Azure injects pipeline variables into the task environment, so an absent variable could be inherited rather than meaning "do not publish".
• Credential check made advisory (continueOnError). It was a hard gate early in Build, so a dead Models token would abort before the WinGet and MS Learn rehearsals — the stages this exists to run.
• WinGet rehearses against the release list, not /releases/latest, which excludes prereleases. Every release here ships as a prerelease, so it was validating an ever-staler version.
• Portable winappcli-<arch>.zip names verified in Release_GitHub, where they're created. The upload glob is winappcli-*.zip and their URLs are hardcoded in the WinGet submission, so a rename would have uploaded fine and 404'd later.
• Asset-name preflight promoted to every build, not just rehearsals — a real release could previously publish versioned names and break the WinGet URLs silently, exactly Fix: Wingetcreate installer count mismatch in release pipeline #568.

Plus a cut list, removing ~90 lines the review found weren't earning their keep: a dead -SkipZip branch (the only caller always passed it), -ExpectedArchitectures, -TreatWarningsAsErrors, the rate-limit check, the redundant product-repo check, and a release-notes gate that structurally could not fire.

Separately, generate-release-notes.ps1 no longer echoes its output to stdout — that text is built from PR titles and model output, and Azure Pipelines executes any ##vso[...] line, so a crafted PR title could set pipeline variables.

AGENTS.md: the corp-machine test guidance was wrong

Unrelated to the pipeline but found while validating this branch. AGENTS.md called the local NuGet test failures "a known limitation, not a flaky test" — so I initially wrote off 69 failing tests as environmental.

They weren't. With the right configuration the full suite passes: 4621 tests, 0 failures.

The reason it was easy to get wrong is that two different mechanisms resolve packages:

• NugetService reads the WINAPP_NUGET_* variables AGENTS.md already documented → fixed 62.
• The end-to-end tests shell out to dotnet add package, which goes through the NuGet client, reads nuget.config, and ignores those variables entirely → the last 7. Those need VSS_NUGET_EXTERNAL_FEED_ENDPOINTS; VSS_NUGET_ACCESSTOKEN alone returns 401 on the service index.

A repo-local nuget.config can't help, because those tests run in %TEMP% outside the repo. AGENTS.md now documents a session-scoped fix (redirect APPDATA at a temp dir holding .pipelines/release-nuget.config) and explicitly says not to register a global package source.

Validation

• Structural audit of the parsed YAML confirms every publishing action is gated — zero ungated.
• The security reviewer's injection payload is refuted by the new guard.
• 4621 unit tests, 0 failures. 74 Pester tests pass.
• New tests are deliberately offline-only: build-cli.ps1 runs scripts/tests during a real release build, so a test reaching api.github.com would let a GitHub outage block a release.

Setup required before this does anything

No new pipeline is needed — the schedule runs on the existing WinDevCLI - Release definition, which already has its variable groups and connections authorized. Two things to check in the ADO UI, both documented in .pipelines/README.md:

1. "Override the YAML schedule" must be off, or the weekly trigger will not fire.
2. Any Branch control check on the shared service connections must allow both refs/heads/rel/v* and refs/heads/main — a main-only filter would block real releases.

Known limitations (documented, not papered over)

Signing is never exercised in a rehearsal, so an ESRP-side break still surfaces only during a real release. Service-connection checks prove authorization, not that the stored secret works. The rehearsal validates the publish steps' preconditions, not the final API calls.

Phase 2 — a scheduled agent prompt that reads each weekly run and posts a summary to the team — is intentionally a follow-up, to be written once there are real runs to summarize.

[Copilot]

Pull request overview

Adds a weekly, non-publishing rehearsal of the release pipeline.

Changes:

• Adds release-fidelity build and credential validation.
• Extracts shared pipeline environment setup.
• Adds asset-staging checks and release-script tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
.pipelines/dryrun.yml	Defines the weekly rehearsal pipeline.
.pipelines/templates/build-env.yaml	Centralizes agent and feed setup.
.pipelines/ci.yml	Uses shared environment setup.
.pipelines/release.yml	Uses shared setup and documents staging parity.
.pipelines/README.md	Documents operation, security, and setup.
AGENTS.md	Adds pipeline maintenance guidance.
scripts/check-release-credentials.ps1	Validates release credentials and connections.
scripts/stage-release-assets.ps1	Renames and verifies release assets.
scripts/generate-release-notes.ps1	Prevents untrusted notes reaching pipeline stdout.
scripts/tests/check-release-credentials.Tests.ps1	Tests offline credential-check behavior.
scripts/tests/stage-release-assets.Tests.ps1	Tests asset naming and verification.

Suppressed comments (1)

scripts/check-release-credentials.ps1:294

• The repository probe has the same false-failure path: any non-404 response, including a timeout or GitHub 5xx, is marked FAIL even though it does not prove the fork or token is unusable. Preserve 404 as the definitive failure for these public repositories and classify other request failures as WARN.

    if (-not $result.Ok) {
        if ($result.StatusCode -eq 404) {
            Add-Result -Status 'FAIL' -Check $Label -Detail "'$Repo' was not found, or the token cannot see it. If the fork was deleted, recreate it - wingetcreate and the docs PR both depend on it."
        }
        else {
            Add-Result -Status 'FAIL' -Check $Label -Detail "GET /repos/$Repo failed with status $($result.StatusCode)."
        }

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Build Metrics Report

Binary Sizes

Artifact	Baseline	Current	Delta
CLI (ARM64)	38.63 MB	38.63 MB	✅ 0.0 KB (0.00%)
CLI (x64)	38.75 MB	38.75 MB	✅ 0.0 KB (0.00%)
MSIX (ARM64)	16.03 MB	16.03 MB	📈 +0.1 KB (+0.00%)
MSIX (x64)	17.02 MB	17.02 MB	📈 +0.0 KB (+0.00%)
NPM Package	33.43 MB	33.43 MB	📉 -0.4 KB (-0.00%)
NuGet Package	33.47 MB	33.47 MB	📈 +0.1 KB (+0.00%)

Test Results

✅ 4619 passed, 5 skipped out of 4624 tests in 643.0s (+0.7s vs. baseline)

Test Coverage

✅ 89.1% line coverage, 82.5% branch coverage · ✅ no change vs. baseline

CLI Startup Time

51ms median (x64, winapp --version) · ✅ -6ms vs. baseline

Try This Build

Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 783

Switching between builds often?

Put the tool on your PATH once:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPath

Then this build is just:

winapp-pr 783

Run winapp-pr with no arguments to pick from a list of open PRs.

---

Updated 2026-09-01 21:33:17 UTC · commit 43cdf55 · workflow run

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Suppressed comments (1)

scripts/check-release-credentials.ps1:192

• A transient GitHub failure is classified as a definitive credential failure. For example, a simulated network outage produces status 0, adds [FAIL] GitHub token, and makes the final rehearsal stage fail, although the script’s contract says an inconclusive probe is WARN. Reserve FAIL for the explicit 401 branch and warn for other statuses so GitHub outages do not create false credential alarms.

            $hasRepoWrite = ($scopeList -contains 'repo') -or ($scopeList -contains 'public_repo')
            if (-not $hasRepoWrite) {
                Add-Result -Status 'FAIL' -Check 'GitHub token scopes' -Detail "Neither 'repo' nor 'public_repo' is granted. WinGet submission and the MS Learn docs PR both push branches and will fail."

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/verify-release-assets.ps1:82

• This does not verify the NuGet asset's expected name; any unversioned package passes. For example, a staging set containing only Wrong.Package.nupkg prints “All expected release assets are present,” so a PackageId/naming regression reaches the release stages undetected. Require exactly Microsoft.Windows.SDK.BuildTools.WinApp.nupkg and update the fixture to use that real contract name.

$nuget = @(Get-AssetNames -Sub 'nuget-packages' -Filter '*.nupkg')
if (-not $nuget) {
    $errors.Add('No .nupkg assets were produced.')
}

.pipelines/release.yml:813

• The rehearsal skips the fork sync above but still clones the fork, so it can validate a stale docs tree. A real release syncs first and then runs against current upstream content; clone $docsUpstream on the rehearsal path so Monday's run exercises the same input without mutating the fork.

                # Clone the fork
                Write-Host "Cloning docs repo fork..."
                $docsDir = "$tempDir\windows-dev-docs-pr"
                & $ghExe repo clone $docsFork $docsDir -- --depth 1

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.pipelines/release.yml:806

• The rehearsal skips syncing the fork but then clones that fork, so it validates whatever stale main the fork last had rather than this week's upstream docs. For example, an upstream change after the last real release is absent from Monday's rehearsal, yet the next real release syncs it and can fail. Keep the production sync, but clone $docsUpstream on the non-publishing path so the rehearsal uses current docs without mutating the fork.

                # Sync the fork with upstream. Skipped in a rehearsal: it mutates the fork, and
                # nothing gets pushed anyway. Push access is asserted by
                # check-release-credentials.ps1 in Build, which checks the permission rather than
                # just reachability.
                if ($env:MSLEARN_PUBLISH -eq 'true') {