diff --git a/.github/instructions/ado-pipeline.instructions.md b/.github/instructions/ado-pipeline.instructions.md index 5718e28098f..9616626ba33 100644 --- a/.github/instructions/ado-pipeline.instructions.md +++ b/.github/instructions/ado-pipeline.instructions.md @@ -35,11 +35,42 @@ Every ADO pipeline in this repo is split into **two YAML files**: `ob_*` job-scope variables and `LinuxContainerImage` still appear here (ADO requires them at job scope), but they are set from `${{ parameters.* }}`, so the raw author never has to know which OneBranch convention they satisfy. -File-pairing convention: a wrapper at `.github/workflows/ado/.yml` pairs with a raw stages template at `.github/workflows/ado/templates/-stages.yml`. **Multiple wrappers MAY share a single raw stages template** — that is in fact a primary motivation for the split: define several ADO pipelines (e.g. a DEV NonOfficial wrapper and a PROD Official wrapper, or per-environment variants) that all run the same stages/jobs/steps but differ in OneBranch variant, `featureFlags`, service connection, variable group, container image, etc. When wrappers share a raw template, name them so the relationship is obvious (e.g. `sources-upload-dev.yml` + `sources-upload-prod.yml` both pointing at `templates/sources-upload-stages.yml`). The variant choice cannot be hoisted into a shared sub-template because ADO requires `extends:` at the root of the entry pipeline — that is exactly why each wrapper exists. +File-pairing convention: a wrapper at `.github/workflows/ado/.yml` pairs with a raw stages template at `.github/workflows/ado/templates/-stages.yml`. **Multiple wrappers MAY share a single raw stages template** — that is in fact a primary motivation for the split: define several ADO pipelines (e.g. a DEV NonOfficial wrapper and a PROD Official wrapper, or per-environment variants) that all run the same stages/jobs/steps but differ in OneBranch variant, `featureFlags`, service connection, variable group, container image, etc. When wrappers share a raw template, name them so the relationship is obvious (e.g. `package-build-dev.yml` + `package-build-prod.yml` both pointing at `templates/package-build-stages.yml`). The variant choice cannot be hoisted into a shared sub-template because ADO requires `extends:` at the root of the entry pipeline — that is exactly why each wrapper exists. -See [.github/workflows/ado/sources-upload.yml](.github/workflows/ado/sources-upload.yml) and [.github/workflows/ado/templates/sources-upload-stages.yml](.github/workflows/ado/templates/sources-upload-stages.yml) for the canonical example. +See [.github/workflows/ado/package-build.yml](.github/workflows/ado/package-build.yml) and [.github/workflows/ado/templates/package-build-stages.yml](.github/workflows/ado/templates/package-build-stages.yml) for the canonical example. -Shared **step sub-templates** live under `.github/workflows/ado/templates/steps/.yml` and are spliced into a job's `steps:` via `- template: steps/.yml` (path relative to the including stages template). Use these to share step sequences across stages templates that differ only in a trailing pipeline-specific step. Splicing as **steps** (not a separate job/stage) keeps job-scoped pipeline variables and on-disk files flowing to the steps that follow — a separate job would force output variables + artifact upload/download. The including job must define any job-scope variables the shared steps reference (e.g. `ob_outputDirectory`). See [.github/workflows/ado/templates/steps/common-steps.yml](.github/workflows/ado/templates/steps/common-steps.yml), shared by the `package-build` and `sources-upload` pipelines. +Shared **step sub-templates** live under `.github/workflows/ado/templates/steps/.yml` and are spliced into a job's `steps:` via `- template: steps/.yml` (path relative to the including stages template). Use these to share step sequences across stages templates that differ only in a trailing pipeline-specific step. Splicing as **steps** (not a separate job/stage) keeps job-scoped pipeline variables and on-disk files flowing to the steps that follow — a separate job would force output variables + artifact upload/download. The including job must define any job-scope variables the shared steps reference (e.g. `ob_outputDirectory`). See the granular templates under [.github/workflows/ado/templates/steps/](.github/workflows/ado/templates/steps/) (e.g. `ensure-full-history.yml`, `install-deps.yml`, `prepare-change-set.yml`) and the composite `get-changes-info.yml`, shared by the `package-build` and `pr-check-ct` pipelines. + +**A template comment describes only what the template itself does** — not how a caller may or may not use it, and not where its parameter *values* originate. Assumptions about the caller (e.g. "passed by the wrapper", "owns the variable group", "the PR check does X") go stale and belong in the caller, not the shared template. The calling convention is documented here, once; do not repeat it in every YAML. + +**Sort lists alphabetically.** Within a file, keep `parameters`, `variables`, and step `env:` entries in alphabetical order by name (case-insensitive). This makes additions and diffs predictable and variables easy to find. Ordered sequences whose order is semantically load-bearing — job `steps:`, script lines — are NOT sorted. + +**Prefer a script file over `inlineScript` for single-command tasks.** A task whose body is one functionally-important command/script should reference a script file (e.g. `Bash@3` `filePath`, or `AzureCLI@2` `scriptLocation: scriptPath`) rather than embedding it inline. **Exception:** if `scriptPath` cannot be used without an artificial wrapper, keep `inlineScript`. (For example, an `AzureCLI@2` step whose body is a non-shell command such as `python3 some_script.py …` — run for its WIF az-login context — cannot use `scriptPath`, which runs the file *through* bash and ignores any `#!` line; adding a throwaway bash wrapper just to satisfy the rule is not worth it.) + +**Comment which variables a composed template sets, above the `- template:` line.** When a template you invoke sets job/output variables that *this* file consumes, list them above the invocation in this fixed format (omit an empty section; list only the variables the caller actually uses): + +```yaml + # Sets job variables: + # - + # - + # + # Sets output job variables: + # - + - template: steps/commit-range-pr.yml +``` + +**Parameterize the names of pipeline variables a template sets.** If a step template sets a job variable (`##vso[task.setvariable variable=...]`), expose `` through a parameter (with the conventional name as the default) so a caller composing several templates can rename it to avoid collisions. Use a consistent suffix scheme: + +- **Same-job variables** (default `##vso[task.setvariable]`, no `isoutput`): parameter `Var` (e.g. `sourceCommitVar`, `changedComponentsFileVar`). +- **Output variables** (`isoutput=true`, visible in other jobs/stages): parameter `OutputVar` for the variable name, AND parameter `OutputVarTask` for the setting task's `name:` (a cross-job output variable is referenced as `.`, so the task name is part of the contract). + +Reading another template's variable follows the same shape: take a `Var` parameter and reference it as `$(${{ parameters.Var }})` in `env:` (e.g. `prepare-change-set.yml`'s `fromCommitVar`/`toCommitVar`). + +**Pass pipeline parameters and variables into scripts through `env:`.** Do not inline `$(someVariable)` or `${{ parameters.X }}` inside an `inlineScript`/`script` body — map them in the step's `env:` block and reference the environment variable in the script. This keeps the script readable, avoids quoting/injection pitfalls, and makes the data flow explicit. This also covers values used to build a `##vso[...]` logging command: route the parameter through `env:` and reference `$ENV_VAR` in the `echo` rather than substituting `${{ parameters.X }}` inline. + +**Predefined variables: reference the auto-exposed env var directly; never re-map them.** ADO auto-exposes every non-secret predefined variable to scripts as an env var — uppercased with `.` → `_` (e.g. `Build.Reason` → `$BUILD_REASON`, `Build.ArtifactStagingDirectory` → `$BUILD_ARTIFACTSTAGINGDIRECTORY`, `Build.Repository.Uri` → `$BUILD_REPOSITORY_URI`, `System.DefinitionId` → `$SYSTEM_DEFINITIONID`). This is the default way to use a predefined variable: reference `$NAME` directly in the script and do NOT add it to `env:` — neither under its own name (the agent silently drops an `env:` entry that shadows a reserved variable) nor under a new alias (redundant, and it hides that the value is predefined). A short inline comment noting a value is auto-exposed prevents a maintainer from "completing" the `env:` block. **The one exception is a *secret* predefined variable** such as `System.AccessToken`, which is NOT auto-exposed and therefore MUST be mapped in `env:`. + +**Surface runtime-consumed parameters as `variables` when the template owns a `variables` section.** When a template defines its own `variables` section (i.e. it declares stages/jobs), convert the input parameters it consumes at runtime into entries in that `variables` section (`- name: foo` / `value: ${{ parameters.foo }}`) and reference them as `$(foo)`. A parameter value may be a runtime `$[ ]` (or `$(macro)`) expression, which is only evaluated inside the `variables` section — referencing `${{ parameters.foo }}` directly elsewhere would emit the literal, unevaluated string. Parameters used only in compile-time structural positions (e.g. `pool.type`, `timeoutInMinutes`, task `name:`/`azureSubscription:`) stay as parameters. **Caveat — do not create a self-referential cycle:** if a parameter's value is a `$(macro)` that references a variable whose name matches the surfaced variable's name (variable names are case-insensitive — e.g. a param passed `apiAudience: $(ApiAudience)` from a variable group), surfacing it as a same-named job variable makes it reference itself, and ADO reports *"Unable to expand variable … A cyclical reference was detected."* In that case reference the parameter directly (`${{ parameters.foo }}`) in `env:` instead, or surface it under a distinct name. ## OneBranch templates (MANDATORY — wrapper only) @@ -236,9 +267,16 @@ Use these as a starting point for a new pipeline. Two files: a wrapper (NonOffic ### Wrapper — `.github/workflows/ado/.yml` ```yaml +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + trigger: none pr: none +# Variables extracted from this group: ApiAudience, ApiBaseUrl. +variables: + - group: + resources: repositories: - repository: templates @@ -256,32 +294,38 @@ extends: stages: - template: /.github/workflows/ado/templates/-stages.yml@self parameters: - outputDirectory: $(Build.ArtifactStagingDirectory)/output + apiAudience: $(ApiAudience) + apiBaseUrl: $(ApiBaseUrl) artifactBaseName: containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0 + outputDirectory: $(Build.ArtifactStagingDirectory)/output poolType: linux serviceConnection: - variableGroup: timeoutInMinutes: # explicit, conservative ``` ### Raw stages — `.github/workflows/ado/templates/-stages.yml` ```yaml +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + parameters: - - name: outputDirectory + - name: apiAudience + type: string + - name: apiBaseUrl type: string - name: artifactBaseName type: string - name: containerImage type: string + - name: outputDirectory + type: string - name: poolType type: string default: linux - name: serviceConnection type: string - - name: variableGroup - type: string - name: timeoutInMinutes type: number @@ -293,13 +337,12 @@ stages: pool: type: ${{ parameters.poolType }} variables: - - group: ${{ parameters.variableGroup }} # audience URI, base URL, etc. - - name: ob_outputDirectory - value: ${{ parameters.outputDirectory }} - - name: ob_artifactBaseName - value: ${{ parameters.artifactBaseName }} - name: LinuxContainerImage value: ${{ parameters.containerImage }} + - name: ob_artifactBaseName + value: ${{ parameters.artifactBaseName }} + - name: ob_outputDirectory + value: ${{ parameters.outputDirectory }} steps: - task: PipAuthenticate@1 displayName: "Authenticate pip" @@ -318,8 +361,11 @@ stages: --api-audience "$API_AUDIENCE" \ --api-base-url "$API_BASE_URL" env: - API_AUDIENCE: $(ApiAudience) - API_BASE_URL: $(ApiBaseUrl) + # Referenced from the parameters directly: the wrapper passes + # $(ApiAudience) / $(ApiBaseUrl) from the group, so a same-named + # job variable would self-reference (cyclical reference). + API_AUDIENCE: ${{ parameters.apiAudience }} + API_BASE_URL: ${{ parameters.apiBaseUrl }} ``` Replace every `<...>` placeholder. diff --git a/.github/workflows/ado/package-build.yml b/.github/workflows/ado/package-build.yml index 1ab1c27a03f..6eda717a931 100644 --- a/.github/workflows/ado/package-build.yml +++ b/.github/workflows/ado/package-build.yml @@ -1,4 +1,5 @@ -# Microsoft Corporation +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. # # Wrapper pipeline — passed to ADO as the entry point for the package-build # pipeline. This file owns all OneBranch-specific wiring (governed templates @@ -7,15 +8,14 @@ # .github/workflows/ado/templates/package-build-stages.yml # # Authenticates via Workload Identity Federation (OIDC) and calls the Control -# Tower APIs to run the v1 post-merge delta build (stopgap until source upload -# is reworked): +# Tower APIs to run the v1 post-merge delta build: # 1. Resolve the (target, source) commit range for this push from the # previous CI build (ADO Builds API). # 2. Submit official package builds for the components that changed across # that range, via Workload Identity Federation (OIDC) to Control Tower. # -# The setup + change-detection + validation steps are shared with the -# source-upload pipeline via templates/steps/common-steps.yml. +# The setup + change-detection + validation steps are composed from the shared +# templates/steps/* step templates via templates/steps/get-changes-info.yml. # # Helper scripts live under: # - scripts/ci/control-tower/ - (Control Tower-specific) @@ -44,6 +44,10 @@ trigger: none pr: none +# Variables extracted from this group: ApiAudience, ApiBaseAFDUrl. +variables: + - group: ControlTower-PRCheck + resources: repositories: - repository: templates @@ -74,14 +78,15 @@ extends: stages: - template: /.github/workflows/ado/templates/package-build-stages.yml@self parameters: - outputDirectory: $(Build.ArtifactStagingDirectory)/output + apiAudience: $(ApiAudience) + apiBaseAFDUrl: $(ApiBaseAFDUrl) artifactBaseName: packagebuild containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0 - poolType: linux - serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV - variableGroup: ControlTower-PRCheck + outputDirectory: $(Build.ArtifactStagingDirectory)/output # Control Tower package target for the 4.0 branch. packageTarget: azl4 + poolType: linux + serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV # Must exceed the script's --poll-timeout-seconds (default 600s = 10m) # with enough headroom for setup steps and the final API call. timeoutInMinutes: 25 diff --git a/.github/workflows/ado/pr-check-ct.yml b/.github/workflows/ado/pr-check-ct.yml new file mode 100644 index 00000000000..e0e4a81b0cb --- /dev/null +++ b/.github/workflows/ado/pr-check-ct.yml @@ -0,0 +1,111 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Wrapper pipeline — passed to ADO as the entry point for the PR Control Tower +# check. It calls Control Tower 'prcheck' (which uploads the changed +# components' missing lookaside sources) and then submits a *scratch* Control +# Tower build of those components, WAITS for that build to finish, and fails +# the check if the build fails (or is rejected). The build runs in Control +# Tower's own sandbox — NO PR-controlled code is built on the CI agent (only +# read-only change detection runs there). +# +# This file owns all OneBranch-specific wiring (governed templates repo, +# NonOfficial variant, featureFlags) and delegates the actual stages/jobs/steps +# to the raw stages template at: +# .github/workflows/ado/templates/pr-check-ct-stages.yml +# +# WHY SCRATCH + REVIEWER-GATED: building unmerged PR code is only safe because +# (a) it is a *scratch* build (throwaway, never persisted to a production +# repo), so a malicious PR cannot poison published artifacts, and +# (b) the check is REVIEWER-GATED: it is wired in ADO so it does NOT auto-run +# on every PR push — a maintainer triggers it after eyeballing the diff, +# which bounds "unmerged code consumes build capacity / runs in CT" to an +# explicit human decision. See the reviewer-gate prerequisite below. +# run_package_build.py enforces the matching invariant: it refuses to submit an +# OFFICIAL (persisted) build for a PR trigger -- scratch is the default, so the +# PR check only ever produces throwaway builds. +# +# NonOfficial: this is PR validation. It calls the Control Tower DEV endpoint +# (via the DEV service connection) and produces only scratch builds; it is not +# a production-classified pipeline. +# +# Helper scripts live under: +# - scripts/ci/control-tower/ - (Control Tower client + submit script). +# - scripts/ci/components/ - cross-pipeline azldev change-set helpers (shared +# with the GitHub Actions PR gates and the Control Tower pipelines). +# +# Prerequisites (ADO / Azure Portal): +# 1. Entra ID App Registration with audience URI +# "api://" (see variable group below). +# 2. Federated identity credential on the app registration for the ADO +# service connection (issuer: https://vstoken.dev.azure.com/, +# subject: sc:////). +# 3. ARM service connection in ADO project settings using Workload Identity +# Federation (manual). +# 4. REVIEWER-GATED trigger (configured in ADO, not here): wire this pipeline +# as a build-validation check that does NOT automatically queue on every +# PR push (e.g. an optional/manual build-validation policy, or a manual +# approval check), so a maintainer must trigger it after reviewing the PR. +# +# Variable Group (ADO Pipelines > Library): +# Name: "ControlTower-PRCheck" +# Required variables: +# - ApiAudience : Entra ID audience URI for the Control Tower app +# - ApiBaseAFDUrl : Base URL of the Control Tower service (Azure Front Door endpoint) + +# Trigger controlled by ADO branch policy — not YAML triggers. +trigger: none + +pr: none + +# Variables extracted from this group: ApiAudience, ApiBaseAFDUrl. +variables: + - group: ControlTower-PRCheck + +resources: + repositories: + - repository: templates + type: git + name: OneBranch.Pipelines/GovernedTemplates + ref: refs/heads/main + +extends: + template: v2/OneBranch.NonOfficial.CrossPlat.yml@templates + parameters: + featureFlags: + golang: + internalModuleProxy: + enabled: true + LinuxHostVersion: + Network: R1 + runOnHost: true + EnableCDPxPAT: false + + # https://aka.ms/obpipelines/sdl + globalSdl: + disableLegacyManifest: true + sbom: + enabled: false + tsa: + enabled: false + + stages: + - template: /.github/workflows/ado/templates/pr-check-ct-stages.yml@self + parameters: + apiAudience: $(ApiAudience) + apiBaseAFDUrl: $(ApiBaseAFDUrl) + artifactBaseName: prcheckct + containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0 + outputDirectory: $(Build.ArtifactStagingDirectory)/output + # Control Tower package target for the 4.0 branch. + packageTarget: azl4 + # This check WAITS for the Control Tower build to finish (pass/fail), + # so the job must cover the full build. pollTimeoutSeconds caps how + # long run_package_build.py waits (21600 = 6h, our worst-case build); + # timeoutInMinutes sits above that plus setup headroom, so the + # script's own clear failure fires before ADO blunt-kills the job. + # Raise both together if a legitimate build is being killed. + pollTimeoutSeconds: 21600 + poolType: linux + serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV + timeoutInMinutes: 420 diff --git a/.github/workflows/ado/pr-package-build.yml b/.github/workflows/ado/pr-package-build.yml deleted file mode 100644 index f81f5a67110..00000000000 --- a/.github/workflows/ado/pr-package-build.yml +++ /dev/null @@ -1,104 +0,0 @@ -# Microsoft Corporation -# -# Wrapper pipeline — passed to ADO as the entry point for the PR package-build -# check. It submits a *scratch* Control Tower build of the components a pull -# request changes, WAITS for that build to finish, and fails the check if the -# build fails (or is rejected). The build runs in Control Tower's own sandbox — -# NO PR-controlled code is built on the CI agent (only read-only change -# detection runs there). -# -# This file owns all OneBranch-specific wiring (governed templates repo, -# NonOfficial variant, featureFlags) and delegates the actual stages/jobs/steps -# to the raw stages template at: -# .github/workflows/ado/templates/pr-package-build-stages.yml -# -# WHY SCRATCH + REVIEWER-GATED: building unmerged PR code is only safe because -# (a) it is a *scratch* build (throwaway, never persisted to a production -# repo), so a malicious PR cannot poison published artifacts, and -# (b) the check is REVIEWER-GATED: it is wired in ADO so it does NOT auto-run -# on every PR push — a maintainer triggers it after eyeballing the diff, -# which bounds "unmerged code consumes build capacity / runs in CT" to an -# explicit human decision. See the reviewer-gate prerequisite below. -# run_package_build.py enforces the matching invariant: it refuses to submit an -# OFFICIAL (persisted) build for a PR trigger -- scratch is the default, so the -# PR check only ever produces throwaway builds. -# -# NonOfficial: this is PR validation. It calls the Control Tower DEV endpoint -# (via the DEV service connection) and produces only scratch builds; it is not -# a production-classified pipeline. -# -# Helper scripts live under: -# - scripts/ci/control-tower/ - (Control Tower client + submit script). -# - scripts/ci/components/ - cross-pipeline azldev change-set helpers (shared -# with the GitHub Actions PR gates and the Control Tower pipelines). -# -# Prerequisites (ADO / Azure Portal): -# 1. Entra ID App Registration with audience URI -# "api://" (see variable group below). -# 2. Federated identity credential on the app registration for the ADO -# service connection (issuer: https://vstoken.dev.azure.com/, -# subject: sc:////). -# 3. ARM service connection in ADO project settings using Workload Identity -# Federation (manual). -# 4. REVIEWER-GATED trigger (configured in ADO, not here): wire this pipeline -# as a build-validation check that does NOT automatically queue on every -# PR push (e.g. an optional/manual build-validation policy, or a manual -# approval check), so a maintainer must trigger it after reviewing the PR. -# -# Variable Group (ADO Pipelines > Library): -# Name: "ControlTower-PRCheck" -# Required variables: -# - ApiAudience : Entra ID audience URI for the Control Tower app -# - ApiBaseAFDUrl : Base URL of the Control Tower service (Azure Front Door endpoint) - -# Trigger controlled by ADO branch policy — not YAML triggers. -trigger: none - -pr: none - -resources: - repositories: - - repository: templates - type: git - name: OneBranch.Pipelines/GovernedTemplates - ref: refs/heads/main - -extends: - template: v2/OneBranch.NonOfficial.CrossPlat.yml@templates - parameters: - featureFlags: - golang: - internalModuleProxy: - enabled: true - LinuxHostVersion: - Network: R1 - runOnHost: true - EnableCDPxPAT: false - - # https://aka.ms/obpipelines/sdl - globalSdl: - disableLegacyManifest: true - sbom: - enabled: false - tsa: - enabled: false - - stages: - - template: /.github/workflows/ado/templates/pr-package-build-stages.yml@self - parameters: - outputDirectory: $(Build.ArtifactStagingDirectory)/output - artifactBaseName: prpackagebuild - containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0 - poolType: linux - serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV - variableGroup: ControlTower-PRCheck - # Control Tower package target for the 4.0 branch. - packageTarget: azl4 - # This check WAITS for the Control Tower build to finish (pass/fail), - # so the job must cover the full build. pollTimeoutSeconds caps how - # long run_package_build.py waits (21600 = 6h, our worst-case build); - # timeoutInMinutes sits above that plus setup headroom, so the - # script's own clear failure fires before ADO blunt-kills the job. - # Raise both together if a legitimate build is being killed. - pollTimeoutSeconds: 21600 - timeoutInMinutes: 420 diff --git a/.github/workflows/ado/pr-package-build.yml b/.github/workflows/ado/pr-package-build.yml new file mode 120000 index 00000000000..7c6fe979da6 --- /dev/null +++ b/.github/workflows/ado/pr-package-build.yml @@ -0,0 +1 @@ +pr-check-ct.yml \ No newline at end of file diff --git a/.github/workflows/ado/sources-upload.yml b/.github/workflows/ado/sources-upload.yml deleted file mode 100644 index 1c3fc1c1849..00000000000 --- a/.github/workflows/ado/sources-upload.yml +++ /dev/null @@ -1,82 +0,0 @@ -# Microsoft Corporation -# -# Wrapper pipeline — passed to ADO as the entry point for the source-upload -# pipeline. This file owns all OneBranch-specific wiring (governed templates -# repo, Official vs NonOfficial variant, featureFlags) and delegates the actual -# stages/jobs/steps to the raw stages template at: -# .github/workflows/ado/templates/sources-upload-stages.yml -# -# Authenticates via Workload Identity Federation (OIDC) and calls the Control -# Tower 'prcheck' API to upload changed component sources. prcheck returns early -# (no upload) on pull-request triggers -- unmerged code should not consume -# capacity. -# -# The setup + change-detection + validation steps are shared with the -# package-build pipeline via templates/steps/common-steps.yml. -# -# Helper scripts live under: -# - scripts/ci/control-tower/ - (Control Tower-specific) -# - scripts/ci/ado/ - (ADO API: commit-range detection) -# - scripts/ci/components/ - (cross-pipeline azldev helpers shared with the GH Actions PR gates). -# -# Prerequisites (ADO / Azure Portal): -# 1. Entra ID App Registration with audience URI -# "api://" (see variable group below). -# 2. Federated identity credential on the app registration for the ADO -# service connection (issuer: https://vstoken.dev.azure.com/, -# subject: sc:////). -# 3. ARM service connection in ADO project settings using Workload Identity -# Federation (manual). -# 4. CI trigger configured (in ADO pipeline settings) to fire on pushes to -# the target branch. -# -# Variable Group (ADO Pipelines > Library): -# Name: "ControlTower-PRCheck" -# Required variables: -# - ApiAudience : Entra ID audience URI for the Control Tower app -# - ApiBaseAFDUrl : Base URL of the Control Tower service (Azure Front Door endpoint) - -# Trigger controlled by ADO branch policy — not YAML triggers. -trigger: none - -pr: none - -resources: - repositories: - - repository: templates - type: git - name: OneBranch.Pipelines/GovernedTemplates - ref: refs/heads/main - -extends: - template: v2/OneBranch.Official.CrossPlat.yml@templates - parameters: - featureFlags: - golang: - internalModuleProxy: - enabled: true - LinuxHostVersion: - Network: R1 - runOnHost: true - EnableCDPxPAT: false - - # https://aka.ms/obpipelines/sdl - globalSdl: - disableLegacyManifest: true - sbom: - enabled: false - tsa: - enabled: false - - stages: - - template: /.github/workflows/ado/templates/sources-upload-stages.yml@self - parameters: - outputDirectory: $(Build.ArtifactStagingDirectory)/output - artifactBaseName: sourceupload - containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0 - poolType: linux - serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV - variableGroup: ControlTower-PRCheck - # Must exceed the script's --poll-timeout-seconds (default 7200s = 120m) - # with enough headroom for setup steps and the final API call. - timeoutInMinutes: 150 diff --git a/.github/workflows/ado/templates/package-build-stages.yml b/.github/workflows/ado/templates/package-build-stages.yml index a3cfc9ce8a2..85403b001ca 100644 --- a/.github/workflows/ado/templates/package-build-stages.yml +++ b/.github/workflows/ado/templates/package-build-stages.yml @@ -1,65 +1,70 @@ -# Microsoft Corporation +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. # # Raw stages template for the package-build pipeline (v1 post-merge delta # build). OneBranch-agnostic: declares the stages/jobs/steps and exposes the # OneBranch-coupled knobs as parameters. The wrapper at # .github/workflows/ado/package-build.yml supplies concrete values. # -# The setup + change-detection + validation steps are shared with the -# source-upload pipeline via templates/steps/common-steps.yml; this template -# appends only the package-build-specific Control Tower call. +# Per push: gather the change-set info (get-changes-info, mode=postmerge: full +# history, deps, commit range, lock + render verification, changed-component +# set), then submit an official Control Tower build for the changed components. parameters: - - name: outputDirectory + - name: apiAudience + type: string + - name: apiBaseAFDUrl type: string - name: artifactBaseName type: string - name: containerImage type: string + - name: outputDirectory + type: string + # Control Tower package target for builds submitted from this pipeline + # (e.g. azl4 for the 4.0 branch, azl5 for 5.0). + - name: packageTarget + type: string - name: poolType type: string default: linux - name: serviceConnection type: string - - name: variableGroup - type: string - name: timeoutInMinutes type: number - # Control Tower package target for builds submitted from this pipeline - # (e.g. azl4 for the 4.0 branch, azl5 for 5.0). Bound per-branch by the - # wrapper so a branch's builds land in the correct target. - - name: packageTarget - type: string stages: - stage: PackageBuild jobs: - job: PackageBuild - # This is a post-merge build: code has already merged, so there is no PR - # or merge queue to gate. The job fails loud -- any failing step turns - # the run red so submission/validation breakage is visible and alertable - # rather than masked as a partial success. The build itself runs - # asynchronously in Control Tower/Koji; this pipeline only submits it and - # confirms acceptance (see run_package_build.py). - # Must exceed the script's --poll-timeout-seconds (default 600s = 10m) - # with enough headroom for setup steps and the final API call. + # Post-merge build: code has already merged, so there is no PR or merge + # queue to gate. Fails loud -- any failing step turns the run red so + # submission/validation breakage is visible rather than masked. The + # build runs asynchronously in Control Tower/Koji; this pipeline only + # submits it and confirms acceptance (see run_package_build.py). timeoutInMinutes: ${{ parameters.timeoutInMinutes }} pool: type: ${{ parameters.poolType }} variables: - - group: ${{ parameters.variableGroup }} - - name: ob_outputDirectory - value: ${{ parameters.outputDirectory }} - - name: ob_artifactBaseName - value: ${{ parameters.artifactBaseName }} - name: LinuxContainerImage value: ${{ parameters.containerImage }} + - name: ob_artifactBaseName + value: ${{ parameters.artifactBaseName }} + - name: ob_outputDirectory + value: ${{ parameters.outputDirectory }} + - name: packageTarget + value: ${{ parameters.packageTarget }} steps: - # Shared setup + change detection + validation (PipAuthenticate, - # install deps, determine commit range, verify locks, prepare change - # set, verify rendered specs). Produces the job-scope variables and - # change-set files the submit step below consumes. - - template: steps/common-steps.yml + # Sets job variables: + # - changedComponentsFile + # - sourceCommit + - template: steps/get-changes-info.yml + parameters: + mode: postmerge + + - template: steps/install-deps.yml + parameters: + installControlTowerClient: true - task: AzureCLI@2 displayName: "Submit package build to Control Tower" @@ -73,17 +78,24 @@ stages: python3 scripts/ci/control-tower/run_package_build.py \ --api-audience "$API_AUDIENCE" \ --api-base-url "$API_BASE_URL" \ - --build-reason "$CT_BUILD_REASON" \ + --build-reason "$BUILD_REASON" \ --changed-components-file "$CHANGED_COMPONENTS_FILE" \ - --package-target "${{ parameters.packageTarget }}" \ + --package-target "$PACKAGE_TARGET" \ --official-build \ --commit-sha "$SOURCE_COMMIT" \ - --repo-uri "$UPSTREAM_REPO_URL" + --repo-uri "$BUILD_REPOSITORY_URI" env: - API_AUDIENCE: $(ApiAudience) - API_BASE_URL: $(ApiBaseAFDUrl) - # Non-reserved name: an `env:` override of the reserved BUILD_REASON var is silently ignored by the agent. - CT_BUILD_REASON: $(Build.Reason) + # apiAudience / apiBaseAFDUrl come from the parameters directly + # (not surfaced as same-named job variables): the wrapper passes + # $(ApiAudience) / $(ApiBaseAFDUrl) from the variable group, and a + # job variable of the same (case-insensitive) name would + # self-reference -> "cyclical reference" and never resolve. + API_AUDIENCE: ${{ parameters.apiAudience }} + API_BASE_URL: ${{ parameters.apiBaseAFDUrl }} CHANGED_COMPONENTS_FILE: $(changedComponentsFile) + PACKAGE_TARGET: $(packageTarget) SOURCE_COMMIT: $(sourceCommit) - UPSTREAM_REPO_URL: $(Build.Repository.Uri) + # Build.Reason and Build.Repository.Uri are predefined and + # auto-exposed as $BUILD_REASON / $BUILD_REPOSITORY_URI; reference + # them directly and do not map them here (env: overrides of + # reserved variables are silently ignored by the agent). diff --git a/.github/workflows/ado/templates/pr-check-ct-stages.yml b/.github/workflows/ado/templates/pr-check-ct-stages.yml new file mode 100644 index 00000000000..9f1b478daae --- /dev/null +++ b/.github/workflows/ado/templates/pr-check-ct-stages.yml @@ -0,0 +1,162 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Raw stages template for the PR Control Tower check (prcheck + scratch package +# build). Wrapper-agnostic: declares the stages/jobs/steps and exposes the +# wrapper-coupled knobs as parameters. The wrapper at +# .github/workflows/ado/pr-check-ct.yml supplies concrete values. +# +# Per PR: gather the change-set info (get-changes-info, mode=pr: full history, +# deps, commit range, lock + render verification, changed-component set), then +# call Control Tower 'prcheck' (uploads the changed components' missing +# lookaside sources) and submit a *scratch* Control Tower build of the PR head +# for those components, waiting for it to finish. NO PR-controlled code is built +# on the CI agent -- only read-only change detection runs here. + +parameters: + - name: apiAudience + type: string + - name: apiBaseAFDUrl + type: string + - name: artifactBaseName + type: string + - name: containerImage + type: string + - name: outputDirectory + type: string + # Control Tower package target for builds submitted from this pipeline + # (e.g. azl4 for the 4.0 branch, azl5 for 5.0). + - name: packageTarget + type: string + # Max seconds run_package_build.py waits for the Control Tower build to reach + # a terminal state. Keep below the job's timeoutInMinutes so the script's own + # clear failure fires before ADO blunt-kills the job. Default 21600 = 6h. + - name: pollTimeoutSeconds + type: number + default: 21600 + - name: poolType + type: string + default: linux + - name: serviceConnection + type: string + - name: timeoutInMinutes + type: number + +stages: + - stage: PRCheckCT + jobs: + - job: PRCheckCT + # Fail-loud: a failed submission, an immediate Control Tower rejection, + # or a build that fails (or never reaches a terminal state) turns the PR + # check red. The build runs in Control Tower's own sandbox -- NOT on this + # agent -- but this pipeline WAITS for it to finish. Size the timeout to + # cover the FULL build: it must exceed pollTimeoutSeconds (6h default). + timeoutInMinutes: ${{ parameters.timeoutInMinutes }} + pool: + type: ${{ parameters.poolType }} + variables: + - name: LinuxContainerImage + value: ${{ parameters.containerImage }} + - name: ob_artifactBaseName + value: ${{ parameters.artifactBaseName }} + - name: ob_outputDirectory + value: ${{ parameters.outputDirectory }} + - name: packageTarget + value: ${{ parameters.packageTarget }} + - name: pollTimeoutSeconds + value: ${{ parameters.pollTimeoutSeconds }} + steps: + # Sets job variables: + # - baseCommit + # - changedComponentsFile + # - sourceCommit + - template: steps/get-changes-info.yml + parameters: + mode: pr + + - template: steps/install-deps.yml + parameters: + installControlTowerClient: true + + # Call Control Tower 'prcheck' BEFORE the build. Among other things, + # prcheck uploads the missing lookaside sources for the changed + # components, so they are available when the scratch build below + # fetches them. + - task: AzureCLI@2 + displayName: "Call Control Tower 'prcheck' API" + inputs: + azureSubscription: ${{ parameters.serviceConnection }} + scriptType: bash + scriptLocation: inlineScript + inlineScript: | + set -euo pipefail + + python3 scripts/ci/control-tower/run_prcheck.py \ + --api-audience "$API_AUDIENCE" \ + --api-base-url "$API_BASE_URL" \ + --build-reason "$BUILD_REASON" \ + --changed-components-file "$CHANGED_COMPONENTS_FILE" \ + --source-commit "$SOURCE_COMMIT" \ + --repo-uri "$BUILD_REPOSITORY_URI" + env: + # apiAudience / apiBaseAFDUrl are referenced from the parameters + # directly (NOT surfaced as same-named job variables): the wrapper + # passes $(ApiAudience) / $(ApiBaseAFDUrl) from the variable group, + # and a job variable of the same (case-insensitive) name would + # self-reference -> "cyclical reference" and never resolve. + API_AUDIENCE: ${{ parameters.apiAudience }} + API_BASE_URL: ${{ parameters.apiBaseAFDUrl }} + # TODO: Base commit is not used yet. Will be needed once we move + # detection of affected components to CT. ADO task: 18816 + BASE_COMMIT: $(baseCommit) + CHANGED_COMPONENTS_FILE: $(changedComponentsFile) + SOURCE_COMMIT: $(sourceCommit) + # Build.Reason and Build.Repository.Uri are predefined and + # auto-exposed as $BUILD_REASON / $BUILD_REPOSITORY_URI; reference + # them directly and do not map them here (env: overrides of + # reserved variables are silently ignored by the agent). + + # Submit a SCRATCH Control Tower build of the PR head for the changed + # components. Scratch = throwaway: it never persists to a production + # repo, so building unmerged PR code is safe. Scratch is the default + # (no --official-build); run_package_build.py additionally refuses an + # OFFICIAL build for a PR trigger. --wait-for-completion blocks until + # the build reaches a terminal state and fails the check on a build + # failure (or on poll-timeout). No PR code is built on this agent. + # + # This step assumes the pipeline is wired as a REVIEWER-GATED check in + # ADO: it should not auto-run on every PR push, so a maintainer + # eyeballs the diff before unmerged code is submitted for a build. + - task: AzureCLI@2 + displayName: "Submit scratch build to Control Tower" + inputs: + azureSubscription: ${{ parameters.serviceConnection }} + scriptType: bash + scriptLocation: inlineScript + inlineScript: | + set -euo pipefail + + python3 scripts/ci/control-tower/run_package_build.py \ + --api-audience "$API_AUDIENCE" \ + --api-base-url "$API_BASE_URL" \ + --build-reason "$BUILD_REASON" \ + --changed-components-file "$CHANGED_COMPONENTS_FILE" \ + --package-target "$PACKAGE_TARGET" \ + --commit-sha "$SOURCE_COMMIT" \ + --repo-uri "$BUILD_REPOSITORY_URI" \ + --wait-for-completion \ + --poll-timeout-seconds "$POLL_TIMEOUT_SECONDS" + env: + # See the prcheck step: apiAudience / apiBaseAFDUrl come from the + # parameters directly to avoid a cyclical self-reference with the + # same-named variable-group variables. + API_AUDIENCE: ${{ parameters.apiAudience }} + API_BASE_URL: ${{ parameters.apiBaseAFDUrl }} + CHANGED_COMPONENTS_FILE: $(changedComponentsFile) + PACKAGE_TARGET: $(packageTarget) + POLL_TIMEOUT_SECONDS: $(pollTimeoutSeconds) + SOURCE_COMMIT: $(sourceCommit) + # Build.Reason and Build.Repository.Uri are predefined and + # auto-exposed as $BUILD_REASON / $BUILD_REPOSITORY_URI; reference + # them directly and do not map them here (env: overrides of + # reserved variables are silently ignored by the agent). diff --git a/.github/workflows/ado/templates/pr-package-build-stages.yml b/.github/workflows/ado/templates/pr-package-build-stages.yml deleted file mode 100644 index 1cc26e256fd..00000000000 --- a/.github/workflows/ado/templates/pr-package-build-stages.yml +++ /dev/null @@ -1,255 +0,0 @@ -# Microsoft Corporation -# -# Raw stages template for the PR package-build check. Wrapper-agnostic: declares -# the stages/jobs/steps and exposes the wrapper-coupled knobs as parameters. The -# wrapper at .github/workflows/ado/pr-package-build.yml supplies concrete -# values. See that wrapper for why this pipeline exists. -# -# What it does, per PR: -# 1. Ensure full git history (rpmautospec + change detection need it). -# 2. Authenticate to the internal pip feed and install host deps: azldev (for -# change detection only -- no mock, no build) and the Control Tower Python -# client. -# 3. Resolve the PR commit range from the merge commit's parents -# (^1 = target-branch tip, ^2 = PR head). -# 4. Compute the changed-component set (shared compute_change_set.sh). -# 5. Submit a *scratch* Control Tower build of the PR head for exactly those -# components (run_package_build.py --wait-for-completion). The build runs -# in Control Tower's own sandbox; this pipeline WAITS for it to reach a -# terminal state and fails the check if the build fails (or does not -# finish within the poll timeout). NO PR-controlled code is built on the -# CI agent -- only read-only change detection runs here. -# -# It deliberately does NOT reuse templates/steps/common-steps.yml: that shared -# step set resolves the commit range via the *previous CI build* (the -# post-merge delta logic), which is wrong for a PR. A PR range comes from the -# merge commit's parents, computed inline below. Reusing common-steps would -# also pull in the lock/render verify steps (already covered by the GitHub -# Actions PR gates) and force a refactor of a file shared by two production -# pipelines. -# -# Because it calls Control Tower, this pipeline needs the WIF service connection -# and the Control Tower variable group (audience + base URL); the wrapper -# supplies both as parameters. - -parameters: - - name: outputDirectory - type: string - - name: artifactBaseName - type: string - - name: containerImage - type: string - - name: poolType - type: string - default: linux - - name: serviceConnection - type: string - - name: variableGroup - type: string - # Control Tower package target for builds submitted from this pipeline - # (e.g. azl4 for the 4.0 branch, azl5 for 5.0). Bound per-branch by the - # wrapper so a branch's builds land in the correct target. - - name: packageTarget - type: string - - name: timeoutInMinutes - type: number - # Max seconds run_package_build.py waits for the Control Tower build to reach - # a terminal state. Keep below the job's timeoutInMinutes (above) so the - # script's own clear failure fires before ADO blunt-kills the job. Default - # 21600 = 6h (our worst-case build); the wrapper passes it alongside - # timeoutInMinutes so the two are raised together. - - name: pollTimeoutSeconds - type: number - default: 21600 - -stages: - - stage: PRPackageBuild - jobs: - - job: PRPackageBuild - # Fail-loud: a failed submission, an immediate Control Tower rejection, - # or a build that fails (or never reaches a terminal state) turns the PR - # check red. The build runs in Control Tower's own sandbox -- NOT on - # this agent -- but this pipeline WAITS for it to finish - # (run_package_build.py --wait-for-completion). Size the timeout to - # cover the FULL build: it must exceed the script's pollTimeoutSeconds - # (6h default) so the script's own clear failure fires before ADO - # blunt-kills the job. - timeoutInMinutes: ${{ parameters.timeoutInMinutes }} - pool: - type: ${{ parameters.poolType }} - variables: - - group: ${{ parameters.variableGroup }} - - name: ob_outputDirectory - value: ${{ parameters.outputDirectory }} - - name: ob_artifactBaseName - value: ${{ parameters.artifactBaseName }} - - name: LinuxContainerImage - value: ${{ parameters.containerImage }} - steps: - # Full history: `azldev component changed` tree-diffs two commits and - # rpmautospec derives Release/changelog from `git log`. The CI - # checkout may be shallow (depth 1); unshallow once, up front. Never - # `git fetch --depth=N` afterwards — that re-shallows a full clone and - # silently corrupts the rpmautospec Release calculation. - - script: | - set -euo pipefail - if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then - echo "##[group]Fetching full git history" - git fetch --unshallow - echo "##[endgroup]" - fi - displayName: "Ensure full git history" - - - task: PipAuthenticate@1 - displayName: "Authenticate pip" - inputs: - artifactFeeds: "azl/ControlTowerFeed" - - # azldev opens the repo with go-git, which rejects a config that - # declares the `worktreeconfig` extension while - # core.repositoryformatversion is still 0: - # "core.repositoryformatversion does not support extension: worktreeconfig" - # Native git tolerates this, and the ADO agent checkout leaves the - # extension set, so strip it before any azldev invocation. Each CI run - # is a fresh checkout so this is safe and self-contained. - # TODO: remove this step once azldev no longer needs the workaround - # (go-git v6 fixes the underlying bug): - # https://github.com/microsoft/azure-linux-dev-tools/issues/241 - - script: | - set -euo pipefail - if git config --get extensions.worktreeConfig >/dev/null 2>&1; then - echo "Removing extensions.worktreeConfig so go-git (azldev) can open the repo" - git config --unset-all extensions.worktreeConfig || true - fi - displayName: "Normalize git config for azldev (go-git)" - - # Host deps for change detection + the Control Tower submission only: - # azldev (`azldev component changed` + git diff -- no mock, no build) - # and the Control Tower Python client. The build itself never runs on - # the agent; it runs asynchronously in Control Tower's own sandbox. - - script: | - set -euo pipefail - echo "##[group]Azldev (host, for change-set)" - # Only the version string comes from the PR checkout; reject a - # malformed/garbage value before it reaches `go install`. - AZLDEV_VERSION="$(tr -d '\n' < .azldev-version)" - if ! printf '%s' "$AZLDEV_VERSION" | grep -Eq '^[0-9A-Za-z._+-]+$'; then - echo "##[error].azldev-version is empty or has unexpected characters" - exit 1 - fi - echo "Installing azldev@${AZLDEV_VERSION}..." - go install "github.com/microsoft/azure-linux-dev-tools/cmd/azldev@${AZLDEV_VERSION}" - - go_bin_path="$(go env GOPATH)/bin" - echo "##vso[task.prependpath]$go_bin_path" - - "$go_bin_path/azldev" --version - echo "##[endgroup]" - - echo "##[group]Python dependencies (Control Tower client)" - pip install -r scripts/ci/control-tower/requirements.txt - echo "##[endgroup]" - displayName: "Install host dependencies" - - # Resolve the PR commit range. A PR-policy build checks out the MERGE - # commit (Build.SourceVersion): parent ^1 is the target-branch tip, - # parent ^2 is the PR head. The PR's true change set is the diff - # between their MERGE BASE (the fork point) and ^2 -- i.e. - # `git diff merge-base(^1,^2)..^2`, the same three-dot set GitHub's - # "Files changed" shows. We deliberately compare against the merge - # base, NOT the target tip: comparing against the tip additionally - # flags every component that landed on the target branch after the PR - # forked (the PR head still carries their pre-update state), which can - # balloon a 1-component PR into dozens. We read the range here and set - # pipeline variables so the wiring stays visible in the YAML. - - script: | - set -euo pipefail - if ! git rev-parse --verify -q "HEAD^2" >/dev/null; then - echo "##[error]HEAD is not a merge commit -- this pipeline must run as a PR build (Build.Reason=PullRequest)." - exit 1 - fi - source_commit="$(git rev-parse HEAD^2)" - # Baseline = the MERGE BASE (fork point) of the target-branch tip - # (HEAD^1) and the PR head (HEAD^2), NOT the target tip itself. - # See the block comment above for why; this is passed as the - # `--target-commit` (from) baseline to the change-set helper. - target_commit="$(git merge-base HEAD^1 HEAD^2)" - # PR-supplied data is untrusted: validate both SHAs before use. - for sha in "$target_commit" "$source_commit"; do - if [[ ! "$sha" =~ ^[0-9a-f]{40}$ ]]; then - echo "##[error]invalid commit SHA: $sha" - exit 1 - fi - done - echo "Resolved range: base(merge-base)=$target_commit source=$source_commit" - echo "##vso[task.setvariable variable=sourceCommit;isreadonly=true]$source_commit" - echo "##vso[task.setvariable variable=targetCommit;isreadonly=true]$target_commit" - displayName: "Determine PR commit range" - - # Compute the changed-component set with the shared, cross-pipeline - # single-source-of-truth helper (also used by the GitHub Actions PR - # gates). changed-components.json holds the per-component change - # records consumed by the Control Tower submit step below. - # compute_change_set.sh hard-fails on the supply-chain drift tripwire - # (sourcesChange without an identity change) -- a guard we want to keep - # on PRs. The script self-prefixes AZLDEV_ALLOW_ROOT=1 internally. - - script: | - set -euo pipefail - change_set_dir="$(Build.ArtifactStagingDirectory)/change-set" - echo "##[group]Preparing change set" - scripts/ci/components/compute_change_set.sh \ - --output-dir "$change_set_dir" \ - --source-commit "$SOURCE_COMMIT" \ - --target-commit "$TARGET_COMMIT" - echo "##[endgroup]" - echo "##vso[task.setvariable variable=changedComponentsFile;isreadonly=true]$change_set_dir/changed-components.json" - env: - SOURCE_COMMIT: $(sourceCommit) - TARGET_COMMIT: $(targetCommit) - displayName: "Prepare change set" - - # Submit a SCRATCH Control Tower build of the PR head for the changed - # components. Scratch = throwaway: it never persists to a production - # repo, so building unmerged PR code is safe. Scratch is the default - # (no --official-build); run_package_build.py additionally refuses an - # OFFICIAL build for a PR trigger. --wait-for-completion makes the - # script block until the build reaches a terminal state and fail the - # check on a build failure (or if it does not finish within - # --poll-timeout-seconds, 6h below -- our worst-case build). No PR - # code is built on this agent. - # - # This step assumes the pipeline is wired as a REVIEWER-GATED check in - # ADO (see the wrapper header): it should not auto-run on every PR - # push, so that a maintainer eyeballs the diff before unmerged code is - # submitted for a build. - - task: AzureCLI@2 - displayName: "Submit scratch build to Control Tower" - inputs: - azureSubscription: ${{ parameters.serviceConnection }} - scriptType: bash - scriptLocation: inlineScript - inlineScript: | - set -euo pipefail - - # --poll-timeout-seconds comes from the pollTimeoutSeconds - # parameter (6h default = our worst-case build). Keep it below - # the job's timeoutInMinutes (wrapper) so the script's own clear - # failure fires before ADO blunt-kills the job. - python3 scripts/ci/control-tower/run_package_build.py \ - --api-audience "$API_AUDIENCE" \ - --api-base-url "$API_BASE_URL" \ - --build-reason "$CT_BUILD_REASON" \ - --changed-components-file "$CHANGED_COMPONENTS_FILE" \ - --package-target "${{ parameters.packageTarget }}" \ - --commit-sha "$SOURCE_COMMIT" \ - --repo-uri "$UPSTREAM_REPO_URL" \ - --wait-for-completion \ - --poll-timeout-seconds ${{ parameters.pollTimeoutSeconds }} - env: - API_AUDIENCE: $(ApiAudience) - API_BASE_URL: $(ApiBaseAFDUrl) - # Non-reserved name: an `env:` override of the reserved BUILD_REASON var is silently ignored by the agent. - CT_BUILD_REASON: $(Build.Reason) - CHANGED_COMPONENTS_FILE: $(changedComponentsFile) - SOURCE_COMMIT: $(sourceCommit) - UPSTREAM_REPO_URL: $(Build.Repository.Uri) diff --git a/.github/workflows/ado/templates/sources-upload-stages.yml b/.github/workflows/ado/templates/sources-upload-stages.yml deleted file mode 100644 index 501f5aff297..00000000000 --- a/.github/workflows/ado/templates/sources-upload-stages.yml +++ /dev/null @@ -1,93 +0,0 @@ -# Microsoft Corporation -# -# Raw stages template for the source-upload pipeline (Control Tower prcheck). -# OneBranch-agnostic: declares the stages/jobs/steps and exposes the -# OneBranch-coupled knobs as parameters. The wrapper at -# .github/workflows/ado/sources-upload.yml supplies concrete values. -# -# The setup + change-detection + validation steps are shared with the -# package-build pipeline via templates/steps/common-steps.yml; this template -# appends only the source-upload-specific Control Tower call (prcheck). -# -# NOTE: source upload is the original Control Tower flow. It runs prcheck, which -# returns early (no upload) on pull-request triggers -- unmerged code should not -# consume capacity. This pipeline is retained for when source upload returns to -# active duty; the v1 stopgap runs builds via package-build.yml instead. - -parameters: - - name: outputDirectory - type: string - - name: artifactBaseName - type: string - - name: containerImage - type: string - - name: poolType - type: string - default: linux - - name: serviceConnection - type: string - - name: variableGroup - type: string - - name: timeoutInMinutes - type: number - -stages: - - stage: SourceUpload - jobs: - - job: SourceUpload - # Non-blocking PR check: failing steps still render red error annotations - # and surface in the build-issues view, but the job resolves to - # SucceededWithIssues and the run to PartiallySucceeded, which the Azure - # Pipelines GitHub integration reports as success to GitHub PR checks and - # the merge queue. Errors and warnings remain visible to the user. - # TODO: Remove `continueOnError` once the pipeline is stable so failures - # block PRs and the merge queue. - # ADO task: 19179 - continueOnError: true - # Must exceed the script's --poll-timeout-seconds (run_prcheck.py default - # 7200s = 120m) with enough headroom for setup steps and the final API call. - timeoutInMinutes: ${{ parameters.timeoutInMinutes }} - pool: - type: ${{ parameters.poolType }} - variables: - - group: ${{ parameters.variableGroup }} - - name: ob_outputDirectory - value: ${{ parameters.outputDirectory }} - - name: ob_artifactBaseName - value: ${{ parameters.artifactBaseName }} - - name: LinuxContainerImage - value: ${{ parameters.containerImage }} - steps: - # Shared setup + change detection + validation (PipAuthenticate, - # install deps, determine commit range, verify locks, prepare change - # set, verify rendered specs). Produces the job-scope variables and - # change-set files the prcheck step below consumes. - - template: steps/common-steps.yml - - - task: AzureCLI@2 - displayName: "Call Control Tower 'prcheck' API" - inputs: - azureSubscription: ${{ parameters.serviceConnection }} - scriptType: bash - scriptLocation: inlineScript - inlineScript: | - set -euo pipefail - - python3 scripts/ci/control-tower/run_prcheck.py \ - --api-audience "$API_AUDIENCE" \ - --api-base-url "$API_BASE_URL" \ - --build-reason "$CT_BUILD_REASON" \ - --changed-components-file "$CHANGED_COMPONENTS_FILE" \ - --source-commit "$SOURCE_COMMIT" \ - --repo-uri "$UPSTREAM_REPO_URL" - env: - API_AUDIENCE: $(ApiAudience) - API_BASE_URL: $(ApiBaseAFDUrl) - # Non-reserved name: an `env:` override of the reserved BUILD_REASON var is silently ignored by the agent. - CT_BUILD_REASON: $(Build.Reason) - CHANGED_COMPONENTS_FILE: $(changedComponentsFile) - SOURCE_COMMIT: $(sourceCommit) - # TODO: Target commit is not used. Will be needed once we move detection of affected components to CT. - # ADO task: 18816 - TARGET_COMMIT: $(targetCommit) - UPSTREAM_REPO_URL: $(Build.Repository.Uri) diff --git a/.github/workflows/ado/templates/steps/commit-range-postmerge.yml b/.github/workflows/ado/templates/steps/commit-range-postmerge.yml new file mode 100644 index 00000000000..81db13e949b --- /dev/null +++ b/.github/workflows/ado/templates/steps/commit-range-postmerge.yml @@ -0,0 +1,63 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Step template: resolve the commit range for a post-merge delta build. +# source = the commit that triggered this run (Build.SourceVersion). +# base = the previous CI build's commit on this branch, from the ADO Builds +# API. Pairing each build with the one immediately before it keeps +# concurrent runs from overlapping and captures every commit a rebase +# merge appends. See scripts/ci/ado/determine_commit_range.py for the +# rationale and the first-run (source^1) fallback. +# +# Sets job variables named by baseCommitVar and sourceCommitVar. + +parameters: + - name: baseCommitVar + type: string + default: baseCommit + - name: sourceCommitVar + type: string + default: sourceCommit + +steps: + - script: | + set -euo pipefail + # The helper prints the resolved range to stdout as two lines: + # sourceCommit= + # baseCommit= + # (all diagnostics go to stderr). We parse them and set the pipeline + # variables HERE so the wiring is visible in the YAML. + range_output="$(python3 scripts/ci/ado/determine_commit_range.py \ + --definition-id "$SYSTEM_DEFINITIONID" \ + --current-build-id "$BUILD_BUILDID" \ + --branch "$BUILD_SOURCEBRANCH" \ + --source-commit "$BUILD_SOURCEVERSION")" + echo "$range_output" + + source_commit="$(sed -n 's/^sourceCommit=//p' <<< "$range_output")" + base_commit="$(sed -n 's/^baseCommit=//p' <<< "$range_output")" + if [[ -z "$source_commit" || -z "$base_commit" ]]; then + echo "##[error]determine_commit_range.py did not emit a source/base commit range." + exit 1 + fi + + # baselineQueryFailed=true means the ADO Builds API query itself failed + # (bad SYSTEM_ACCESSTOKEN scope, network, SDK misconfig) -- not a benign + # first run. A failed query silently degrades every run to a single-commit + # delta, so raise a visible warning here. + if [[ "$(sed -n 's/^baselineQueryFailed=//p' <<< "$range_output")" == "true" ]]; then + echo "##vso[task.logissue type=warning]Delta build could not query previous builds; degraded to a single-commit delta (base=source^1). Check the SYSTEM_ACCESSTOKEN scope and ADO Builds API connectivity. The weekly true-up job covers any gap." + fi + + echo "##vso[task.setvariable variable=$BASE_COMMIT_VAR;isreadonly=true]$base_commit" + echo "##vso[task.setvariable variable=$SOURCE_COMMIT_VAR;isreadonly=true]$source_commit" + env: + # System.AccessToken is a secret -- NOT auto-exposed -- so it must be + # mapped here. Every other predefined variable the script uses + # (Build.BuildId, Build.SourceBranch, Build.SourceVersion, + # System.CollectionUri, System.DefinitionId, System.TeamProject) is + # auto-exposed as $NAME and referenced directly, so it is not mapped here. + BASE_COMMIT_VAR: ${{ parameters.baseCommitVar }} + SOURCE_COMMIT_VAR: ${{ parameters.sourceCommitVar }} + SYSTEM_ACCESSTOKEN: $(System.AccessToken) + displayName: "Determine source and base commit range" diff --git a/.github/workflows/ado/templates/steps/commit-range-pr.yml b/.github/workflows/ado/templates/steps/commit-range-pr.yml new file mode 100644 index 00000000000..45cfdf42c79 --- /dev/null +++ b/.github/workflows/ado/templates/steps/commit-range-pr.yml @@ -0,0 +1,56 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Step template: resolve a PR's commit range from the merge commit's parents. +# A PR-policy build checks out the MERGE commit: parent ^1 is the target-branch +# tip, parent ^2 is the PR head. The change set is the diff between their MERGE +# BASE (fork point) and ^2 -- the same three-dot set GitHub's "Files changed" +# shows. Comparing against the merge base rather than the target tip avoids +# flagging components that merely landed on the target branch after the fork. +# +# Sets job variables named by baseCommitVar (merge base) and sourceCommitVar +# (PR head). + +parameters: + - name: baseCommitVar + type: string + default: baseCommit + - name: sourceCommitVar + type: string + default: sourceCommit + +steps: + # Fail fast, at compile time, when this is not a PR build: the merge-commit + # parent logic only makes sense for a PullRequest trigger. + - ${{ if ne(variables['Build.Reason'], 'PullRequest') }}: + - script: | + echo "##[error]This pipeline must run as a PR build (Build.Reason=PullRequest); got '$BUILD_REASON'." + exit 1 + displayName: "Guard: PR build only" + + - script: | + set -euo pipefail + # Defense in depth: a PullRequest trigger should always check out a merge + # commit, but verify the ^2 parent exists before relying on it. + if ! git rev-parse --verify -q "HEAD^2" >/dev/null; then + echo "##[error]HEAD is not a merge commit -- expected a PR merge build." + exit 1 + fi + source_commit="$(git rev-parse HEAD^2)" + # Baseline = the MERGE BASE (fork point) of the target-branch tip (HEAD^1) + # and the PR head (HEAD^2), NOT the target tip itself. + base_commit="$(git merge-base HEAD^1 HEAD^2)" + # PR-supplied data is untrusted: validate both SHAs before use. + for sha in "$base_commit" "$source_commit"; do + if [[ ! "$sha" =~ ^[0-9a-f]{40}$ ]]; then + echo "##[error]invalid commit SHA: $sha" + exit 1 + fi + done + echo "Resolved range: base(merge-base)=$base_commit source=$source_commit" + echo "##vso[task.setvariable variable=$BASE_COMMIT_VAR;isreadonly=true]$base_commit" + echo "##vso[task.setvariable variable=$SOURCE_COMMIT_VAR;isreadonly=true]$source_commit" + env: + BASE_COMMIT_VAR: ${{ parameters.baseCommitVar }} + SOURCE_COMMIT_VAR: ${{ parameters.sourceCommitVar }} + displayName: "Determine PR commit range" diff --git a/.github/workflows/ado/templates/steps/common-steps.yml b/.github/workflows/ado/templates/steps/common-steps.yml deleted file mode 100644 index 03fbc386975..00000000000 --- a/.github/workflows/ado/templates/steps/common-steps.yml +++ /dev/null @@ -1,242 +0,0 @@ -# Microsoft Corporation -# -# Shared step template for the Control Tower integration pipelines (package -# build + source upload). Spliced into each pipeline's single job via: -# -# steps: -# - template: steps/common-steps.yml -# - -# -# Keeping these as STEPS (not a separate job/stage) is deliberate: the -# change-detection steps emit job-scoped pipeline variables (sourceCommit, -# targetCommit, changedComponentsFile, renderSetFile) and write the change-set -# files to disk. -# Those flow freely to later steps in the SAME job but would need output -# variables + artifact upload/download to cross a job boundary. Splicing the -# common steps inline keeps that wiring trivial. -# -# Contract for the including job: -# * It MUST define the job-scope variable `ob_outputDirectory` (both stages -# templates set it from their `outputDirectory` parameter); the verify-locks -# and prepare-change-set steps publish triage artifacts there. -# * These steps assume normal step sequencing -- a failed step stops the ones -# after it (the default; do NOT set step-level `continueOnError` on them). -# Later steps depend on earlier ones, e.g. the render step trusts that -# "Prepare change set" ran and set renderSetFile to a real path. A job-level -# `continueOnError` (which only affects how the JOB result is reported, not -# intra-job step flow) is fine -- the source-upload pipeline uses one. -# -# The steps below are identical across both pipelines: ensure full git history, -# authenticate + install dependencies, resolve the (target, source) commit -# range, and validate (lock files + rendered specs). The pipeline-specific -# Control Tower call (package build or prcheck) is appended by the including -# stages template. - -steps: - # Ensure the full git history is present, ONCE, before anything that needs it. - # The OneBranch checkout may be shallow (depth 1), but lock resolution, spec - # rendering (rpmautospec derives Release/changelog from `git log` over the - # lock files), and commit-range detection all require the complete history. - # Doing the unshallow here means the helper scripts can assume full history - # and never fetch themselves -- a `git fetch --depth=N` inside a script would - # re-shallow a full clone, which silently corrupts the rpmautospec Release - # calculation (and is a footgun when running the scripts locally). - - script: | - set -euo pipefail - if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then - echo "##[group]Fetching full git history" - git fetch --unshallow - echo "##[endgroup]" - fi - displayName: "Ensure full git history" - - - task: PipAuthenticate@1 - displayName: "Authenticate pip" - inputs: - artifactFeeds: "azl/ControlTowerFeed" - - - script: | - set -euo pipefail - - echo "##[group]Mock" - tdnf install -y mock mock-rpmautospec python3-chardet - sudo usermod -aG mock "$(whoami)" - echo "##[endgroup]" - - echo "##[group]Azldev" - AZLDEV_VERSION=$(cat .azldev-version) - echo "Installing azldev@${AZLDEV_VERSION}..." - go install "github.com/microsoft/azure-linux-dev-tools/cmd/azldev@${AZLDEV_VERSION}" - - go_bin_path="$(go env GOPATH)/bin" - echo "##vso[task.prependpath]$go_bin_path" - - "$go_bin_path/azldev" --version - echo "##[endgroup]" - - echo "##[group]Python dependencies" - pip install -r scripts/ci/control-tower/requirements.txt - pip install -r scripts/ci/ado/requirements.txt - echo "##[endgroup]" - displayName: "Install dependencies" - - # Resolve the (target, source) commit range for this post-merge build: - # source = the commit that triggered this run (Build.SourceVersion). - # target = the previous CI build's commit on this branch, looked up - # via the ADO Builds API. Pairing each build with the - # one immediately before it keeps concurrent runs from - # overlapping and captures every commit a rebase merge - # appends. Runs after the dependency-install step because - # it imports the azure-devops SDK. See - # scripts/ci/ado/determine_commit_range.py for the full - # rationale and the first-run (source^1) fallback. - - script: | - set -euo pipefail - # The helper prints the resolved range to stdout as two lines: - # sourceCommit= - # targetCommit= - # (all diagnostics go to stderr). We parse them and set the - # pipeline variables HERE so the variable wiring is visible in - # the YAML rather than hidden in the script. - range_output="$(python3 scripts/ci/ado/determine_commit_range.py \ - --definition-id "$SYSTEM_DEFINITIONID" \ - --current-build-id "$BUILD_BUILDID" \ - --branch "$DELTA_QUERY_BRANCH" \ - --source-commit "$BUILD_SOURCEVERSION")" - echo "$range_output" - - source_commit="$(sed -n 's/^sourceCommit=//p' <<< "$range_output")" - target_commit="$(sed -n 's/^targetCommit=//p' <<< "$range_output")" - if [[ -z "$source_commit" || -z "$target_commit" ]]; then - echo "##[error]determine_commit_range.py did not emit a source/target commit range." - exit 1 - fi - - # The helper emits baselineQueryFailed=true ONLY when the ADO Builds API - # query itself failed (bad SYSTEM_ACCESSTOKEN scope, network, SDK - # misconfig) -- as opposed to a successful query that simply found no - # prior build (a benign first run). A failed query silently degrades EVERY - # run to a single-commit delta, with the weekly true-up as the sole safety - # net, so raise a visible pipeline warning here rather than letting it pass - # unnoticed. - if [[ "$(sed -n 's/^baselineQueryFailed=//p' <<< "$range_output")" == "true" ]]; then - echo "##vso[task.logissue type=warning]Delta build could not query previous builds; degraded to a single-commit delta (target=source^1). Check the SYSTEM_ACCESSTOKEN scope and ADO Builds API connectivity. The weekly true-up job covers any gap." - fi - - # Mark the resolved range readonly: it is computed once here and only - # consumed downstream, so locking it prevents a later step from clobbering - # the commit range (matches changedComponentsFile / renderSetFile below). - echo "##vso[task.setvariable variable=sourceCommit;isreadonly=true]$source_commit" - echo "##vso[task.setvariable variable=targetCommit;isreadonly=true]$target_commit" - env: - SYSTEM_COLLECTIONURI: $(System.CollectionUri) - SYSTEM_TEAMPROJECT: $(System.TeamProject) - SYSTEM_ACCESSTOKEN: $(System.AccessToken) - SYSTEM_DEFINITIONID: $(System.DefinitionId) - BUILD_BUILDID: $(Build.BuildId) - DELTA_QUERY_BRANCH: $(Build.SourceBranch) - BUILD_SOURCEVERSION: $(Build.SourceVersion) - displayName: "Determine source and target commit range" - - # Verify lock files are current. --check-only validates without - # writing, exits nonzero if any lock would change. - - script: | - set -euo pipefail - scripts/ci/control-tower/verify_locks.sh \ - --output-file "$(Build.ArtifactStagingDirectory)/lock-update.json" \ - --publish-dir "$(ob_outputDirectory)" - displayName: "Verify lock files are up to date" - - # Detect changed components AND prepare the render set. The same - # changed-components JSON is consumed by the downstream - # render-verify step (below) and by the CT API calls (appended by - # the including stages template -- prcheck or package-build), so we - # compute it once here and publish it as a triage artifact on every - # exit path. - # - # azldev hard-fails if any component has sourcesChange == true - # without a corresponding identity change (changeType not in - # {added, changed, deleted}). That combination would let - # attacker-supplied bytes ride into the lookaside under an - # unchanged component identity, so we treat it as hostile and - # fail closed (supply-chain drift tripwire). --include-unchanged - # (set inside compute_changed.sh) ensures the full component - # list is available for downstream consumers. - - script: | - set -euo pipefail - change_set_dir="$(Build.ArtifactStagingDirectory)/change-set" - json_file="$change_set_dir/changed-components.json" - render_set_file="$change_set_dir/render-set.txt" - - # Publish the changed-components JSON for post-mortem triage on - # EVERY exit path, not just success -- if azldev hard-fails on a - # consistency tripwire the partial JSON is exactly what an - # operator needs to investigate. - publish_artifact() { - if [[ -s "$json_file" ]]; then - mkdir -p "$(ob_outputDirectory)/changed-components" - cp "$json_file" "$(ob_outputDirectory)/changed-components/" || true - fi - } - trap publish_artifact EXIT - - echo "##[group]Preparing change set" - scripts/ci/components/compute_change_set.sh \ - --output-dir "$change_set_dir" \ - --source-commit "$SOURCE_COMMIT" \ - --target-commit "$TARGET_COMMIT" - echo "##[endgroup]" - - echo "##[group]Upload set (sourcesChange == true, changeType in {added, changed})" - upload_count=$(jq -r '[.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed")))] | length' "$json_file") - jq -r '.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed"))) | .component' "$json_file" | sort - echo "Total: $upload_count component(s) to upload." - echo "##[endgroup]" - - # Publish the downstream pipeline variables now that the change set is - # ready (consumed by the render-verify step and the trailing Control - # Tower call). - echo "##vso[task.setvariable variable=changedComponentsFile;isreadonly=true]$json_file" - echo "##vso[task.setvariable variable=renderSetFile;isreadonly=true]$render_set_file" - env: - SOURCE_COMMIT: $(sourceCommit) - TARGET_COMMIT: $(targetCommit) - displayName: "Prepare change set" - - # Render the components flagged by the change set above - # (azldev-flagged plus any with hand-edited spec files) in - # --check-only mode: azldev renders to a staging area and diffs - # against the on-disk output without writing. Exits nonzero if - # any component's rendered output would change, catching both - # stale renders and direct hand-edits. - - script: | - set -euo pipefail - # A missing render-set file means "Prepare change set" did not complete; - # fail loud rather than silently skipping the render check. - if [[ ! -f "$RENDER_SET_FILE" ]]; then - echo "##[error]render-set file '$RENDER_SET_FILE' is missing -- 'Prepare change set' did not complete cleanly." - exit 1 - fi - # An empty render set just means no PR-scoped components changed. - if [[ ! -s "$RENDER_SET_FILE" ]]; then - echo "No changed components -- skipping render." - exit 0 - fi - echo "##[group]Render set" - sed 's/^/ - /' "$RENDER_SET_FILE" - echo "##[endgroup]" - echo "##[group]Specs rendering + verification" - # `-x` fails loudly if the arg list would split into multiple - # xargs invocations -- a multi-batch render-check would - # silently mask drift in the later batches. `-n 50000` is - # required for `-x` to have any effect (without `-n`/`-L`, - # GNU xargs never splits). `--` ends azldev option parsing so - # component names beginning with `-` (none today) are - # unambiguous. `env AZLDEV_ALLOW_ROOT=1` is inline per - # .github/instructions/ado-pipeline.instructions.md. - xargs -x -n 50000 -d '\n' env AZLDEV_ALLOW_ROOT=1 azldev component render --check-only -- \ - < "$RENDER_SET_FILE" - echo "##[endgroup]" - env: - RENDER_SET_FILE: $(renderSetFile) - displayName: "Verify rendered specs are up to date" diff --git a/.github/workflows/ado/templates/steps/ensure-full-history.yml b/.github/workflows/ado/templates/steps/ensure-full-history.yml new file mode 100644 index 00000000000..aac374c264e --- /dev/null +++ b/.github/workflows/ado/templates/steps/ensure-full-history.yml @@ -0,0 +1,16 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Step template: unshallow the git checkout when it is shallow, so the full +# history is available to later steps. Fetches with --unshallow exactly once; +# a `git fetch --depth=N` would otherwise re-shallow a full clone. + +steps: + - script: | + set -euo pipefail + if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then + echo "##[group]Fetching full git history" + git fetch --unshallow + echo "##[endgroup]" + fi + displayName: "Ensure full git history" diff --git a/.github/workflows/ado/templates/steps/get-changes-info.yml b/.github/workflows/ado/templates/steps/get-changes-info.yml new file mode 100644 index 00000000000..d3e455480be --- /dev/null +++ b/.github/workflows/ado/templates/steps/get-changes-info.yml @@ -0,0 +1,76 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Composite step template: gather a build's change-set information. Given a +# `mode` (pr | postmerge) it resolves the commit range, installs the host tools +# it needs (azldev + mock, plus the ADO API client in postmerge mode and the +# go-git config normalization in pr mode), validates source origins, verifies +# lock files, computes the changed-component set, and verifies rendered specs. +# +# Its install-deps covers only these internal needs. A caller that needs other +# host tools (e.g. the Control Tower client for a submission step) installs them +# with its own install-deps.yml step. +# +# Requires the job-scope variable ob_outputDirectory. Sets the change-info +# variables named by the *Var parameters. + +parameters: + - name: baseCommitVar + type: string + default: baseCommit + - name: changedComponentsFileVar + type: string + default: changedComponentsFile + - name: mode + type: string + values: + - pr + - postmerge + - name: renderSetFileVar + type: string + default: renderSetFile + - name: sourceCommitVar + type: string + default: sourceCommit + +steps: + - template: ensure-full-history.yml + + - template: install-deps.yml + parameters: + installAdoRequirements: ${{ eq(parameters.mode, 'postmerge') }} + installAzldev: true + installMock: true + normalizeGoGitConfig: ${{ eq(parameters.mode, 'pr') }} + + # Sets job variables: + # - baseCommit + # - sourceCommit + - ${{ if eq(parameters.mode, 'pr') }}: + - template: commit-range-pr.yml + parameters: + baseCommitVar: ${{ parameters.baseCommitVar }} + sourceCommitVar: ${{ parameters.sourceCommitVar }} + - ${{ if eq(parameters.mode, 'postmerge') }}: + - template: commit-range-postmerge.yml + parameters: + baseCommitVar: ${{ parameters.baseCommitVar }} + sourceCommitVar: ${{ parameters.sourceCommitVar }} + + - template: verify-source-origin.yml + + - template: verify-locks.yml + + # Sets job variables: + # - changedComponentsFile + # - renderSetFile + - template: prepare-change-set.yml + parameters: + changedComponentsFileVar: ${{ parameters.changedComponentsFileVar }} + fromCommitVar: ${{ parameters.baseCommitVar }} + renderSetFileVar: ${{ parameters.renderSetFileVar }} + toCommitVar: ${{ parameters.sourceCommitVar }} + + - template: verify-rendered-specs.yml + parameters: + renderSetFileVar: ${{ parameters.renderSetFileVar }} diff --git a/.github/workflows/ado/templates/steps/install-deps.yml b/.github/workflows/ado/templates/steps/install-deps.yml new file mode 100644 index 00000000000..17380e131f7 --- /dev/null +++ b/.github/workflows/ado/templates/steps/install-deps.yml @@ -0,0 +1,96 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Step template: authenticate to the internal pip feed, then install the +# requested host tools. Each tool is opt-in via a boolean parameter. azldev's +# version comes from the committed .azldev-version and is validated before use. + +parameters: + # azure-devops SDK (scripts/ci/ado/requirements.txt). + - name: installAdoRequirements + type: boolean + default: false + # azldev CLI (change detection + spec render). + - name: installAzldev + type: boolean + default: false + # Control Tower client (scripts/ci/control-tower/requirements.txt). + - name: installControlTowerClient + type: boolean + default: false + # mock + mock-rpmautospec (spec render / lock work). + - name: installMock + type: boolean + default: false + # strip extensions.worktreeConfig so azldev's go-git can open the checkout. + - name: normalizeGoGitConfig + type: boolean + default: false + +steps: + - task: PipAuthenticate@1 + displayName: "Authenticate pip" + inputs: + artifactFeeds: "azl/ControlTowerFeed" + + - ${{ if parameters.normalizeGoGitConfig }}: + # azldev opens the repo with go-git, which rejects a config that declares + # the `worktreeconfig` extension while core.repositoryformatversion is still + # 0: "core.repositoryformatversion does not support extension: worktreeconfig". + # Native git tolerates this and the ADO agent checkout leaves the extension + # set, so strip it. Each CI run is a fresh checkout so this is self-contained. + # TODO: remove once azldev no longer needs the workaround (go-git v6 fixes + # the underlying bug): https://github.com/microsoft/azure-linux-dev-tools/issues/241 + - script: | + set -euo pipefail + if git config --get extensions.worktreeConfig >/dev/null 2>&1; then + echo "Removing extensions.worktreeConfig so go-git (azldev) can open the repo" + git config --unset-all extensions.worktreeConfig || true + fi + displayName: "Normalize git config for azldev (go-git)" + + - ${{ if parameters.installMock }}: + - script: | + set -euo pipefail + echo "##[group]Mock" + tdnf install -y mock mock-rpmautospec python3-chardet + sudo usermod -aG mock "$(whoami)" + echo "##[endgroup]" + displayName: "Install mock" + + - ${{ if parameters.installAzldev }}: + - script: | + set -euo pipefail + echo "##[group]Azldev (host)" + # Only the version string comes from the checkout; reject a + # malformed/garbage value before it reaches `go install`. + AZLDEV_VERSION="$(tr -d '\n' < .azldev-version)" + if ! printf '%s' "$AZLDEV_VERSION" | grep -Eq '^[0-9A-Za-z._+-]+$'; then + echo "##[error].azldev-version is empty or has unexpected characters" + exit 1 + fi + echo "Installing azldev@${AZLDEV_VERSION}..." + go install "github.com/microsoft/azure-linux-dev-tools/cmd/azldev@${AZLDEV_VERSION}" + + go_bin_path="$(go env GOPATH)/bin" + echo "##vso[task.prependpath]$go_bin_path" + + "$go_bin_path/azldev" --version + echo "##[endgroup]" + displayName: "Install azldev" + + - ${{ if parameters.installControlTowerClient }}: + - script: | + set -euo pipefail + echo "##[group]Python dependencies (Control Tower client)" + pip install -r scripts/ci/control-tower/requirements.txt + echo "##[endgroup]" + displayName: "Install Control Tower Python client" + + - ${{ if parameters.installAdoRequirements }}: + - script: | + set -euo pipefail + echo "##[group]Python dependencies (ADO API client)" + pip install -r scripts/ci/ado/requirements.txt + echo "##[endgroup]" + displayName: "Install ADO API Python client" diff --git a/.github/workflows/ado/templates/steps/prepare-change-set.yml b/.github/workflows/ado/templates/steps/prepare-change-set.yml new file mode 100644 index 00000000000..847ab29ae03 --- /dev/null +++ b/.github/workflows/ado/templates/steps/prepare-change-set.yml @@ -0,0 +1,67 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Step template: detect changed components and prepare the render set, then +# publish changed-components.json as a triage artifact on every exit path. +# compute_change_set.sh diffs from -> to and hard-fails on the supply-chain +# drift tripwire (sourcesChange == true without an identity change in +# {added, changed, deleted}). Requires the job-scope variable ob_outputDirectory. +# +# Reads the (from, to) commit range from the variables named by fromCommitVar / +# toCommitVar; sets the variables named by changedComponentsFileVar and +# renderSetFileVar. + +parameters: + - name: changedComponentsFileVar + type: string + default: changedComponentsFile + - name: fromCommitVar + type: string + default: baseCommit + - name: renderSetFileVar + type: string + default: renderSetFile + - name: toCommitVar + type: string + default: sourceCommit + +steps: + - script: | + set -euo pipefail + change_set_dir="$BUILD_ARTIFACTSTAGINGDIRECTORY/change-set" + json_file="$change_set_dir/changed-components.json" + render_set_file="$change_set_dir/render-set.txt" + + # Publish the changed-components JSON for post-mortem triage on EVERY exit + # path, not just success -- if azldev hard-fails on a consistency tripwire + # the partial JSON is exactly what an operator needs to investigate. + publish_artifact() { + if [[ -s "$json_file" ]]; then + mkdir -p "$OB_OUTPUT_DIR/changed-components" + cp "$json_file" "$OB_OUTPUT_DIR/changed-components/" || true + fi + } + trap publish_artifact EXIT + + echo "##[group]Preparing change set" + scripts/ci/components/compute_change_set.sh \ + --output-dir "$change_set_dir" \ + --from-commit "$FROM_COMMIT" \ + --to-commit "$TO_COMMIT" + echo "##[endgroup]" + + echo "##[group]Upload set (sourcesChange == true, changeType in {added, changed})" + upload_count=$(jq -r '[.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed")))] | length' "$json_file") + jq -r '.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed"))) | .component' "$json_file" | sort + echo "Total: $upload_count component(s) to upload." + echo "##[endgroup]" + + echo "##vso[task.setvariable variable=$CHANGED_COMPONENTS_FILE_VAR;isreadonly=true]$json_file" + echo "##vso[task.setvariable variable=$RENDER_SET_FILE_VAR;isreadonly=true]$render_set_file" + env: + CHANGED_COMPONENTS_FILE_VAR: ${{ parameters.changedComponentsFileVar }} + FROM_COMMIT: $(${{ parameters.fromCommitVar }}) + OB_OUTPUT_DIR: $(ob_outputDirectory) + RENDER_SET_FILE_VAR: ${{ parameters.renderSetFileVar }} + TO_COMMIT: $(${{ parameters.toCommitVar }}) + displayName: "Prepare change set" diff --git a/.github/workflows/ado/templates/steps/verify-locks.yml b/.github/workflows/ado/templates/steps/verify-locks.yml new file mode 100644 index 00000000000..8cb44e88a2f --- /dev/null +++ b/.github/workflows/ado/templates/steps/verify-locks.yml @@ -0,0 +1,16 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Step template: verify lock files are current. --check-only validates without +# writing and exits nonzero if any lock would change; publishes a lock-update +# diff to ob_outputDirectory. + +steps: + - script: | + set -euo pipefail + scripts/ci/control-tower/verify_locks.sh \ + --output-file "$BUILD_ARTIFACTSTAGINGDIRECTORY/lock-update.json" \ + --publish-dir "$OB_OUTPUT_DIR" + env: + OB_OUTPUT_DIR: $(ob_outputDirectory) + displayName: "Verify lock files are up to date" diff --git a/.github/workflows/ado/templates/steps/verify-rendered-specs.yml b/.github/workflows/ado/templates/steps/verify-rendered-specs.yml new file mode 100644 index 00000000000..22928e19da2 --- /dev/null +++ b/.github/workflows/ado/templates/steps/verify-rendered-specs.yml @@ -0,0 +1,44 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Step template: render the components in the render set (--check-only) and fail +# if any component's rendered output would change, catching stale renders and +# direct hand-edits. Reads the render-set path from the variable named by +# renderSetFileVar. + +parameters: + - name: renderSetFileVar + type: string + default: renderSetFile + +steps: + - script: | + set -euo pipefail + # A missing render-set file means the change set was not prepared; fail + # loud rather than silently skipping the render check. + if [[ ! -f "$RENDER_SET_FILE" ]]; then + echo "##[error]render-set file '$RENDER_SET_FILE' is missing -- the change set was not prepared cleanly." + exit 1 + fi + # An empty render set just means no components changed. + if [[ ! -s "$RENDER_SET_FILE" ]]; then + echo "No changed components -- skipping render." + exit 0 + fi + echo "##[group]Render set" + sed 's/^/ - /' "$RENDER_SET_FILE" + echo "##[endgroup]" + echo "##[group]Specs rendering + verification" + # `-x` fails loudly if the arg list would split into multiple xargs + # invocations -- a multi-batch render-check would silently mask drift in + # the later batches. `-n 50000` is required for `-x` to have any effect + # (without `-n`/`-L`, GNU xargs never splits). `--` ends azldev option + # parsing so component names beginning with `-` (none today) are + # unambiguous. `env AZLDEV_ALLOW_ROOT=1` is inline per + # .github/instructions/ado-pipeline.instructions.md. + xargs -x -n 50000 -d '\n' env AZLDEV_ALLOW_ROOT=1 azldev component render --check-only -- \ + < "$RENDER_SET_FILE" + echo "##[endgroup]" + env: + RENDER_SET_FILE: $(${{ parameters.renderSetFileVar }}) + displayName: "Verify rendered specs are up to date" diff --git a/.github/workflows/ado/templates/steps/verify-source-origin.yml b/.github/workflows/ado/templates/steps/verify-source-origin.yml new file mode 100644 index 00000000000..b6f0f640ef7 --- /dev/null +++ b/.github/workflows/ado/templates/steps/verify-source-origin.yml @@ -0,0 +1,13 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Step template: validate that the resolved source origins (Fedora dist-git, +# lookaside, and repo base URIs) match the trusted allowlist before any source +# fetch or lock update runs, so a PR cannot redirect source fetches to an +# untrusted origin. Requires azldev on PATH (install-deps installAzldev). + +steps: + - script: | + set -euo pipefail + python3 scripts/ci/render-specs-check/source_origin_guard.py + displayName: "Verify source origins" diff --git a/.github/workflows/check-rendered-specs.yml b/.github/workflows/check-rendered-specs.yml index e11cf604d53..705019c9641 100644 --- a/.github/workflows/check-rendered-specs.yml +++ b/.github/workflows/check-rendered-specs.yml @@ -437,8 +437,8 @@ jobs: # current head. /scripts-components/compute_change_set.sh \ --output-dir /output/change-set \ - --source-commit "$PR_HEAD_SHA" \ - --target-commit "$PR_BASE_SHA" + --from-commit "$PR_BASE_SHA" \ + --to-commit "$PR_HEAD_SHA" # Components removed in this PR no longer have a comp.toml, so # `azldev component render` will not touch (or clean) their old diff --git a/scripts/ci/ado/README.md b/scripts/ci/ado/README.md index 2c857b3ce56..6a2bb14f089 100644 --- a/scripts/ci/ado/README.md +++ b/scripts/ci/ado/README.md @@ -6,7 +6,7 @@ build metadata) from ADO YAML pipelines, using the official | File | Purpose | | ---- | ------- | -| `determine_commit_range.py` | Resolves the `(target, source)` commit range for the post-merge delta build and prints it to stdout as `sourceCommit=` / `targetCommit=` lines. The calling step sets the pipeline variables. | +| `determine_commit_range.py` | Resolves the `(base, source)` commit range for the post-merge delta build and prints it to stdout as `sourceCommit=` / `baseCommit=` lines. The calling step sets the pipeline variables. | | `requirements.txt` | Python dependencies (`azure-devops`), installed from the internal feed. | ## Conventions @@ -29,17 +29,17 @@ build metadata) from ADO YAML pipelines, using the official pipeline's "Ensure full git history" step fetches it once up front) and never fetches — a `git fetch --depth=N` would re-shallow a full clone. - **Output:** two `key=value` lines on **stdout** (`sourceCommit=` and - `targetCommit=`); all diagnostics go to **stderr**. The caller parses + `baseCommit=`); all diagnostics go to **stderr**. The caller parses stdout and owns the `##vso[task.setvariable]` wiring, so pipeline-variable assignment stays visible in the YAML. It is best-effort: if the previous build cannot be found (first run) or the ADO -query fails, it falls back to `target = source^1` (warning on stderr) rather +query fails, it falls back to `base = source^1` (warning on stderr) rather than failing the pipeline. A hard failure (invalid source SHA, or no parent found for the fallback) exits non-zero so the calling step fails. ## Callers -- `templates/steps/common-steps.yml` "Determine source and target commit range" - step → `determine_commit_range.py` (shared by the package-build and - source-upload pipelines). +- `templates/steps/commit-range-postmerge.yml` "Determine source and base + commit range" step → `determine_commit_range.py` (used by the post-merge + package-build pipeline). diff --git a/scripts/ci/ado/determine_commit_range.py b/scripts/ci/ado/determine_commit_range.py index 00dd0b9077c..ee2ce167a45 100644 --- a/scripts/ci/ado/determine_commit_range.py +++ b/scripts/ci/ado/determine_commit_range.py @@ -1,9 +1,9 @@ -"""Resolve the ``(target, source)`` commit range for a post-merge delta build. +"""Resolve the ``(base, source)`` commit range for a post-merge delta build. -Strategy (see ``.github/workflows/ado/templates/steps/common-steps.yml``): +Strategy (see ``.github/workflows/ado/templates/steps/commit-range-postmerge.yml``): * ``source`` is the commit that triggered this run (``Build.SourceVersion``). -* ``target`` is the ``sourceVersion`` of the immediately-preceding CI build of +* ``base`` is the ``sourceVersion`` of the immediately-preceding CI build of this pipeline definition on the same branch, selected by build id — regardless of that build's result. @@ -19,7 +19,7 @@ current tip), it captures EVERY commit a rebase merge appends, not just the tip. ``azldev component changed`` then tree-diffs the two endpoints. -Fallback (first run, or no prior CI build found): ``target = source^1``. That +Fallback (first run, or no prior CI build found): ``base = source^1``. That run only builds the single tip commit's components, self-correcting on the next push. Two distinct situations trigger this fallback and are NOT treated the same: a successful query that simply found no prior build (benign -- a genuine @@ -29,7 +29,7 @@ and the calling step raises a visible pipeline warning. The resolved hashes are printed to stdout as two ``key=value`` lines -(``sourceCommit=`` and ``targetCommit=``), plus an optional +(``sourceCommit=`` and ``baseCommit=``), plus an optional ``baselineQueryFailed=true`` line when the build-history query failed. The calling pipeline step reads them, sets the corresponding ADO pipeline variables, and raises a warning on query failure -- so the variable wiring and the warning @@ -74,19 +74,19 @@ def _log(message: str) -> None: print(message, file=sys.stderr) -def _emit_range(source_commit: str, target_commit: str, *, query_failed: bool = False) -> None: +def _emit_range(source_commit: str, base_commit: str, *, query_failed: bool = False) -> None: """Print the resolved range to stdout as ``key=value`` lines. The calling pipeline step parses these lines and sets the ``sourceCommit`` / - ``targetCommit`` pipeline variables, so the variable wiring lives in the YAML + ``baseCommit`` pipeline variables, so the variable wiring lives in the YAML rather than in this script. When ``query_failed`` is True an extra ``baselineQueryFailed=true`` line is emitted so the caller can raise a pipeline warning -- this marks the actionable "Builds API query failed" case and is deliberately NOT emitted for a benign "no prior build" first run. """ - _log(f"Resolved range: target={target_commit} source={source_commit}") + _log(f"Resolved range: base={base_commit} source={source_commit}") print(f"sourceCommit={source_commit}") - print(f"targetCommit={target_commit}") + print(f"baseCommit={base_commit}") if query_failed: print("baselineQueryFailed=true") @@ -196,7 +196,7 @@ def main() -> int: _log(f"ERROR: --source-commit is not a 40-character hex SHA: {source_commit!r}") return 1 - target_commit: str | None = None + base_commit: str | None = None # Distinguish a *failed* build-history query (actionable) from a successful # query that simply found no prior build (benign). Only the former sets this. query_failed = False @@ -207,7 +207,7 @@ def main() -> int: branch=args.branch, top=args.top, ) - target_commit = _select_baseline(builds, args.current_build_id) + base_commit = _select_baseline(builds, args.current_build_id) except (ClientException, OSError, RuntimeError) as exc: # The query itself failed (bad token scope, network, SDK misconfig). # Fall back to a single-commit delta so the run still makes progress, @@ -216,33 +216,33 @@ def main() -> int: query_failed = True _log(f"WARNING: Could not query previous builds ({exc}); falling back to source^1.") - if target_commit is None: + if base_commit is None: if not query_failed: # Benign: the query succeeded but there is genuinely no prior CI # build (e.g. the first run on a new branch). _log( "INFO: No previous CI build found for this branch; building only " - "the tip commit (target = source^1). The weekly true-up job covers " + "the tip commit (base = source^1). The weekly true-up job covers " "any gap." ) - target_commit = _parent_commit(source_commit) - if target_commit is None: + base_commit = _parent_commit(source_commit) + if base_commit is None: _log("ERROR: Unable to determine a parent of the source commit; cannot compute a build range.") return 1 - _emit_range(source_commit, target_commit, query_failed=query_failed) + _emit_range(source_commit, base_commit, query_failed=query_failed) return 0 # Both endpoints must be present for the downstream tree diff in the # change-set step. Full history is fetched once by the pipeline before this # step runs, so we only sanity-check presence here (no fetching). - missing = [commit for commit in (target_commit, source_commit) if not _commit_present(commit)] + missing = [commit for commit in (base_commit, source_commit) if not _commit_present(commit)] if missing: _log( f"WARNING: commit(s) not present locally: {', '.join(missing)}; the change-set step may be " "unable to diff them. Ensure the full git history was fetched before this step." ) - _emit_range(source_commit, target_commit) + _emit_range(source_commit, base_commit) return 0 diff --git a/scripts/ci/components/README.md b/scripts/ci/components/README.md index 8f16d9a8ce7..a29ecc76ec0 100644 --- a/scripts/ci/components/README.md +++ b/scripts/ci/components/README.md @@ -1,17 +1,16 @@ # Shared azldev component helpers Pipeline-agnostic shell + Python helpers consumed by the GitHub Actions -PR gates (`.github/workflows/check-rendered-specs.yml`), the ADO -Control Tower integration pipeline -(`.github/workflows/ado/templates/sources-upload-stages.yml`), and the ADO -PR package-build check -(`.github/workflows/ado/templates/pr-package-build-stages.yml`). +PR gates (`.github/workflows/check-rendered-specs.yml`), the ADO post-merge +package-build pipeline +(`.github/workflows/ado/templates/package-build-stages.yml`), and the ADO +PR Control Tower check +(`.github/workflows/ado/templates/pr-check-ct-stages.yml`). | Script | Purpose | | ------ | ------- | -| `compute_changed.sh` | Wraps `azldev component changed --from --to -O json`. | | `compute_render_set.py` | Computes the union of (azldev-flagged components) and (components with hand-edited rendered specs), then drops deleted entries. | -| `compute_change_set.sh` | Orchestrates the two above: writes `changed-components.json`, `specs-diff.txt`, and `render-set.txt` into a caller-chosen output directory. | +| `compute_change_set.sh` | Runs `azldev component changed --from --to ` inline, unions the result with `compute_render_set.py`, and writes `changed-components.json`, `specs-diff.txt`, and `render-set.txt` into a caller-chosen output directory. | ## Conventions @@ -22,12 +21,12 @@ PR package-build check `AZLDEV_ALLOW_ROOT=1` prefix per [`ado-pipeline.instructions.md`](../../../.github/instructions/ado-pipeline.instructions.md). Callers do **not** set this at step scope. -- **Single source of truth.** Both pipelines should call these scripts +- **Single source of truth.** Every consumer should call these scripts rather than re-implementing the change-set computation. A regression - here breaks both gates simultaneously, so changes need extra care. + here breaks every gate simultaneously, so changes need extra care. ## Callers - `check-rendered-specs.yml` `render` job → `compute_change_set.sh` -- `sources-upload-stages.yml` "Prepare change set" step → `compute_change_set.sh` -- `pr-package-build-stages.yml` "Prepare change set" step → `compute_change_set.sh` +- `package-build-stages.yml` (via `steps/prepare-change-set.yml`) → `compute_change_set.sh` +- `pr-check-ct-stages.yml` (via `steps/prepare-change-set.yml`) → `compute_change_set.sh` diff --git a/scripts/ci/components/compute_change_set.sh b/scripts/ci/components/compute_change_set.sh index 32f11bf54c3..b7988aa5e01 100755 --- a/scripts/ci/components/compute_change_set.sh +++ b/scripts/ci/components/compute_change_set.sh @@ -24,19 +24,44 @@ set -euo pipefail cd "$(git rev-parse --show-toplevel)" usage() { - echo "Usage: $0 --output-dir DIR --source-commit SHA --target-commit SHA" >&2 + echo "Usage: $0 --output-dir DIR --from-commit SHA --to-commit SHA \\" >&2 + echo " [--changed-components-file NAME] [--specs-diff-file NAME] [--render-set-file NAME]" >&2 + echo " --from-commit baseline commit (merge-base / fork point)" >&2 + echo " --to-commit newer commit (e.g. PR head)" >&2 exit 1 } +# Output file names default to the canonical set but can be overridden by a +# caller that needs to control them (e.g. to stage several change sets side by +# side). Only the names are configurable; they always land under --output-dir. +changed_components_file_name="changed-components.json" +specs_diff_file_name="specs-diff.txt" +render_set_file_name="render-set.txt" + while [[ $# -gt 0 ]]; do case "$1" in - --output-dir) output_dir="$2"; shift 2 ;; - --source-commit) source_commit="$2"; shift 2 ;; - --target-commit) target_commit="$2"; shift 2 ;; + --output-dir) output_dir="$2"; shift 2 ;; + --from-commit) from_commit="$2"; shift 2 ;; + --to-commit) to_commit="$2"; shift 2 ;; + --changed-components-file) changed_components_file_name="$2"; shift 2 ;; + --specs-diff-file) specs_diff_file_name="$2"; shift 2 ;; + --render-set-file) render_set_file_name="$2"; shift 2 ;; *) usage ;; esac done -[[ -z "${output_dir:-}" || -z "${source_commit:-}" || -z "${target_commit:-}" ]] && usage +[[ -z "${output_dir:-}" || -z "${from_commit:-}" || -z "${to_commit:-}" ]] && usage + +# The output-file-name params are joined onto --output-dir verbatim, so a value +# containing a path separator or traversal (e.g. `../foo`) would escape it. +# Reject anything that is not a plain file name. +for name in "$changed_components_file_name" "$specs_diff_file_name" "$render_set_file_name"; do + case "$name" in + */* | "" | . | ..) + echo "refusing unsafe output file name: '$name' (must be a plain file name)" >&2 + exit 2 + ;; + esac +done # Defensive guard: the script owns --output-dir exclusively for the duration # of the invocation (it does `rm -rf` below to clean up stale state). Refuse @@ -60,14 +85,27 @@ mkdir -p "$output_dir" script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -changed_file="$output_dir/changed-components.json" -specs_diff_file="$output_dir/specs-diff.txt" -render_set_file="$output_dir/render-set.txt" +changed_file="$output_dir/$changed_components_file_name" +specs_diff_file="$output_dir/$specs_diff_file_name" +render_set_file="$output_dir/$render_set_file_name" + +# Compute the set of changed components between two commits. Writes one JSON +# entry per component to $out_file with fields: component, changeType, +# sourcesChange. azldev hard-fails if any component has sourcesChange == true +# without a corresponding identity change (added/changed/deleted) -- a +# supply-chain drift guard. Inline AZLDEV_ALLOW_ROOT=1 so CI agents running as +# root work without lifting the restriction at step scope (see +# .github/instructions/ado-pipeline.instructions.md). +compute_changed() { + local out_file="$1" from="$2" to="$3" + mkdir -p "$(dirname "$out_file")" + AZLDEV_ALLOW_ROOT=1 azldev component changed --from "$from" --to "$to" \ + -a --include-unchanged -O json > "$out_file" + echo "Changed components (non-unchanged):" + jq -r '.[] | select(.changeType != "unchanged") | " \(.changeType)\t\(.component)"' "$out_file" | sort +} -"$script_dir/compute_changed.sh" \ - --output-file "$changed_file" \ - --source-commit "$source_commit" \ - --target-commit "$target_commit" +compute_changed "$changed_file" "$from_commit" "$to_commit" # azldev's renderedSpecsDir is absolute. Translate to repo-relative so it # matches git's output (`git diff --name-only` always emits repo-relative @@ -80,7 +118,7 @@ specs_dir="$(realpath --relative-to="$(pwd)" "$specs_dir_abs")" # `azldev component changed` misses). --no-renames prevents collapse of # delete+add into a rename entry, which would lose the old path; the # Python script filters out deleted/unknown components. -git diff --no-renames --name-only "$target_commit" "$source_commit" \ +git diff --no-renames --name-only "$from_commit" "$to_commit" \ -- "$specs_dir" > "$specs_diff_file" python3 "$script_dir/compute_render_set.py" \ diff --git a/scripts/ci/components/compute_changed.sh b/scripts/ci/components/compute_changed.sh deleted file mode 100755 index 48f7f935f71..00000000000 --- a/scripts/ci/components/compute_changed.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env bash -# Compute the set of changed components between source and target commits. -# -# Writes one JSON entry per component to , with fields: -# - component: name -# - changeType: 'added' | 'changed' | 'unchanged' | 'deleted' -# - sourcesChange: bool (rendered 'sources' file differs across commits) -# -# azldev hard-fails if any component has sourcesChange == true without a -# corresponding identity change (added/changed/deleted) -- supply-chain -# drift protection. -# -# Callers are responsible for log grouping and artifact publication. -# `azldev` is invoked with an inline `AZLDEV_ALLOW_ROOT=1` prefix so CI -# agents running as root work without callers having to lift the -# restriction at step scope (see -# .github/instructions/ado-pipeline.instructions.md). - -set -euo pipefail - -usage() { echo "Usage: $0 --output-file FILE --source-commit SHA --target-commit SHA" >&2; exit 1; } - -while [[ $# -gt 0 ]]; do - case "$1" in - --output-file) output_file="$2"; shift 2 ;; - --source-commit) source_commit="$2"; shift 2 ;; - --target-commit) target_commit="$2"; shift 2 ;; - *) usage ;; - esac -done -[[ -z "${output_file:-}" || -z "${source_commit:-}" || -z "${target_commit:-}" ]] && usage - -mkdir -p "$(dirname "$output_file")" - -AZLDEV_ALLOW_ROOT=1 azldev component changed --from "$target_commit" --to "$source_commit" -a --include-unchanged -O json > "$output_file" - -echo "Changed components (non-unchanged):" -jq -r '.[] | select(.changeType != "unchanged") | " \(.changeType)\t\(.component)"' "$output_file" | sort diff --git a/scripts/ci/control-tower/run_prcheck.py b/scripts/ci/control-tower/run_prcheck.py index 20aaef5090d..f5f3982bca8 100644 --- a/scripts/ci/control-tower/run_prcheck.py +++ b/scripts/ci/control-tower/run_prcheck.py @@ -167,10 +167,6 @@ def main() -> None: print("Payload:") print(json.dumps(payload, indent=2)) - if args.build_reason == "PullRequest": - print("Skipping Control Tower call - pull request triggers are not supported, yet.") - return - if not components: print("No affected components detected between source and target commits; skipping Control Tower call.") return diff --git a/scripts/ci/control-tower/verify_locks.sh b/scripts/ci/control-tower/verify_locks.sh index 8dfe3d478b7..fe90da033ae 100755 --- a/scripts/ci/control-tower/verify_locks.sh +++ b/scripts/ci/control-tower/verify_locks.sh @@ -23,8 +23,8 @@ git config --unset extensions.worktreeConfig || true # NOTE: full git history (needed for lock resolution and rpmautospec Release # calculation) is ensured ONCE by the pipeline's "Ensure full git history" step -# (.github/workflows/ado/templates/steps/common-steps.yml) before this script -# runs. This script assumes it is present and does not fetch. +# (.github/workflows/ado/templates/steps/ensure-full-history.yml) before this +# script runs. This script assumes it is present and does not fetch. mkdir -p "$(dirname "$output_file")"