diff --git a/examples/README.md b/examples/README.md index 2ffc05cb5..afd700875 100644 --- a/examples/README.md +++ b/examples/README.md @@ -22,6 +22,10 @@ Examples and templates for using Codex Security: validation script against a deliberately vulnerable API using synthetic data. Follow the demo's setup instructions, and do not deploy the example app. +- [Azure Pipelines with Amazon Bedrock](azure-pipelines/README.md): centrally run + manual full or committed-diff scans against Azure Repos, with OIDC credentials, + report artifacts, and optional native SARIF publishing. + ## npm package This top-level `examples/` directory is available in the repository only; it is diff --git a/examples/azure-pipelines/README.md b/examples/azure-pipelines/README.md new file mode 100644 index 000000000..ce2c36fa4 --- /dev/null +++ b/examples/azure-pipelines/README.md @@ -0,0 +1,106 @@ +# Azure Pipelines with Amazon Bedrock + +This example runs Codex Security from a centrally owned Azure Pipelines YAML +file against a configured Azure Repos Git repository. It supports manual full +and committed-diff scans, short-lived AWS credentials through OIDC, report +artifacts, and optional SARIF publishing to the target repository's Advanced +Security view. No Codex Security runtime changes or custom extension are needed. + +It targets Azure DevOps Services and Microsoft-hosted Linux agents. It does not +enroll repositories, create branch policies, or automatically validate PRs. +Use a trusted tooling repository for the pipeline definition; the target +repository does not need its own pipeline file. + +## Setup + +1. Copy [`azure-pipelines.yml`](azure-pipelines.yml) into the tooling repository. + Set `resources.repositories.target.name` to the approved `Project/Repository` + and adjust the default `targetRef`. Keep the repository name fixed in reviewed + YAML instead of accepting arbitrary scan targets at queue time. +2. Install the current [AWS Toolkit for Azure DevOps](https://github.com/aws/aws-toolkit-azure-devops/releases) + extension in your organization. Create an AWS service connection named + `codex-security-bedrock` with **Use OIDC** and an AWS role ARN; leave access + keys unset. Follow the [AWS OIDC setup guide](https://aws.amazon.com/blogs/modernizing-with-aws/how-to-federate-into-aws-from-azure-devops-using-openid-connect/). + Restrict the role trust to the service connection's actual issuer, audience, + and subject. Authorize only this pipeline to use the connection. +3. Set `awsRegion` and `bedrockModelId` to an available, approved Bedrock model or + inference profile. Grant the role only the required Bedrock invocation + permissions for those resources. The requested AWS session is one hour, + matching the job timeout; configure the role to allow that duration. +4. Create a pipeline pointing at the copied YAML. Grant its project build + service identity read access to the target repository and authorize the + repository resource. Cross-project checkout needs explicit access in the + target project; do not disable project-scoped job authorization to work around + a missing grant. See [multi-repository checkout permissions](https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops). +5. Keep `publishToAdvancedSecurity: false` for artifact-only runs. To enable the + native findings view, enable the required GitHub Code Security / Advanced + Security entitlement on the **target** repository and grant the pipeline + identity permission to publish its results. Then opt in when queuing a run. + See [third-party SARIF publishing](https://learn.microsoft.com/en-us/azure/devops/repos/security/github-advanced-security-code-scanning-third-party?view=azure-devops). + +The YAML pins the CLI, Node, and Python versions and uses current task majors. +Azure Pipelines resolves compatible task updates within those majors. Review +the pinned versions when adopting or upgrading the example. + +## Run a scan + +Choose **Run pipeline** from the trusted tooling branch, then set: + +| Parameter | Meaning | +| --------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| `targetRef` | Target branch or tag, defaulting to `refs/heads/main`. | +| `scanMode` | `full` for a baseline, or `diff` for changes relative to a base revision. | +| `baseRevision` | Used only for `diff`; defaults to `HEAD^` (the preceding commit). Prefer an exact base commit SHA for reproducibility. | +| `failOnSeverity` | `none` is report-only; select a severity to fail on findings at or above it. | +| `publishToAdvancedSecurity` | Opt in to native SARIF publishing after completing its setup. | + +Start with a full scan. For a later committed-diff scan, select the target branch +and a base revision present in that repository's history. The checkout fetches +full history; Codex Security resolves the diff against the checked-out commit. +This scans committed changes, not an automatically discovered PR. + +The optional publisher uses the `target` repository resource metadata, not the +tooling repository's build metadata. Keep its alias and +`advancedsecurity.publish.repository` in sync when adapting the YAML. This is +Microsoft's [explicit multi-repository publishing mechanism](https://learn.microsoft.com/en-us/azure/devops/release-notes/2025/sprint-253-update#multi-repository-publishing-scenarios-supported-for-github-advanced-security-for-azure-devops). +Full and diff results use separate categories so a partial diff does not replace +the full-scan baseline. A diff result is still a snapshot of that diff, not a +complete repository inventory; use full scans to track the baseline over time. + +## Results and failures + +The `codex-security` pipeline artifact contains `result.json`, available +`report.md`, `coverage.json`, and `findings.json`, plus `results.sarif` after a +successful export. Check the JSON/report for findings, coverage, runtime, and +usage; missing or incomplete coverage is not a clean scan. + +| Scan exit | Pipeline behavior | +| ------------- | ---------------------------------------------------------------------------------------------- | +| `0` | Completed scan; export SARIF and retain reports. | +| `1` | Severity policy failed; still export SARIF and retain reports, but keep the job failed. | +| Other nonzero | Scan failed or was incomplete; retain available reports without exporting or publishing SARIF. | + +An export or publishing failure also fails the job. Cancellation skips the +post-scan steps, so artifacts are not guaranteed for canceled runs. Native +publishing waits for processing; verify the target repository and commit in the +first live run. Local validation cannot prove your organization's OIDC trust, +permissions, model access, or SARIF ingestion. + +## Trust and rollout boundaries + +- Restrict who can edit or queue this privileged pipeline and use its AWS service + connection. Treat scanned code and model output as untrusted. OIDC avoids + long-lived keys; it does not make running untrusted code with credentials safe. +- The CLI is installed before target checkout, outside the repository, with npm + lifecycle scripts disabled. Checkout does not persist its credential. The AWS + task supplies temporary credentials only to the scan step; no PAT or Azure + access token is explicitly passed to the CLI. +- Reports can contain source excerpts and vulnerability details. Restrict artifact + access and configure your project's pipeline-run retention policy. The example + retains selected reports, not raw scan state, transcripts, or authentication + files. +- For automatic Azure Repos PR checks, configure a target-branch **build validation + policy** and a PR-aware checkout/pipeline design. YAML `pr:` triggers do not + enable Azure Repos PR validation. Attaching this manual pipeline unchanged to + a policy would still scan its configured target ref, not necessarily the PR. + See [Azure Repos PR triggers](https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops#pr-triggers). diff --git a/examples/azure-pipelines/azure-pipelines.yml b/examples/azure-pipelines/azure-pipelines.yml new file mode 100644 index 000000000..71d085dad --- /dev/null +++ b/examples/azure-pipelines/azure-pipelines.yml @@ -0,0 +1,163 @@ +# Setup and permissions: examples/azure-pipelines/README.md +trigger: none + +parameters: + - name: targetRef + type: string + default: refs/heads/main + - name: scanMode + type: string + default: full + values: [full, diff] + - name: baseRevision + type: string + default: HEAD^ + - name: failOnSeverity + type: string + default: none + values: [none, low, medium, high, critical] + - name: publishToAdvancedSecurity + type: boolean + default: false + +resources: + repositories: + - repository: target + type: git + name: ExampleProject/example-app # Replace with the approved project/repository. + ref: ${{ parameters.targetRef }} + trigger: none + +variables: + codexSecurityVersion: "0.1.27" + awsServiceConnection: codex-security-bedrock + awsRegion: us-east-1 + bedrockModelId: REPLACE_WITH_APPROVED_MODEL_ID + aws.rolecredential.maxduration: "3600" + targetDirectory: $(Pipeline.Workspace)/s/scan-target + scanDirectory: $(Agent.TempDirectory)/codex-security/scan + reportDirectory: $(Agent.TempDirectory)/codex-security/reports + scanExitCode: not-started + sarifReady: "false" + +jobs: + - job: scan + displayName: Codex Security + timeoutInMinutes: 60 + pool: + vmImage: ubuntu-24.04 + variables: + # Attribute alerts to the scanned repository, not the pipeline's repository. + advancedsecurity.publish.repository: $[ convertToJson(resources.repositories['target']) ] + steps: + - task: UseNode@1 + inputs: + version: "24.21.0" + - task: UsePythonVersion@0 + inputs: + versionSpec: "3.14.7" + + # Install before checkout so repository npm configuration is not consulted. + - task: Bash@3 + name: installCli + displayName: Install Codex Security + inputs: + targetType: inline + workingDirectory: $(Agent.TempDirectory) + script: | + set -euo pipefail + npm install --prefix "$CLI_DIRECTORY" --ignore-scripts --no-audit --no-fund \ + --registry=https://registry.npmjs.org/ \ + "@openai/codex-security@$CODEX_SECURITY_VERSION" + printf '##vso[task.prependpath]%s\n' "$CLI_DIRECTORY/node_modules/.bin" + env: + CLI_DIRECTORY: $(Agent.TempDirectory)/codex-security-cli + CODEX_SECURITY_VERSION: $(codexSecurityVersion) + + - checkout: target + path: s/scan-target + fetchDepth: 0 + persistCredentials: false + + - task: AWSShellScript@1 + name: runScan + displayName: Scan with Bedrock OIDC credentials + inputs: + awsCredentials: $(awsServiceConnection) + regionName: $(awsRegion) + scriptType: inline + disableAutoCwd: true + workingDirectory: $(targetDirectory) + inlineScript: | + set -euo pipefail + mkdir -p "$REPORT_DIRECTORY" + args=(scan "$TARGET_DIRECTORY" --provider amazon-bedrock + --model "$BEDROCK_MODEL_ID" --mode standard --effort high + --output-dir "$SCAN_DIRECTORY" --json) + if [[ "$SCAN_MODE" == diff ]]; then + args+=(--diff "$BASE_REVISION") + fi + if [[ "$FAIL_ON_SEVERITY" != none ]]; then + args+=(--fail-on-severity "$FAIL_ON_SEVERITY") + fi + status=0 + codex-security "${args[@]}" > "$REPORT_DIRECTORY/result.json" || status=$? + printf '##vso[task.setvariable variable=scanExitCode]%s\n' "$status" + exit "$status" + env: + TARGET_DIRECTORY: $(targetDirectory) + SCAN_DIRECTORY: $(scanDirectory) + REPORT_DIRECTORY: $(reportDirectory) + BEDROCK_MODEL_ID: $(bedrockModelId) + SCAN_MODE: ${{ parameters.scanMode }} + BASE_REVISION: ${{ parameters.baseRevision }} + FAIL_ON_SEVERITY: ${{ parameters.failOnSeverity }} + + - task: Bash@3 + name: exportSarif + displayName: Export completed scan to SARIF + condition: and(succeededOrFailed(), in(variables['scanExitCode'], '0', '1')) + inputs: + targetType: inline + script: | + set -euo pipefail + codex-security export "$SCAN_DIRECTORY" --export-format sarif \ + --source-root "$TARGET_DIRECTORY" --output "$REPORT_DIRECTORY/results.sarif" + printf '##vso[task.setvariable variable=sarifReady]true\n' + env: + TARGET_DIRECTORY: $(targetDirectory) + SCAN_DIRECTORY: $(scanDirectory) + REPORT_DIRECTORY: $(reportDirectory) + + - ${{ if parameters.publishToAdvancedSecurity }}: + - task: AdvancedSecurity-Publish@1 + displayName: Publish SARIF to the target repository + condition: and(succeededOrFailed(), eq(variables['sarifReady'], 'true')) + inputs: + SarifsInputDirectory: $(reportDirectory) + Category: codex-security/${{ parameters.scanMode }} + WaitForProcessing: true + + - task: Bash@3 + name: stageReports + displayName: Collect reports without raw scan state + condition: and(succeededOrFailed(), ne(variables['scanExitCode'], 'not-started')) + inputs: + targetType: inline + script: | + set -euo pipefail + for report in report.md coverage.json findings.json; do + if [[ -f "$SCAN_DIRECTORY/$report" ]]; then + cp "$SCAN_DIRECTORY/$report" "$REPORT_DIRECTORY/$report" + fi + done + env: + SCAN_DIRECTORY: $(scanDirectory) + REPORT_DIRECTORY: $(reportDirectory) + + - task: PublishPipelineArtifact@1 + displayName: Retain scan reports + condition: and(succeededOrFailed(), ne(variables['scanExitCode'], 'not-started')) + inputs: + targetPath: $(reportDirectory) + artifact: codex-security diff --git a/sdk/typescript/tests-ts/azure-pipelines-example.test.ts b/sdk/typescript/tests-ts/azure-pipelines-example.test.ts new file mode 100644 index 000000000..d1c53deb4 --- /dev/null +++ b/sdk/typescript/tests-ts/azure-pipelines-example.test.ts @@ -0,0 +1,231 @@ +import { spawnSync } from "node:child_process"; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, test } from "bun:test"; +import { parse } from "yaml"; + +interface Step { + name?: string; + task?: string; + checkout?: string; + condition?: string; + fetchDepth?: number; + persistCredentials?: boolean; + env?: Record; + inputs?: Record; +} + +const pipeline = parse( + readFileSync( + new URL( + "../../../examples/azure-pipelines/azure-pipelines.yml", + import.meta.url, + ), + "utf8", + ), +) as { + trigger: string; + resources: { repositories: { repository: string; trigger: string }[] }; + jobs: { + variables: Record; + steps: (Step | Record)[]; + }[]; +}; +const job = pipeline.jobs[0]!; +const steps = job.steps.flatMap((step) => + step.task || step.checkout ? [step] : Object.values(step).flat(), +) as Step[]; +const scan = steps.find((step) => step.name === "runScan")!; +const sarif = steps.find((step) => step.name === "exportSarif")!; +const temporaryDirectories: string[] = []; + +afterEach(() => { + for (const directory of temporaryDirectories.splice(0)) { + rmSync(directory, { recursive: true, force: true }); + } +}); + +function runStep( + step: Step, + overrides: Record = {}, + scanFiles: string[] = [], +) { + const directory = mkdtempSync(join(tmpdir(), "codex azure example ")); + temporaryDirectories.push(directory); + for (const child of ["repository with spaces", "scan", "reports"]) { + mkdirSync(join(directory, child)); + } + for (const file of scanFiles) { + writeFileSync(join(directory, "scan", file), "synthetic fixture"); + } + // Git Bash accepts forward-slash drive paths on Windows. + const root = directory.replaceAll("\\", "/"); + const bash = + process.platform === "win32" + ? join( + process.env["ProgramFiles"] ?? "C:/Program Files", + "Git/bin/bash.exe", + ) + : "bash"; + const result = spawnSync( + bash, + [ + "--noprofile", + "--norc", + "-c", + `codex-security() { + printf '%s\\0' "$@" > "$ARGUMENTS_PATH" + printf '{"mock":true}\\n' + return "$MOCK_EXIT_CODE" +} +${step.inputs?.["inlineScript"] ?? step.inputs?.["script"]}`, + ], + { + cwd: directory, + encoding: "utf8", + env: { + PATH: process.env["PATH"], + SYSTEMROOT: process.env["SYSTEMROOT"], + ARGUMENTS_PATH: `${root}/arguments`, + TARGET_DIRECTORY: `${root}/repository with spaces`, + SCAN_DIRECTORY: `${root}/scan`, + REPORT_DIRECTORY: `${root}/reports`, + BEDROCK_MODEL_ID: "example.model", + SCAN_MODE: "full", + BASE_REVISION: "HEAD^", + FAIL_ON_SEVERITY: "none", + MOCK_EXIT_CODE: "0", + ...overrides, + }, + }, + ); + if (result.error) throw result.error; + const argumentsPath = join(directory, "arguments"); + const args = existsSync(argumentsPath) + ? readFileSync(argumentsPath, "utf8").split("\0").slice(0, -1) + : []; + return { ...result, directory, root, args }; +} + +test("runs a report-only full scan outside the target checkout", () => { + const result = runStep(scan); + expect(result.status).toBe(0); + expect(result.args).toEqual([ + "scan", + `${result.root}/repository with spaces`, + "--provider", + "amazon-bedrock", + "--model", + "example.model", + "--mode", + "standard", + "--effort", + "high", + "--output-dir", + `${result.root}/scan`, + "--json", + ]); +}); + +test("passes a diff revision literally, without evaluating shell syntax", () => { + const base = "revision with spaces; $(exit 99)"; + const result = runStep(scan, { SCAN_MODE: "diff", BASE_REVISION: base }); + expect(result.status).toBe(0); + expect(result.args.slice(-2)).toEqual(["--diff", base]); +}); + +for (const status of [0, 1, 2, 130, 143]) { + test(`preserves scan exit ${status} and records it for subsequent tasks`, () => { + const result = runStep(scan, { + MOCK_EXIT_CODE: String(status), + FAIL_ON_SEVERITY: "high", + }); + expect(result.status).toBe(status); + expect(result.args.slice(-2)).toEqual(["--fail-on-severity", "high"]); + expect(result.stdout).toBe( + `##vso[task.setvariable variable=scanExitCode]${status}\n`, + ); + expect( + JSON.parse( + readFileSync(join(result.directory, "reports/result.json"), "utf8"), + ), + ).toEqual({ mock: true }); + }); +} + +test("exports fingerprints and only signals readiness after a successful export", () => { + const result = runStep(sarif); + expect(result.status).toBe(0); + expect(result.args).toEqual([ + "export", + `${result.root}/scan`, + "--export-format", + "sarif", + "--source-root", + `${result.root}/repository with spaces`, + "--output", + `${result.root}/reports/results.sarif`, + ]); + expect(result.stdout).toContain( + "##vso[task.setvariable variable=sarifReady]true\n", + ); + const failed = runStep(sarif, { MOCK_EXIT_CODE: "2" }); + expect(failed.status).toBe(2); + expect(failed.stdout).not.toContain("variable=sarifReady"); + expect(sarif.condition).toBe( + "and(succeededOrFailed(), in(variables['scanExitCode'], '0', '1'))", + ); + const publisher = steps.find( + (step) => step.task === "AdvancedSecurity-Publish@1", + )!; + expect(publisher.condition).toBe( + "and(succeededOrFailed(), eq(variables['sarifReady'], 'true'))", + ); +}); + +test("retains selected reports without copying raw state or credentials", () => { + const collect = steps.find((step) => step.name === "stageReports")!; + const reports = ["coverage.json", "findings.json", "report.md"]; + const result = runStep(collect, {}, [ + ...reports, + "auth.json", + "transcript.jsonl", + "workbench.sqlite3", + ]); + expect(result.status).toBe(0); + expect(readdirSync(join(result.directory, "reports")).sort()).toEqual( + reports, + ); + expect(runStep(collect).status).toBe(0); +}); + +test("uses the explicit target resource and keeps setup separate from scanning", () => { + expect(pipeline.trigger).toBe("none"); + expect(pipeline.resources.repositories).toMatchObject([ + { repository: "target", trigger: "none" }, + ]); + expect(job.variables["advancedsecurity.publish.repository"]).toBe( + "$[ convertToJson(resources.repositories['target']) ]", + ); + const checkout = steps.findIndex((step) => step.checkout === "target"); + expect(steps.findIndex((step) => step.name === "installCli")).toBeLessThan( + checkout, + ); + expect(steps[checkout]).toMatchObject({ + fetchDepth: 0, + persistCredentials: false, + }); + expect(scan.inputs?.["disableAutoCwd"]).toBe(true); + for (const step of steps) { + expect(step.env ?? {}).not.toHaveProperty("SYSTEM_ACCESSTOKEN"); + } +});