Conversation
…-package v0.79.0 The CI sync (elastic#20542) replaced with_docker_compose (standalone binary installed into BIN_FOLDER) with with_docker_compose_plugin (CLI plugin installed into ~/.docker/cli-plugins). This caused elastic-package v0.79.0, which hardcodes exec.Command("docker-compose", ...), to pick up the system's docker-compose v2.24.7 (bundled with Docker 26.1.2) instead of the pinned version. Docker Compose v2.23+ changed the YAML output of `docker compose config` so that the environment field is serialized as a sequence instead of a map, which elastic-package v0.79.0 cannot unmarshal (map[string]string). Restore with_docker_compose so the pinned standalone binary is placed in BIN_FOLDER (first in PATH) and called before the system version. Call it from test_one_package.sh and test_integrations_with_serverless.sh alongside the existing plugin installation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…TTP errors Without --fail, curl exits 0 even on 4xx responses, saving the HTML error body silently. This caused the docker-compose binary to contain an HTML page, making it fail at execution time. With --fail, HTTP errors produce a non-zero exit code so retry 5 can kick in properly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This parameter was not available until elastic-package v0.96.0
5 tasks
Contributor
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
forwarded |
997.01 | 417.89 | -579.12 (-58.09%) | 💔 |
powershell_operational |
2631.58 | 1901.14 | -730.44 (-27.76%) | 💔 |
sysmon_operational |
2710.03 | 711.24 | -1998.79 (-73.76%) | 💔 |
To see the full report comment with /test benchmark fullreport
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate findings affect workflow execution, shell safety, and backport correctness.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR syncs CI and backport automation from main to the backport branch.
Changes:
- Adds backport CLI, inventory, changelog, ownership, and checklist tooling.
- Updates Buildkite and GitHub automation.
- Updates validation fixtures, scripts, coverage tooling, and dependencies.
File summaries
| File | Change |
|---|---|
tools.go |
Updates development tooling. |
go.mod |
Updates Go dependencies. |
dev/scripts/README.md |
Updates script documentation. |
dev/scripts/get_release_commit.sh |
Adds release commit lookup logic. |
dev/scripts/backport_apply.sh |
Adds the backport application wrapper. |
dev/packagenames/testdata/no_duplicates/p2/manifest.yml |
Package-name validation fixture. |
dev/packagenames/testdata/no_duplicates/p1/manifest.yml |
Package-name validation fixture. |
dev/packagenames/testdata/nested/no_duplicates/technology/p2/manifest.yml |
Nested package validation fixture. |
dev/packagenames/testdata/nested/no_duplicates/technology/p1/manifest.yml |
Nested package validation fixture. |
dev/packagenames/testdata/nested/no_duplicates/p3/manifest.yml |
Nested package validation fixture. |
dev/packagenames/testdata/nested/duplicates/technology/p2/manifest.yml |
Duplicate package validation fixture. |
dev/packagenames/testdata/nested/duplicates/technology/p1/manifest.yml |
Duplicate package validation fixture. |
dev/packagenames/testdata/nested/duplicates/p3/manifest.yml |
Duplicate package validation fixture. |
dev/packagenames/testdata/invalid_manifests/p2/manifest.yml |
Invalid manifest fixture. |
dev/packagenames/testdata/invalid_manifests/p1/manifest.yml |
Invalid manifest fixture. |
dev/packagenames/testdata/duplicates/p2/manifest.yml |
Duplicate package fixture. |
dev/packagenames/testdata/duplicates/p1/manifest.yml |
Duplicate package fixture. |
dev/packagenames/packagenames.go |
Implements package-name validation. |
dev/packagenames/packagenames_test.go |
Tests package-name validation. |
dev/import-beats/variables.go |
Updates import-beats YAML handling. |
dev/import-beats/variables_compact.go |
Updates compact YAML handling. |
dev/import-beats/README.md |
Updates import-beats documentation. |
dev/import-beats/packages.go |
Updates package processing. |
dev/import-beats/fields.go |
Updates field processing. |
dev/import-beats/elasticsearch.go |
Updates Elasticsearch processing. |
dev/gitutil/git.go |
Adds Git utilities. |
dev/coverage/testdata/test-coverage-3.xml |
Coverage merge fixture. |
dev/coverage/testdata/test-coverage-2.xml |
Coverage merge fixture. |
dev/coverage/testdata/test-coverage-1.xml |
Coverage merge fixture. |
dev/coverage/testdata/expected-test-coverage.xml |
Expected coverage output. |
dev/coverage/coverage.go |
Implements coverage merging. |
dev/coverage/coverage_test.go |
Tests coverage merging. |
dev/codeowners/testdata/test_packages/package_1/manifest.yml |
CODEOWNERS validation fixture. |
dev/codeowners/testdata/test_packages/package_1/data_stream/stream_2/.keep |
Data stream fixture. |
dev/codeowners/testdata/test_packages/package_1/data_stream/stream_1/.keep |
Data stream fixture. |
dev/codeowners/testdata/nested_packages/package_top/manifest.yml |
Nested package fixture. |
dev/codeowners/testdata/nested_packages/category/package_nested_2/manifest.yml |
Nested package fixture. |
dev/codeowners/testdata/nested_packages/category/package_nested_1/manifest.yml |
Nested package fixture. |
dev/codeowners/testdata/nested_packages/category/package_nested_1/data_stream/stream_2/.keep |
Nested data stream fixture. |
dev/codeowners/testdata/nested_packages/category/package_nested_1/data_stream/stream_1/.keep |
Nested data stream fixture. |
dev/codeowners/testdata/devexp/manifest.yml |
CODEOWNERS validation fixture. |
dev/codeowners/testdata/CODEOWNERS-streams-valid |
Valid ownership fixture. |
dev/codeowners/testdata/CODEOWNERS-streams-multiple-owners |
Multiple-owner fixture. |
dev/codeowners/testdata/CODEOWNERS-streams-missing-owners |
Missing-owner fixture. |
dev/codeowners/testdata/CODEOWNERS-owners-trailing-slash |
Trailing-slash fixture. |
dev/codeowners/testdata/CODEOWNERS-owners-packages-datastreams |
Package and data stream ownership fixture. |
dev/codeowners/testdata/CODEOWNERS-nested-valid |
Nested ownership fixture. |
dev/codeowners/testdata/CODEOWNERS-nested-streams-valid |
Nested stream ownership fixture. |
dev/codeowners/testdata/CODEOWNERS-nested-streams-missing-owners |
Nested missing-owner fixture. |
dev/codeowners/testdata/CODEOWNERS-nested-missing-owner |
Nested missing-owner fixture. |
dev/codeowners/testdata/CODEOWNERS-nested-category-owner |
Nested category ownership fixture. |
dev/citools/subscription.go |
Adds subscription utilities. |
dev/citools/packages.go |
Adds package utilities. |
dev/citools/packagemanifest.go |
Adds manifest utilities. |
dev/citools/logsdb.go |
Adds LogsDB utilities. |
dev/citools/logsdb_test.go |
Tests LogsDB utilities. |
dev/citools/kibana.go |
Adds Kibana utilities. |
dev/citools/kibana_test.go |
Tests Kibana utilities. |
dev/citools/gomod.go |
Adds Go module utilities. |
dev/citools/gomod_test.go |
Tests Go module utilities. |
cmd/backport/tools.go |
Updates backport tooling. |
cmd/backport/magefile.go |
Adds backport build targets. |
cmd/backport/go.mod |
Defines the standalone backport module. |
cmd/backport/gitutil/git.go |
Adds backport Git utilities. |
cmd/backport/citools/packages.go |
Adds backport package utilities. |
cmd/backport/citools/packagemanifest.go |
Adds backport manifest utilities. |
cmd/backport/backports/packages/detect.go |
Detects changed packages. |
cmd/backport/backports/owners/check.go |
Checks package ownership. |
cmd/backport/backports/owners/check_test.go |
Tests ownership checks. |
cmd/backport/backports/changelog/update.go |
Updates changelogs. |
cmd/backport/backports/changelog/update_test.go |
Tests changelog updates. |
cmd/backport/backports/changelog/sync_test.go |
Tests changelog synchronization. |
cmd/backport/backports/changelog/resolve.go |
Resolves changelog entries. |
cmd/backport/backports/changelog/resolve_test.go |
Tests changelog resolution. |
cmd/backport/backports/changelog/insert.go |
Inserts changelog entries. |
cmd/backport/backports/changelog/insert_test.go |
Tests changelog insertion. |
cmd/backport/backports/changelog/extract.go |
Extracts changelog entries. |
cmd/backport/backports/changelog/extract_test.go |
Tests changelog extraction. |
cmd/backport/backports/changelog/comment.go |
Handles changelog comments. |
cmd/backport/.go-version |
Pins the backport toolchain. |
.gitignore |
Updates ignored files. |
.github/workflows/validate-yaml-dashboards.requirements.txt |
Updates workflow requirements. |
.github/workflows/validate-package-docs.yml |
Validates package documentation. |
.github/workflows/vale-report.yml |
Reports Vale results. |
.github/workflows/vale-lint.yml |
Runs Vale linting. |
.github/workflows/updatecli/values.d/scm.yml |
Updates Updatecli SCM values. |
.github/workflows/updatecli/updatecli.d/sync-packages-to-bug-issue-template.yml |
Syncs package issue templates. |
.github/workflows/updatecli/updatecli.d/bump-latest-9x-snapshot-version.yml |
Updates snapshot versions. |
.github/workflows/updatecli/updatecli.d/bump-latest-8x-snapshot-version.yml |
Updates snapshot versions. |
.github/workflows/updatecli/updatecli.d/bump-latest-7x-version.yml |
Updates release versions. |
.github/workflows/trigger-text-auditor.yml |
Triggers text auditing. |
.github/workflows/trigger-stale-issues.yml |
Triggers stale issue handling. |
.github/workflows/trigger-pr-review.yml |
Triggers PR review automation. |
.github/workflows/trigger-pr-actions-detective.yml |
Triggers PR action detection. |
.github/workflows/trigger-package-tests-security-ml.yml |
Triggers package tests. |
.github/workflows/trigger-newbie-contributor-patrol.yml |
Triggers contributor patrol. |
.github/workflows/trigger-mention-in-pr.yml |
Handles PR mentions. |
.github/workflows/trigger-mention-in-issue.yml |
Handles issue mentions. |
.github/workflows/trigger-issue-triage.yml |
Triggers issue triage. |
.github/workflows/trigger-duplicate-issue-detector.yml |
Detects duplicate issues. |
.github/workflows/trigger-docs-patrol.yml |
Triggers documentation patrol. |
.github/workflows/trigger-bug-hunter.yml |
Triggers bug-hunter automation. |
.github/workflows/trigger-breaking-change-detector.yml |
Detects breaking changes. |
.github/workflows/sweep-ingest-pipeline-safety.yml |
Runs ingest pipeline checks. |
.github/workflows/sweep-httpjson-pagination.yml |
Runs HTTP JSON checks. |
.github/workflows/sweep-field-mapping-conflicts.yml |
Checks field mapping conflicts. |
.github/workflows/sweep-dashboard-data-scope.yml |
Checks dashboard data scope. |
.github/workflows/requires-update.yml |
Handles required updates. |
.github/workflows/pr-buildkite-detective.yml |
Integrates Buildkite detection. |
.github/workflows/notify-package-docs-failure.yml |
Reports documentation failures. |
.github/workflows/docs-preview-cleanup.yml |
Cleans up documentation previews. |
.github/workflows/docs-deploy.yml |
Deploys documentation. |
.github/workflows/docs-build.yml |
Builds documentation. |
.github/workflows/ci-comment.yml |
Handles CI comments. |
.github/workflows/catalog-info.yml |
Publishes catalog information. |
.github/workflows/bump-elastic-stack-version.yml |
Updates Elastic Stack versions. |
.github/workflows/backport-packages-detect.yml |
Detects packages for backporting. |
.github/actions/sync-backport-changelog/action.yml |
Runs changelog synchronization. |
.buildkite/scripts/trigger_backport.sh |
Triggers backport workflows. |
.buildkite/scripts/test_one_package.sh |
Tests an individual package. |
.buildkite/scripts/test_integrations_with_serverless.sh |
Tests serverless integrations. |
.buildkite/scripts/test_helpers.sh |
Provides test helpers. |
.buildkite/scripts/test_check_backport_owners.sh |
Tests ownership checks. |
.buildkite/scripts/run_dev_scripts_tests.sh |
Runs developer script tests. |
.buildkite/scripts/run_buildkite_scripts_tests.sh |
Runs Buildkite script tests. |
.buildkite/scripts/requirements-ci-python-scripts.txt |
Defines CI Python requirements. |
.buildkite/scripts/report_issues.sh |
Reports CI issues. |
.buildkite/scripts/process_benchmarks.sh |
Processes benchmarks. |
.buildkite/scripts/packages/security_detection_engine.sh |
Runs security package checks. |
.buildkite/scripts/packages/crowdstrike.sh |
Runs CrowdStrike package checks. |
.buildkite/scripts/notify_backport_pr.sh |
Notifies backport PRs. |
.buildkite/scripts/non_package_patterns.txt |
Defines non-package patterns. |
.buildkite/scripts/check_sources.sh |
Checks package sources. |
.buildkite/scripts/check_changelog_versions_in_main.sh |
Checks changelog versions. |
.buildkite/scripts/check_backports_inventory.sh |
Checks backport inventory. |
.buildkite/scripts/check_backport_tool.sh |
Checks the backport tool. |
.buildkite/scripts/build_packages.sh |
Builds packages. |
.buildkite/scripts/backport_branch_lib.sh |
Provides backport branch helpers. |
.buildkite/pull-requests.json |
Configures pull request pipelines. |
.buildkite/pipeline.serverless.yml |
Configures serverless pipelines. |
.buildkite/pipeline.schedule-weekly.yml |
Configures weekly scheduled pipelines. |
.buildkite/pipeline.schedule-daily.yml |
Configures daily scheduled pipelines. |
.buildkite/pipeline.publish.yml |
Configures publishing pipelines. |
.buildkite/pipeline.backport.yml |
Configures backport pipelines. |
.buildkite/pipeline.backport-dispatch.yml |
Configures backport dispatch. |
.buildkite/hooks/pre-exit |
Adds the pre-exit hook. |
.buildkite/hooks/post-checkout |
Adds the post-checkout hook. |
Review details
Suppressed comments (9)
cmd/backport/backports/changelog/collect.go:222
- This collects every changelog path, including deletions, but
collectChangelogEntryimmediately requires the current checkout to contain that package'smanifest.yml. Deleting a package (or its changelog) therefore fails the sync workflow even though there is no new entry to backport; restrict the diff to added/modified/renamed changelogs before processing.
cmd/backport/backports/changelog/collect.go:261 - A failure from
git show origin/main:<path>is assumed to mean that the changelog file does not exist onmain, but the same error also occurs whenorigin/mainis missing or the repository is otherwise unavailable. The sync then treats every version as new and can create duplicate entries; distinguish a missing path from other Git errors and propagate the latter.
cmd/backport/backports/changelog/comment.go:107 - Every
gh apifailure is treated as a missing branch, so authentication, permission, rate-limit, and network failures are reported as if the working branch was not pushed. Only a 404 should be converted tofalse; return other errors so the retry/comment path surfaces the real failure.
cmd/backport/backports/inventory.go:210 AddEntryderives a deterministic branch name from package and major/minor but never checks whether that branch already exists in the inventory. Adding another release for an existing series (for example,aws2.5.x whenbackport-aws-2.5is already present) will silently create a duplicate branch entry, leaving branch creation and active-branch lookups ambiguous. Reject an existing branch before inserting the entry.
cmd/backport/backports/packages/detect.go:40- These file names come from
git diff --name-onlyand use/, whilepkgs[i].Pathis built with the platform-specificfilepathseparator. On Windows the prefix check never matches package files, sodetect-packagesandcheck-ownerssilently return no packages; normalize the Git path before comparing.
cmd/backport/main.go:14 - Because
cmd/backportnow has its owngo.mod, running the documented command from the repository root (go build ./cmd/backport/) does not build this nested module. Document the module-local command (for example,go build -C cmd/backport -o build/backport .) so contributors can actually build the CLI as described.
cmd/backport/main.go:321 - The Buildkite step is configured to run when
.github/CODEOWNERSchanges, butdiffPackagesonly maps files underpackages/to package names. A CODEOWNERS-only change therefore produces an emptypkgslist andCheckPackagesreports success without checking the affected package(s), allowing an owner mismatch to bypass this check. Include package paths represented by the CODEOWNERS diff (or check all packages when that file changes).
dev/scripts/README.md:7 - This says
dev/scripts/is listed incommon.sh, but the actual pattern is maintained in.buildkite/scripts/non_package_patterns.txt(line 54);common.shonly reads that file. Point the README at the source-of-truth pattern file so future edits know where to update the CI behavior.
dev/scripts/backport_apply.sh:70 - The documented local fallback builds
cmd/backportwith the caller'sgobinary, but this repository pins the root toolchain to Go 1.19.1 while the standalone module requires Go 1.26 and usesgo build -C. Running this wrapper locally withoutBACKPORT_BINtherefore fails under the repository's normal Go setup instead of building the tool. Use the backport toolchain (or require a prebuilt binary) before invokinggo build.
- Files reviewed: 173/192 changed files
- Comments generated: 16
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+243
to
+250
| echo "--- Copying .github/workflows from $SOURCE_BRANCH..." | ||
| git rm -r --cached ".github/workflows" 2>/dev/null || true | ||
| git checkout "$SOURCE_BRANCH" -- ".github/workflows" | ||
| git add .github/workflows | ||
|
|
||
| echo "--- Copying .github/actions from $SOURCE_BRANCH..." | ||
| git rm -r --cached ".github/actions" 2>/dev/null || true | ||
| git checkout "$SOURCE_BRANCH" -- ".github/actions" |
| PR_AUTHOR: ${{ steps.resolve.outputs.pr_author }} | ||
| COMMENT_ID: ${{ steps.find-comment.outputs.comment_id }} | ||
| BODY_FILE: ${{ steps.find-comment.outputs.body_file }} | ||
| run: .github/scripts/backport/process-checked-branches.sh |
Comment on lines
+73
to
+74
| # Escape quotes and special characters for safe command generation | ||
| CHANGELOG_DESC=$(echo "$CUSTOM_DESC" | sed 's/"/\\"/g' | sed "s/'/\\'/g") |
| # the backport tool can parse previously-checked branches and preserve | ||
| # their state. Prints nothing when no package has any active branch. | ||
| NEW_BODY=$(printf '%s' "$EXISTING_BODY" | \ | ||
| "$GITHUB_WORKSPACE/build/backport" render-checklist "$ARTIFACT") |
| } | ||
|
|
||
| workingBranch := workingBranchName(opts.Package, branchName, sha8) | ||
| if err := a.prepareWorkingBranch(remote, branchName, workingBranch); err != nil { |
Comment on lines
+157
to
+160
| if echo "${stripped}" | grep -qE '^.+-[0-9]+\.[0-9]+(\.[0-9]+)?$'; then | ||
| pkg="$(echo "${stripped}" | sed -E 's/-([0-9]+\.[0-9]+(\.[0-9]+)?)$//')" | ||
| ver_suffix="$(echo "${stripped}" | sed -E 's/^.*-([0-9]+\.[0-9]+(\.[0-9]+)?)$/\1/')" | ||
| fi |
| --remote) remote="$2"; shift 2 ;; | ||
| --repository) repository="$2"; shift 2 ;; | ||
| --packages-dir) packages_dir="$2"; shift 2 ;; | ||
| -h|--help) usage ;; |
|
|
||
| After running, manually review the output and fill in maintained_until dates. | ||
| EOF | ||
| exit 1 |
Comment on lines
+2
to
+3
| # Finds the commit on main where a package version was released. | ||
| # Usage: ./get_release_commit.sh <package_name> <version> |
Comment on lines
+19
to
+22
| h) | ||
| usage | ||
| exit 0 | ||
| ;; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Syncs
.buildkiteCI configuration frommainto thebackport-cloud_security_posture-1.1backport branch.This brings updated CI scripts and pipeline fixes that only work when they are
present on the target branch itself.
What does this PR do?
Backports the following CI configuration changes to the
backport-cloud_security_posture-1.1branch:shellinitcommandwith_docker_composeto pin standalone binary for elastic-package v0.79.0--failto curl inwith_docker_composeto enable retries on HTTP errors--coverage-formatparameterELASTIC_PACKAGE_REPOSITORY_LICENSEvariableProposed commit message
```
Sync CI configuration with main branch for backport-cloud_security_posture-1.1.
Keep CI pipelines and tooling consistent between active backport branches and
`main`, so that automated workflows and tooling improvements land on all
supported branches.
WHY:
Buildkite pipeline scripts are resolved from the branch where the event occurs,
not from `main`. CI fixes and improvements must be present on the backport branch
itself to run correctly.
```
Author's Checklist
main.How to test this PR locally
Verify that Buildkite pipelines run correctly on the
backport-cloud_security_posture-1.1branch after merge.Related issues