Skip to content

[backport-cloud_security_posture-1.0] Sync CI configuration with main branch - #21194

Open
mrodm wants to merge 9 commits into
elastic:backport-cloud_security_posture-1.0from
mrodm:sync-ci-configuration-cloud_security_posture-1.0
Open

[backport-cloud_security_posture-1.0] Sync CI configuration with main branch#21194
mrodm wants to merge 9 commits into
elastic:backport-cloud_security_posture-1.0from
mrodm:sync-ci-configuration-cloud_security_posture-1.0

Conversation

@mrodm

@mrodm mrodm commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

TL;DR

Syncs .buildkite and .github CI configuration from main to the
backport-cloud_security_posture-1.0 backport branch.

This brings automated backport workflows, new CI scripts, Go tooling, and
GitHub Actions that only work when they are present on the target branch itself.

Proposed commit message

```
Sync CI configuration with main branch for backport-cloud_security_posture-1.0.

Keep CI pipelines, GitHub Actions, and tooling consistent between active backport
branches and main, so that automated backport workflows and tooling improvements
land on all supported branches.

WHY:
GitHub Actions resolves workflow files from the branch where the event occurs,
not from main. Backport-branch workflows (changelog sync, post-backport
checklist, owner checks, auto-backport) must be present on the backport branch
itself to fire correctly.
```

Author's Checklist

  • CI configuration files have been synced verbatim from main.

Related issues

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

mrodm and others added 7 commits September 11, 2026 17:04
@mrodm
Sync CI configuration with main branch
c16da9b
@mrodm
Add shellinit command
fe6470d
@mrodm
Restore docker-compose version
864ff7f
@mrodm @claude
fix: restore with_docker_compose to pin standalone binary for elastic…
0e36ae7 
…-package v0.79.0

The CI sync (elastic#20542) replaced with_docker_compose (standalone binary
installed into BIN_FOLDER) with with_docker_compose_plugin (CLI plugin
installed into ~/.docker/cli-plugins). This caused elastic-package
v0.79.0, which hardcodes exec.Command("docker-compose", ...), to pick
up the system's docker-compose v2.24.7 (bundled with Docker 26.1.2)
instead of the pinned version.

Docker Compose v2.23+ changed the YAML output of `docker compose config`
so that the environment field is serialized as a sequence instead of a
map, which elastic-package v0.79.0 cannot unmarshal (map[string]string).

Restore with_docker_compose so the pinned standalone binary is placed
in BIN_FOLDER (first in PATH) and called before the system version.
Call it from test_one_package.sh and test_integrations_with_serverless.sh
alongside the existing plugin installation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@mrodm @claude
fix: add --fail to curl in with_docker_compose to enable retries on H…
45b8365 
…TTP errors

Without --fail, curl exits 0 even on 4xx responses, saving the HTML
error body silently. This caused the docker-compose binary to contain
an HTML page, making it fail at execution time. With --fail, HTTP
errors produce a non-zero exit code so retry 5 can kick in properly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@mrodm
Remove --coverage-format parameter
d916bbb 
This parameter was not available until elastic-package v0.96.0
@mrodm
Test subset of packages
106fef2
@mrodm mrodm self-assigned this Sep 11, 2026
@mrodm mrodm mentioned this pull request Sep 11, 2026
Open
5 tasks
@mrodm
mrodm marked this pull request as ready for review September 14, 2026 08:21
@mrodm
mrodm requested a review from a team as a code owner September 14, 2026 08:21
Copilot AI lite review requested due to automatic review settings September 14, 2026 08:21
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Sep 14, 2026 •
edited
Loading

Copy link
Copy Markdown

💔 Build Failed

Failed CI Steps

History

cc @mrodm

Copilot AI reviewed Sep 14, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR synchronizes CI and backport tooling from main onto the backport branch.

Changes:

  • Adds backport workflows, Buildkite checks, and standalone backport tooling.
  • Updates Go dependencies and YAML tooling.
  • Adds package-name validation and developer script tests.
File summaries
File Description
tools.go Updated as part of this pull request.
go.mod Updated as part of this pull request.
dev/scripts/README.md Updated as part of this pull request.
dev/scripts/get_release_commit.sh Updated as part of this pull request.
dev/scripts/backport_apply.sh Updated as part of this pull request.
dev/packagenames/testdata/no_duplicates/p2/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/no_duplicates/p1/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/nested/no_duplicates/technology/p2/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/nested/no_duplicates/technology/p1/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/nested/no_duplicates/p3/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/nested/duplicates/technology/p2/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/nested/duplicates/technology/p1/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/nested/duplicates/p3/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/invalid_manifests/p2/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/invalid_manifests/p1/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/duplicates/p2/manifest.yml Updated as part of this pull request.
dev/packagenames/testdata/duplicates/p1/manifest.yml Updated as part of this pull request.
dev/packagenames/packagenames.go Updated as part of this pull request.
dev/packagenames/packagenames_test.go Updated as part of this pull request.
dev/import-beats/variables.go Updated as part of this pull request.
dev/import-beats/variables_compact.go Updated as part of this pull request.
dev/import-beats/README.md Updated as part of this pull request.
dev/import-beats/packages.go Updated as part of this pull request.
dev/import-beats/fields.go Updated as part of this pull request.
dev/import-beats/elasticsearch.go Updated as part of this pull request.
dev/gitutil/git.go Updated as part of this pull request.
dev/coverage/testdata/test-coverage-3.xml Updated as part of this pull request.
dev/coverage/testdata/test-coverage-2.xml Updated as part of this pull request.
dev/coverage/testdata/test-coverage-1.xml Updated as part of this pull request.
dev/coverage/testdata/expected-test-coverage.xml Updated as part of this pull request.
dev/coverage/coverage.go Updated as part of this pull request.
dev/coverage/coverage_test.go Updated as part of this pull request.
dev/codeowners/testdata/test_packages/package_1/manifest.yml Updated as part of this pull request.
dev/codeowners/testdata/test_packages/package_1/data_stream/stream_2/.keep Updated as part of this pull request.
dev/codeowners/testdata/test_packages/package_1/data_stream/stream_1/.keep Updated as part of this pull request.
dev/codeowners/testdata/nested_packages/package_top/manifest.yml Updated as part of this pull request.
dev/codeowners/testdata/nested_packages/category/package_nested_2/manifest.yml Updated as part of this pull request.
dev/codeowners/testdata/nested_packages/category/package_nested_1/manifest.yml Updated as part of this pull request.
dev/codeowners/testdata/nested_packages/category/package_nested_1/data_stream/stream_2/.keep Updated as part of this pull request.
dev/codeowners/testdata/nested_packages/category/package_nested_1/data_stream/stream_1/.keep Updated as part of this pull request.
dev/codeowners/testdata/devexp/manifest.yml Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-streams-valid Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-streams-multiple-owners Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-streams-missing-owners Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-owners-trailing-slash Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-owners-packages-datastreams Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-nested-valid Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-nested-streams-valid Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-nested-streams-missing-owners Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-nested-missing-owner Updated as part of this pull request.
dev/codeowners/testdata/CODEOWNERS-nested-category-owner Updated as part of this pull request.
dev/citools/subscription.go Updated as part of this pull request.
dev/citools/packages.go Updated as part of this pull request.
dev/citools/packagemanifest.go Updated as part of this pull request.
dev/citools/logsdb.go Updated as part of this pull request.
dev/citools/logsdb_test.go Updated as part of this pull request.
dev/citools/kibana.go Updated as part of this pull request.
dev/citools/kibana_test.go Updated as part of this pull request.
dev/citools/gomod.go Updated as part of this pull request.
dev/citools/gomod_test.go Updated as part of this pull request.
cmd/backport/tools.go Updated as part of this pull request.
cmd/backport/magefile.go Updated as part of this pull request.
cmd/backport/go.mod Updated as part of this pull request.
cmd/backport/gitutil/git.go Updated as part of this pull request.
cmd/backport/citools/packages.go Updated as part of this pull request.
cmd/backport/citools/packagemanifest.go Updated as part of this pull request.
cmd/backport/backports/packages/detect.go Updated as part of this pull request.
cmd/backport/backports/owners/check.go Updated as part of this pull request.
cmd/backport/backports/owners/check_test.go Updated as part of this pull request.
cmd/backport/backports/changelog/update.go Updated as part of this pull request.
cmd/backport/backports/changelog/update_test.go Updated as part of this pull request.
cmd/backport/backports/changelog/sync_test.go Updated as part of this pull request.
cmd/backport/backports/changelog/resolve.go Updated as part of this pull request.
cmd/backport/backports/changelog/resolve_test.go Updated as part of this pull request.
cmd/backport/backports/changelog/insert.go Updated as part of this pull request.
cmd/backport/backports/changelog/insert_test.go Updated as part of this pull request.
cmd/backport/backports/changelog/extract.go Updated as part of this pull request.
cmd/backport/backports/changelog/extract_test.go Updated as part of this pull request.
cmd/backport/backports/changelog/comment.go Updated as part of this pull request.
cmd/backport/.go-version Updated as part of this pull request.
.gitignore Updated as part of this pull request.
.github/workflows/validate-yaml-dashboards.requirements.txt Updated as part of this pull request.
.github/workflows/validate-package-docs.yml Updated as part of this pull request.
.github/workflows/vale-report.yml Updated as part of this pull request.
.github/workflows/vale-lint.yml Updated as part of this pull request.
.github/workflows/updatecli/values.d/scm.yml Updated as part of this pull request.
.github/workflows/updatecli/updatecli.d/sync-packages-to-bug-issue-template.yml Updated as part of this pull request.
.github/workflows/updatecli/updatecli.d/bump-latest-9x-snapshot-version.yml Updated as part of this pull request.
.github/workflows/updatecli/updatecli.d/bump-latest-8x-snapshot-version.yml Updated as part of this pull request.
.github/workflows/updatecli/updatecli.d/bump-latest-7x-version.yml Updated as part of this pull request.
.github/workflows/trigger-text-auditor.yml Updated as part of this pull request.
.github/workflows/trigger-stale-issues.yml Updated as part of this pull request.
.github/workflows/trigger-pr-review.yml Updated as part of this pull request.
.github/workflows/trigger-pr-actions-detective.yml Updated as part of this pull request.
.github/workflows/trigger-package-tests-security-ml.yml Updated as part of this pull request.
.github/workflows/trigger-newbie-contributor-patrol.yml Updated as part of this pull request.
.github/workflows/trigger-mention-in-pr.yml Updated as part of this pull request.
.github/workflows/trigger-mention-in-issue.yml Updated as part of this pull request.
.github/workflows/trigger-issue-triage.yml Updated as part of this pull request.
.github/workflows/trigger-duplicate-issue-detector.yml Updated as part of this pull request.
.github/workflows/trigger-docs-patrol.yml Updated as part of this pull request.
.github/workflows/trigger-bug-hunter.yml Updated as part of this pull request.
.github/workflows/trigger-breaking-change-detector.yml Updated as part of this pull request.
.github/workflows/sweep-ingest-pipeline-safety.yml Updated as part of this pull request.
.github/workflows/sweep-httpjson-pagination.yml Updated as part of this pull request.
.github/workflows/sweep-field-mapping-conflicts.yml Updated as part of this pull request.
.github/workflows/sweep-dashboard-data-scope.yml Updated as part of this pull request.
.github/workflows/requires-update.yml Updated as part of this pull request.
.github/workflows/pr-buildkite-detective.yml Updated as part of this pull request.
.github/workflows/notify-package-docs-failure.yml Updated as part of this pull request.
.github/workflows/docs-preview-cleanup.yml Updated as part of this pull request.
.github/workflows/docs-deploy.yml Updated as part of this pull request.
.github/workflows/docs-build.yml Updated as part of this pull request.
.github/workflows/ci-comment.yml Updated as part of this pull request.
.github/workflows/catalog-info.yml Updated as part of this pull request.
.github/workflows/bump-elastic-stack-version.yml Updated as part of this pull request.
.github/workflows/backport-packages-detect.yml Updated as part of this pull request.
.github/actions/sync-backport-changelog/action.yml Updated as part of this pull request.
.buildkite/scripts/trigger_backport.sh Updated as part of this pull request.
.buildkite/scripts/test_one_package.sh Updated as part of this pull request.
.buildkite/scripts/test_integrations_with_serverless.sh Updated as part of this pull request.
.buildkite/scripts/test_helpers.sh Updated as part of this pull request.
.buildkite/scripts/test_check_backport_owners.sh Updated as part of this pull request.
.buildkite/scripts/run_dev_scripts_tests.sh Updated as part of this pull request.
.buildkite/scripts/run_buildkite_scripts_tests.sh Updated as part of this pull request.
.buildkite/scripts/requirements-ci-python-scripts.txt Updated as part of this pull request.
.buildkite/scripts/report_issues.sh Updated as part of this pull request.
.buildkite/scripts/process_benchmarks.sh Updated as part of this pull request.
.buildkite/scripts/packages/security_detection_engine.sh Updated as part of this pull request.
.buildkite/scripts/packages/crowdstrike.sh Updated as part of this pull request.
.buildkite/scripts/notify_backport_pr.sh Updated as part of this pull request.
.buildkite/scripts/non_package_patterns.txt Updated as part of this pull request.
.buildkite/scripts/check_sources.sh Updated as part of this pull request.
.buildkite/scripts/check_changelog_versions_in_main.sh Updated as part of this pull request.
.buildkite/scripts/check_backports_inventory.sh Updated as part of this pull request.
.buildkite/scripts/check_backport_tool.sh Updated as part of this pull request.
.buildkite/scripts/build_packages.sh Updated as part of this pull request.
.buildkite/scripts/backport_branch_lib.sh Updated as part of this pull request.
.buildkite/pull-requests.json Updated as part of this pull request.
.buildkite/pipeline.serverless.yml Updated as part of this pull request.
.buildkite/pipeline.schedule-weekly.yml Updated as part of this pull request.
.buildkite/pipeline.schedule-daily.yml Updated as part of this pull request.
.buildkite/pipeline.publish.yml Updated as part of this pull request.
.buildkite/pipeline.backport.yml Updated as part of this pull request.
.buildkite/pipeline.backport-dispatch.yml Updated as part of this pull request.
.buildkite/hooks/pre-exit Updated as part of this pull request.
.buildkite/hooks/post-checkout Updated as part of this pull request.
Review details

Suppressed comments (10)

.buildkite/scripts/backport_branch.sh:172

  • git rm --cached removes these entries only from the index and leaves files that were deleted from main in the worktree. The following git add .buildkite then stages those stale files, so the new backport branch can retain CI files that no longer exist on main instead of being synchronized verbatim. Remove the directory from the worktree (and apply the same change to the analogous dev/.github copies) before checking out main's version.
    .buildkite/scripts/backport_branch.sh:251
  • Even after adding the script referenced by auto-backport.yml, this branch-sync routine would omit it: it checks out .github/workflows and .github/actions but never copies .github/scripts. Newly created backport branches would therefore still have a broken auto-backport workflow; include the script directory in the synchronized paths.
    .github/workflows/post-backport-checklist.yml:82
  • MATCHING is a JSON array, but first.id reads an id field from the array itself rather than the first array element, so this expression yields no comment ID. After the first checklist post, subsequent runs will never update it and will create a duplicate; the duplicate guard then aborts future updates. Index the array explicitly with .[0].id.
    .github/workflows/post-backport-checklist.yml:120
  • The existing checklist comment is deleted before the replacement POST. If the POST fails transiently, the user’s checklist (and its checked-branch state) is lost, and the next run cannot recover it. Update the existing comment with PATCH instead, using POST only when no comment ID exists.
    .github/workflows/post-backport-checklist.yml:107
  • When all active branches disappear, render-checklist returns an empty body, but this early exit leaves the existing checklist comment in place. That stale comment still shows archived branches and can be acted on even though BuildComment says stale sections should disappear on recompute. Delete the existing comment when COMMENT_ID is set before exiting.
    cmd/backport/backports/changelog/collect.go:222
  • A deleted changelog.yml is still returned by this diff, but collectChangelogEntry then reads its sibling manifest from the current worktree and errors when the package/changelog was removed. Exclude deleted paths (or explicitly skip them) so a package removal on a backport branch does not fail the entire sync workflow.
    cmd/backport/backports/packages/detect.go:35
  • Git paths returned by git diff --name-only always use /, while filepath.Separator and the paths from WalkDir use \\ on Windows. On that platform no changed file matches a discovered package, so detection returns an empty list. Normalize discovered paths with filepath.ToSlash before prefix matching.
    cmd/backport/main.go:249
  • The workflow passes the current PR base SHA and head SHA, but this two-dot diff compares their endpoint trees. If main advances after the PR branch is created, it includes files changed only on main, so the checklist can contain unrelated packages and offer incorrect backports. Use a merge-base/three-dot diff for PR-scoped changes, as changedChangelogs already does.
    dev/scripts/README.md:7
  • The README points maintainers to .buildkite/scripts/common.sh, but that file only reads the patterns; the ^dev/scripts/ entry is configured in .buildkite/scripts/non_package_patterns.txt. Update the referenced path so changes to the package-test exclusion are made in the file that actually defines it.
    dev/scripts/backport_apply.sh:70
  • cmd/backport requires Go 1.26.0 and is pinned to 1.26.2, while the repository root is pinned to Go 1.19.1. This fallback build runs the active root toolchain, so on a normal checkout go build -C is unsupported and the module's minimum Go version is unmet; use the pinned backport toolchain (as with_backport does) before building.
  • Files reviewed: 173/192 changed files
  • Comments generated: 6
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .buildkite/scripts/backport_branch_lib.sh
Comment on lines +34 to +35
yq -r '.requires.input[].package' "${manifest}" 2>/dev/null || true
yq -r '.requires.content[].package' "${manifest}" 2>/dev/null || true
Comment thread .github/workflows/auto-backport.yml
PR_AUTHOR: ${{ steps.resolve.outputs.pr_author }}
COMMENT_ID: ${{ steps.find-comment.outputs.comment_id }}
BODY_FILE: ${{ steps.find-comment.outputs.body_file }}
run: .github/scripts/backport/process-checked-branches.sh
Comment thread dev/scripts/backport_apply.sh
--remote) remote="$2"; shift 2 ;;
--repository) repository="$2"; shift 2 ;;
--packages-dir) packages_dir="$2"; shift 2 ;;
-h|--help) usage ;;
Comment thread .buildkite/pipeline.yml
- label: ":busts_in_silhouette: Check backport package owners"
key: "check-backport-owners"
command: ".buildkite/scripts/check_backport_owners.sh"
soft_fail: true
Comment thread cmd/backport/backports/apply/apply.go
}

// branchRE matches valid backport branch names (mirrors cmd/backport/backports/inventory.go).
var branchRE = regexp.MustCompile(`^backport-[a-zA-Z0-9_]+-[0-9][0-9.]*x?$`)
Comment thread dev/scripts/get_release_commit.sh
Comment on lines +20 to +21
usage
exit 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@mrodm mrodm

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mrodm