[backport] Auto-create backport PRs, fix changelog links, and sync changelogs to main by mrodm · Pull Request #20197 · elastic/integrations

mrodm · 2026-07-17T16:12:00Z

Proposed commit message

This PR implements the automation described in #19214 (and related meta-issue #19016). It adds three tightly-coupled features:

1. Auto-backport PR creation (auto-backport.yml)

A new auto-backport.yml workflow fires on push to main and on issue_comment: edited. When a PR merges into main:

It reads the backport checklist comment on the merged PR.
For each checked branch (- [x] backport-*), it calls mage applyBackport to cherry-pick the fix onto a working branch and open a backport PR.
The checklist comment is updated in real time (✅ / ⚠️) as each branch completes.
When a user edits the checklist (checking a previously-unchecked branch), the same workflow detects the change and creates the missing backport PR.
Early-exit guards: skips bot-authored comment edits, non-PR issue comments, and non-merged PRs, to avoid spurious API calls.

2. Changelog link sentinel + post-creation fix (apply.go, update.go)

Cherry-picked changelog entries previously copied the source PR's link verbatim, pointing readers at the wrong PR. This is fixed by:

Injecting a sentinel URL (https://github.com/<repo>/pull/REPLACE_ME) for every link: field when Apply() writes the changelog entry.
After the backport PR is opened, fixChangelogLink() replaces the sentinel with the real backport PR URL, stages and commits the change as a second commit ("Fix changelog link to backport PR"), and pushes — no force-push required.
A new changelog.UpdateEntryLinks(path, version, newURL) helper handles the line-level rewrite.

3. Sync-changelog-to-main workflow (sync-backport-changelog.yml)

When a backport PR is merged into a backport-* branch, a new sync-backport-changelog.yml workflow fires:

It detects the new changelog entry (version + entry block) using mage collectChangelog.
It opens a PR against main adding that entry, so package changelogs on the main branch stay up to date.
Retries the GitHub API call that resolves the backport PR number (with exponential backoff: 0 s, 5 s, 15 s, 30 s) to handle propagation delay after merge.

4. Buildkite check for duplicate changelog versions (check_changelog_versions_in_main.sh)

A new Buildkite step runs on PRs targeting backport-* branches and fails if a changelog version introduced by the PR already exists in main. This catches sync collisions early. Implemented via:

changelog.CheckVersionsAgainstMain(git, before, after) in collect.go
mage checkChangelogVersionsInMain <baseBranch> target in magefile.go

5. Backport checklist improvements (checklist.go)

BuildComment now renders - [ ] / - [x] checkboxes (not plain list items) and updated intro text explaining that PRs are created automatically.
New ParseChecklistItems and UpdateBranchStatus helpers maintain per-branch status without disturbing the rest of the comment body.
post-backport-checklist.yml recreates (delete + POST) the checklist comment on each push, ensuring authors always receive a notification.

Other fixes:

Apply() redirects git stdout to stderr so the JSON result stays clean.
Restore original branch after Apply() succeeds.
mage ApplyBackport *string params now accept positional args correctly.
backport_apply.sh is now executable.
Various early-exit guards in workflow jobs to skip redundant runs.

Merge commit message

[backport] Auto-create backport PRs, fix changelog links, and sync changelogs to main

Add three automation workflows for the backport pipeline:

- auto-backport.yml: creates backport PRs when a PR merges into main with checked
  backport branches; also handles manually checked entries via issue_comment edits.
- sync-backport-changelog.yml: fix API propagation delay when resolving the backport
  PR number (retry with exponential backoff) and correct working-branch base mismatch.
- post-backport-checklist.yml: the checklist comment is now deleted and recreated on
  every push, ensuring authors receive a notification each time.
- Buildkite check: catches changelog version collisions early on PRs targeting
  backport-* branches.

Changelog entries in backport PRs now use a sentinel link that is replaced
automatically with the real backport PR URL after creation.

Author's Checklist

Remove debug commit 9b3c252501 ("Skip trigger integration tests - to be removed") which disables the integration test trigger step in .buildkite/pipeline.yml
Remove debug commit ad8e1cf7d6 ("Tests for debugging in fork") which adds a test branch trigger to auto-backport.yml and changes the sync.go checkout base to add_workflows_backport_prs-test instead of main
Remove debug commit 9d376f8131 ("Add backport entry") which adds a temporary entry to .backports.yml used for testing
Verify auto-backport.yml fires correctly on both push and issue_comment edited events
Verify the checklist comment is recreated (not just patched) on each push
Verify the sentinel link is replaced by the real backport PR URL in a second commit
Verify sync-backport-changelog.yml opens a PR to main after a backport branch merge
Verify the Buildkite check-changelog-versions-in-main step blocks on duplicate versions

How to test this PR locally

# Build and run unit tests for the changed packages
go test ./dev/backports/...

# Test the changelog link rewrite locally (dry-run)
mage applyBackport <sha> <package> <target-branch> true packages elastic/integrations .

# Test the changelog version check
mage checkChangelogVersionsInMain <base-branch>

The full end-to-end flow has been validated in a personal fork. The following three PRs demonstrate each stage of the automation:

PR adding a change to the default branch (triggers auto-backport.yml on merge):
Test kubernets package update - including owners script mrodm/integrations#53
Backport PR created automatically after merging the above (created by auto-backport.yml, changelog link fixed to point at this PR):
[kubernetes] Backport Test backport (1.83.4) mrodm/integrations#54
Sync-changelog PR updating the default branch changelog (created automatically by sync-backport-changelog.yml after the backport PR was merged):
changelog: kubernetes 1.83.4 (backport sync from PR #54) mrodm/integrations#55

Related issues

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

Inject a sentinel URL (https://github.com/<repo>/pull/REPLACE_ME) into every changelog link field of a backport entry, instead of copying the original PR link. The sentinel is replaced with the actual backport PR URL in a follow-up commit once the PR is opened. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ersion Adds UpdateEntryLinks(path, version, newURL) which replaces every link: value in a given changelog version entry with newURL, leaving all other entries untouched. Used to replace the sentinel URL with the real backport PR URL once the PR is opened. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add fixChangelogLink to apply.go, which runs after maybeOpenPR() returns the real PR URL. It calls UpdateEntryLinks to replace the sentinel, then commits "Fix changelog link to backport PR" and pushes as a second commit on the working branch. Failures are non-fatal: a warning is logged to stderr and Apply() still returns success with the PRURL populated. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

BuildComment now renders `- [ ] \`branch\`` / `- [x] \`branch\`` using the checked map instead of plain list items. Intro text updated to: "PRs will be created automatically on merge, or when you update this checklist after merge." Tests updated to cover unchecked/checked rendering and the new sentence. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

New ChecklistItem struct (Package, Branch, Checked, Processed) and ParseChecklistItems(body string) []ChecklistItem in checklist.go. Parses **package** section headers and checkbox branch lines; sets Processed=true when a ✅ or ⚠️ status suffix is present. 10 test cases covering: empty/marker-free body, checked/unchecked, success/conflict status markers, maintained_until suffix combinations, and multiple packages with multiple branches. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

New pure function UpdateBranchStatus(body, branch, status string) string in checklist.go. Appends " — <status>" to the matching checkbox branch line; no-ops if the line already carries a ✅ or ⚠️ suffix (defence-in- depth against workflow self-trigger loops). 7 test cases covering: success/conflict status appended, no-op on existing ✅/⚠️ suffix, branch not found, only matching line updated, and maintained_until suffix preserved. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… mage targets ParseBackportChecklist reads a checklist comment body from a file and prints a JSON array of ChecklistItem to stdout. UpdateChecklistBranchStatus reads a body from a file, appends a status suffix to the given branch line, and prints the updated body to stdout. Both are used by the upcoming auto-backport.yml workflow. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Triggers on push to main and issue_comment edited events. For each merged PR that has a backport checklist, creates backport PRs for every checked, unprocessed branch. Updates the checklist comment with ✅ #PR or ⚠️ conflict status after each branch. Key behaviours: - Job-level actor guard breaks the self-trigger loop from bot comment edits - Permission check (write/admin) gates issue_comment runs before checkout - backport:sync-changelog PRs are silently skipped - Dedup via working-branch name prevents duplicate PRs on re-runs - Checklist patched per-branch so partial progress survives cancellation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add a lightweight pr-check step (just gh pr view --json state) that runs before checkout for issue_comment events. Checkout, Go setup, mage install, and resolve now gate on pr-check.outputs.merged so unmerged-PR comment edits exit without spinning up the full toolchain. The redundant state check inside the resolve step is removed since pr-check already guarantees the PR is merged at that point. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Resolve merge SHA and PR metadata now runs before checkout, Go setup, and mage install. All three gate on steps.resolve.outputs.pr_number so push events with no matching PR or a backport:sync-changelog label exit after a single fast API call without cloning the repository. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

After a successful non-dry-run Apply(), the working tree was left on the auto-backport working branch (based on the target backport branch), not on the branch that was active when Apply() started. The auto-backport workflow calls mage targets (updateChecklistBranchStatus) after backport_apply.sh returns; those targets only exist on the calling branch, so the "Unknown target specified" errors seen in CI were caused by this missing restore. The error path already ran `git checkout -` via the deferred cleanup — this commit adds the same restore to the success path, just before the `success = true` that skips the defer. Also adds TestApplyIntegration_RestoresBranchAfterSuccess to cover this. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ut clean gitutil.Git.Run() wires cmd.Stdout to os.Stdout. When backport_apply.sh calls `exec mage ApplyBackport --json`, mage's os.Stdout is captured by the shell's $() subshell into $RESULT. Every a.git.Run("commit", ...) inside Apply() therefore mixed git commit summaries into $RESULT, making it invalid JSON. jq then failed to parse the status field, causing the workflow to always treat a successful backport as a conflict (⚠️ instead of ✅). Fix: add gitutil.Git.RunToStderr(), which redirects both stdout and stderr to os.Stderr. Switch all 16 a.git.Run() calls in apply.go to RunToStderr() so that git informational output never reaches the JSON capture buffer. The original Run() is untouched and still used by other callers (sync.go, etc.) where stdout contamination is not a concern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…Backport mage *string parameters require -name=value flag syntax, not positional arguments. backport_apply.sh was passing remote, repository and packagesDir as positional args after the bool flags, so mage never set them (they stayed nil) and treated the leftover tokens as extra targets to run — producing "Unknown target specified: """ errors and causing repository to default to "elastic/integrations" regardless of the actual repo. Fix: add all three *string params to the flags array using -name=value format, only when non-empty (omitting the flag leaves the param nil in mage, which Apply() handles as "use default"). Also update the ApplyBackport doc comment to document all optional flags with their correct format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Switch post-backport-checklist.yml from PATCH (edit in place) to DELETE + POST so the checklist always floats to the bottom of the PR thread and notifies subscribers on each update. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The commits/{sha}/pulls API may not immediately associate a merge commit with its PR after a push event fires. Add a retry loop with delays of 0s, 5s, 15s, and 30s before giving up so transient API propagation lag no longer causes the sync-changelog step to skip. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add a PR check that fails when a backport PR introduces a changelog version that already exists in origin/main, catching duplicate entries before merge. Only runs on PRs targeting backport-* branches where **/changelog.yml changed. - CheckVersionsAgainstMain in collect.go: diffs the PR's changes against its base branch and checks each new version against main - CheckChangelogVersionsInMain mage target: takes the base branch as a parameter (passed from $BUILDKITE_PULL_REQUEST_BASE_BRANCH) - check_changelog_versions_in_main.sh: Buildkite script wrapper - pipeline.yml: new step with PR + backport-* base branch guards - collect_test.go: 3 integration tests (no conflict, conflict, no changelog changes) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Exit code 1 (hard fail) is now reserved for changelog entries containing the REPLACE_ME placeholder, which indicates the auto-fix step in Apply() failed to push the corrected link. Regular link mismatches (wrong but real PR URL) exit with code 2 (soft fail). The pipeline step is updated to only soft-fail on exit 2, so a REPLACE_ME link blocks the backport PR from merging instead of just annotating it. Generated with Claude Sonnet 4.6

When a push event and an issue_comment event fire at nearly the same time (e.g. a checklist item is checked just before the PR is merged), both runs pass the dedup guard simultaneously and race to push the working branch. The losing run's push is rejected, which previously surfaced as a false ⚠️ conflict on the checklist. The failure path now does a recovery lookup: if a PR already exists for the working branch, the run posts ✅ with that PR number instead of ⚠️. Generated with Claude Sonnet 4.6

Moves the inline shell script from the "Process checked branches" step in auto-backport.yml into .github/scripts/backport/process-checked-branches.sh. The workflow step is reduced to a single run: call. No behaviour change. Also adds ^\.github/scripts/ to non_package_patterns.txt so changes to this new directory do not trigger package tests. Generated with Claude Sonnet 4.6

The dedup check was extracting the version with grep -oP '\d+\.\d+$', which fails for branches like backport-kubernetes-1.x (VERSION becomes empty, producing auto-backport/kubernetes--<sha>). Now mirrors workingBranchName in apply.go exactly: strings.TrimPrefix(branchName, "backport-"+pkg+"-") using bash prefix stripping: ${BRANCH#backport-${PKG}-} This correctly handles both numeric (6.14) and non-numeric (1.x) suffixes. Generated with Claude Sonnet 4.6

Generated with Claude Sonnet 4.6

…ly changes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm · 2026-07-22T15:25:39Z

Latest tests run in my own fork:

Change targeting default branch: Test updating kubernetes package - changes on collect mrodm/integrations#63
PR created in the backport branch selected: [kubernetes] Backport Test backport (1.83.5) mrodm/integrations#64
Sync of the changelog entry in default branch: changelog: kubernetes 1.83.5 (backport sync from PR #64) mrodm/integrations#65

The default branch has been changed in my own fork for debugging purposes.

mrodm · 2026-07-22T15:32:21Z

Hi @elastic/observablt-ci ,

adding you as a reviewers in case there is anything that should be changed or improved regarding the new Github Action workflows introduced here (permissions, steps, concurrency groups...) to automate backport processes or if you find anything else.

Thanks in advance!

mallendem · 2026-07-27T13:39:14Z

      - step: "check-buildkite-scripts"
        allow_failure: false
+    if: |
+      build.branch == "nonexisting"


I guess this is a leftover?

Yes, I'll remove it before merging to avoid triggering a new step for each package defined in the repo (460 or more steps).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

teresaromero

🚀

mergify · 2026-07-28T08:20:45Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

infra-vault-gh-plugin-prod · 2026-07-28T14:27:00Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: dbdb669

Failed CI Steps

History

💚 Build #46692 succeeded 47285b3
💚 Build #46688 succeeded f3b5a1e
💚 Build #46468 succeeded 01ff583
💚 Build #46455 succeeded 9428899
💚 Build #46454 succeeded 24b1355

cc @mrodm

elastic-vault-github-plugin-prod · 2026-07-28T17:32:46Z

🚀 Benchmarks report

Package `1password` 👍(1) 💚(1) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`item_usages`	15873.02	7874.02	-7999 (-50.39%)	💔

Package `akamai` 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`siem`	1277.14	940.73	-336.41 (-26.34%)	💔

Package `atlassian_jira` 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`audit`	3401.36	2617.8	-783.56 (-23.04%)	💔

Package `mimecast` 👍(4) 💚(3) 💔(4)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`ttp_ip_logs`	16393.44	12048.19	-4345.25 (-26.51%)	💔
`dlp_logs`	8620.69	5494.51	-3126.18 (-36.26%)	💔
`siem_logs`	3759.4	2941.18	-818.22 (-21.76%)	💔
`threat_intel_malware_grid`	7407.41	6211.18	-1196.23 (-16.15%)	💔

To see the full report comment with /test benchmark fullreport

…ckages Update docs to reflect the now-merged auto-backport automation (PR elastic#20197) and upcoming skip_checklist_packages field (PR elastic#20384): - Replace "Coming soon" note with description of auto-backport.yml: on merge, the workflow reads checked branches from the checklist and creates backport PRs automatically; post-merge checklist edits also trigger it. - Update checklist comment section: describe checkboxes and auto-backport trigger, update example to show [ ] checkbox format and new intro text, and note that the comment is now recreated (delete+POST) on each push. - Add "Suppressing a package from the checklist" block documenting the new top-level skip_checklist_packages field (elastic#20384). - ci_pipelines.md: remove stale "coming soon" qualifier from the check-changelog-versions-in-main step — it shipped with elastic#20197. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm and others added 28 commits July 15, 2026 17:01

Add backport entry

9d376f8

Tests for debugging in fork

ad8e1cf

Make backport_apply.sh executable

f3dd5fd

[backport] Skip job early when edited comment is not the checklist

e5e9a65

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

[backport] Log before PATCH and silence gh api response

88375f1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Update sync to avoid showing git output in stdout

98690cb

Create step for git credentials

a681dbd

Add error messages in sync process

193aacd

Update error message

f1ccbd7

Update if condition in workflow

f41dce8

Skip trigger integration tests - to be removed

9b3c252

Merge upstream/main into add_workflows_backport_prs

12bdefe

mrodm self-assigned this Jul 17, 2026

mrodm commented Jul 17, 2026

View reviewed changes

Comment thread dev/backports/changelog/sync.go Resolved

mrodm and others added 7 commits July 21, 2026 12:12

Reorder early-exit checks: merged before actor permission

24b1355

Generated with Claude Sonnet 4.6

Remove debug branch references for add_workflows_backport_prs-test

9428899

Use three-dot diff in changedChangelogs and gitDiff to scope to PR-on…

01ff583

…ly changes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm marked this pull request as ready for review July 22, 2026 15:29

mrodm requested a review from a team as a code owner July 22, 2026 15:29

mrodm requested review from a team and teresaromero July 22, 2026 15:32

mrodm mentioned this pull request Jul 24, 2026

[Docs][Backports] Update developer documentation to reflect new automated backport workflow (#19284) #20351

Closed

1 task done

mallendem reviewed Jul 27, 2026

View reviewed changes

mrodm and others added 2 commits July 27, 2026 16:41

Add check_changelog_versions_in_main.sh to non_package_patterns.txt

f3b5a1e

Add error message before sentinel exit in check_changelog_entries.sh

47285b3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm requested a review from mallendem July 27, 2026 16:01

teresaromero approved these changes Jul 28, 2026

View reviewed changes

mallendem approved these changes Jul 28, 2026

View reviewed changes

Revert skip triggering integration tests

dbdb669

mrodm mentioned this pull request Jul 28, 2026

Skip security_detection_engine from backport checklist comment #20384

Merged

2 tasks done

mrodm merged commit 23a9590 into elastic:main Jul 29, 2026
8 checks passed

mrodm deleted the add_workflows_backport_prs branch July 29, 2026 08:29

This was referenced Jul 30, 2026

[Docs][Backports] Update developer documentation to reflect new automated backport workflow (#19284) mrodm/integrations#67

Closed

[Docs][Backports] Update developer documentation to reflect new automated backport workflow #20413

Merged

Navigation Menu

Repository navigation

[backport] Auto-create backport PRs, fix changelog links, and sync changelogs to main - #20197

[backport] Auto-create backport PRs, fix changelog links, and sync changelogs to main#20197
mrodm merged 41 commits into
elastic:mainfrom
mrodm:add_workflows_backport_prs

mrodm commented Jul 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

mrodm commented Jul 22, 2026

Uh oh!

mrodm commented Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mallendem Jul 27, 2026

Uh oh!

mrodm Jul 27, 2026

Uh oh!

Uh oh!

teresaromero left a comment

Uh oh!

mergify Bot commented Jul 28, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mrodm commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Merge commit message

Author's Checklist

How to test this PR locally

Related issues

Uh oh!

Uh oh!

mrodm commented Jul 22, 2026

Uh oh!

mrodm commented Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mallendem Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

mrodm Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

teresaromero left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jul 28, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⏳ Build in-progress, with failures

Failed CI Steps

History

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 28, 2026

🚀 Benchmarks report

Package 1password 👍(1) 💚(1) 💔(1)

Package akamai 👍(0) 💚(0) 💔(1)

Package atlassian_jira 👍(0) 💚(0) 💔(1)

Package mimecast 👍(4) 💚(3) 💔(4)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented Jul 17, 2026 •

edited

Loading

infra-vault-gh-plugin-prod Bot commented Jul 28, 2026 •

edited

Loading

Package `1password` 👍(1) 💚(1) 💔(1)

Package `akamai` 👍(0) 💚(0) 💔(1)

Package `atlassian_jira` 👍(0) 💚(0) 💔(1)

Package `mimecast` 👍(4) 💚(3) 💔(4)