[CI][Backports] add integrations-backport-dispatch private pipeline for triggering backport creation by mrodm · Pull Request #20284 · elastic/integrations

mrodm · 2026-07-22T14:00:11Z

Proposed commit message

[CI][Backports] add integrations-backport-dispatch private pipeline

Move backport branch creation trigger from the public integrations pipeline
to a new private integrations-backport-dispatch pipeline, preventing failures
caused by public-to-private pipeline trigger restrictions in Buildkite.

The public integrations pipeline can fail to trigger the private
integrations-backport pipeline in some scenarios due to public→private
pipeline restrictions in Buildkite. This PR introduces a new dedicated
private pipeline (integrations-backport-dispatch) that reliably triggers
integrations-backport on merges to main.

WHAT:

New .buildkite/pipeline.backport-dispatch.yml: private pipeline that
runs check-backports-inventory and trigger-backport-create on pushes
to main when .backports.yml changes.
pipeline.yml: removed the trigger-backport-create step (moved to
dispatch pipeline); narrowed check-backports-inventory to PR-only since
the main-push case is now owned by the dispatch pipeline.
pipeline.backport.yml: added integrations-backport-dispatch as a
valid trigger source in the check-ui guard; added failure email
notifications scoped to automated runs from the dispatch pipeline.
trigger_backport.sh: added a safeguard that exits 1 if backport branch
creation is attempted from any pipeline other than
integrations-backport-dispatch.
catalog-info.yaml: registered the new pipeline as a Backstage resource.
pull-requests.json: added the new pipeline entry and
pipeline.backport-dispatch.yml to skip_ci_on_only_changed.
non_package_patterns.txt: added pipeline.backport-dispatch.yml so
changes to it alone do not trigger package integration tests.

WHY:
Buildkite restricts public pipelines from triggering private pipelines in
certain scenarios (e.g. fork builds, specific webhook configurations). By
moving the backport branch creation trigger to a private pipeline, we
eliminate that failure mode while keeping the dry-run validation in the
public integrations pipeline for PR builds.

An example of the failure this addresses can be seen in:
https://buildkite.com/elastic/integrations/builds/46280

Author's Checklist

Revert commit 0c3d9a6f47 ("Skip integrations tests - to be removed") before merging — it was added for testing purposes only.

Related issues

Closes backport: make branch creation PR-driven via .backports.yml #19211
Follow-up from [CI] Add automated backport changelog sync to main #19273

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

…ing backport creation The public integrations pipeline can fail to trigger the private integrations-backport pipeline in some scenarios due to public→private pipeline restrictions in Buildkite. Move the backport branch creation trigger to a new dedicated private pipeline (integrations-backport-dispatch) that reliably triggers integrations-backport on merges to main. Changes: - Add .buildkite/pipeline.backport-dispatch.yml with inventory validation and backport creation trigger steps - Remove trigger-backport-create step from pipeline.yml; narrow check-backports-inventory to PR-only in that pipeline - Allow integrations-backport-dispatch as a valid trigger source in pipeline.backport.yml and add failure email notifications - Add pipeline slug safeguard in trigger_backport.sh so branch creation only runs from integrations-backport-dispatch - Register the new pipeline in catalog-info.yaml, pull-requests.json, and non_package_patterns.txt Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm · 2026-07-22T14:08:06Z

+    if: |
+      build.branch == "nonexisting"


To be reverted before merging

I guess this is some leftover?

Yes, I'll remove it before merging to avoid triggering a new step for each package defined in the repo (460 or more steps).

mrodm · 2026-07-22T14:09:52Z

+        if [[ "${BUILDKITE_PIPELINE_SLUG}" != "integrations-backport-dispatch" ]]; then
+            echo "Backport branch creation can only run from the 'integrations-backport-dispatch' pipeline (got: ${BUILDKITE_PIPELINE_SLUG})"
+            exit 1
+        fi


If this pipeline is triggered from main (and therefore it would create a new backport branch), it must be running from the new integrations-backport-dispatch pipeline.

mrodm · 2026-07-22T15:23:50Z

Hi @elastic/observablt-ci , adding you as a reviewers in case there is anything that should be changed or improved regarding the changes in Buildkite pipelines.
Thanks in advance!

mallendem · 2026-07-27T13:25:46Z

+    if: |
+      build.branch == "nonexisting"


I guess this is some leftover?

mallendem · 2026-07-27T13:28:52Z

+        build_tags: false
+        filter_enabled: true
+        filter_condition: >-
+          build.pull_request.id == null || (build.creator.name == 'elasticmachine' && build.pull_request.id != null && build.source == 'api')


If we don't build PRs, build.pull_request.id == null is always true, so the second part is never evaluated.

Just copied from the other catalog-info entries.
I did the same, just in case at some point it is updated to run in Pull Requests as a safe guard, even if it is not supposed to do so.

mallendem · 2026-07-27T13:30:19Z

+        ecosystem:
+          access_level: MANAGE_BUILD_AND_READ
+        everyone:
+          access_level: READ_ONLY


backport pipeline has

everyone: access_level: BUILD_AND_READ

Not against lowering the permissions, double checking to see if it is intended.

It's expected to have that value 👍
This would allow to run this pipeline for any user that merges Pull Requests in the integrations.

Used the same settings as integrations-publish pipeline for this new pipeline.

teresaromero · 2026-07-28T08:31:11Z

            echo "Not on main branch (branch: ${BUILDKITE_BRANCH}), skipping"
            exit 0
        fi
+        if [[ "${BUILDKITE_PIPELINE_SLUG}" != "integrations-backport-dispatch" ]]; then


i wonder if we could keep the value integrations-backport-dispatch in a shared variable across all the places is used so in case the string is changed (probably not) no typo error can occur? i see it hardcoded in a number a places

Most of the usages are in the buildkite pipeline definitions, and there I think it cannot be used environment variables for that.
I will leave it as it is, that pipeline name should not change.

teresaromero

lgtm

mergify · 2026-07-28T08:33:33Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

mrodm · 2026-07-28T11:37:25Z

/test

infra-vault-gh-plugin-prod · 2026-07-28T12:11:27Z

💔 Build Failed

Buildkite Build
Commit: 104d6e0

Failed CI Steps

History

💚 Build #46471 succeeded b9fac20

cc @mrodm

elastic-vault-github-plugin-prod · 2026-07-28T15:02:43Z

🚀 Benchmarks report

Package `azure_frontdoor` 👍(2) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`access`	2518.89	2127.66	-391.23 (-15.53%)	💔

Package `bitdefender` 👍(1) 💚(0) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`push_configuration`	37037.04	24390.24	-12646.8 (-34.15%)	💔
`push_statistics`	76923.08	58823.53	-18099.55 (-23.53%)	💔

Package `jamf_protect` 👍(1) 💚(3) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`telemetry_legacy`	1792.11	1474.93	-317.18 (-17.7%)	💔

Package `mimecast` 👍(5) 💚(5) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`threat_intel_malware_grid`	7407.41	5524.86	-1882.55 (-25.41%)	💔

To see the full report comment with /test benchmark fullreport

mrodm · 2026-07-29T07:35:47Z

Merging since CI failures are unrelated to this change.

mrodm and others added 3 commits July 21, 2026 18:32

Skip integrations tests - to be removed

f8da308

split check-backports-inventory if condition onto two lines

b9fac20

mrodm self-assigned this Jul 22, 2026

mrodm commented Jul 22, 2026

View reviewed changes

mrodm marked this pull request as ready for review July 22, 2026 14:25

mrodm requested a review from a team as a code owner July 22, 2026 14:25

mrodm requested a review from a team July 22, 2026 14:25

mrodm mentioned this pull request Jul 24, 2026

[Docs][Backports] Update developer documentation to reflect new automated backport workflow (#19284) #20351

Closed

1 task done

mallendem reviewed Jul 27, 2026

View reviewed changes

mrodm requested a review from mallendem July 27, 2026 15:55

teresaromero reviewed Jul 28, 2026

View reviewed changes

teresaromero approved these changes Jul 28, 2026

View reviewed changes

mallendem approved these changes Jul 28, 2026

View reviewed changes

Revert skip integrations tests

104d6e0

mrodm merged commit 11a2f7c into elastic:main Jul 29, 2026
8 of 9 checks passed

mrodm deleted the create_new_dispatch_backport_pipeline branch July 29, 2026 07:36

This was referenced Jul 30, 2026

[Docs][Backports] Update developer documentation to reflect new automated backport workflow (#19284) mrodm/integrations#67

Closed

[Docs][Backports] Update developer documentation to reflect new automated backport workflow #20413

Merged

Conversation

mrodm commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Author's Checklist

Related issues

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm commented Jul 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teresaromero left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jul 28, 2026

Uh oh!

mrodm commented Jul 28, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 28, 2026

🚀 Benchmarks report

Package azure_frontdoor 👍(2) 💚(0) 💔(1)

Package bitdefender 👍(1) 💚(0) 💔(2)

Package jamf_protect 👍(1) 💚(3) 💔(1)

Package mimecast 👍(5) 💚(5) 💔(1)

Uh oh!

mrodm commented Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented Jul 22, 2026 •

edited

Loading

infra-vault-gh-plugin-prod Bot commented Jul 28, 2026 •

edited

Loading

Package `azure_frontdoor` 👍(2) 💚(0) 💔(1)

Package `bitdefender` 👍(1) 💚(0) 💔(2)

Package `jamf_protect` 👍(1) 💚(3) 💔(1)

Package `mimecast` 👍(5) 💚(5) 💔(1)