[CI] Add Buildkite step to validate changelog PR links by mrodm · Pull Request #19171 · elastic/integrations

mrodm · 2026-05-22T13:41:42Z

Proposed commit message

[CI] Add Buildkite step to validate changelog PR links

Add a Buildkite step and script to verify that newly added changelog.yml
entries in a PR point to the correct PR link. Issue links are accepted as-is.
The validation logic is extracted into testable shell functions with a full
test suite. The shell scripts test runner is also refactored into a thin
orchestrator with each script having its own dedicated test file.

WHAT:

Add a new Buildkite CI step that inspects changelog.yml files modified in a pull request and verifies that newly added link: entries point to the current PR URL. The check:

Detects any changelog.yml file changed at any directory depth (supporting both flat and nested package layouts).
Accepts issue links (/issues/<n>) without requiring them to match the PR URL.
Posts a Buildkite annotation when a mismatch is found.
Only runs on PR builds and only when at least one changelog.yml is part of the changed files (if_changed).

The validation logic is extracted into discrete, testable shell functions (github_repo_path, get_added_links, check_changelog_file) with a BASH_SOURCE main guard so the script can be sourced by tests without executing.

A dedicated test file (test_check_changelog_entries.sh) is added covering: SSH/HTTPS URL parsing, correct PR links, wrong PR links, issue link exemption, no new entries, deleted files, and multiple-error accumulation.

As part of this work, the existing run_dev_scripts_tests.sh is refactored into a thin orchestrator (run_shell_scripts_tests.sh) and the get_release_commit.sh tests are moved to their own self-contained file (dev/scripts/test_get_release_commit.sh), following the same pattern.

WHY:

Changelog entries with wrong PR links (e.g. copied from another PR) are a recurring source of noise in the release process. Automating this check at CI time catches the mistake before merge without requiring manual review of every changelog entry.

Author's Checklist

How to test this PR locally

Run the shell script test suite:

.buildkite/scripts/run_shell_scripts_tests.sh

To simulate the changelog link check against a PR branch manually (this is based on the remote origin and it must be updated):

BUILDKITE_PULL_REQUEST=<pr_number> \
BUILDKITE_PULL_REQUEST_BASE_BRANCH=main \
BUILDKITE_REPO=git@github.com:elastic/integrations.git \
.buildkite/scripts/check_changelog_entries.sh

Related issues

Relates Improve package backport tooling for routine multi-version support #19016

Screenshots

Buildkite annotation describing the failures:

GitHub comment with the failures:

Set pipeline as soft_fail:

GitHub comment after running successfully:

Add message about Github label to skip validation:

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

Add a new script `.buildkite/scripts/check_changelog_entries.sh` that inspects any `changelog.yml` files modified in a pull request and verifies that newly added `link:` entries point to the current PR. Issue links (`/issues/<n>`) are accepted as-is. A matching Buildkite step is wired into `.buildkite/pipeline.yml`, gated on PR builds and triggered only when a `changelog.yml` file is changed (`if_changed`). Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Extract the validation logic of check_changelog_entries.sh into discrete, testable functions (github_repo_path, get_added_links, check_changelog_file) guarded by a BASH_SOURCE main guard so the script can be sourced by tests without executing. Fix the common.sh source path to be relative to the script's own directory. Add test_check_changelog_entries.sh covering URL parsing, correct PR links, wrong PR links, issue link exemption, no new entries, deleted files, and multiple-error accumulation. Hook the new test file into run_dev_scripts_tests.sh so it runs in the existing CI step. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Extract all tests for get_release_commit.sh from run_dev_scripts_tests.sh into a dedicated dev/scripts/test_get_release_commit.sh, following the same pattern established for test_check_changelog_entries.sh. The runner run_dev_scripts_tests.sh is now a thin orchestrator that delegates to each test file in sequence. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Rename run_dev_scripts_tests.sh to run_shell_scripts_tests.sh and update the Buildkite step label and key accordingly (check-shell-scripts). Update all references in common.sh and the test file comments. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

elastic-vault-github-plugin-prod · 2026-05-22T13:59:33Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

mrodm · 2026-05-22T14:50:48Z

/test

…iles Refactor notify_changelog_mismatch to accept a combined message and a fixed annotation context, then accumulate all failing changelog files into an array during the loop and call the function once after it with a bulleted list of every affected file instead of one notification per file. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add a depends_on on check-shell-scripts (allow_failure: false) to the check-changelog-pr-links step so it only runs once the shell unit tests have completed successfully. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add edit_gh_pr_comment_if_exists to common.sh next to add_or_edit_gh_pr_comment: it edits an existing PR comment identified by its marker, and does nothing if no matching comment is found. Use it in check_changelog_entries.sh to update the mismatch comment with a resolved message when all changelog links are correct, avoiding a spurious success comment on clean runs. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add delete_and_create_gh_pr_comment to common.sh: it deletes any existing comment with the given marker and posts a fresh one, moving the status to the bottom of the PR thread for visibility. Use it in check_changelog_entries.sh on the success path instead of edit_gh_pr_comment_if_exists, so the resolved message appears as a new comment rather than an in-place edit. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace add_or_edit_gh_pr_comment with delete_and_create_gh_pr_comment in notify_changelog_mismatch so that repeated failures always post a fresh comment at the bottom of the PR thread instead of editing in place. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

infra-vault-gh-plugin-prod · 2026-06-01T12:03:37Z

Pinging @elastic/ecosystem (Team:Ecosystem)

This reverts commit 4c7d767.

Move duplicated assert_equals and assert_exit_code functions from test_check_changelog_entries.sh and test_get_release_commit.sh into a shared test_helpers.sh file to avoid repetition. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

elastic-vault-github-plugin-prod · 2026-06-01T14:35:44Z

Changelog link mismatch — expected https://github.com/elastic/integrations/pull/19171 in the following file(s):

packages/elastic_package_registry/changelog.yml

Tip

Add the changelog-link-check:skip label to skip this check.

View Buildkite build
@mrodm

elasticmachine · 2026-06-01T15:16:07Z

💔 Build Failed

Buildkite Build
Commit: d43517e

Failed CI Steps

Check integrations netbox

History

💚 Build #43888 succeeded 925ad9b
💔 Build #43887 failed 35ad3d6
💔 Build #43886 failed 6db6161
💔 Build #43884 failed 2d3dfdc
💔 Build #43875 failed d8dc983

cc @mrodm

github-actions · 2026-06-01T15:25:32Z

TL;DR

The failing Buildkite step (Check integrations netbox) appears to be caused by this PR broadening package selection in .buildkite/scripts/common.sh, which made CI run netbox checks that were previously out of scope for this branch. Immediate fix: restore package filtering (or otherwise scope this pipeline) before re-running.

Remediation

In .buildkite/scripts/common.sh:920, revert list_all_directories() to the previous filtered behavior (only packages/elastic_package_registry) or add an equivalent pipeline-specific guard.
Re-run Buildkite for this PR after scoping is restored; if netbox still needs to run intentionally, collect and attach the full failing test XML/log excerpt for root-causeing the netbox test itself.

Investigation details

Root Cause

The PR head commit d43517e4b487a46016ea87c00928d325758001ee changes list_all_directories() from a filtered list to all packages:

Before: .buildkite/scripts/common.sh:920 used mage ... | grep "^packages/elastic_package_registry$"
After: .buildkite/scripts/common.sh:920 uses mage ... listPackages (no filter)

That change expands downstream Buildkite scripts that consume list_all_directories() and explains why this build failed on netbox, even though the branch intent is around changelog-link checks.

Evidence

Build: https://buildkite.com/elastic/integrations/builds/43892
Job/step: Check integrations netbox
Commit diff evidence: .buildkite/scripts/common.sh line 920 changed from filtered package selection to all packages.
Key log excerpt:
- --- [netbox] failed
- 🚨 Error: The command exited with status 1
- user command error: exit status 1

Verification

Executed the same step command locally (.buildkite/scripts/test_one_package.sh packages/netbox origin/main d43517e...) but local run stops early due missing CI env var (YQ_VERSION), so direct local reproduction of the netbox assertion was not possible in this environment.

Follow-up

If running all packages is intentional, this PR likely also needs a separate fix for the netbox failure (or a baseline retry/data point proving flake), because the current build logs available here are truncated to teardown and do not include the underlying test assertion.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

[CI] Add Buildkite step to validate changelog PR links #19171 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
#19171 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

mrodm · 2026-06-03T09:44:13Z

Created changelog-link-skip:check label in the integrations repository

mrodm and others added 6 commits May 22, 2026 15:14

Test changelog

c298efa

Test subset of packages

0c20c86

mrodm self-assigned this May 22, 2026

This comment has been minimized.

Sign in to view

mrodm added 2 commits May 22, 2026 16:33

Add github comment

4c44845

Show error

d370bb9

This comment has been minimized.

Sign in to view

Add tools to post comments

261a3f1

This comment has been minimized.

Sign in to view

mrodm and others added 12 commits May 22, 2026 16:54

Add mention

f313165

Refactor notification into a new method

3f820f1

Add another newline

38b4069

Enable soft_fail in the new step

2953319

Test with two packages failing

7102c24

Add buildkite build link in github annotation

6585ae1

[CI] Add collapsible sections in Buildkite output for notifications

93feb1f

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

andrewkroh added Integration:elastic_package_registry Elastic Package Registry Integration:zipkin_input_otel Zipkin OpenTelemetry Input Package labels May 22, 2026

andrewkroh added Integration:elastic_package_registry Elastic Package Registry Team:Ecosystem Packages Ecosystem team [elastic/ecosystem] labels Jun 1, 2026

mrodm added the skip-changelog label Jun 1, 2026

github-actions Bot added 5 commits June 1, 2026 15:02

Test skip label regex

70f06df

Test with env variable with double dollar sign

f5d97e0

Just one dollar sign for environment variable

881cdb1

Move environment variable to step

4c7d767

Revert "Move environment variable to step"

6f0d6ba

This reverts commit 4c7d767.

mrodm removed the skip-changelog label Jun 1, 2026

Update comment

d8dc983

This comment has been minimized.

Sign in to view

github-actions Bot and others added 4 commits June 1, 2026 15:51

Restore Github label

2d3dfdc

Test multiple changes

6db6161

Both changes in the same file wrong links

35ad3d6

github-actions Bot added 2 commits June 1, 2026 16:37

Revert changes in elastic_package_registry

925ad9b

test all packages

d43517e

mrodm requested a review from jsoriano June 1, 2026 14:56

andrewkroh removed the Integration:elastic_package_registry Elastic Package Registry label Jun 1, 2026

jsoriano approved these changes Jun 2, 2026

View reviewed changes

mrodm merged commit e385cb3 into elastic:main Jun 3, 2026
10 of 11 checks passed

mrodm deleted the add_check_changelog_link branch June 3, 2026 10:13

mrodm mentioned this pull request Jun 4, 2026

[CI] Improve changelog link check TIP message to mention issue links #19379

Merged

1 task done

kcreddy mentioned this pull request Jun 4, 2026

[axonius] Initial release of Axonius #19376

Merged

5 tasks done

Navigation Menu

Repository navigation

[CI] Add Buildkite step to validate changelog PR links - #19171

[CI] Add Buildkite step to validate changelog PR links#19171
mrodm merged 57 commits into
elastic:mainfrom
mrodm:add_check_changelog_link

mrodm commented May 22, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented May 22, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

This comment has been minimized.

mrodm commented May 22, 2026

Uh oh!

This comment has been minimized.

infra-vault-gh-plugin-prod Bot commented Jun 1, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented Jun 1, 2026

Uh oh!

elasticmachine commented Jun 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 1, 2026

Root Cause

Evidence

Verification

Follow-up

Uh oh!

mrodm commented Jun 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mrodm commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

This comment has been minimized.

This comment has been minimized.

mrodm commented May 22, 2026

Uh oh!

This comment has been minimized.

infra-vault-gh-plugin-prod Bot commented Jun 1, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented Jun 1, 2026

Uh oh!

elasticmachine commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

github-actions Bot commented Jun 1, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

Uh oh!

mrodm commented Jun 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mrodm commented May 22, 2026 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented May 22, 2026 •

edited

Loading

elasticmachine commented Jun 1, 2026 •

edited

Loading