Update sync-packages workflow to support nested package directories by mrodm · Pull Request #18549 · elastic/integrations

mrodm · 2026-04-21T17:31:01Z

Proposed commit message

Update the GitHub Workflow to update the issue templates (sync-packages-to-bug-issue-template via updatecli) to support packages that are located in nested directories.

WHAT: Update the sync-packages-to-bug-issue-template updatecli script and the bump-elastic-stack-version workflow to support packages living in multiple (nested) directories.

Replace the manual ls packages/ loop with mage listPackages, which returns the full path to each package regardless of which directory it lives in.
Read title from each manifest.yml using the path returned by mage listPackages, and display the full package path (e.g. packages/foo/bar) in the issue template options list instead of just the top-level folder name.
Add Go and Mage installation steps to the bump-elastic-stack-version workflow so that mage listPackages is available at runtime.

WHY: The existing script assumed all packages live directly under packages/, breaking for integrations that use nested directories. Delegating package discovery to mage listPackages makes the workflow resilient to any directory layout.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Verify that the bump-elastic-stack-version workflow correctly installs Go and Mage before the Updatecli step runs.
Confirm the issue template options list shows full package paths after this change.

How to test this PR locally

Ensure mage is installed and mage listPackages returns the expected list of package paths (including any nested ones).

Copy the script in the .github/workflows/updatecli/updatecli.d/sync-packages-to-bug-issue-template.yml locally to a script, and try to run it.

cat <<EOF > sync_issues.sh
#!/usr/bin/env bash
set -eu

package_paths=$(mage listPackages)

pkgs=$(echo "$package_paths}" | while IFS= read -r package_path; do
  title=$(yq .title "${package_path}/manifest.yml")
  echo "- $title [$package_path]"
done | LC_ALL=C sort -s -f | yq -o json .)
     
yq eval ".body[1].attributes.options = $pkgs" -i '.github/ISSUE_TEMPLATE/integration_bug.yml'
yq eval ".body[1].attributes.options = $pkgs" -i '.github/ISSUE_TEMPLATE/integration_feature_request.yml'
   
git diff --name-only HEAD
EOF

bash sync_issues.sh
git diff .github/ISSUE_TEMPLATE

Inspect .github/ISSUE_TEMPLATE/integration_bug.yml and integration_feature_request.yml to confirm
the options list contains - <title> [<package_path>] entries for all packages, including nested ones.

Related issues

This PR was generated with the assistance of Claude.

…ctories Replace the ls-based flat package loop with find to discover manifests at up to two levels of nesting under packages/. Use yq .name from the manifest instead of the directory name as the package identifier. Keep LC_ALL=C on sort for consistent locale-independent ordering. Generated with the assistance of Claude Sonnet 4.6

Add Go and mage setup steps to bump-elastic-stack-version.yml so that mage listPackages can be used for package discovery in the updatecli pipeline, replacing the find-based approach and ensuring consistent behaviour with the rest of the CI tooling. Generated with the assistance of Claude Sonnet 4.6

Prevent mage listPackages failures from silently producing an empty package list that would wipe all options from the issue template dropdowns. Generated with the assistance of Claude Sonnet 4.6

Replace `set -euo pipefail` with `set -eu` and extract `mage listPackages` into a separate variable assignment so its exit code is caught by `set -e`, avoiding the "Illegal option -o pipefail" error in sh environments. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm · 2026-04-21T17:38:30Z

+        pkgs=$(echo "$package_paths" | while IFS= read -r package_path; do
+          title=$(yq .title "${package_path}/manifest.yml")
+          echo "- $title [$package_path]"
+        done | LC_ALL=C sort -s -f | yq -o json .)


Instead of writing the package folder name (e.g 1password), I propose to use the full package path (e.g. packages/1password) instead here.

As there could be packages in nested directories, that could help to locate the corresponding package. Moreover, it could happen that the same package folder name is repeated in a different nested directory even if the package names (from manifest.yml file).

WDYT @andrewkroh @teresaromero @kpollich ?

Example:

Current approach in main:

- type: dropdown id: integration_name attributes: label: Integration Name description: Which integration package are you having issues with? options: - 1Password [1password] - Abnormal AI [abnormal_security] - abuse.ch [ti_abusech] - Active Directory Entity Analytics [entityanalytics_ad] - ActiveMQ OpenTelemetry Assets [activemq_otel]

Change proposed in this PR:

- type: dropdown id: integration_name attributes: label: Integration Name description: Which integration package are you having issues with? options: - 1Password [packages/1password] - Abnormal AI [packages/abnormal_security] - abuse.ch [packages/ti_abusech] - Active Directory Entity Analytics [packages/entityanalytics_ad] - ActiveMQ OpenTelemetry Assets [packages/activemq_otel]

cc @lucabelluccini for awareness in case it is changed that list in the templates

I think including the full path is relatively harmless and only adds clarity in cases where integration names might collide w/ composable packages. +1 from me on switching to packages/* for the options here.

That seems good if it helps remove ambiguity, given the new option to put packages in more places.

+1 to clarity. However, reading the parent issue i find this example

so for example the nginx_input_otel package can be stored under packages/nginx, as input_otel, along with other packages related to nginx.

in this case, the name on the manifest will be telling us the folder structure, is this coincidence of the example or could it lead to other combination? i mean this because, if the naming convention is like that, the name at the manifest will be unique? cc @jsoriano

if the naming convention is like that, the name at the manifest will be unique?

Package names in the manifest are ensured to be unique via the changes introduced in this PR (dev/packagenames tool).
#18187

in this case, the name on the manifest will be telling us the folder structure, is this coincidence of the example or could it lead to other combination?

Currently, there is no restriction about the folder names it could be created for new packages. That's why it was introduced this new validation, specially when they could be created in nested directories.

For instance, it could be created the nested packages as (even it could be redundant to repeat nginx) or other different folder names:

packages/nginx/nginx

packages/nginx/nginx_input_otel

@teresaromero would this answer your doubts? Not sure if I'm missing anything

mrodm · 2026-04-21T17:38:51Z

+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: .go-version
+
+      - name: Install mage
+        run: go install github.com/magefile/mage
+


Required to execute mage listPackages in the updatecli workflow.

Should we pin a version of mage here?

I think this is not needed, since this step is running after the checkout action. I think in that case, this action is running in the root of the working directory and therefore the go install command would install the mage version specified in the go.mod file.

I did a test in this PR by forcing to use another mage version (v1.17.1) instead of the latest one (v1.17.2) here: bc1f22b

And the workflow/step installed the v1.17.1 (workflow link):

Run go install github.com/magefile/mage go install github.com/magefile/mage shell: /usr/bin/bash -e {0} env: GOTOOLCHAIN: local go: downloading github.com/magefile/mage v1.17.1

Ah you are right, it is already in the go.mod 👍

…ctories-gh-workflows

elasticmachine · 2026-04-28T17:58:08Z

💚 Build Succeeded

Buildkite Build
Commit: bc1f22b

cc @mrodm

This reverts commit bc1f22b.

jsoriano · 2026-04-29T10:00:11Z

+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: .go-version
+
+      - name: Install mage
+        run: go install github.com/magefile/mage
+


Ah you are right, it is already in the go.mod 👍

…lastic#18549) Update the GitHub Workflow to update the issue templates (sync-packages-to-bug-issue-template via updatecli) to support packages that are located in nested directories. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm and others added 6 commits April 21, 2026 16:08

Use folder_name

cca27aa

[github] Add pipefail to sync-packages updatecli script

d103e61

Prevent mage listPackages failures from silently producing an empty package list that would wipe all options from the issue template dropdowns. Generated with the assistance of Claude Sonnet 4.6

Show the title and package path in the list

b706154

mrodm self-assigned this Apr 21, 2026

mrodm commented Apr 21, 2026

View reviewed changes

mrodm changed the title ~~Allow multiple directories gh workflows~~ Update sync-packages workflow to support nested package directories Apr 21, 2026

mrodm marked this pull request as ready for review April 21, 2026 17:40

mrodm requested a review from a team as a code owner April 21, 2026 17:40

mrodm mentioned this pull request Apr 21, 2026

Add support for an additional level of directories under packages/ #17403

Closed

3 tasks done

mrodm requested a review from andrewkroh April 21, 2026 18:10

mrodm added 3 commits April 28, 2026 19:34

Update comment

ab12bed

Merge remote-tracking branch 'upstream/main' into allow-multiple-dire…

8f9160d

…ctories-gh-workflows

Test another mage version - to be removed

bc1f22b

macroscopeapp Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread .buildkite/scripts/common.sh Resolved

Comment thread .buildkite/scripts/common.sh Resolved

Revert "Test another mage version - to be removed"

df1f1c4

This reverts commit bc1f22b.

mrodm requested review from jsoriano and teresaromero April 28, 2026 18:14

jsoriano approved these changes Apr 29, 2026

View reviewed changes

teresaromero approved these changes Apr 29, 2026

View reviewed changes

mrodm merged commit f8e677d into elastic:main Apr 29, 2026
12 checks passed

mrodm deleted the allow-multiple-directories-gh-workflows branch April 29, 2026 14:08

Conversation

mrodm commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Apr 28, 2026

💚 Build Succeeded

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mrodm commented Apr 21, 2026 •

edited

Loading