Add support for an additional level of directories under packages/

In the context of composable packages project, and with the introduction of OTel packages, there are going to be more packages related to the same technology.

Allow to have an additional level of directories, so packages can be grouped by technology. See a POC in #16966.

The main changes will be:

Allow to store packages with an additional nesting level under the packages directory, so for example the nginx_input_otel package can be stored under packages/nginx, as input_otel, along with other packages related to nginx.
Code owners can be defined at the level of a directory containing multiple packages for the same technology. So if the packages/nginx directory contains multiple packages related to NGINX, the CODEOWNERS file only needs to specify a single entry, for packages/nginx, instead of an entry for each package. Each package still needs to define its owner accordingly in the manifest.
It should be still possible to define more specific owners, so it is possible to define a different owner for a single package, even if the rest of packages for the same technology are owned by another team.
In PRs, selective triggering tests per package should also trigger tests of dependent packages.
Packages can be released individually from their directory, independently of its nested position.
Packages will be grouped by technology with the help of their maintainer teams.
Previous model should keep working.

This approach relies on existing features in elastic-package, and only requires changes in the scripts in the integrations repository. These changes need to take into account backwards compatibility with backport branches.

Update dev tools to validate package names and codeowners. [CI] Add ListPackages and support nested package directories in dev tools #18543
Update Buildkite scripts to trigger tests, publish and create backport branches. [CI] Support nested package directories in CI scripts #18577

Update github workflow to automate the creation of the issue templates. Update sync-packages workflow to support nested package directories #18549

integrations/.github/workflows/updatecli/updatecli.d/sync-packages-to-bug-issue-template.yml

Lines 18 to 43 in a710cd1

    
             integration_issue_templates_yml: 
        
               name: update pkgs in .github/ISSUE_TEMPLATE/integration_*.yml 
        
               kind: shell 
        
               scmid: default 
        
               spec: 
        
                 environments: 
        
                   - name: PATH 
        
                   - name: HOME 
        
                 # This script collects titles and package names (as determined by their 
        
                 # directory name) from source. Then updates the issue template. It uses 
        
                 # 'git diff' to satisfy the conditions needed by updatecli to determine 
        
                 # if any changes occurred. 
        
                 command: | 
        
                   #!/usr/bin/env bash 
        
                   set -eu 
        
                   pkgs=$(for i in $(LC_ALL=C ls packages); do 
        
                     title=$(yq .title packages/$i/manifest.yml) 
        
                     pkg=$i 
        
                     echo "- $title [$pkg]" 
        
                   done | sort -s -f | yq -o json .) 
        
                   yq eval ".body[1].attributes.options = $pkgs" -i '.github/ISSUE_TEMPLATE/integration_bug.yml' 
        
                   yq eval ".body[1].attributes.options = $pkgs" -i '.github/ISSUE_TEMPLATE/integration_feature_request.yml' 
        
                   git diff --name-only HEAD

@jsoriano

Claude finished @jsoriano's task in 3m 24s —— View job

Analysis Complete

Based on my investigation, I've analyzed the feature request and the referenced POC in PR #16966.

Recommendation

Follow the approach demonstrated in PR #16966. The POC shows a working implementation that leverages elastic-package find (Technical Preview) to discover nested packages, requiring modifications to 4 key files: .buildkite/scripts/common.sh, .buildkite/scripts/trigger_integrations_in_parallel.sh, dev/codeowners/codeowners.go, and .github/CODEOWNERS.

Findings

Current Architecture

The repository uses a flat package structure under packages/:

Package discovery via list_all_directories() in common.sh:831 uses find . -maxdepth 1 -mindepth 1 -type d
CODEOWNERS validation in dev/codeowners/codeowners.go expects packages at exactly one level deep
Test triggering in trigger_integrations_in_parallel.sh iterates flat packages
Change detection in is_pr_affected() uses git diff with pattern ^packages/${package}/

Key Constraints

elastic-package find already supports nested packages via --depth flag (default: 2, meaning it can find packages/nginx/input_otel)
Backwards compatibility required for backport branches that don't have these changes
CODEOWNERS supports hierarchical patterns - parent directory rules apply to nested paths
CI pipelines must handle package paths with slashes (e.g., nginx/input_otel instead of nginx_input_otel)

POC Analysis (PR #16966)

The POC makes these changes:

list_all_directories(): Changed from find to elastic-package find --depth 2 --output-info dir_name
Pipeline step keys: Changed to sanitize slashes (${package/\\//-}) since Buildkite step keys can't contain /
codeowners.go: Added filepath.WalkDir() to recursively find packages and findOwnerForFile() to traverse parent directories for matching CODEOWNERS rules
CODEOWNERS file: Consolidated nginx-related packages under /packages/nginx/

Detailed Action Plan

Phase 1: Shell Script Changes

File: .buildkite/scripts/common.sh

Modify list_all_directories() (line 831-833):

list_all_directories() {
    ${ELASTIC_PACKAGE_BIN} find -C . --depth 2 --output-info dir_name | sort
}

Update is_pr_affected() (line 778) to handle nested package paths:

# Change from:
if git diff --name-only "${commit_merge}" "${to}" | grep -E "^packages/${package}/" > /dev/null
# To support nested paths like nginx/input_otel:
if git diff --name-only "${commit_merge}" "${to}" | grep -E "^packages/${package}(/|$)" > /dev/null

Update process_package() (line 1034) to handle the nested path when pushd:

pushd "${package}" > /dev/null  # This already works if package="nginx/input_otel"

File: .buildkite/scripts/trigger_integrations_in_parallel.sh

Add use_elastic_package call after line 11 to ensure binary is available

Update pipeline step key generation (around line 79-81):

# Sanitize package name for step key (replace / with -)
local step_key="${package//\//-}"
- label: "Check integrations ${package}"
  key: "test-integrations-${step_key}"

File: .buildkite/scripts/test_one_package.sh

Update pushd packages > /dev/null (line 28) to handle nested paths:

# The package arg could be "nginx/input_otel", so:
pushd "packages/${package}" > /dev/null  # Instead of pushd packages then process_package

File: .buildkite/scripts/build_packages.sh

Replace find . -maxdepth 1 -mindepth 1 -type d (line 53) with elastic-package find:

for it in $(${ELASTIC_PACKAGE_BIN} find -C . --depth 2 --output-info absolute_path); do

Phase 2: Go Code Changes

File: dev/codeowners/codeowners.go

Update validatePackages() (line 65-97) to use filepath.WalkDir():

func validatePackages(codeowners *githubOwners, packagesDir string) error {
    return filepath.WalkDir(packagesDir, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }
        if !d.IsDir() {
            return nil
        }
        manifestPath := filepath.Join(path, "manifest.yml")
        if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
            return nil // Not a package directory
        }
        if err := codeowners.checkManifest(manifestPath); err != nil {
            return err
        }
        return codeowners.checkDataStreams(path)
    })
}

Add findOwnerForFile() helper function:

func (codeowners *githubOwners) findOwnerForFile(filePath string) ([]string, bool) {
    // Try exact match first, then traverse up the directory tree
    for p := filePath; p != "/" && p != "."; p = filepath.Dir(p) {
        slashPath := "/" + filepath.ToSlash(p)
        if owners, found := codeowners.owners[slashPath]; found {
            return owners, true
        }
    }
    return nil, false
}

Update checkManifest() to use the new helper:

func (codeowners *githubOwners) checkManifest(path string) error {
    pkgDir := filepath.Dir(path)
    owners, found := codeowners.findOwnerForFile(pkgDir)
    if !found {
        return fmt.Errorf("there is no owner for %q in %q", pkgDir, codeowners.path)
    }
    // ... rest of validation
}

Update PackageOwners() function (line 33-54) for nested package lookup

Phase 3: CODEOWNERS Migration (Done Per-Technology)

For each technology grouping (starting with nginx as the POC):

Create new directory structure: packages/nginx/input_otel/
Move package contents

Update CODEOWNERS:

/packages/nginx @elastic/obs-infraobs-integrations

Individual packages can still override:

/packages/nginx/special_package @elastic/different-team

Phase 4: Testing & Validation

Verify elastic-package find discovers nested packages correctly
Run mage check to validate CODEOWNERS
Test selective test triggering with a PR modifying a nested package
Test build and publish pipelines
Verify backport compatibility by testing on a backport branch

Backwards Compatibility Considerations

Check elastic-package version in go.mod (currently v0.118.0) - ensure find command is available
Use mage isElasticPackageDependencyLessThan to conditionally fall back to old behavior on backport branches
The elastic-package find command is in Technical Preview but functional

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
elastic/integrations	PR #16966	POC implementation demonstrating the nested directory support with nginx packages

Related Files

Repository	File	Relevance	Lines
elastic/integrations	.buildkite/scripts/common.sh	`list_all_directories()` and `is_pr_affected()` need updates for nested paths	691-784, 831-833
elastic/integrations	.buildkite/scripts/trigger_integrations_in_parallel.sh	Pipeline generation needs elastic-package and step key sanitization	12-14, 79-81
elastic/integrations	.buildkite/scripts/build_packages.sh	`build_packages()` uses `find` that needs replacement	50-82
elastic/integrations	.buildkite/scripts/test_one_package.sh	Package path handling may need adjustment	28-36
elastic/integrations	.buildkite/scripts/test_integrations_with_serverless.sh	Uses `list_all_directories()` for package iteration	79-107
elastic/integrations	dev/codeowners/codeowners.go	`validatePackages()` needs recursive discovery, `checkManifest()` needs parent traversal	61-97, 172-209
elastic/integrations	.github/CODEOWNERS	Will need reorganization as packages are grouped by technology	All
elastic/integrations	magefile.go	Imports codeowners package for `mage check`	23, 50

Why is Claude responding? | Type @claude to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Once the support for the additional level of directories is added, it needs to be revisited also the CI workflows/pipelines that rely on just one level of directories.
As example of one of these workflows: #17879

Check other Github Workflows in case they need to be updated too.

@alaudazzi

@alaudazzi @jmikell821 @bmorelli25 FYI. Docs locations could change.

[ci] Support nested package paths in docs validation trigger

[ci] Add blocking PR check for package docs using docs-builder (#17879)

@theletterf

@theletterf @jsoriano
Is this going to impact the content in the integrations-docs repo?

I'm checking.

@jsoriano

In theory, it should hold. @jsoriano is EPR going to change the way they expose files as a result of this?

If nested packages use slashes in their name (e.g., nginx/input_otel), that could break the filename construction in src/util.js:makeFilepath(), which builds paths like
docs/reference/${name}.md. A name with a / would create unexpected directory nesting. But EPR package names typically use hyphens, not slashes.
If the restructuring causes existing packages to be renamed or removed from EPR, redirects would be needed in docs/redirects.yml.

@jsoriano is EPR going to change the way they expose files as a result of this?

No, EPR responses won't change, and neither will do the package names.

This change only offers new options to store the source files in this repository, nothing should change once packages are built and published.

It should be taken into account that the changes included in .buildkite folder and others are also copied into the process of creating new backport branches created. It could happen that these folder do not support elastic-package find command because in those branches are installing an older elastic-package version that does not support that command.

It should be taken into account that the changes included in .buildkite folder and others are also copied into the process of creating new backport branches created. It could happen that these folder do not support elastic-package find command because in those branches are installing an older elastic-package version that does not support that command.

These older branches wouldn't either be using the new structure, so they won't need this functionality. But it is true that we might need to wrap it so they fall back to the old functionality when elastic-package find is not available.

@jsoriano

In PRs, selective triggering tests per package should also trigger tests of dependent packages.

@jsoriano Not sure how to check for these dependent packages.

Should it be taken into account the require definitions in the package manifest:
https://github.com/elastic/package-spec/blob/8c7fcac97545da93310e4edd301a697fe271d113/test/packages/good_requires/manifest.yml#L12-L22

Or the ones in the test configuration:

https://github.com/elastic/package-spec/blob/8c7fcac97545da93310e4edd301a697fe271d113/test/packages/good_requires/_dev/test/config.yml#L1-L9

One option would be to consider the list from the package manifest. But maybe those dependent packages are not affected by the changes in the given package if they use pinned versions of the packages.

Considering the ones from the test configuration, it could be problematic to check whether or not they refer to the same package since there could be relative paths.

WDYT ?

This would require some refactors in the buildkite scripts (bash) to achieve this.

@mrodm

@mrodm at the end input package dependencies are going to be pinned, and resolved at build time, so it is not so risky to check dependent packages, they will be checked when they update their dependencies.

After #17404 dependabot-like PRs will be automatically created, and the affected packages will be tested before being upgraded.

So we can probably skip this step, and reconsider the situation after #17404. Maybe an option is to create a daily job, similar to what we do with snapshot stack versions.

Closing issue after being merged this PR #18577
Updated:

tools to validate package names and codeowners.
Github Workflows to sync issue templates about bugs and feature requests.
CI scripts related to backport branches, testing in pull request, publish packages and test packages with Serverless.

Navigation Menu

Repository navigation

Add support for an additional level of directories under packages/ #17403

Analysis Complete

Recommendation

Current Architecture

Key Constraints

POC Analysis (PR #16966)

Phase 1: Shell Script Changes

Phase 2: Go Code Changes

Phase 3: CODEOWNERS Migration (Done Per-Technology)

Phase 4: Testing & Validation

Backwards Compatibility Considerations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	integration_issue_templates_yml:
	name: update pkgs in .github/ISSUE_TEMPLATE/integration_*.yml
	kind: shell
	scmid: default
	spec:
	environments:
	- name: PATH
	- name: HOME
	# This script collects titles and package names (as determined by their
	# directory name) from source. Then updates the issue template. It uses
	# 'git diff' to satisfy the conditions needed by updatecli to determine
	# if any changes occurred.
	command: \|
	#!/usr/bin/env bash
	set -eu

	pkgs=$(for i in $(LC_ALL=C ls packages); do
	title=$(yq .title packages/$i/manifest.yml)
	pkg=$i
	echo "- $title [$pkg]"
	done \| sort -s -f \| yq -o json .)

	yq eval ".body[1].attributes.options = $pkgs" -i '.github/ISSUE_TEMPLATE/integration_bug.yml'
	yq eval ".body[1].attributes.options = $pkgs" -i '.github/ISSUE_TEMPLATE/integration_feature_request.yml'

	git diff --name-only HEAD

Add support for an additional level of directories under packages/ #17403

Description

Activity

github-actions commented on Feb 12, 2026

Analysis Complete

Recommendation

Current Architecture

Key Constraints

POC Analysis (PR #16966)

Phase 1: Shell Script Changes

Phase 2: Go Code Changes

Phase 3: CODEOWNERS Migration (Done Per-Technology)

Phase 4: Testing & Validation

Backwards Compatibility Considerations

mrodm commented on Mar 25, 2026

theletterf commented on Mar 25, 2026

alaudazzi commented on Mar 25, 2026

theletterf commented on Mar 25, 2026

theletterf commented on Mar 25, 2026

jsoriano commented on Mar 26, 2026

mrodm commented on Mar 31, 2026

jsoriano commented on Apr 6, 2026

mrodm commented on Apr 16, 2026

jsoriano commented on Apr 17, 2026

mrodm commented on Apr 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions