Skip to content

[Stack 9.5.0-SNAPSHOT] Too many packages failing in daily job #18803

New issue
New issue
Closed
Closed
[Stack 9.5.0-SNAPSHOT] Too many packages failing in daily job#18803
Labels
Team:EcosystemPackages Ecosystem team [elastic/ecosystem]automationflaky-testUnstable or unreliable test cases.

Description

@elastic-vault-github-plugin-prod
elastic-vault-github-plugin-prod
bot
opened on May 5, 2026
  • Stack version: 9.5.0-SNAPSHOT
  • Packages:
    • admin_by_request_epm
    • apache_input_otel
    • atlassian_cloud
    • atlassian_confluence
    • atlassian_jira
    • azure_blob_storage
    • azure_network_watcher_nsg
    • azure_network_watcher_vnet
    • cisco_asa
    • cisco_secure_endpoint
    • claude_code
    • claude_cowork
    • docker_input_otel
    • elasticapm_input_otel
    • elasticsearch
    • endace
    • entityanalytics_ad
    • filelog_otel
    • forcepoint_web
    • hostmetrics_input_otel
    • httpcheck_otel
    • iis_input_otel
    • istio
    • jaeger_input_otel
    • kafka_input_otel
    • kubeletstats_input_otel
    • kubernetes
    • mongodb
    • mysql_input_otel
    • network_traffic
    • nginx_input_otel
    • okta
    • oracle
    • otlp_input_otel
    • prometheus_input_otel
    • prometheus_input_otel_raw
    • rapid7_insightvm
    • redis_input_otel
    • sql_server_input_otel
    • statsd_input_otel
    • system
    • ti_opencti
    • vectra_rux
    • verifier_otel
    • zipkin_input_otel
  • Owners:
    • @elastic/ecosystem

Latest issue closed for the same test: #18575

First build failed: https://buildkite.com/elastic/integrations/builds/42293

Latest 5 failed builds:

Activity

  1. elastic-vault-github-plugin-prod
    added
    automation
    flaky-testUnstable or unreliable test cases.
    Team:EcosystemPackages Ecosystem team [elastic/ecosystem]
    on May 5, 2026

  2. mrodm commented on May 5, 2026

    @mrodm
    mrodm
    on May 5, 2026
    Collaborator

    There were some packages failing due to network glitches:

    • activemq
    • journald
    • iptables
    • cassandra
    • f5_bigip
    • trellix_edr_cloud
    • symnatec_endpoint_security
    • cisco_umbrella
    • aws (test related to apt-get failures)
    • cloudfare_logpush (test related to apt-get failures)

    After retry those packages, those tests failed due to apt-get were running successfully or at least without failures related to temporal network failures.

  3. mrodm commented on May 8, 2026

    @mrodm
    mrodm
    on May 8, 2026
    Collaborator

    Latest build has introduced a lot of errors for could not find the expected hits:
    https://buildkite.com/elastic/integrations/builds/42520

  4. teresaromero commented on May 8, 2026

    @teresaromero
    teresaromero
    on May 8, 2026
    Contributor

    After some investigation i got the following output for the majority of the packages. They might be failing because a cel error related to the otel collector. Assisted by claude

    Summary

    elastic/beats#48909 (merged 2026-02-18) was supposed to fix request-tracer path validation under the managed agent for the cel, httpjson, http_endpoint, and entityanalytics inputs. The fix is present on main and on the 9.4 release branch (the active branch when the PR landed). However the bug reproduces deterministically on stack 9.5.0-SNAPSHOT (verified locally 2026-05-08), because in 9.5 the elastic-agent runs filebeat as a receiver inside elastic-otel-collector (otelcol.component.id: filebeatreceiver/_agent-component/cel-default), and paths.Resolve(paths.Logs, input) in that runtime does not produce a path ending in the input name.

    The integrations CI's daily build against 9.5.0-SNAPSHOT is currently flagging ~95 packages with the same "could not find expected hits" symptom (see elastic/integrations#18803). A static audit of elastic/integrations shows 75 of those 95 packages share this exact root cause, plus an additional 4 packages that match the audit signature but aren't yet in CI's failing list (latent / will start failing as soon as their tracer-enabled tests run on 9.5+).

    Reproduction

    Stack 9.5.0-SNAPSHOT, elastic-package v0.123.0:

    elastic-package stack up -d -v --version=9.5.0-SNAPSHOT
cd packages/akamai
elastic-package test system -v --setup --config-file data_stream/siem/_dev/test/system/test-emulator-config.yml
docker logs $(docker ps --format '{{.Names}}' | grep agent-akamai-siem) 2>&1 | grep -i 'request tracer'

    Observed: the cel-default component goes STARTING → FAILED with:

    Permanent: Failed to start crawler: starting input failed: error while initializing input:
  request tracer path must be within "cel" path accessing 'filebeat.inputs.0.resource'

    Input never executes; no HTTP requests reach the test emulator; ES logs-akamai.siem-* count stays at 0; system test asserts 144 hits and fails.

    The compiled cel input config sent by Fleet (extracted from /api/fleet/agent_policies?full=true):

    "resource": {
  "url": "http://akamai-siem-emulator:9903/siem/v1/configs/123456",
  "tracer": {
    "enabled": true,
    "filename": "../../logs/cel/http-request-trace-*.ndjson",
    "maxbackups": 5
  }
}

    That ../../logs/cel/... relative pattern is emitted by every cel.yml.hbs / httpjson.yml.hbs / http_endpoint.yml.hbs template in elastic/integrations (473 data-stream templates across 146 packages — counts in the linked audit).

    Why #48909 doesn't cover this

    The error is generated at x-pack/filebeat/input/cel/config.go:327:

    ok, err := httplog.IsPathInLogsFor(inputName, c.Tracer.Filename)
if err != nil { return err }
if !ok {
    return fmt.Errorf("request tracer path must be within %q path",
                       paths.Resolve(paths.Logs, inputName))
}

    IsPathInLogsFor (in x-pack/filebeat/input/internal/httplog/roundtripper.go) uses paths.Resolve(paths.Logs, "cel") as the validation root. Under the legacy elastic-agent runtime, paths.Logs is /usr/share/elastic-agent/data/elastic-agent-XXX/logs and the join produces …/logs/cel, which is the same dir the tracer ultimately writes to — so the relative ../../logs/cel/foo resolves inside it and validation passes.

    Under the OTel-collector receiver runtime in 9.5, the filebeat receiver is hosted by a different binary (/usr/share/elastic-agent/data/elastic-agent-XXX/components/elastic-otel-collector). paths.Logs in that process apparently does not point at the elastic-agent's logs/ tree, so the join produces a root that doesn't contain the resolved tracer path, and IsPathIn returns false.

    Per #48909's body the same code path also runs when the tracer is disabled (tracerConfig.enabled() == nil || true is the default), which is consistent with the audit finding that several packages in #18803 have NO tests setting enable_request_tracer: true yet still fail.

  5. mrodm commented on May 8, 2026

    @mrodm
    mrodm
    on May 8, 2026
    Collaborator

    Thanks @teresaromero !

    Buildkite build: https://buildkite.com/elastic/integrations/builds/42520 was running with the following stack:

    ╭──────────────────┬────────────────┬───────────────────┬───────────────────┬────────────╮
│     SERVICE      │    VERSION     │      STATUS       │ IMAGE BUILD DATE  │  VCS REF   │
├──────────────────┼────────────────┼───────────────────┼───────────────────┼────────────┤
│ elastic-agent    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-07T03:05Z │ 7a962179d6 │
│ elasticsearch    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-07T22:54Z │ 84580e8c70 │
│ fleet-server     │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-07T03:05Z │ 7a962179d6 │
│ kibana           │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-07T11:19Z │ 4eeb07e04b │
│ package-registry │ latest         │ running (healthy) │                   │            │
╰──────────────────┴────────────────┴───────────────────┴───────────────────┴────────────╯

    Example of the error found for akamai:

    {"log.level":"error","@timestamp":"2026-05-08T03:53:24.400Z","message":"error starting filebeat receiver{error 26 0  beat receiver run error: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource'}","service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":29,"file.name":"fbreceiver/receiver.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/fbreceiver.(*filebeatReceiver).Start.func1"},"resource":{"service.instance.id":"8a494810-ec70-4971-a69a-58e9518ddc30","service.name":"/usr/share/elastic-agent/data/elastic-agent-7a9621/components/elastic-otel-collector","service.version":"9.5.0"},"otelcol.component.id":"filebeatreceiver/_agent-component/cel-default","otelcol.signal":"logs","otelcol.component.kind":"receiver","log":{"source":"cel-default"},"component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"ecs.version":"1.6.0"}

    Example of the error found for auth0:

    {"log.level":"error","@timestamp":"2026-05-08T03:51:47.927Z","message":"error starting filebeat receiver{error 26 0  beat receiver run error: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource'}","otelcol.signal":"logs","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"service.name":"filebeat","resource":{"service.instance.id":"773c0bf1-6a9b-4c02-b124-ffb067d7b7aa","service.name":"/usr/share/elastic-agent/data/elastic-agent-7a9621/components/elastic-otel-collector","service.version":"9.5.0"},"otelcol.component.id":"filebeatreceiver/_agent-component/cel-default","otelcol.component.kind":"receiver","log":{"source":"cel-default"},"ecs.version":"1.6.0","log.origin":{"file.line":29,"file.name":"fbreceiver/receiver.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/fbreceiver.(*filebeatReceiver).Start.func1"},"ecs.version":"1.6.0"}

    According to this, that elastic-agent commit: https://github.com/elastic/elastic-agent/commits/7a962179d6db1d402f0475f3db0a98d272b4d2d3

    That snapshot version contains this PR that could be related since it looks they are using the otel runtime now elastic/elastic-agent#13991 (cc @leehinman )

    As it was also mentioned by the previous comment, this looks similar to the issue that was fixed a few weeks ago here elastic/beats#48909

    cc @elastic/ecosystem @cmacknz

  6. cmacknz commented on May 8, 2026

    @cmacknz
    cmacknz
    on May 8, 2026
    Member

    As it was also mentioned by the previous comment, this looks similar to the issue that was fixed a few weeks ago here elastic/beats#48909

    CC @efd6 the request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource' error is back, though I am confused about why we no longer see this bug in agentless when CEL is run as a receiver.

  7. efd6 commented on May 11, 2026

    @efd6
    efd6
    on May 11, 2026
    Contributor

    @cmacknz The root cause is that NewBeatForReceiver in x-pack/libbeat/cmd/instance/beat.go intentionally creates a separate *paths.Path instance for each filebeat receiver and stores it in b.Info.Paths, but does not update the global paths.Paths singleton (because multiple receivers can coexist in the same collector process with different paths). However, httplog.IsPathInLogsFor and ResolvePathInLogsFor use the global paths.Resolve(paths.Logs, input) for validation. Under the OTel receiver runtime, it seems that the global is never initialized, so paths.Logs is empty and paths.Resolve(paths.Logs, "cel") returns just "cel" — a bare relative path. The tracer filename's ../../logs/cel/... then resolves outside that root and validation fails.

    PR #48909's fix still works for the process runtime (where the global is correctly set) and for disabled tracers (the early return skips validation). The 9.5 regression only fires when both: (a) the OTel receiver runtime is used, and (b) the tracer is enabled. In the integrations CI, every system test config sets enable_request_tracer: true, which is why ~75 packages fail.

    Re: why agentless doesn't show it: it is likely latent there too, but no test or real user enables request tracing in agentless, so the !c.Tracer.enabled() guard returns early before the broken path validation is reached.

    The fix would be: (elastic/beats#50581)

    • Remove the IsPathInLogsFor check from all Validate() methods (these run during config unpacking and can't access the correct per-receiver paths).
    • Change ResolvePathInLogsFor to accept a *paths.Path parameter instead of using the global.
    • Update all runtime callers (cel, httpjson, http_endpoint, entityanalytics/okta, jamf, azuread) to pass ctx.Agent.Paths, the correct per-instance paths from beat.Info.

    @mrodm Can we have the sha refs for failures included in the original report?

  8. mrodm commented on May 11, 2026

    @mrodm
    mrodm
    on May 11, 2026
    Collaborator

    @mrodm Can we have the sha refs for failures included in the original report?

    @efd6 If you refer about the SHA of the docker images running in the stack, that information was added a few months ago (starting with elastic-package v0.117.0)) in the Buildkite output itself (linked in the description). This is shown as part of the elastic-package stack status command in the VCS REF column
    https://buildkite.com/elastic/integrations/builds/42614#019e1520-730c-4bd7-8d0b-f0642f771d11/L745-L753

    ╭──────────────────┬────────────────┬───────────────────┬───────────────────┬────────────╮
│     SERVICE      │    VERSION     │      STATUS       │ IMAGE BUILD DATE  │  VCS REF   │
├──────────────────┼────────────────┼───────────────────┼───────────────────┼────────────┤
│ elastic-agent    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-10T03:11Z │ eac42baa7d │
│ elasticsearch    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-08T16:38Z │ 78c313d002 │
│ fleet-server     │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-10T03:11Z │ eac42baa7d │
│ kibana           │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-10T11:27Z │ 1dfe3071d2 │
│ package-registry │ latest         │ running (healthy) │                   │            │
╰──────────────────┴────────────────┴───────────────────┴───────────────────┴────────────╯

  9. mrodm commented on May 11, 2026

    @mrodm
    mrodm
    on May 11, 2026
    Collaborator

    @efd6 maybe it could be added this information into the jUnit XML files directly.
    There is a properties block that could be used for that:

    Example copied from https://github.com/testmoapp/junitxml#properties-for-suites-and-cases

    <testsuite name="Tests.Registration" time="7.005871">
    <properties>
        <property name="version" value="1.774" />
        <property name="commit" value="ef7bebf" />
        <property name="browser" value="Google Chrome" />
        <property name="ci" value="https://github.com/actions/runs/1234" />
        <property name="config"> <!-- Some tools support properties with text content -->
            Config line #1
            Config line #2
            Config line #3
        </property>
    </properties>

    <!-- Test cases here -->
</testsuite>

    Created issues for that:

  10. efd6 commented on May 11, 2026

    @efd6
    efd6
    on May 11, 2026
    Contributor

    @mrodm I'm thinking for here where currently it says "Stack version: 9.5.0-SNAPSHOT". The details should be in the report, rather than just in buildkite logs where they increase friction for finding the information.

  11. mrodm commented on May 12, 2026

    @mrodm
    mrodm
    on May 12, 2026
    Collaborator

    @mrodm I'm thinking for here where currently it says "Stack version: 9.5.0-SNAPSHOT". The details should be in the report, rather than just in buildkite logs where they increase friction for finding the information.

    👍 As the script in charge of creating all these Github issues related to flaky-tests reads all the data from the jUnit files (XML files), it is required to add first that metadata as properties in those files.

  12. mrodm commented on Jun 3, 2026

    @mrodm
    mrodm
    on Jun 3, 2026
    Collaborator

    Hi!
    Latest daily builds are still failing even after being merged this fix a few days ago elastic/beats#50581 cc @efd6

    Checking some packages failing in the latest build at the moment of writing this comment https://buildkite.com/elastic/integrations/builds/44038 same error appears in elastic-agent logs:

    request tracer path must be within \"httpjson\" path accessing 'filebeat.inputs.0.request'

request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource'

    These are the full log messages for the packages that I checked:

    • 1password:
    {"log.level":"error","@timestamp":"2026-06-03T03:48:35.695Z","message":"error starting filebeat receiver{error 26 0  beat receiver run error: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"httpjson\" path accessing 'filebeat.inputs.0.request'}","otelcol.component.id":"filebeatreceiver/_agent-component/httpjson-default","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":29,"file.name":"fbreceiver/receiver.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/fbreceiver.(*filebeatReceiver).Start.func1"},"resource":{"service.instance.id":"5e06ff16-d5e1-4165-8698-9929a57b498a","service.name":"/usr/share/elastic-agent/data/elastic-agent-d1e89d/components/elastic-otel-collector","service.version":"9.5.0"},"otelcol.component.kind":"receiver","otelcol.signal":"logs","log":{"source":"httpjson-default"},"ecs.version":"1.6.0"}
    • akamai
    {"log.level":"error","@timestamp":"2026-06-03T03:49:49.385Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":1160},"message":"Unit state changed cel-default-cel-akamai-95281fe1-b0f5-491b-abf5-7b2f9d8f31a0 (STARTING->FAILED): Permanent: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource'","log.source":"elastic-agent","component":{"id":"cel-default","state":"FAILED"},"unit":{"id":"cel-default-cel-akamai-95281fe1-b0f5-491b-abf5-7b2f9d8f31a0","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
    • github
    {"log.level":"error","@timestamp":"2026-06-03T04:57:03.832Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":1160},"message":"Unit state changed httpjson-default-httpjson-github-1f11021e-8f0a-4130-aaa7-0a12e38c3fb4 (STARTING->FAILED): Permanent: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"httpjson\" path accessing 'filebeat.inputs.0.request'","log.source":"elastic-agent","component":{"id":"httpjson-default","state":"FAILED"},"unit":{"id":"httpjson-default-httpjson-github-1f11021e-8f0a-4130-aaa7-0a12e38c3fb4","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

    The stack used for testing in both packages is:

    ╭──────────────────┬────────────────┬───────────────────┬───────────────────┬────────────╮
│     SERVICE      │    VERSION     │      STATUS       │ IMAGE BUILD DATE  │  VCS REF   │
├──────────────────┼────────────────┼───────────────────┼───────────────────┼────────────┤
│ elastic-agent    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-06-02T03:28Z │ d1e89db2e8 │
│ elasticsearch    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-06-03T00:40Z │ 2997e39d1e │
│ fleet-server     │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-06-02T03:28Z │ d1e89db2e8 │
│ kibana           │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-06-02T11:22Z │ 2860842a14 │
│ package-registry │ latest         │ running (healthy) │                   │            │
╰──────────────────┴────────────────┴───────────────────┴───────────────────┴────────────╯

  13. mrodm commented on Jun 15, 2026

    @mrodm
    mrodm
    on Jun 15, 2026
    Collaborator

    Checked again the results from this build https://buildkite.com/elastic/integrations/builds/44457 and there are still the same errors found in Elastic Agent logs that I think it causes that no documents are ingested cc @efd6 @cmacknz

    Some examples:

    • 1password:
    "error starting filebeat receiver{error 26 0  beat receiver run error: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"httpjson\" path accessing 'filebeat.inputs.0.request'}"
    • akamai:
    "error starting filebeat receiver{error 26 0  beat receiver run error: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource'}"

  14. efd6 commented on Jun 16, 2026

    @efd6
    efd6
    on Jun 16, 2026
    Contributor

    All httpjson and CEL-based integration system tests on 9.5.0-SNAPSHOT were failing from ~June 2 to June 15 with:

    request tracer path must be within "httpjson" path accessing 'filebeat.inputs.0.request'
request tracer path must be within "cel" path accessing 'filebeat.inputs.0.resource'

    100+ packages affected.

    Root Cause

    Under the OTel receiver runtime (how elastic-agent runs filebeat since elastic-agent#13991), NewBeatForReceiver creates a per-instance *paths.Path stored in b.Info.Paths but never updates the global paths.Paths singleton.

    Both IsPathInLogsFor (in Validate()) and ResolvePathInLogsFor (at input start) used the global. So paths.Resolve(paths.Logs, "httpjson") returned a bare relative name like "httpjson", and the tracer filename's ../../logs/httpjson/ pattern resolved outside that root.

    Integration system tests set enable_request_tracer: true in their test configs, exercising this code path.

    The Fix

    elastic/beats#50581 (merged May 29):

    1. Removes IsPathInLogsFor from all Validate() methods (they run during config unpacking and can't access per-receiver paths)
    2. Changes ResolvePathInLogsFor to accept a *paths.Path parameter so callers pass the correct per-instance paths

    Why It Took 21 Days to Reach 9.5.0-SNAPSHOT

    The elastic-agent main branch beats submodule was stuck at 71d0f09b5402 (May 25) because the automated update pipeline kept failing on unrelated issues:

    PR Dates Outcome
    #14565 May 27 → June 5 Closed: internal/edot/README.md generated-file drift + Windows filebeatreceiver queue init conflict
    #14832 June 8 → June 12 Closed: same CI failures + merge conflicts → Mergify auto-close
    #14894 June 15 → June 15 Merged — beats updated to 53223a456c82

    Meanwhile, the 9.4 branch was fine — its beats update automation picked up the backport (#51028) on June 3.

    Why the Earlier Fix (PR #49655) Didn't Help

    elastic/beats#49655 (March 2026) skips path validation when tracing is disabled. But system tests explicitly enable tracing, so the broken code path is still exercised. PR #50581 was needed to fix the path resolution itself.

    Sequence Diagram

    sequenceDiagram
    participant BK as Beats (main)
    participant EA as Elastic-Agent (main)
    participant CI as Integrations Daily CI

    Note over BK: May 25: beats at 71d0f09b
    EA->>BK: Submodule update (PR #14498)
    Note over BK: May 29: Fix merged (ec6e82f8)
    EA--xBK: PR #14565 fails CI (May 27–Jun 5)
    CI->>EA: Jun 2: Snapshot built without fix
    Note over CI: Jun 3–15: All httpjson/CEL tests fail
    EA--xBK: PR #14832 fails CI (Jun 8–12)
    EA->>BK: PR #14894 merges (Jun 15)
    Note over CI: Jun 16+: Should pass
    Loading

  15. mrodm commented on Jun 18, 2026

    @mrodm
    mrodm
    on Jun 18, 2026
    Collaborator

    It looks like that starting with buildkite build: https://buildkite.com/elastic/integrations/builds/44607 , packages failing due to the issue elastic/beats#50581 was already fixed. For instance 1password and akamai packages are running successfully, thanks @efd6 !

    However, there are now new packages failing. In that buildkite build, some of the packages failing are otel input packages whose policy tests (~72 tests) are failing (e.g. docker_input_otel, otlp_input_otel).

    These policy tests failing are related to the changes introduced in these two PRs in kibana:

    One of the main concerns here is that those policy tests cannot be fixed for the minimum stack supported by these packages. For instance, docker_input_otel supports 9.2.0, but that version does not build those policies including those suffixes (component IDs). Therefore, if policy tests are updated as suggested in these reports (testing with 9.5.0-SNAPSHOT stack), those tests will fail when packages are tested with 9.2.0 or 9.4.0 (on the main branch). cc @elastic/ecosystem

    There are also other packages failing totally unrelated to these policy tests from OTel input packages like admin_by_request_epm, auditd, azure_blob_storage, cursor, dga, google_scc, kubernetes, lastpass, ...

  16. mrodm commented on Jun 26, 2026

    @mrodm
    mrodm
    on Jun 26, 2026
    Collaborator

    The errors from OTel input packages caused by elastic/kibana#270487 could be solved by re-generating the expected files. However, if those files are re-generated they will fail with the minimum stack supported (e.g. 9.2.0 or 9.4.0 depending on the package).

    Tested with filelog_otel:

    cd packages/filelog_otel

## Testing with 9.5.0-SNAPSHOT
elastic-package stack up -v -d --version 9.5.0-SNAPSHOT

# regenerate file to add the component ID to the `forward` processor
elastic-package test policy -v --generate

elastic-package test policy -v  # this command now runs successfully after re-generating the expected files

elastic-package stack down -v

## Testing with 9.2.0 the same package with the new generated expected files from 9.5.0-SNAPSHOT

elastic-package stack up -v -d --version 9.2.0

elastic-package test policy -v  # this command fails with the re-generated expected files

elastic-package stack down -v

  17. efd6 commented on Jul 6, 2026

    @efd6
    efd6
    on Jul 6, 2026
    Contributor

    @mrodm The alternative is for elastic-package to normalise the policy test output. We have #19952 which is caused by elastic/kibana#270487. We could bump the kibana version constraint, but that would drop 9.4.0-9.4.2 users purely to avoid the flaky test failure.

    Something like

    Details 
    diff --git a/internal/testrunner/runners/policy/policy.go b/internal/testrunner/runners/policy/policy.go
index 4a9736f0..94d52684 100644
--- a/internal/testrunner/runners/policy/policy.go
+++ b/internal/testrunner/runners/policy/policy.go
@@ -236,6 +236,71 @@ func isOTelVariableKey(key string) bool {
        return strings.Contains(key, "/")
 }
 
+// ottlConditionalDataStreamAttr matches " where attributes["data_stream.<field>"] == nil"
+// suffixes appended by Fleet since kibana#274993. Stripping them makes policy comparison
+// stable across Fleet versions that do and don't add the guard.
+var ottlConditionalDataStreamAttr = regexp.MustCompile(` where attributes\["data_stream\.\w+"\] == nil$`)
+
+// preNormalizePolicy rewrites the decoded policy tree before component-ID
+// normalization to absorb Fleet version differences that would otherwise
+// cause spurious policy test failures.
+func preNormalizePolicy(root map[string]any) {
+       // Collapse all forward connector keys ("forward" and "forward/<outputId>") to
+       // "forward/_bare". Fleet added an output-ID suffix in 9.4.3 (kibana#270487);
+       // after normalization every variant becomes "forward/componentid-N".
+       if connectors, ok := toMap(root["connectors"]); ok {
+               for k, v := range connectors {
+                       if k == "forward" || strings.HasPrefix(k, "forward/") {
+                               delete(connectors, k)
+                               connectors["forward/_bare"] = v
+                       }
+               }
+       }
+
+       // Rename bare pipeline keys (e.g. "logs", "metrics") to "<signal>/_bare" so
+       // they participate in normalization the same way suffixed keys do. Fleet
+       // started suffixing these with the output ID in 9.4.3 (kibana#270487).
+       if service, ok := toMap(root["service"]); ok {
+               if pipelines, ok := toMap(service["pipelines"]); ok {
+                       for k, v := range pipelines {
+                               if !strings.Contains(k, "/") {
+                                       delete(pipelines, k)
+                                       pipelines[k+"/_bare"] = v
+                               }
+                       }
+               }
+       }
+
+       // Walk string elements in arrays to:
+       //   - replace forward connector refs with "forward/_bare" (pipeline arrays)
+       //   - strip conditional "where ... == nil" OTTL suffixes from set() statements
+       preNormalizeNode(root)
+}
+
+// preNormalizeNode recursively walks the tree. String elements inside slices
+// are rewritten; map keys and scalar map values are left to the later
+// normalization pass.
+func preNormalizeNode(node any) {
+       switch n := node.(type) {
+       case map[string]any:
+               for _, v := range n {
+                       preNormalizeNode(v)
+               }
+       case []any:
+               for i, elem := range n {
+                       if s, ok := elem.(string); ok {
+                       s = ottlConditionalDataStreamAttr.ReplaceAllString(s, "")
+                       if s == "forward" || strings.HasPrefix(s, "forward/") {
+                               s = "forward/_bare"
+                       }
+                               n[i] = s
+                       } else {
+                               preNormalizeNode(elem)
+                       }
+               }
+       }
+}
+
 // normalizePolicyToCanonical rewrites OTel component IDs to deterministic type/componentid-N
 // and updates all references. It works on the decoded tree and sorts variable keys by
 // canonical value so that equivalent policies with different map key order normalize to
@@ -246,6 +311,8 @@ func normalizePolicyToCanonical(policy []byte) ([]byte, error) {
                return nil, fmt.Errorf("failed to decode policy: %w", err)
        }
 
+       preNormalizePolicy(root)
+
        // Build mapping oldKey -> newKey (e.g. "elasticsearch/default" -> "elasticsearch/componentid-0")
        // by processing each variable-key section with deterministic (value-based) key order.
        idMapping := make(map[string]string)
diff --git a/internal/testrunner/runners/policy/policy_test.go b/internal/testrunner/runners/policy/policy_test.go
index 86f76b57..7e12e78e 100644
--- a/internal/testrunner/runners/policy/policy_test.go
+++ b/internal/testrunner/runners/policy/policy_test.go
@@ -227,6 +227,86 @@ service:
                assert.Contains(t, string(out), "- elasticsearch/componentid-0")
        })
 
+       t.Run("bare forward connector normalizes to forward/componentid-N", func(t *testing.T) {
+               policy := `
+connectors:
+  forward: {}
+service:
+  pipelines:
+    logs/my-stream:
+      receivers:
+        - otlp/my-stream
+      exporters:
+        - forward
+    logs:
+      receivers:
+        - forward
+      exporters:
+        - elasticsearch/default
+`
+               out, err := normalizePolicyToCanonical([]byte(policy))
+               assert.NoError(t, err)
+               assert.Contains(t, string(out), "forward/componentid-0")
+               assert.NotContains(t, string(out), "forward: {}")
+               assert.NotContains(t, string(out), "_bare")
+       })
+
+       t.Run("bare and suffixed forward normalize to same result", func(t *testing.T) {
+               bare := `
+connectors:
+  forward: {}
+service:
+  pipelines:
+    logs/my-stream:
+      exporters:
+        - forward
+    logs:
+      receivers:
+        - forward
+      exporters:
+        - elasticsearch/default
+`
+               suffixed := `
+connectors:
+  forward/default: {}
+service:
+  pipelines:
+    logs/my-stream:
+      exporters:
+        - forward/default
+    logs/default:
+      receivers:
+        - forward/default
+      exporters:
+        - elasticsearch/default
+`
+               outBare, err := normalizePolicyToCanonical([]byte(bare))
+               assert.NoError(t, err)
+               outSuffixed, err := normalizePolicyToCanonical([]byte(suffixed))
+               assert.NoError(t, err)
+               assert.Equal(t, string(outBare), string(outSuffixed))
+       })
+
+       t.Run("strips OTTL where clause from data_stream set statements", func(t *testing.T) {
+               policy := `
+processors:
+  transform/abc:
+    log_statements:
+      - context: log
+        statements:
+          - set(attributes["data_stream.type"], "logs")
+          - set(attributes["data_stream.dataset"], "my_pkg.events") where attributes["data_stream.dataset"] == nil
+          - set(attributes["data_stream.namespace"], "ep") where attributes["data_stream.namespace"] == nil
+`
+               out, err := normalizePolicyToCanonical([]byte(policy))
+               assert.NoError(t, err)
+               s := string(out)
+               assert.Contains(t, s, `set(attributes["data_stream.dataset"], "my_pkg.events")`)
+               assert.Contains(t, s, `set(attributes["data_stream.namespace"], "ep")`)
+               assert.NotContains(t, s, "where")
+               assert.NotContains(t, s, "== nil")
+       })
+
        t.Run("order-independent: same components different key order normalize to same result", func(t *testing.T) {
                policyA := `
 exporters:

  18. mrodm commented on Jul 6, 2026

    @mrodm
    mrodm
    on Jul 6, 2026
    Collaborator

    Thanks @efd6 !
    We were also thinking in another approach elastic/elastic-package#3710 (still in discussion) to create different expected policy tests depending on the stack version.

    Currently, I'm not sure if that approach in elastic/elastic-package#3710 could introduce some problems rather than help. For instance, how to decide when to create a new policy test for a specific range stack versions or if it should be considered an actual error.
    cc @teresaromero

    We could bump the kibana version constraint, but that would drop 9.4.0-9.4.2 users purely to avoid the flaky test failure.

    As you mentioned, I would like some approach to avoid bumping the kibana version.

  19. teresaromero commented on Aug 6, 2026

    @teresaromero
    teresaromero
    on Aug 6, 2026
    Contributor

    OTel policy test failures resolved by elastic-package v0.126.0 (PR #20512). Serverless builds #2012 and #2013 (2026-08-06) pass cleanly.

  20. teresaromero
    closed this as completedon Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Team:EcosystemPackages Ecosystem team [elastic/ecosystem]automationflaky-testUnstable or unreliable test cases.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions