[aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-7.1 by brijesh-elastic · Pull Request #21129 · elastic/integrations

brijesh-elastic · 2026-09-08T11:53:24Z

What

Backport of #20403 to backport-aws-7.1 so that users on the 9.4.x–9.5.x Kibana
stacks (AWS package 7.1.x line, kibana: ^9.4.0) receive the CloudTrail ingest
pipeline improvements and bugfixes. This brings the cloudtrail data stream to
parity with main.

Why

main has moved on to the 7.2+/8.x line (kibana: ^9.6.0), and the 6.x
backport (#20972) covers Kibana 8.19.4/9.2–9.3. Users pinned to the 7.1.x
line — Kibana 9.4.x and 9.5.x — currently resolve to 7.1.2, which predates
#20403, so they do not get these CloudTrail fixes. This PR closes that gap.

What this includes vs. `main`

The cloudtrail ingest pipeline, fields.yml, sample_event.json, data-stream
manifest, docs, and pipeline test fixtures are byte-identical to the merged
main change (a3291a9 / [aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes #20403).
Package version bumped to 7.1.3 (format_version kept at 3.6.1).
SVR00009 kept excluded in validation.yml (pre-existing pipeline predates
that check).

Scope

Limited to the cloudtrail data stream and package-level files (changelog,
manifest, validation). No other data streams are modified.
Unlike the 6.x backport ([aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-6.x #20972), the assumed-role session-name change ([aws] cloudtrail: move assumed-role session name from user.changes.* to related.user #20318)
is not folded in here — it was already released on this line in 7.1.0, so
the code already contains it and no extra changelog entry is needed.

Enhancements

Replace every ignore_failure: true with an ignore_missing flag, a condition
guard, or an explicit on_failure handler reporting to error.message, so real
faults surface instead of being silently swallowed.
Add support for CloudTrail fields that were previously dropped.
Populate the cloud.provider, cloud.service.name and related.* ECS fields.

Bugfixes

Stop populating file.hash.sha256 on digest records, where it held the hash of
the previous digest file while file.path pointed at the current one. The value
is still available as aws.cloudtrail.digest.previous_hash_value and in
related.hash.
Fix DeleteObject using event.type: delete, which is not a valid ECS value.
It is now deletion.
Fix Publish, SendCommand and Converse producing no event.type. They now
use info, start and info respectively.
Fix retain: none behaving like retain: all. It now drops both the keyword and
the flattened copies as documented.
Fix retain: keyword and retain: minimal discarding digest and Insights records.
The setting governs API call payloads and no longer affects either record type.
Fix aws.cloudtrail.service_event_details surviving retain: flattened and
retain: none. It is now governed like the other payload fields.

How to test this PR locally

Check out this branch and start a stack on a supported 9.4+ version, e.g.
elastic-package stack up -d --version 9.4.0.
From packages/aws, run: elastic-package test pipeline --data-streams cloudtrail -v

This commit updates the aws.cloudtrail ingest pipeline with the following enhancements and bug fixes: Enhancements: - Replace every `ignore_failure: true` with an `ignore_missing` flag, a condition guard, or an explicit `on_failure` handler reporting to `error.message`, so real faults surface instead of being silently swallowed. - Add support for CloudTrail fields that were previously dropped. - Populate the `cloud.provider`, `cloud.service.name` and `related.*` ECS fields. Bugfixes: - Stop populating `file.hash.sha256` on digest records, where it held the hash of the previous digest file while `file.path` pointed at the current one. The value is still available as `aws.cloudtrail.digest.previous_hash_value` and in `related.hash`. - Fix `DeleteObject` using `event.type: delete`, which is not a valid ECS value. It is now `deletion`. - Fix `Publish`, `SendCommand` and `Converse` producing no `event.type`. They now use `info`, `start` and `info` respectively. - Fix `retain: none` behaving like `retain: all`. It now drops both the keyword and the flattened copies as documented. - Fix `retain: keyword` and `retain: minimal` discarding digest and Insights records. The setting governs API call payloads and no longer affects either record type. - Fix `aws.cloudtrail.service_event_details` surviving `retain: flattened` and `retain: none`. It is now governed like the other payload fields.

infra-vault-gh-plugin-prod · 2026-09-08T11:53:30Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

github-actions · 2026-09-08T11:55:08Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

elastic-vault-github-plugin-prod · 2026-09-08T16:58:43Z

🚀 Benchmarks report

Package `aws` 👍(10) 💚(6) 💔(6)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`firewall_logs`	4464.29	3571.43	-892.86 (-20%)	💔
`lambda_logs`	10000	8130.08	-1869.92 (-18.7%)	💔
`rds`	33333.33	27777.78	-5555.55 (-16.67%)	💔
`s3access`	4830.92	3508.77	-1322.15 (-27.37%)	💔
`apigateway_logs`	28571.43	15384.62	-13186.81 (-46.15%)	💔
`ec2_logs`	41666.67	33333.33	-8333.34 (-20%)	💔

To see the full report comment with /test benchmark fullreport

vera-review-bot · 2026-09-09T10:16:37Z

Review summary

Issues found across the latest commits 00af686 — 1 medium

Package-level:

🟡 external_id is added to the aws-s3 template but not to the CloudWatch template (aws-cloudwatch.yml.hbs) for the cloudtrail data stream

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

elastic-vault-github-plugin-prod · 2026-09-09T10:18:16Z

✅ All changelog entries have the correct PR link.

elastic-vault-github-plugin-prod · 2026-09-09T10:18:44Z

✅ Package owners are in sync with main.

brijesh-elastic · 2026-09-09T10:42:44Z

Review summary

Issues found across the latest commits 00af686 — 1 medium
Package-level:

🟡 external_id is added to the aws-s3 template but not to the CloudWatch template (aws-cloudwatch.yml.hbs) for the cloudtrail data stream

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

The reason for adding external_id to aws-s3.yml.hbs is that it was mistakenly removed in earlier commits. For aws-cloudwatch.yml.hbs, it was never there in the first place.

infra-vault-gh-plugin-prod · 2026-09-09T12:30:14Z

💚 Build Succeeded

Buildkite Build
Commit: 00af686

History

💚 Build #49315 succeeded 3b20b58
💔 Build #49293 failed dfa1020

cc @brijesh-elastic

efd6

This backport is wider than justified. It should only include the bugfixes.

brijesh-elastic · 2026-09-11T02:52:26Z

This backport is wider than justified. It should only include the bugfixes.

I've been asked to backport the entire change related to the ingest pipeline quality work.

I've already raised a PR targeting the backport-aws-6x branch, which covers users on 8.19.x and 9.3.x. This PR will cover users on 9.4.x through 9.5.x.

So, this change also addresses the SDH alongside the other work.

efd6 · 2026-09-17T20:59:11Z

+        - append:
+            tag: append_error_message_extract_session_name_from_arn
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'


Suggested change

value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'

value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'

(throughout)

We are in default.

brijesh-elastic self-assigned this Sep 8, 2026

brijesh-elastic requested review from a team as code owners September 8, 2026 11:53

update changelog entry

dfa1020

brijesh-elastic requested a review from efd6 September 8, 2026 11:55

run tests in 9.4.0

3b20b58

MichaelKatsoulis reviewed Sep 9, 2026

View reviewed changes

Comment thread packages/aws/data_stream/cloudtrail/agent/stream/aws-s3.yml.hbs

address comments

00af686

brijesh-elastic requested a review from MichaelKatsoulis September 9, 2026 10:39

MichaelKatsoulis approved these changes Sep 9, 2026

View reviewed changes

brijesh-elastic mentioned this pull request Sep 10, 2026

aws.cloudtrail: widen ARN partition grok pattern to include aws-cn and aws-iso* (7.1.x) #21160

Open

efd6 reviewed Sep 10, 2026

View reviewed changes

brijesh-elastic requested a review from efd6 September 11, 2026 05:24

kcreddy mentioned this pull request Sep 16, 2026

aws.cloudtrail: Ingest pipeline & ECS mapping improvements #20040

Open

12 tasks

efd6 reviewed Sep 17, 2026

View reviewed changes

Navigation Menu

[aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-7.1 - #21129

[aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-7.1#21129
brijesh-elastic wants to merge 4 commits into
elastic:backport-aws-7.1from
brijesh-elastic:backport-aws-7.1-cloudtrail-pipeline

brijesh-elastic commented Sep 8, 2026 •

edited by efd6

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

vera-review-bot Bot commented Sep 9, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 9, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 9, 2026

Uh oh!

brijesh-elastic commented Sep 9, 2026

Review summary

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 9, 2026

Uh oh!

efd6 left a comment

Uh oh!

brijesh-elastic commented Sep 11, 2026

Uh oh!

efd6 Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
	value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'

Conversation

brijesh-elastic commented Sep 8, 2026 • edited by efd6 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

What this includes vs. main

Scope

Enhancements

Bugfixes

How to test this PR locally

Related

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package aws 👍(10) 💚(6) 💔(6)

Uh oh!

Uh oh!

vera-review-bot Bot commented Sep 9, 2026

Review summary

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 9, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 9, 2026

Uh oh!

brijesh-elastic commented Sep 9, 2026

Review summary

Uh oh!

infra-vault-gh-plugin-prod Bot commented Sep 9, 2026

💚 Build Succeeded

History

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

brijesh-elastic commented Sep 11, 2026

Uh oh!

efd6 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

brijesh-elastic commented Sep 8, 2026 •

edited by efd6

Loading

What this includes vs. `main`

elastic-vault-github-plugin-prod Bot commented Sep 8, 2026 •

edited

Loading

Package `aws` 👍(10) 💚(6) 💔(6)