[aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes by brijesh-elastic · Pull Request #20403 · elastic/integrations

brijesh-elastic · 2026-07-30T05:40:27Z

Proposed commit message

aws.cloudtrail: ingest pipeline improvements and fixes

This commit updates the aws.cloudtrail ingest pipeline with the following enhancements and bug fixes:

Enhancements:
- Replace every `ignore_failure: true` with an `ignore_missing` flag, a condition guard, or
  an explicit `on_failure` handler reporting to `error.message`, so real faults surface
  instead of being silently swallowed.
- Add support for CloudTrail fields that were previously dropped.
- Populate the `cloud.provider`, `cloud.service.name` and `related.*` ECS fields.

Bugfixes:
- Stop populating `file.hash.sha256` on digest records, where it held the hash of the previous
  digest file while `file.path` pointed at the current one. The value is still available as
  `aws.cloudtrail.digest.previous_hash_value` and in `related.hash`.
- Fix `DeleteObject` using `event.type: delete`, which is not a valid ECS value. It is now `deletion`.
- Fix `Publish`, `SendCommand` and `Converse` producing no `event.type`. They now use
  `info`, `start` and `info` respectively.
- Fix `retain: none` behaving like `retain: all`. It now drops both the keyword and the
  flattened copies as documented.
- Fix `retain: keyword` and `retain: minimal` discarding digest and Insights records. The setting
  governs API call payloads and no longer affects either record type.
- Fix `aws.cloudtrail.service_event_details` surviving `retain: flattened` and `retain: none`. It is
  now governed like the other payload fields.

Note

These changes aren't classified as breaking because fixing a feature that never worked correctly is considered a bugfix, not a breaking-change. With that in mind, none of the changes above fall into the breaking-change category.

To Reviewers:

This PR is best reviewed commit-by-commit.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/aws directory.
Run the following command to run tests.

elastic-package test -v

Related issues

Relates aws.cloudtrail: Ingest pipeline & ECS mapping improvements #20040

…r condition guard where failure was expected and an explicit `on_failure` handler reporting to `error.message` where it wasn't, so routine cases no longer throw per document and real faults surface instead of being silently swallowed

…ix file.hash.sha256 mapping to use previousDigestHashValue instead of previousDigestSignature

elastic-vault-github-plugin-prod · 2026-08-10T09:36:22Z

🚀 Benchmarks report

Package `aws` 👍(8) 💚(8) 💔(6)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`firewall_logs`	3937.01	3289.47	-647.54 (-16.45%)	💔
`inspector`	1763.67	1356.85	-406.82 (-23.07%)	💔
`rds`	29411.76	24390.24	-5021.52 (-17.07%)	💔
`route53_public_logs`	15873.02	12195.12	-3677.9 (-23.17%)	💔
`route53_resolver_logs`	10309.28	7299.27	-3010.01 (-29.2%)	💔
`securityhub_insights`	1114.83	904.98	-209.85 (-18.82%)	💔

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-08-11T05:56:44Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2026-08-12T21:14:26Z

/test

brijesh-elastic · 2026-08-18T12:42:06Z

/test

efd6

Nits then LGTM

efd6 · 2026-08-23T21:04:25Z

 errors:
  exclude_checks:
    - SVR00004 # References in dashboards.
+    - SVR00009


Add a comment for this.

elastic-vault-github-plugin-prod · 2026-08-24T05:37:39Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-08-24T05:49:09Z

🟢 Reviewed the latest commits 75bf34c — nothing new beyond already posted comments.

Review summary

Issues found across earlier commits 039d3b8…4b0b230 (120 commits) — 1 medium

Package-level:

🟡 Eleven of the newly mapped CloudTrail fields have no pipeline test fixture, so a wrong JSON path in one of the new nested renames would silently no-op forever

Issues found across earlier commits aa0d611, 33d0044 — 1 medium, 1 low

🟡 Dropping file.hash.sha256 from digest records removes a field users may query on, so it should be its own changelog entry with type breaking-change rather than being bundled into the bugfix list. (link) (Resolved)
🔵 The on_failure message template uses {{{_ingest.pipeline}}}, which is not an ingest metadata field and renders empty (link) (Resolved)

Issues found across earlier commits fb56839 — 2 medium

🟡 The new append_related_hosts_source_address processor puts AWS service principal names such as config.amazonaws.com into related.hosts (link) (Resolved)
🟡 file.hash.sha256 is now populated from previousDigestHashValue, which hashes the previous digest file, while file.path is set from digestS3Object, the current one (link) (Resolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

infra-vault-gh-plugin-prod · 2026-08-24T07:54:16Z

💚 Build Succeeded

Buildkite Build
Commit: 75bf34c

History

💚 Build #48342 succeeded 0d9d831
💔 Build #48273 failed f27b03a
💚 Build #48142 succeeded 4b0b230
💚 Build #48046 succeeded 33d0044
💔 Build #47765 failed 33d0044
💔 Build #47702 failed 33d0044

cc @brijesh-elastic

efd6

LGTM but please wait for @kcreddy

mergify · 2026-08-25T03:37:36Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

kcreddy

LGTM

elastic-vault-github-plugin-prod · 2026-08-25T10:04:20Z

Package aws - 7.4.0 containing this change is available at https://epr.elastic.co/package/aws/7.4.0/

Backport of #20403 to `backport-aws-6.x` so that users on the 8.x Kibana stack (package 6.x, Kibana `^8.19.4 || ^9.2.1`) receive the CloudTrail ingest pipeline improvements. This brings the `cloudtrail` data stream to parity with `main`. Because `backport-aws-6.x` did not have the intermediate CloudTrail change from #20318, that change is included here as well (it is a prerequisite for the #20403 pipeline and is part of the same CloudTrail quality effort). Enhancements: - Replace every `ignore_failure: true` with an `ignore_missing` flag, a condition guard, or an explicit `on_failure` handler reporting to `error.message`, so real faults surface instead of being silently swallowed. - Add support for CloudTrail fields that were previously dropped. - Populate the `cloud.provider`, `cloud.service.name` and `related.*` ECS fields. - Stop storing the assumed-role ARN session name in `user.changes.*`. The session name is now added to `related.user` for cross-source correlation; when it is an email, both the full email and the local-part prefix are added. `user.name` continues to hold the IAM role name for detection rules. (#20318) Bugfixes: - Stop populating `file.hash.sha256` on digest records, where it held the hash of the previous digest file while `file.path` pointed at the current one. The value is still available as `aws.cloudtrail.digest.previous_hash_value` and in `related.hash`. - Fix `DeleteObject` using `event.type: delete`, which is not a valid ECS value. It is now `deletion`. - Fix `Publish`, `SendCommand` and `Converse` producing no `event.type`. They now use `info`, `start` and `info` respectively. - Fix `retain: none` behaving like `retain: all`. It now drops both the keyword and the flattened copies as documented. - Fix `retain: keyword` and `retain: minimal` discarding digest and Insights records. The setting governs API call payloads and no longer affects either record type. - Fix `aws.cloudtrail.service_event_details` surviving `retain: flattened` and `retain: none`. It is now governed like the other payload fields.

brijesh-elastic self-assigned this Jul 30, 2026

brijesh-elastic added 3 commits July 30, 2026 11:10

update changelog entry

6584de8

Merge branch 'main' into aws_cloudtrail-ingest_pipeline-improvements

d8f26cb

Add support for new CloudTrail fields that were previously dropped, F…

1046295

…ix file.hash.sha256 mapping to use previousDigestHashValue instead of previousDigestSignature

Improve ECS mapping and fix event categorization bugs

fb56839

brijesh-elastic force-pushed the aws_cloudtrail-ingest_pipeline-improvements branch from a46b662 to fb56839 Compare August 10, 2026 12:41

brijesh-elastic added the bugfix Pull request that fixes a bug issue label Aug 11, 2026

brijesh-elastic marked this pull request as ready for review August 11, 2026 05:56

brijesh-elastic requested review from a team as code owners August 11, 2026 05:56

vera-review-bot Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml Outdated

Comment thread packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml Outdated

efd6 reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/aws/data_stream/cloudtrail/sample_event.json

Comment thread ...ages/aws/data_stream/cloudtrail/_dev/test/pipeline/test-update-lambda-json.log-expected.json

Comment thread packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml Outdated

brijesh-elastic added 2 commits August 12, 2026 11:43

address comments

aa0d611

Fix few bugs around retain option

33d0044

brijesh-elastic changed the title ~~[aws.cloudtrail] Ingest Pipeline Improvments~~ [aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes Aug 12, 2026

brijesh-elastic requested a review from efd6 August 12, 2026 07:34

vera-review-bot Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/aws/changelog.yml

Comment thread packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml Outdated

Merge branch 'main' into aws_cloudtrail-ingest_pipeline-improvements

4b0b230

kcreddy reviewed Aug 19, 2026

View reviewed changes

brijesh-elastic added 3 commits August 20, 2026 16:04

address comments from krishna

f27b03a

Merge branch 'main' into aws_cloudtrail-ingest_pipeline-improvements

415e580

exclude error check SVR00009

0d9d831

brijesh-elastic requested a review from kcreddy August 21, 2026 06:43

efd6 reviewed Aug 23, 2026

View reviewed changes

trim the test log file and add proper comments in validation.yml

75bf34c

efd6 approved these changes Aug 24, 2026

View reviewed changes

kcreddy approved these changes Aug 25, 2026

View reviewed changes

zmoog approved these changes Aug 25, 2026

View reviewed changes

brijesh-elastic merged commit a3291a9 into elastic:main Aug 25, 2026
10 checks passed

brijesh-elastic added the Category: Integration quality Category: Quality used for SI planning label Aug 27, 2026

brijesh-elastic mentioned this pull request Aug 31, 2026

[aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-6.x #20972

Merged

5 tasks

bryans3c mentioned this pull request Sep 2, 2026

[AWS]: Map userIdentity.onBehalfOf for IAM Identity Center events #21018

Open

brijesh-elastic mentioned this pull request Sep 8, 2026

[aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-7.1 #21129

Open

Uh oh!

Conversation

brijesh-elastic commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package aws 👍(8) 💚(8) 💔(6)

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Aug 12, 2026

Uh oh!

brijesh-elastic commented Aug 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efd6 Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 24, 2026

Uh oh!

vera-review-bot Bot commented Aug 24, 2026

Review summary

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 24, 2026

💚 Build Succeeded

History

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 25, 2026

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

brijesh-elastic commented Jul 30, 2026 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Aug 10, 2026 •

edited

Loading

Package `aws` 👍(8) 💚(8) 💔(6)