[snowflake] Initial release with access_history, data_transfer_history, and stages data streams by mohitjha-elastic · Pull Request #21995 · elastic/integrations

mohitjha-elastic · 2026-10-14T09:09:53Z

Proposed commit message

snowflake: Initial release with access_history, data_transfer_history, and stages data streams.

This release includes the access_history, data_transfer_history, and stages data streams.
The fields are mapped to their corresponding ECS fields where possible.
Field Schemas and Test samples were derived from live data samples and documentation[1], which were 
subsequently sanitized.

[1] https://docs.snowflake.com/en/sql-reference/snowflake-db

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

To test the integration:

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/snowflake directory.
Run the following command to run tests.

elastic-package test -v

…a_transfer_history, and stages data streams.

infra-vault-gh-plugin-prod · 2026-10-14T09:10:01Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

github-actions · 2026-10-14T09:11:42Z

Elastic Docs Style Checker (Vale)

Summary: 1 suggestion found

💡 Suggestions (1): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/snowflake/changelog.yml	1	Elastic.Versions	Use 'later versions' instead of 'newer versions' when referring to versions.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

elastic-vault-github-plugin-prod · 2026-10-14T10:02:57Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

1. Update cel code and refactor them. 2. Update script in pipeline.

kcreddy

This release includes access_history, data_transfer_history, and stages data streams,
associated dashboard and visualizations.

@mohitjha-elastic, I don't see any dashboards in this PR. Are they included somewhere else?

ECS git@v9.5.0 / ecs.version: 9.5.0 datastore category Dropped async=true; POST is sync. HTTP 202 polling is still there as a safety path Full batch_size pages continue from the watermark without lag until the backlog drains Duplicate-row wording in the docs User-stage objectName → related.user cloud.origin / cloud.target on data transfer Stage COMMENT → message; STAGE_OWNER → related.user; STAGE_REGION → cloud.region; s3/azure/gcs → cloud.provider

mohitjha-elastic · 2026-10-16T10:51:37Z

This release includes access_history, data_transfer_history, and stages data streams,
associated dashboard and visualizations.

@mohitjha-elastic, I don't see any dashboards in this PR. Are they included somewhere else?

Dashboards are not included in this PR, as the recommended dashboards outlined in the main issue cannot be created using these three data streams alone due to the limited number of available fields. I’ll explore creating a combined dashboard with visualizations from multiple data streams in a follow-up PR once we have a useful set of views.

Updated the commit message.

Reframe the SQL statement.

efd6 · 2026-10-20T02:28:38Z

Please reformat this; there have been nesting changes that change indentation.

@efd6 Reformatted the cel code and also ran celfmt command. Thanks for pointing out.

elastic-vault-github-plugin-prod · 2026-10-20T07:33:28Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-10-20T07:39:04Z

🟢 No issues across the latest commits 6ff7c89.

Review summary

Issues found across earlier commits fc8ce20 — 1 medium

🟡 The data_transfer_history catch-up predicate compares COALESCE(col, '') but both statements ORDER BY the raw nullable columns, so the stored continuation key disagrees with the sort order when cloud/region columns are NULL (link) (Outdated)

Issues found across earlier commits c4518bc — 2 medium, 1 low

🟡 The catching-up POST resumes with a strict > max(last_timestamp), so rows beyond the LIMIT that tie with or sort below the watermark are skipped. Continue from the last returned row's ORDER BY key instead of the max watermark. (link) (Outdated)
🟡 STAGE_OWNER is a role or application name (SYSADMIN, ACCOUNTADMIN, SNOWFLAKE), not a user, so remove the append_stage_owner_to_related processor instead of copying it into related.user. (link) (Outdated)
🔵 The new # data_transfer_history comment at line 160 heads access_history's partition GET rules, and the new # stages comment at line 397 heads data_transfer_history's. Relabel each one for the stream whose statement handle it serves. (link) (Outdated)

Issues found across earlier commits fbe3577 — 1 high, 1 medium, 2 low

🟡 Pipeline fixtures cover only happy-path rows (link)
🔵 beats.yml declares log.offset, which the CEL input never emits and no pipeline sets (link) (Outdated)

Package-level:

🟠 Missing test-common-config.yml in the access_history, data_transfer_history and stages pipeline test directories

Add test-common-config.yml to each of the three _dev/test/pipeline/ directories, then regenerate the expected outputs with elastic-package test pipeline --generate:
```
fields:
  tags:
    - preserve_original_event
```

🔵 Proposed commit message

snowflake: initial release with access_history, data_transfer_history, and stages data streams

Add the Snowflake integration, which uses the CEL input to query
SNOWFLAKE.ACCOUNT_USAGE views through the Snowflake SQL API with a
programmatic access token. The access_history data stream collects object
and column access events, data_transfer_history collects hourly
cross-cloud and cross-region transfer volumes, and stages collects stage
inventory changes. Each stream keeps a watermark cursor and re-queries a
configurable lag window so that rows reaching the views late are not
missed. Fields are mapped to ECS where possible.

[1] https://docs.snowflake.com/en/sql-reference/snowflake-db

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

infra-vault-gh-plugin-prod · 2026-10-20T07:51:59Z

💚 Build Succeeded

Buildkite Build
Commit: 6ff7c89

History

💚 Build #51743 succeeded cbff278
💚 Build #51739 succeeded fc8ce20
💚 Build #51706 succeeded c4518bc
💚 Build #51652 succeeded da324b8
💚 Build #51590 succeeded 6df1a22
💔 Build #51588 failed fbe3577

cc @mohitjha-elastic

kcreddy

LGTM, please wait for @efd6

mergify · 2026-10-20T10:59:25Z

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

elastic-vault-github-plugin-prod · 2026-10-21T08:16:37Z

Package snowflake - 0.1.0 containing this change is available at https://epr.elastic.co/package/snowflake/0.1.0/

Initial release of the Snowflake integration with access_history, dat…

bc1a0ca

…a_transfer_history, and stages data streams.

mohitjha-elastic self-assigned this Oct 14, 2026

mohitjha-elastic added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Oct 14, 2026

mohitjha-elastic requested a review from a team as a code owner October 14, 2026 09:09

Update changelog entry

fbe3577

mohitjha-elastic mentioned this pull request Oct 14, 2026

[New Integration] Snowflake #11617

Closed

8 tasks

vera-review-bot Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread ...flake/data_stream/access_history/_dev/test/pipeline/test-snowflake-access-history-sample.log Resolved

Comment thread packages/snowflake/data_stream/access_history/fields/beats.yml Resolved

Resolve vera bot comment

6df1a22

efd6 reviewed Oct 15, 2026

View reviewed changes

Resolve review comments by Dan.

da324b8

1. Update cel code and refactor them. 2. Update script in pipeline.

mohitjha-elastic requested a review from efd6 October 15, 2026 11:36

kcreddy reviewed Oct 15, 2026

View reviewed changes

kcreddy mentioned this pull request Oct 16, 2026

[snowflake] Initial release with login_history, sessions, and grants_to_users Data Streams #21885

Merged

5 tasks done

mohitjha-elastic requested a review from kcreddy October 16, 2026 10:58

vera-review-bot Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread packages/snowflake/data_stream/access_history/agent/stream/cel.yml.hbs Resolved

Comment thread packages/snowflake/data_stream/stages/elasticsearch/ingest_pipeline/default.yml Resolved

Comment thread packages/snowflake/_dev/deploy/docker/files/config.yml Resolved

Resolve vera bot review comment

fc8ce20

vera-review-bot Bot reviewed Oct 19, 2026

View reviewed changes

Comment thread packages/snowflake/data_stream/data_transfer_history/agent/stream/cel.yml.hbs Resolved

Resolve vera bot comment.

cbff278

Reframe the SQL statement.

efd6 reviewed Oct 20, 2026

View reviewed changes

Fix indentation in cel program

6ff7c89

mohitjha-elastic requested a review from efd6 October 20, 2026 07:45

mohitjha-elastic added a commit to mohitjha-elastic/integrations that referenced this pull request Oct 20, 2026

Resolve comments as per PR elastic#21995

6f0502e

kcreddy approved these changes Oct 20, 2026

View reviewed changes

efd6 approved these changes Oct 21, 2026

View reviewed changes

mohitjha-elastic merged commit d598877 into elastic:main Oct 21, 2026
16 checks passed

mohitjha-elastic deleted the snowflake-0.2.0 branch October 21, 2026 07:58

Conversation

mohitjha-elastic commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Uh oh!

infra-vault-gh-plugin-prod Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Elastic Docs Style Checker (Vale)

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Oct 14, 2026

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mohitjha-elastic commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

mohitjha-elastic Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Oct 20, 2026

Uh oh!

vera-review-bot Bot commented Oct 20, 2026

Review summary

Uh oh!

infra-vault-gh-plugin-prod Bot commented Oct 20, 2026

💚 Build Succeeded

History

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Oct 21, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

mohitjha-elastic commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

mohitjha-elastic commented Oct 16, 2026 •

edited

Loading

mergify Bot commented Oct 20, 2026 •

edited

Loading