You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[snowflake] Add Support for query_history, events, and external_access_history Data Streams - #22142
snowflake: add support for query_history, events, and external_access_history data streams.
This release includes query_history, events, and external_access_history data streams.
The fields are mapped to their corresponding ECS fields where possible.
Field Schemas and Test samples were derived from live data samples and documentation[1], which
were subsequently sanitized.
Bumped the minimum kibana constraint to 9.4.0.
[1] https://docs.snowflake.com/en/sql-reference/snowflake-db
These three fields produce the same _id for distinct logs that share a timestamp and message, even when they belong to different snow.query.id values or executables. Consequently, one document is rejected as a duplicate even when both rows arrive in the same result page. Include the remaining source/record context in the fingerprint so deduplication removes replayed rows rather than unrelated handler logs.
A JSON-formatted handler message such as {"operation":"sync","result":"failed"} becomes a Map here, but the later message processors only handle Maps with formattedMessage or non-Map values. Cleanup then deletes the payload, losing the message content when original-event preservation is disabled. Parse into a temporary field and replace VALUE only for recognized Openflow envelopes, leaving other messages unchanged for the existing fallback.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The attribute merge can silently discard nested custom values when scalar and dotted keys conflict; normalize conflicting subtrees and add regression coverage for an undocumented prefix.
The reason will be displayed to describe this comment to others. Learn more.
destination.domain is a single-value ECS field, so the first hostname is always mapped, and every hostname still goes to related.hosts. Multi-value lists also keep the raw vendor field (the later remove only runs when the vendor value equals that single domain). I updated the script description to match that.
The reason will be displayed to describe this comment to others. Learn more.
This is Intended. The heatmap is the full weekday/hour grid so off-hours cells are visible against business-hours activity. A WHERE on off-hours would leave a sparse grid and hide that contrast. The panel title is “Off-Hours Login Activity (UTC)” in that sense — it highlights unusual hours rather than filtering to them.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed for unpublished findings involving pagination cursor loss during throttling and misclassification of security-sensitive user changes.
Issues found across the latest commits 0387d92 — 2 low
🔵 If RESOURCE_ATTRIBUTES, SCOPE, RECORD or RECORD_ATTRIBUTES fails to parse as JSON, the raw string is still renamed into an object-mapped snowflake.events.* field, so Elasticsearch rejects the document. Guard the four renames on instanceof Map (or drop the unparsed column in each on_failure). (link)
🔵 snowflake.events.resource_attributes.k8s.container.name is now always removed after it is copied to container.name, but events/fields.yml (and the README field table) still declare it. Delete that leaf, keep restart_count, and rebuild the README. (link)
Issues found across earlier commits 03edd7e — 1 high
Package-level:
🟠 The new events, query_history and external_access_history pipeline test directories have no test-common-config.yml. Add one per stream that sets the preserve_original_event tag.
Add _dev/test/pipeline/test-common-config.yml to each of the three new data streams, then regenerate the expected outputs with elastic-package test pipeline --data-streams events,query_history,external_access_history --generate:
fields:
tags:
- preserve_original_event
Issues found across earlier commits 077b2a4 — 1 medium, 1 low
🟡 The events keyset compares the SQL value COALESCE(TO_VARCHAR(VALUE), '') with a cursor last_value copied from the API cell for the VARIANT VALUE column, and these two strings can differ. Build the tie-break key in SQL, either by folding VALUE into ROW_KEY or by returning a TO_VARCHAR(VALUE) key column. (link) (Outdated)
🔵 The 'lag-window resume' TIEBREAK mock rule can never match, because the earlier 16:00:00.123002000 empty-page rule also matches the lag request. Move this rule above that one, or anchor the 123002000 rule on binding 1. (link) (Outdated)
Issues found across earlier commits e81b3b1 — 6 medium, 4 low
🟡 events/fields.yml declares many fields that only SPAN, SPAN_EVENT, METRIC and EVENT rows carry, but the pipeline drops every non-LOG row, so nothing ever writes them. Remove these declarations, or stop dropping those row types. (link) (Outdated)
🟡 The events _id fingerprint uses only TIMESTAMP, RECORD_TYPE and VALUE, so separate LOG rows with the same message in the same millisecond get the same _id and all but one are rejected. Add the per-row context columns to the fingerprint. (link)
🟡 host.name is set to the external destination hostname, which duplicates destination.domain and misuses the ECS host field. Drop the host.name set and populate related.hosts from destination.domain. (link) (Outdated)
🟡 No events fixture covers the Openflow structured-VALUE branch (formattedMessage, throwable, mdc) or the malformed-JSON VALUE fallback. Add fixtures for both. (link)
🟡 Handler-defined RECORD_ATTRIBUTES keys are indexed unmapped under snowflake.events.record_attributes, which risks mapping conflicts and field-count growth. Route undocumented keys into a flattened field. (link) (Outdated)
🔵 set_event_duration runs whenever TOTAL_ELAPSED_TIME is non-null, even after the convert to long has failed, so a bad value turns the document into a pipeline_error. Guard the script on a numeric type. (link) (Outdated)
🔵 The mock rules for the empty page after the last LIMIT 2 page can never match, because their regexes expect the table name before the binding timestamp. Make the timestamp-first ordering the matching alternative, as the 09:00 rules do. (link) (Outdated)
🔵 The lag-window cursor is parsed with the strict -0700 layout, while the package's newer streams parse with Z0700. Use Z0700 so a Z-suffixed cursor cannot stall the stream. (link) (Outdated)
🔵 Non-LOG event-table rows are dropped only after the fingerprint, five JSON parses, three dot_expanders and three date parses. Move the RECORD_TYPE drop to just after parse_json. (link) (Outdated)
Package-level:
🟡 test-common-config.yml is missing from the new events, query_history and external_access_history pipeline test directories. Add one to each with tags: [preserve_original_event].
Add test-common-config.yml to each of the three new pipeline test directories, then regenerate the expected outputs:
fields:
tags:
- preserve_original_event
A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because pagination and watermark interactions can replay processed results during rate-limited retries or permanently skip late-arriving rows within the configured lag window.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
qcorporation
added
the
documentation
kcreddy
left a comment
Proposed commit message
Checklist
changelog.ymlfile.How to test this PR locally
To test the integration:
Screenshots