[ti_*] Add `is_ioc_transform_source` label so Threat Intel rules skip raw indices by sharad-elastic · Pull Request #22249 · elastic/integrations

sharad-elastic · 2026-10-30T14:05:16Z

Proposed commit message

ti_cyware_intel_exchange,ti_flashpoint,ti_google_threat_intelligence,ti_greynoise: add is_ioc_transform_source label

The built-in Threat Intel Indicator Match detection rules exclude raw
source indices with `not labels.is_ioc_transform_source:"true"`. These
four packages labelled their source data streams and latest transform
destinations only with the non-standard `labels.is_transform_source`
field, so the rules matched their raw `logs-ti_*` indices alongside the
intended `_latest` destinations.

This adds the family-standard `labels.is_ioc_transform_source`
constant_keyword next to the existing field: "true" on the raw source
data streams and "false" on the latest transform destinations. The
non-standard `labels.is_transform_source` is kept but marked deprecated
in its field description and the README, and will be removed in a
future major release.

No ingest pipeline, CEL, saved object, or fleet_transform_version
changes are made, so the change is non-breaking and does not rebuild
existing transform destinations. Backing indices created before the
upgrade cannot be remapped and keep matching the rules until they age
out of the rules' 30-day window.

Closes #22147

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices — N/A, no dashboards are added or changed.

How to test this PR locally

For each affected package (ti_google_threat_intelligence, ti_greynoise, ti_flashpoint, ti_cyware_intel_exchange):

cd packages/<package>
elastic-package check

To confirm the fix end to end:

Install a package (for example ti_greynoise) and ingest into a source data stream.
Check the raw index mapping includes the new field set to true:
GET logs-ti_greynoise.ip-*/_mapping/field/labels.is_ioc_transform_source
and that the latest transform destination (logs-ti_greynoise_latest.ip-*) maps it to false.
Confirm the built-in "Threat Intel Indicator Match" rules no longer match the raw source index, because its documents now satisfy labels.is_ioc_transform_source:"true" and are excluded by the rules' not labels.is_ioc_transform_source:"true" clause.

Related issues

Closes Some ti_*: Source IOC data streams are matched by the built-in Threat Intel Indicator Match rules #22147

… currently missing

infra-vault-gh-plugin-prod · 2026-10-30T14:05:21Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

github-actions · 2026-10-30T14:06:29Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

Copilot

🟡 Changes recommended

Release-note links in four package changelogs point to #1 or #99999 instead of the implementing PR (#22249) and need correction.

1 open issue

Release notes link to unrelated pull requests · New

What changed in this PR

This PR aligns four threat intelligence integrations with the label used by Elastic’s built-in Indicator Match rules to exclude raw source data, addressing #22147 while retaining the existing field for compatibility.

Changes:

Add labels.is_ioc_transform_source as "true" on 19 source data streams and "false" on 12 transform destinations.
Document deprecation of labels.is_transform_source and upgrade limitations without rebuilding existing destinations.
Bump package versions and add release notes.

File	Description
`packages/ti_greynoise/manifest.yml`	Bump version to 0.12.0.
`packages/ti_greynoise/elasticsearch/transform/ip/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_greynoise/elasticsearch/transform/ip/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_greynoise/docs/README.md`	Document new label, deprecation, and upgrade limits.
`packages/ti_greynoise/data_stream/ip/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_greynoise/data_stream/ip/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_greynoise/changelog.yml`	Record label fix and upgrade caveat.
`packages/ti_greynoise/_dev/build/docs/README.md`	Add deprecation and upgrade guidance to template.
`packages/ti_google_threat_intelligence/manifest.yml`	Bump version to 1.6.0.
`packages/ti_google_threat_intelligence/elasticsearch/transform/vulnerability/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/vulnerability/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/elasticsearch/transform/url_ioc/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/url_ioc/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/elasticsearch/transform/url_ioc_st/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/url_ioc_st/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/elasticsearch/transform/ip_ioc/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/ip_ioc/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/elasticsearch/transform/ip_ioc_st/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/ip_ioc_st/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/elasticsearch/transform/file_ioc/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/file_ioc/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/elasticsearch/transform/file_ioc_st/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/file_ioc_st/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/elasticsearch/transform/domain_ioc/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/domain_ioc/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/elasticsearch/transform/domain_ioc_st/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_google_threat_intelligence/elasticsearch/transform/domain_ioc_st/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_google_threat_intelligence/docs/README.md`	Update field tables and explain deprecation and upgrades.
`packages/ti_google_threat_intelligence/data_stream/vulnerability/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/vulnerability/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/vulnerability_weaponization/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/vulnerability_weaponization/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/trending/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/trending/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/threat_actor/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/threat_actor/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/ransomware/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/ransomware/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/phishing/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/phishing/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/osx/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/osx/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/mobile/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/mobile/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/malware/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/malware/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/malicious_network_infrastructure/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/malicious_network_infrastructure/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/linux/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/linux/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/iot/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/iot/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/ioc_stream/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/ioc_stream/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/infostealer/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/infostealer/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/first_stage_delivery_vectors/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/first_stage_delivery_vectors/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/data_stream/cryptominer/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_google_threat_intelligence/data_stream/cryptominer/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_google_threat_intelligence/changelog.yml`	Record label fix and upgrade caveat.
`packages/ti_google_threat_intelligence/_dev/build/docs/README.md`	Add deprecation and upgrade guidance to template.
`packages/ti_flashpoint/manifest.yml`	Bump version to 0.7.0.
`packages/ti_flashpoint/elasticsearch/transform/latest_indicator/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_flashpoint/elasticsearch/transform/latest_indicator/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_flashpoint/docs/README.md`	Document new label, deprecation, and upgrade limits.
`packages/ti_flashpoint/data_stream/indicator/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_flashpoint/data_stream/indicator/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_flashpoint/changelog.yml`	Record label fix and upgrade caveat.
`packages/ti_flashpoint/_dev/build/docs/README.md`	Add deprecation and upgrade guidance to template.
`packages/ti_cyware_intel_exchange/manifest.yml`	Bump version to 0.8.0.
`packages/ti_cyware_intel_exchange/elasticsearch/transform/latest_ioc/fields/is-transform-source-false.yml`	Deprecate legacy destination label in description.
`packages/ti_cyware_intel_exchange/elasticsearch/transform/latest_ioc/fields/is-ioc-transform-source-false.yml`	Add destination label set to `"false"`.
`packages/ti_cyware_intel_exchange/docs/README.md`	Document new label, deprecation, and upgrade limits.
`packages/ti_cyware_intel_exchange/data_stream/indicator/fields/is-transform-source-true.yml`	Deprecate legacy source label in description.
`packages/ti_cyware_intel_exchange/data_stream/indicator/fields/is-ioc-transform-source-true.yml`	Add source label set to `"true"`.
`packages/ti_cyware_intel_exchange/changelog.yml`	Record label fix and upgrade caveat.
`packages/ti_cyware_intel_exchange/_dev/build/docs/README.md`	Add deprecation and upgrade guidance to template.

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Fleet’s mapping update on existing source write indices remains unverified, requiring an end-to-end upgrade check to confirm new raw writes are excluded while existing transform destinations remain queryable without rebuilding.

No open issues 🎉

1 resolved since last review

Release notes link to unrelated pull requests

_{Review effort: Balanced}

elastic-vault-github-plugin-prod · 2026-10-30T14:58:40Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Copilot

🔵 Needs a closer look

Human assessment is needed because upgrade guidance across four packages incorrectly ties label availability to write time and rollover rather than each index’s mapping.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

All 16 CEL templates still report v1.6.0 in their User-Agent despite the package bump to 1.7.0, violating the documented version synchronization requirement.

1 open issue

Update CEL template User-Agent versions to v1.7.0 · New

_{Review effort: Balanced}

elastic-vault-github-plugin-prod · 2026-11-03T13:08:26Z

✅ All changelog entries have the correct PR link.

Copilot

🔵 Needs a closer look

Human assessment is needed to evaluate documentation inaccuracies across four integrations concerning rollover, historical indicators, and index-wide constant_keyword behavior.

1 open issue

Update CEL template User-Agent versions to v1.7.0

_{Review effort: Balanced}

vera-review-bot · 2026-11-03T13:16:23Z

🟢 No issues across the latest commits 43b8907.

Review summary

Issues found across earlier commits 6ad7125 — 2 low

🔵 The GTI version is bumped to 1.6.0 but the CEL User-Agent is still "v1.5.0", despite the manifest comment requiring them to match. Update the User-Agent in the CEL stream templates. (link) (Outdated)

Package-level:

🔵 Proposed commit message

ti_cyware_intel_exchange,ti_flashpoint,ti_google_threat_intelligence,ti_greynoise: add is_ioc_transform_source label

The built-in Threat Intel Indicator Match detection rules exclude raw
source indices with `not labels.is_ioc_transform_source:"true"`. These
four packages labelled their source data streams and latest transform
destinations only with the non-standard `labels.is_transform_source`
field, so the rules matched their raw `logs-ti_*` indices alongside the
intended `_latest` destinations.

This adds the family-standard `labels.is_ioc_transform_source`
constant_keyword next to the existing field: "true" on the raw source
data streams and "false" on the latest transform destinations. The
non-standard `labels.is_transform_source` is kept but marked deprecated
in its field description and the README, and will be removed in a
future major release.

No ingest pipeline, CEL, saved object, or fleet_transform_version
changes are made, so the change is non-breaking and does not rebuild
existing transform destinations. Backing indices created before the
upgrade cannot be remapped and keep matching the rules until they age
out of the rules' 30-day window.

Closes #22147

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.4 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

infra-vault-gh-plugin-prod · 2026-11-03T13:52:59Z

💚 Build Succeeded

Buildkite Build
Commit: 43b8907

History

💚 Build #52576 succeeded 17143b3
💚 Build #52562 succeeded 6ad7125

cc @sharad-elastic

marc-gr

The change itself looks consistent. I went through all 94 files: the "true" label files, the "false" label files and the description edits each follow one pattern, every raw data stream that feeds a latest transform is labelled, and the rules' filter matches what the PR says. My comments are about the text and a few loose ends.

1. The upgrade sentence is not right (README and the 4 changelog entries).
On upgrade, Fleet tries to add the new mapping to the current write index of each data stream, in place, and only rolls over if that fails (write_index_only: true in Kibana's fleet/server/services/epm/elasticsearch/template/template.ts). Because the field is a constant_keyword, every document in that index gets the label at once, old and new. Older backing indices never get it, and a rollover does not change that.
So "only newly written source documents carry the label ... until they roll over" and "backing indices created before the upgrade cannot be relabeled" are both off. Suggested wording: "the current write index gets the label on upgrade; older backing indices keep matching the rules until their documents leave the 30-day window". I read this from source and did not run an upgrade.

2. On upgraded installs the destination label does not arrive, so the README advice gives empty results.
fleet_transform_version is not bumped, so Fleet skips the destination templates (transform/install.ts, if (!transformVersionChanged) return;). Existing _latest indices only get the field after a later bump or a rebuild. A user who reads the new note and queries labels.is_ioc_transform_source: "false" on an upgraded install gets nothing. "How to test" step 2 only holds on a fresh install.
The rules are fine, since they only exclude "true". Not bumping is allowed and the issue chose it. So the docs should match: suggest NOT labels.is_ioc_transform_source: "true" for latest data (as step 1.4 of the issue does) and say the destination field arrives on fresh install or the next rebuild.

3. The PR text says "No ingest pipeline, CEL ... changes", but 16 GTI cel.yml.hbs files change.
They bump the User-Agent from v1.6.0 to v1.7.0. The change is right (it matches the manifest), so only the text needs fixing.

4. Question: is Closes #22147 intended?
Step 2 in the issue (remove labels.is_transform_source in a later major version) is still open, so merging would close the issue and lose that tracker. A follow-up issue for step 2, or "Part of #22147", would avoid that.

5. Question: should #22273 be folded into this PR?
#22273 ("Document overlap with Elastic built-in Threat Intel rules") is open, touches 20 files and edits the same GTI files (CEL User-Agent lines, both READMEs, changelog, manifest, with GTI at 1.6.1). Whichever merges second gets conflicts and a version renumber. It also says Closes #22147. Either fold it in here, or agree a merge order and drop Closes from it.

Nits

The GTI and GreyNoise READMEs still show labels.is_transform_source: true/false example queries right under the new deprecation note. Switch them to the new label, or mark them as the deprecated field.
The new field description has a typo copied from other packages: "or the in latest destination index". ti_ticura and ti_whoisfreaks already say "or in the latest".
Out of scope, only for information: ti_anyrun's raw is-ioc-transform-source-false.yml holds the value "true", and ti_rapid7's alert and vulnerability streams feed transforms with no label (harmless, they carry no indicators).

I did not run a live upgrade, so 1 and 2 come from reading the Fleet source. elastic-package check passes in all four packages and the generated READMEs match.

Add the field to the source data streams of ti_* packages where it is…

5f2e195

… currently missing

sharad-elastic self-assigned this Oct 30, 2026

sharad-elastic added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Oct 30, 2026

Copilot AI balanced review requested due to automatic review settings October 30, 2026 14:05

sharad-elastic requested a review from a team as a code owner October 30, 2026 14:05

sharad-elastic added the Integration:ti_flashpoint Flashpoint label Oct 30, 2026

sharad-elastic requested a review from a team as a code owner October 30, 2026 14:05

Copilot started reviewing on behalf of sharad-elastic October 30, 2026 14:05

View session

update changelog entry

6ad7125

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Comment thread packages/ti_greynoise/changelog.yml Resolved

Copilot AI balanced review requested due to automatic review settings October 30, 2026 14:10

Copilot started reviewing on behalf of sharad-elastic October 30, 2026 14:10

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

vera-review-bot Bot reviewed Oct 30, 2026

View reviewed changes

Comment thread packages/ti_google_threat_intelligence/manifest.yml Resolved

qcorporation added bugfix Pull request that fixes a bug issue documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Oct 30, 2026

Update user agent version of gti

17143b3

Copilot AI balanced review requested due to automatic review settings November 1, 2026 07:53

Copilot started reviewing on behalf of sharad-elastic November 1, 2026 07:54

View session

Copilot AI reviewed Nov 1, 2026

View reviewed changes

Merge branch 'main' into add-ioc-transform-label-22147

622f15b

Copilot AI balanced review requested due to automatic review settings November 3, 2026 12:51

Copilot started reviewing on behalf of sharad-elastic November 3, 2026 12:51

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

Comment thread packages/ti_google_threat_intelligence/manifest.yml Resolved

sync user agent version with package manifest

43b8907

Copilot AI balanced review requested due to automatic review settings November 3, 2026 13:04

Copilot started reviewing on behalf of sharad-elastic November 3, 2026 13:04

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

marc-gr reviewed Nov 3, 2026

View reviewed changes

Conversation

sharad-elastic commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

infra-vault-gh-plugin-prod Bot commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Oct 30, 2026

🚀 Benchmarks report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Nov 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

vera-review-bot Bot commented Nov 3, 2026

Review summary

Uh oh!

infra-vault-gh-plugin-prod Bot commented Nov 3, 2026

💚 Build Succeeded

History

Uh oh!

marc-gr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sharad-elastic commented Oct 30, 2026 •

edited

Loading