ti_threatconnect: fix precision loss of indicator IDs above 2^53 by kcreddy · Pull Request #19110 · elastic/integrations

kcreddy · 2026-05-21T09:17:07Z

Proposed commit message

ti_threatconnect: fix precision loss of indicator IDs above 2^53

The ThreatConnect API returns indicator IDs as JSON numbers. When
these exceed the IEEE 754 safe integer limit (2^53), the CEL
program's decode_json() rounds them through float64, causing
distinct IDs like 9007199254918487, 9007199254918488, and
9007199254918489 to all collapse to 9007199254918488.

Replace decode_json() with decode_json_string_numbers() so that
all JSON numbers are preserved as their literal string form. The
ingest pipeline already uses convert processors with explicit type
coercions for numeric fields, so string-valued numbers are handled
correctly downstream.

Add a script test that verifies indicator IDs above 2^53 are
preserved correctly. The test mock returns two Address indicators
whose IDs both round to the same float64 value and asserts that
both distinct values appear in threat_connect.indicator.id after
ingestion.

The minimum stack version is raised to 8.19.0/9.1.0, which is
the first version shipping a mito release that includes the
decode_json_string_numbers function.

Fixes elastic/integrations#19109

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Added script test (with large numbers) passes.

--- Test results for package: ti_threatconnect - START ---
╭──────────────────┬─────────────┬───────────┬────────────────────┬────────┬─────────────────╮
│ PACKAGE          │ DATA STREAM │ TEST TYPE │ TEST NAME          │ RESULT │    TIME ELAPSED │
├──────────────────┼─────────────┼───────────┼────────────────────┼────────┼─────────────────┤
│ ti_threatconnect │ indicator   │ script    │ env                │ PASS   │      46.96475ms │
│ ti_threatconnect │ indicator   │ script    │ large_id_precision │ PASS   │ 1m26.988814666s │
╰──────────────────┴─────────────┴───────────┴────────────────────┴────────┴─────────────────╯
--- Test results for package: ti_threatconnect - END   ---
Done

Related issues

Closes ti_threatconnect: indicator IDs above 2^53 lose precision due to JSON float64 decoding #19109

Screenshots

Before (id is rounded)

After (id is precise)

The ThreatConnect API returns indicator IDs as JSON numbers. When these exceed the IEEE 754 safe integer limit (2^53), the CEL program's decode_json() rounds them through float64, causing distinct IDs like 9007199254918487, 9007199254918488, and 9007199254918489 to all collapse to 9007199254918488. Replace decode_json() with decode_json_string_numbers() so that all JSON numbers are preserved as their literal string form. The ingest pipeline already uses convert processors with explicit type coercions for numeric fields, so string-valued numbers are handled correctly downstream. The minimum stack version is raised to 8.19.0/9.1.0, which is the first version shipping a mito release that includes the decode_json_string_numbers function. Fixes elastic#19109

github-actions · 2026-05-21T09:33:48Z

✅ Vale Linting Results

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

infra-vault-gh-plugin-prod · 2026-05-21T09:50:37Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2026-05-21T09:55:06Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2026-05-21T09:59:54Z

          }
      }))).do_request().as(resp, resp.StatusCode == 200 ?
-          bytes(resp.Body).decode_json().as(body, {
+          bytes(resp.Body).decode_json_string_numbers().as(body, {


Suggested change

bytes(resp.Body).decode_json_string_numbers().as(body, {

resp.Body.decode_json_string_numbers().as(body, {

Updated in 74e813f

efd6 · 2026-05-21T10:02:27Z

              },
              {
-                "id": 736759,
+                "id": 9007199254918489,


I think we might need a script test to ensure that this is being handled correctly; as it is, we don't know whether we're getting a numeric truncation.

Done in 74e813f

Add a script test that verifies indicator IDs above 2^53 are preserved without float64 precision loss. The test mock returns two Address indicators with IDs 9007199254918487 and 9007199254918489 (both round to 9007199254918488 under IEEE 754 double-precision) and asserts that both distinct values appear in threat_connect.indicator.id after ingestion. Also remove the unnecessary bytes() wrapper from the decode_json_string_numbers call per review feedback.

elasticmachine · 2026-05-21T10:41:42Z

💚 Build Succeeded

Buildkite Build
Commit: 74e813f

History

💚 Build #43215 succeeded 1723ea0
💔 Build #43209 failed 9ff424f

cc @kcreddy

efd6

LGTM (float64 collision confirmation here)

elastic-vault-github-plugin-prod · 2026-05-22T03:48:41Z

Package ti_threatconnect - 2.1.1 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/2.1.1/

…stic#19110) The ThreatConnect API returns indicator IDs as JSON numbers. When these exceed the IEEE 754 safe integer limit (2^53), the CEL program's decode_json() rounds them through float64, causing distinct IDs like 9007199254918487, 9007199254918488, and 9007199254918489 to all collapse to 9007199254918488. Replace decode_json() with decode_json_string_numbers() so that all JSON numbers are preserved as their literal string form. The ingest pipeline already uses convert processors with explicit type coercions for numeric fields, so string-valued numbers are handled correctly downstream. Add a script test that verifies indicator IDs above 2^53 are preserved correctly. The test mock returns two Address indicators whose IDs both round to the same float64 value and asserts that both distinct values appear in threat_connect.indicator.id after ingestion. The minimum stack version is raised to 8.19.0/9.1.0, which is the first version shipping a mito release that includes the decode_json_string_numbers function. Fixes elastic#19109

kcreddy added 3 commits May 21, 2026 14:44

update pr number

06c71f4

Fix README

1723ea0

kcreddy self-assigned this May 21, 2026

kcreddy added bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Integration:ti_threatconnect ThreatConnect labels May 21, 2026

kcreddy marked this pull request as ready for review May 21, 2026 09:50

kcreddy requested review from a team as code owners May 21, 2026 09:50

efd6 reviewed May 21, 2026

View reviewed changes

kcreddy requested a review from efd6 May 21, 2026 10:43

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label May 21, 2026

efd6 approved these changes May 21, 2026

View reviewed changes

kcreddy merged commit 32d1f41 into elastic:main May 22, 2026
12 checks passed

github-actions Bot mentioned this pull request May 22, 2026

[breaking-change] Undocumented breaking changes detected (2026-05-22) #19172

Open

1 of 3 tasks

Navigation Menu

Repository navigation

ti_threatconnect: fix precision loss of indicator IDs above 2^53 - #19110

ti_threatconnect: fix precision loss of indicator IDs above 2^53#19110
kcreddy merged 4 commits into
elastic:mainfrom
kcreddy:ti_threatconnect-precision-fix

kcreddy commented May 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 21, 2026 •

edited

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented May 21, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented May 21, 2026

Uh oh!

efd6 May 21, 2026

Uh oh!

kcreddy May 21, 2026

Uh oh!

efd6 May 21, 2026

Uh oh!

kcreddy May 21, 2026

Uh oh!

elasticmachine commented May 21, 2026

Uh oh!

efd6 left a comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented May 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	bytes(resp.Body).decode_json_string_numbers().as(body, {
	resp.Body.decode_json_string_numbers().as(body, {

Conversation

kcreddy commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

github-actions Bot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Vale Linting Results

Uh oh!

infra-vault-gh-plugin-prod Bot commented May 21, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented May 21, 2026

🚀 Benchmarks report

Uh oh!

efd6 May 21, 2026

Choose a reason for hiding this comment

Uh oh!

kcreddy May 21, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 May 21, 2026

Choose a reason for hiding this comment

Uh oh!

kcreddy May 21, 2026

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 21, 2026

💚 Build Succeeded

History

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented May 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kcreddy commented May 21, 2026 •

edited

Loading

github-actions Bot commented May 21, 2026 •

edited

Loading