Repository navigation
[ti_abusech] Add YARAify data stream and commercial API for URLhaus - #20380
Conversation
moxarth-rathod
added
documentation
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
Elastic Docs Style Checker (Vale)Summary: 10 suggestions found 💡 Suggestions (10): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| "Content-Type": ["application/json"], | ||
| ?"Auth-Key": has(state.auth_key) ? | ||
| optional.of([state.auth_key]) | ||
| state.?api_type.orValue("community") == "commercial" ? |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/data_stream/url/agent/stream/cel.yml.hbs:38
The CEL guard that rejected the deprecated /downloads/json URLhaus endpoint was deleted, so policies still holding that URL now silently collect zero events instead of raising a configuration error; restore the guard (or update/remove the script test that still asserts it).
Details
The previous program opened with a state.url.contains("/downloads/json") branch that emitted a configuration_error event telling the user to move to https://urlhaus-api.abuse.ch/v1/urls/recent/. This PR replaces the top of the program with the api_type dispatch and drops that branch entirely. The community path now issues GET against whatever URL is configured; for /downloads/json the response has no urls key, so has(body.urls) is false, the program returns [], and the user gets no data and no error.
This is not hypothetical: packages/ti_abusech/data_stream/url/_dev/test/scripts/deprecated_url.txt still exists unchanged in the branch and asserts stdout 'deprecated full export endpoint' on error.message plus exactly one error document. That test is now inconsistent with the program. The upgrade note in docs/README.md ("Upgrading to v4.0.0") also still tells users the old URL must be changed manually, which is exactly the population this guard protected. Making url show_user: true in this PR widens the set of users who can end up on a wrong URL, and the same silent-failure applies when a user switches API Type to Commercial but leaves the community default URL in place.
Recommendation:
Re-add the deprecated-endpoint guard ahead of the api_type dispatch, and keep deprecated_url.txt valid:
If dropping the guard is intentional, delete _dev/test/scripts/deprecated_url.txt, remove the stale upgrade instruction from the docs, and add a changelog entry describing the behaviour change.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Add Commercial API support for the URLhaus data stream. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20134 |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/changelog.yml:6
Both new changelog entries link to the placeholder PR #1; point them at this PR (20380) instead.
Details
The two entries added for version 4.2.0 (Add Commercial API support for the URLhaus data stream. at line 6 and Add YARAify data stream with Community and Commercial API support. at line 9) both carry link: https://github.com/elastic/integrations/pull/1. PR 1 is unrelated to this change. The already-merged MalwareBazaar entry directly below correctly references pull/20199, so the placeholder was simply not replaced before pushing. Changelog links are surfaced in the published package changelog, so users following them land on the wrong PR.
Recommendation:
Point both entries at this PR:
- version: "4.2.0"
changes:
- description: Add Commercial API support for the URLhaus data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380
- description: Add YARAify data stream with Community and Commercial API support.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| community: | ||
| - name: api_type | ||
| value: community | ||
| - name: auth_key |
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/ti_abusech/manifest.yml:67
Selecting Commercial API stops Fleet from requiring Auth Key, but threatfox, malware, sslblacklist and ja3_fingerprints always use the Community API and still need it; add auth_key to the commercial required_vars group and fix the contradictory description.
Details
required_vars lists auth_key only under the community group. When a user picks api_type: commercial to collect URLhaus/MalwareBazaar/YARAify, Fleet no longer requires Auth Key, yet the other four data streams in this same policy template (threatfox, malware, sslblacklist, ja3_fingerprints) have no commercial branch in their CEL/httpjson templates and keep calling the community endpoints, which reject unauthenticated requests.
The manifest also contradicts itself about this. The api_type description (line 79) says "Other data streams always use the Community API", while the auth_key description (line 97) says "Not used when API Type is Commercial API" and enumerates only "Threat Fox, Malware Bazaar, URLhaus, YARAify" - omitting SSL Blacklist, JA3 Fingerprints and Malware, all of which need the key regardless of API Type.
Recommendation:
Require auth_key in both groups and correct the description:
required_vars:
community:
- name: api_type
value: community
- name: auth_key
commercial:
- name: api_type
value: commercial
- name: auth_key
- name: username
- name: password - name: auth_key
type: password
title: Auth Key (Community)
multi: false
required: false
show_user: true
secret: true
description: >-
abuse.ch Auth Key for the Community API. Always required: SSL Blacklist, JA3 Fingerprints,
Malware and Threat Fox only support the Community API, and URLhaus, MalwareBazaar and YARAify
use it when API Type is Community API. See [documentation](https://abuse.ch/blog/community-first/) for details.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| #### YARAify | ||
|
|
||
| {{fields "yaraify"}} |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/_dev/build/docs/README.md:199
The Example event section renders {{fields "yaraify"}} instead of {{event "yaraify"}}, so the generated README duplicates the exported-fields table and never shows the YARAify sample event.
Details
Line 169 correctly places {{fields "yaraify"}} under the exported-fields section, but line 199 - added under ### Example event, directly after {{event "url"}} - repeats {{fields "yaraify"}}. The rendered docs/README.md confirms the effect: the identical YARAify field table appears twice (once around line 400 and again around line 947), and no YARAify example event is shown even though data_stream/yaraify/sample_event.json was added in this PR.
Recommendation:
Render the sample event in the Example event section:
Then regenerate docs/README.md with elastic-package build.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| - Configure **Auth Key**. | ||
| - Configure **Auth Key** for Community API datasets. | ||
| - For ThreatFox Commercial API collection, set **API Type** to **Commercial API**, set the **URL** to the commercial API base URL (for example `https://api.spamhaus.com`), and configure **Username** and **Password**. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/_dev/build/docs/README.md:90
The doc template credits Commercial API support to ThreatFox, which this PR does not add, and it no longer matches the committed docs/README.md - so the next build will regress the rendered README.
Details
The template says "For ThreatFox Commercial API collection, set API Type to Commercial API ...". ThreatFox has no commercial branch: data_stream/threatfox/agent/stream/cel.yml.hbs is untouched by this PR and the root manifest's own api_type description says Commercial applies to "URLhaus, MalwareBazaar, and YARAify".
The generated packages/ti_abusech/docs/README.md (line 89 in the rendered file) already reads "For URLhaus, MalwareBazaar, or YARAify Commercial API collection ...". Because the two files disagree, the committed README was not produced from this template - the next elastic-package build will overwrite it with the incorrect ThreatFox wording.
Recommendation:
Fix the template so it matches the rendered docs, then rebuild:
- Configure **Auth Key** for Community API datasets.
- For URLhaus, MalwareBazaar, or YARAify Commercial API collection, set **API Type** to **Commercial API**, set the **URL** to the commercial API base URL (for example `https://api.spamhaus.com`), and configure **Username** and **Password**.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| conditions: | ||
| kibana: | ||
| version: "^8.19.0 || ^9.1.0" | ||
| version: "^9.1.0" |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/manifest.yml:12
conditions.kibana.version is narrowed from "^8.19.0 || ^9.1.0" to "^9.1.0", dropping 8.19 support in a minor release with no feature that requires it and no changelog note; keep the 8.19 range or document the drop.
Details
Nothing added by this PR requires Kibana 9.1+. The only new manifest capability is required_vars on a policy-template input, which package-spec gates on format_version >= 3.3.2 (spec/integration/manifest.spec.yml removes .../inputs/items/properties/required_vars only before: 3.3.2); the package was already at 3.3.2 before this change, and the spec ties it to no Kibana version. The rest of the change is CEL, ingest-pipeline, fields and dashboard work.
The effect is user-visible: every 8.19.x deployment on ti_abusech 4.1.x stops receiving upgrades at 4.2.0. Both changelog entries are typed enhancement, so nothing signals the compatibility reduction.
Recommendation:
Restore the existing range unless a concrete 9.1-only feature is being used:
conditions:
kibana:
version: "^8.19.0 || ^9.1.0"If 8.19 really must be dropped, add a changelog entry stating it, for example:
- description: Drop support for Kibana 8.19; the package now requires 9.1.0 or later.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -1,5 +1,5 @@ | |||
| { | |||
| "@timestamp": "2026-05-11T18:17:40.972Z", | |||
| "@timestamp": "2026-07-27T09:15:15.628Z", | |||
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/data_stream/url/sample_event.json:2
The url sample event was regenerated on an 8.19.0 stack and lost event.module, threat.feed.name, threat.feed.dashboard_id and labels.is_ioc_transform_source; regenerate it on a stack matching the package's new ^9.1.0 requirement.
Details
The regenerated sample event drops four fields that the package still defines and that the previous sample contained: event.module (constant_keyword ti_abusech from data_stream/url/fields/base-fields.yml), threat.feed.name (AbuseCH URL) and threat.feed.dashboard_id (both from the same unchanged file), and labels.is_ioc_transform_source (from fields/is-ioc-transform-source-true.yml, also unchanged). threat.feed.dashboard_id and threat.feed.name are what the Kibana CTI UI uses to link indicators back to the feed dashboard, and labels.is_ioc_transform_source is what distinguishes source-data-stream documents from latest_url destination documents.
The stack version also went backwards: agent.version/elastic_agent.version changed from 9.3.1 to 8.19.0, below the ^9.1.0 floor this same PR sets in the root manifest. The yaraify sample event added in this PR was generated on 9.1.0 and does contain all four fields, so the two sample events in one PR disagree. This propagates into docs/README.md, whose url example event now omits them too.
Recommendation:
Re-run the url system test on a stack that satisfies the package's declared conditions.kibana.version and commit the regenerated sample so it matches the yaraify sample:
{
"event": {
"module": "ti_abusech"
},
"labels": {
"is_ioc_transform_source": "true"
},
"threat": {
"feed": {
"dashboard_id": "ti_abusech-c0d8d1f0-3b20-11ec-ae50-2fdf1e96c6a6",
"name": "AbuseCH URL"
}
}
}Then regenerate docs/README.md.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| description: > | ||
| The ID of the indicator. | ||
|
|
||
| - name: first_seen |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/ti_abusech/data_stream/url/fields/fields.yml:10
abusech.url.first_seen is declared as a field and documented, but the pipeline removes it unconditionally so it can never appear in a document; drop the declaration or stop removing the field.
Details
abusech.url.first_seen is added here and in elasticsearch/transform/latest_url/fields/fields.yml, and it is now listed in the generated README exported-fields table. However elasticsearch/ingest_pipeline/default.yml adds abusech.url.first_seen to the remove_97e5a966 processor, which has no if condition, so the field is stripped from every document after being mapped to threat.indicator.first_seen. The PR's own fixture confirms this: _dev/test/pipeline/test-commercial-fields.log-expected.json contains threat.indicator.first_seen but no abusech.url.first_seen in either expected document. The other always-removed keys (date_added, dateadded, url, host) are correspondingly absent from fields.yml.
Recommendation:
Drop the unused declaration from both data_stream/url/fields/fields.yml and elasticsearch/transform/latest_url/fields/fields.yml:
- name: id
type: keyword
description: >
The ID of the indicator.
- name: last_online
type: dateAlternatively, if the raw value should be retained, remove abusech.url.first_seen from the remove_97e5a966 processor's field list and update the expected fixtures.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Add Commercial API support for the URLhaus data stream. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20134 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/changelog.yml:6
Changelog links a different PR number
Details
This changelog entry's link: points at pull/1, but it was added in PR #20380. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/20134 | ||
| - description: Add YARAify data stream with Community and Commercial API support. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20134 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/changelog.yml:9
Changelog links a different PR number
Details
This changelog entry's link: points at pull/1, but it was added in PR #20380. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| : | ||
| string(resp.Status) + ' (' + string(resp.StatusCode) + ')' | ||
| ), | ||
| duration(state.interval_value) > duration("72h") ? |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/data_stream/url/agent/stream/cel.yml.hbs:162
The rewrite dropped the /downloads/json deprecated-endpoint guard from the community branch; restore it ahead of the interval check so the documented configuration error is still emitted.
Details
Before this change the program's first branch was state.url.contains("/downloads/json"), which emitted a configuration_error telling the user to move off the deprecated URLhaus full-export endpoint. The rewritten program only keeps the 72h interval guard in the community branch, so that check is gone.
Two things break as a result:
packages/ti_abusech/data_stream/url/_dev/test/scripts/deprecated_url.txt(unchanged by this PR, still in the tree) configuresurl: http://abusech-mock:8080/downloads/json, waits for exactly one document and assertsstdout 'deprecated full export endpoint'. With the guard removed no error event is produced at all — the mock returns{},has(body.urls)is false, and the program yields zero events — soget_docs -want 1times out and the test fails.packages/ti_abusech/docs/README.mdline 120 still tells upgrading users "the integration will log an error:the URL is set to the deprecated full export endpoint (/downloads/json) which is no longer supported". That statement is now false, and a user who upgraded from 3.x and never edited the URL setting silently collects nothing instead of getting a diagnosable error.
Recommendation:
Reinstate the guard as the first test inside the community branch, before the interval check:
:
state.url.contains("/downloads/json") ?
{
"events": {
"error": {
"code": "400",
"id": "configuration_error",
"message": "the URL is set to the deprecated full export endpoint (/downloads/json) which is no longer supported. Update the URL setting to https://urlhaus-api.abuse.ch/v1/urls/recent/",
},
},
"want_more": false,
}
: duration(state.interval_value) > duration("72h") ?
{
"events": {
"error": {
"code": "400",
"id": "configuration_error",
"message": "the polling interval (" + state.interval_value + ") exceeds the 3-day window of the URLhaus recent API. Reduce the interval to 72h or less to avoid missing data.",
},
},
"want_more": false,
}
:
request("GET", state.url).with({
Remember to regenerate the three affected _dev/test/policy/*.expected files after the change.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| - Configure **Auth Key**. | ||
| - Configure **Auth Key** for Community API datasets. | ||
| - For ThreatFox Commercial API collection, set **API Type** to **Commercial API**, set the **URL** to the commercial API base URL (for example `https://api.spamhaus.com`), and configure **Username** and **Password**. |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/_dev/build/docs/README.md:90
The README source template and the generated docs/README.md have diverged in two places; update the template so a rebuild reproduces the committed README.
Details
docs/README.md is generated from this template, so the two must agree. They currently do not:
- Template line 90 says "For ThreatFox Commercial API collection..." while generated
docs/README.mdline 90 says "For URLhaus, MalwareBazaar, or YARAify Commercial API collection...". ThreatFox commercial support is PR 20134, a different branch; it is not part of this package version's commercial support set. - Template line 21 still has the old MalwareBazaar bullet (
Collects malware payloads from MalwareBazaar via [MalwareBazaar API]...) while generateddocs/README.mdline 21 has the Community/Commercial wording.
Because the template is the source of truth, elastic-package build regenerates docs/README.md from it and the docs-up-to-date check fails. If it is regenerated as-is, the shipped README loses the MalwareBazaar commercial note and tells users the API Type toggle applies to ThreatFox, which this version does not support.
Recommendation:
Bring the template in line with the intended (and already-generated) content, then re-run elastic-package build so the two files match:
- `malwarebazaar`: Collects malware payloads from MalwareBazaar via the Community [MalwareBazaar API](https://bazaar.abuse.ch/api/#latest_additions) or the Commercial API (`GET /malwarebazaar/v1/samples`). Community API uses Auth Key; Commercial API requires username and password from the [Spamhaus Customer Portal](https://portal.spamhaus.com). - For URLhaus, MalwareBazaar, or YARAify Commercial API collection, set **API Type** to **Commercial API**, set the **URL** to the commercial API base URL (for example `https://api.spamhaus.com`), and configure **Username** and **Password**.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| #### YARAify | ||
|
|
||
| {{fields "yaraify"}} |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/_dev/build/docs/README.md:199
The Example event section for YARAify calls {{fields "yaraify"}} instead of {{event "yaraify"}}, so the field table is rendered twice and the sample event never appears.
Details
Line 169 already emits {{fields "yaraify"}} under the "Exported fields" section. Line 199 sits under ### Example event, after {{event "url"}}, but repeats {{fields "yaraify"}}. The generated docs/README.md confirms the effect: the identical YARAify field table is rendered twice (once at line ~404 and again at line ~950) and no YARAify example event is shown, even though packages/ti_abusech/data_stream/yaraify/sample_event.json exists and is ready to be rendered.
Recommendation:
Use the event helper in the Example event section:
#### YARAify
{{event "yaraify"}}Then re-run elastic-package build to regenerate docs/README.md.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Add Commercial API support for the URLhaus data stream. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20134 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/changelog.yml:6
Both new changelog entries link to PR 20134 (the ThreatFox commercial-API PR) rather than this PR; point them at 20380.
Details
Lines 6 and 9 both set link: https://github.com/elastic/integrations/pull/20134. PR 20134 is titled "[AbuseCH] Add Commercial API support for the ThreatFox data stream" and is a separate open PR — it does not contain the URLhaus commercial support or the YARAify data stream added here. The changelog link is what users follow from the package changelog to find the change, so both entries currently point at unrelated work. The neighbouring MalwareBazaar entry correctly references its own PR (20199), which shows the intended convention.
Recommendation:
Point both new entries at this PR:
- version: "4.2.0"
changes:
- description: Add Commercial API support for the URLhaus data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380
- description: Add YARAify data stream with Community and Commercial API support.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| conditions: | ||
| kibana: | ||
| version: "^8.19.0 || ^9.1.0" | ||
| version: "^9.1.0" |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/manifest.yml:12
The Kibana constraint is narrowed from ^8.19.0 || ^9.1.0 to ^9.1.0, dropping 8.19 support with no changelog entry explaining it.
Details
This is an existing package (4.1.x releases already shipped) and the change makes 4.2.0 uninstallable and un-upgradable on 8.19 stacks that run 4.1.2 today. Nothing in the diff obviously forces the bump: required_vars is accepted from package-spec 3.3.2, which this package already declared before the format_version bump to 3.4.2, and it was already paired with ^8.19.0 || ^9.1.0 on main. The PR's own packages/ti_abusech/data_stream/url/sample_event.json was regenerated on an 8.19.0 agent, which suggests 8.19 is still an active test target.
The changelog for 4.2.0 contains only two enhancement entries and says nothing about raising the minimum stack version, so 8.19 users get no warning about why the upgrade disappears.
Recommendation:
Either keep 8.19 in the constraint:
conditions:
kibana:
version: "^8.19.0 || ^9.1.0"or, if a 9.1-only feature really is required, record the drop explicitly in changelog.yml and name the feature:
- version: "4.2.0"
changes:
- description: Raise the minimum Kibana version to 9.1.0.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| "name": "elastic-agent-40794", | ||
| "type": "filebeat", | ||
| "version": "9.3.1" | ||
| "version": "8.19.0" |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/data_stream/url/sample_event.json:22
The URL sample event was regenerated on an 8.19 stack and lost event.module, labels.is_ioc_transform_source and threat.feed.*; regenerate it on the same 9.x stack used for the YARAify sample.
Details
The agent version in this file moved from 9.3.1 to 8.19.0, and in the same regeneration the document lost event.module, labels.is_ioc_transform_source and the whole threat.feed object (name, dashboard_id). Those are the constant_keyword values declared in data_stream/url/fields/base-fields.yml and fields/is-ioc-transform-source-true.yml; they only materialise into _source on stacks where the logs data stream uses synthetic source, which is why the older 9.3.1 capture had them and the 8.19.0 capture does not.
The result is a README that documents threat.feed.name, threat.feed.dashboard_id, event.module and labels.is_ioc_transform_source as exported fields for url while the rendered example event no longer shows any of them. It is also internally inconsistent with the new data_stream/yaraify/sample_event.json added in this same PR, which was captured on a 9.1.0 agent and does still contain all four.
Recommendation:
Re-run the URL system test on the same 9.x stack used for YARAify and commit the resulting sample so the block again carries the constant_keyword fields:
"labels": {
"is_ioc_transform_source": "true"
},
"threat": {
"feed": {
"dashboard_id": "ti_abusech-c0d8d1f0-3b20-11ec-ae50-2fdf1e96c6a6",
"name": "AbuseCH URL"
},
"indicator": {If the URL sample did not actually need to change in this PR, reverting it is also fine.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| if: ctx.threat?.indicator?.marking?.tlp == null | ||
|
|
||
| - set: | ||
| field: threat.indicator.type |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/data_stream/yaraify/elasticsearch/ingest_pipeline/default.yml:263
threat.indicator.type is only set when a yarahub_reference_md5 is present, so rules without one are indexed with no indicator type; add an unknown fallback.
Details
The only processor that sets threat.indicator.type is guarded by if: ctx.threat?.indicator?.file?.hash?.md5 != null. YARAify rules frequently omit yarahub_reference_md5 — two of the three documents in _dev/test/pipeline/test-yaraify-sample.log-expected.json and one of the two in test-yaraify-community-sample.log-expected.json come out with a populated threat.indicator.* object but no threat.indicator.type.
Those documents still flow through elasticsearch/transform/latest_yaraify into logs-ti_abusech_latest.yaraify, where the Security CTI views and indicator-match rules key off threat.indicator.type to decide how to interpret an indicator. Untyped indicators are silently skipped there. threat.indicator.type is also declared in elasticsearch/transform/latest_yaraify/fields/ecs.yml, so the mapping expects it to be present.
Recommendation:
Keep the file mapping and add a default immediately after it, mirroring how threat.indicator.marking.tlp already defaults to CLEAR in this pipeline:
- set:
field: threat.indicator.type
tag: set_threat_indicator_type_file
value: file
if: ctx.threat?.indicator?.file?.hash?.md5 != null
- set:
field: threat.indicator.type
tag: set_threat_indicator_type_default
value: unknown
if: ctx.threat?.indicator?.type == null && ctx.threat?.indicator?.name != nullUpdate the two pipeline expected-output files accordingly.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ).do_request().as(resp, (resp.StatusCode == 200) ? | ||
| resp.Body.decode_json().as(body, | ||
| { | ||
| "events": has(body.data) ? |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/ti_abusech/data_stream/yaraify/agent/stream/cel.yml.hbs:194
The YARAify community branch ignores query_status, so an API-level error returned with HTTP 200 produces zero events and no error.message; check it as the MalwareBazaar program does.
Details
The community request handler only does has(body.data) ? body.data.map(...) : []. The YARAify Community API answers with HTTP 200 and a query_status field, and for anything other than ok (for example an unknown or rejected query, or an auth problem surfaced in the body) it returns no data key. The program therefore treats it as an empty poll: no events, no error.message, nothing in the request tracer output for an operator to find. The data stream just looks idle.
The sibling data_stream/malwarebazaar/agent/stream/cel.yml.hbs already handles this correctly (lines 170-186), branching on body.?query_status == optional.of("ok") / "no_results" and emitting an error event otherwise. This is new code, so it should follow the better of the two existing patterns in the package.
Recommendation:
Branch on query_status before mapping the payload:
).do_request().as(resp, (resp.StatusCode == 200) ?
resp.Body.decode_json().as(body, body.?query_status == optional.of("ok") ?
{
"events": body.data.map(item, { "message": item.encode_json() }),
"want_more": false,
}
: body.?query_status == optional.of("no_result") ?
{
"events": [],
"want_more": false,
}
:
{
"events": {
"error": {
?"id": body.?query_status,
"message": "POST " + state.url.trim_right("/") + ": " + string(resp.Body),
},
},
"want_more": false,
}
)
Regenerate the three _dev/test/policy/*.expected files after the change.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Add Commercial API support for the URLhaus data stream. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20134 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/changelog.yml:6
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20134, but it was added in PR #20380. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/20134 | ||
| - description: Add YARAify data stream with Community and Commercial API support. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20134 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/changelog.yml:9
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20134, but it was added in PR #20380. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
andrewkroh
added
the
dashboard
| params: | ||
| "TLP:WHITE": CLEAR | ||
| "WHITE": CLEAR | ||
| "TLP:GREEN": GREEN | ||
| "GREEN": GREEN | ||
| "TLP:AMBER": AMBER | ||
| "AMBER": AMBER | ||
| "TLP:RED": RED | ||
| "RED": RED | ||
| source: |- | ||
| String tlp = ctx.abusech.yaraify.yarahub_rule_matching_tlp; | ||
| if (params.containsKey(tlp)) { | ||
| ctx.threat = ctx.threat ?: [:]; | ||
| ctx.threat.indicator = ctx.threat.indicator ?: [:]; | ||
| ctx.threat.indicator.marking = ctx.threat.indicator.marking ?: [:]; | ||
| ctx.threat.indicator.marking.tlp = params[tlp]; | ||
| } |
There was a problem hiding this comment.
This can be done more cheaply by prefix-trimming "TLP:" from the string and conditionally changing "WHITE" to "CLEAR", assuming that the params map is the full set of values that can be in the input field.
String tlp = ctx.abusech.yaraify.yarahub_rule_matching_tlp;
if (tlp.startsWith("TLP:")) {
tlp = tlp.substring("TLP:".length());
}
if (tlp == "WHITE") {
tlp = "CLEAR";
}
…
There was a problem hiding this comment.
You will need to elastic-package build.
| "field": "event.module", | ||
| "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", | ||
| "key": "event.module", | ||
| "key": "threat.indicator.type", |
There was a problem hiding this comment.
This appears to be ANDed with other values for threat.indicator.type resulting in a tautology. Can you check this?
There was a problem hiding this comment.
Sure, let me check
| "alias": null, | ||
| "disabled": false, | ||
| "field": "event.module", | ||
| "field": "threat.indicator.type", |
| "field": "threat.indicator.type", | ||
| "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[4].meta.index", | ||
| "key": "threat.indicator.file.hash.ssdeep", | ||
| "key": "threat.indicator.type", |
| # newer versions go on top | ||
| - version: "4.4.0" | ||
| changes: | ||
| - description: Add Commercial API support for the URLhaus data stream. |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/changelog.yml:4
These changes are appended to version 4.2.0, which is already released on main - add a new - version: "4.3.0" block instead and bump version in manifest.yml to match.
Details
Version 4.2.0 is the version currently on main: I fetched packages/ti_abusech/manifest.yml from the default branch and it reads version: "4.2.0" (blob sha c0eaa5b92d1, identical to this PR's base blob), and the 4.2.0 changelog block already carries the MalwareBazaar entry from PR 20199, which is MERGED. This PR appends two new entries to that released block and leaves manifest.yml at 4.2.0, so the package content changes without a version change. Republishing an already-published version is rejected by the publishing pipeline. Separately, this PR narrows conditions.kibana.version from "^8.19.0 || ^9.1.0" to "^9.1.0" (manifest.yml line 12), dropping Kibana/stack 8.19 support, and that user-visible support drop is not recorded anywhere in the changelog.
Recommendation:
Open a new version block and bump manifest.yml to match. A new data stream plus a dropped stack version warrants at least a minor bump:
# newer versions go on top
- version: "4.3.0"
changes:
- description: Add Commercial API support for the URLhaus data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380
- description: Add YARAify data stream with Community and Commercial API support.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380
- description: Require Kibana 9.1.0 or newer. Support for Kibana 8.19 is dropped.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380
- version: "4.2.0"
changes:
- description: Add Commercial API support for the MalwareBazaar data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/20199and in packages/ti_abusech/manifest.yml:
version: "4.3.0"🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Add Commercial API support for the URLhaus data stream. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20380 |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/changelog.yml:6
Both new changelog entries link to PR 20134 instead of this PR (20380) - the check_changelog_entries CI step fails on this, so update both link: values.
Details
Lines 6 and 9 both point at https://github.com/elastic/integrations/pull/20134. I checked that PR: it is a different, still-OPEN pull request titled "[AbuseCH] Add Commercial API support for the ThreatFox data stream". .buildkite/scripts/check_changelog_entries.sh greps every added link: line and hard-fails (exit 1) with "unexpected link" whenever it is not exactly the URL of the PR under test, so this build cannot go green as written. The neighbouring 4.2.0 entry correctly carries its own originating PR (20199), confirming the one-link-per-PR convention.
Recommendation:
Point both entries at this PR:
- description: Add Commercial API support for the URLhaus data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380
- description: Add YARAify data stream with Community and Commercial API support.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ) | ||
| ) | ||
| : | ||
| state.url.contains("/downloads/json") ? |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/data_stream/url/agent/stream/cel.yml.hbs:162
The rewrite drops the /downloads/json deprecation guard from the community branch, which breaks the existing deprecated_url.txt system test - re-add the guard ahead of the 72h interval check.
Details
The previous program began the community path with state.url.contains("/downloads/json") ? {..."id": "configuration_error", "message": "the URL is set to the deprecated full export endpoint..."}. The new community branch starts directly at the 72h interval check and the guard is not re-added anywhere in the file. packages/ti_abusech/data_stream/url/_dev/test/scripts/deprecated_url.txt is not touched by this PR and still asserts stdout 'deprecated full export endpoint' against error.message after configuring url: http://abusech-mock:8080/downloads/json; that assertion now has nothing to match. The migration note this guard backs is still live in docs/README.md lines 113-117 and in the 4.0.0 changelog entry, so users who never migrated now get a JSON decode failure on a ZIP body instead of the actionable configuration error.
Recommendation:
Restore the guard as the first branch of the community path:
:
state.url.contains("/downloads/json") ?
{
"events": {
"error": {
"code": "400",
"id": "configuration_error",
"message": "the URL is set to the deprecated full export endpoint (/downloads/json) which is no longer supported. Update the URL setting to https://urlhaus-api.abuse.ch/v1/urls/recent/",
},
},
"want_more": false,
}
: duration(state.interval_value) > duration("72h") ?
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -0,0 +1,31 @@ | |||
| - name: abusech.yaraify | |||
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/ti_abusech/data_stream/yaraify/fields/fields.yml:1
The new yaraify data stream has no fields/ecs.yml, so the date-typed ECS field threat.indicator.first_seen it populates is undeclared - add an ecs.yml mirroring the sibling data streams.
Details
data_stream/yaraify/fields/ contains only base-fields.yml, beats.yml, fields.yml and is-ioc-transform-source-true.yml. Every other data stream in this package (url, malwarebazaar, malware, ja3_fingerprints, ...) ships a fields/ecs.yml, and each declares threat.indicator.first_seen as external: ecs precisely because date-typed ECS fields are not covered by the string dynamic templates. The yaraify pipeline sets threat.indicator.first_seen (default.yml line 92) yet nothing in the data stream declares it, so it is mapped from a string value rather than as a date. The mismatch is visible inside this PR: the companion elasticsearch/transform/latest_yaraify/fields/ecs.yml was created and does declare threat.indicator.first_seen plus 21 other ECS fields, so the destination index and the source data stream disagree on the mapping.
Recommendation:
Add packages/ti_abusech/data_stream/yaraify/fields/ecs.yml matching the sibling data streams (data_stream/url/fields/ecs.yml):
- external: ecs
name: threat.indicator.first_seenIf you want the source data stream to declare the same surface as the new transform, mirror the ECS entries from elasticsearch/transform/latest_yaraify/fields/ecs.yml as well.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| #### YARAify | ||
|
|
||
| {{event "yaraify"}} |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/_dev/build/docs/README.md:199
The YARAify entry under "Example event" uses {{fields "yaraify"}} instead of {{event "yaraify"}}, so the generated README repeats the exported-fields table and never shows the sample event.
Details
Line 199 sits in the ### Example event section, directly after {{event "url"}} on line 195, but uses the fields helper. Every other entry in that section uses event. The mistake has already propagated into the committed output: packages/ti_abusech/docs/README.md renders the full YARAify exported-fields table a second time under ### Example event, and no YARAify sample event appears anywhere in the README even though data_stream/yaraify/sample_event.json exists.
Recommendation:
Use the event helper, then regenerate docs/README.md with elastic-package build:
#### YARAify
{{event "yaraify"}}🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| - Configure **Auth Key**. | ||
| - Configure **Auth Key** for Community API datasets. | ||
| - For Commercial API collection, set **API Type** to **Commercial API**, set the **URL** to the commercial API base URL (for example `https://api.spamhaus.com`), and configure **Username** and **Password**. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/_dev/build/docs/README.md:90
The new setup step tells users to enable the Commercial API "For ThreatFox", but ThreatFox has no commercial support here - name URLhaus, MalwareBazaar and YARAify instead.
Details
data_stream/threatfox/ contains no reference to api_type, username or password; its CEL template is community-only. The package's own variable descriptions contradict this line: the root manifest.yml api_type description says "Select the abuse.ch API to use for URLhaus, MalwareBazaar, and YARAify... Other data streams always use the Community API", and the username/password descriptions scope commercial to the same three. The ThreatFox wording appears to have been carried over from PR 20134, which is the separate open PR that adds ThreatFox commercial support. The identical line also exists at packages/ti_abusech/docs/README.md line 90 and needs regenerating.
Recommendation:
Name the data streams that actually support the Commercial API in this version:
- For URLhaus, MalwareBazaar, or YARAify Commercial API collection, set **API Type** to **Commercial API**, set the **URL** to the commercial API base URL (for example `https://api.spamhaus.com`), and configure **Username** and **Password**.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| value: file | ||
| if: ctx.threat?.indicator?.file?.hash?.md5 != null | ||
|
|
||
| - append: |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/data_stream/yaraify/elasticsearch/ingest_pipeline/default.yml:263
threat.indicator.type is only set when a reference MD5 is present, so YARA rules without one are indexed with no indicator type - add a default so every document is typed.
Details
set_threat_indicator_type_file is gated on ctx.threat?.indicator?.file?.hash?.md5 != null, and yarahub_reference_md5 is optional in the feed. This PR's own fixtures prove the gap: in _dev/test/pipeline/test-yaraify-community-sample.log-expected.json the second document (win_othermw) has threat.indicator.name, description, first_seen and marking.tlp but no threat.indicator.type at all, and the same holds for the second and third documents in test-yaraify-sample.log-expected.json and for sample_event.json. Indicators without threat.indicator.type are not usable by the Threat Intelligence UI or by indicator-match rules, and the latest_yaraify transform declares the field, so the destination index carries a column that is empty for most rows. Every other data stream in this package sets an unconditional type (the url pipeline sets threat.indicator.type: url outright).
Recommendation:
Keep the file specialisation, then fall back to an allowed ECS value for untyped rules:
- set:
field: threat.indicator.type
tag: set_threat_indicator_type_file
value: file
if: ctx.threat?.indicator?.file?.hash?.md5 != null
- set:
field: threat.indicator.type
tag: set_threat_indicator_type_unknown
value: unknown
if: ctx.threat?.indicator?.type == null🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Add Commercial API support for the URLhaus data stream. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20380 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/changelog.yml:6
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20134, but it was added in PR #20380. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/20380 | ||
| - description: Add YARAify data stream with Community and Commercial API support. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20380 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/changelog.yml:9
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20134, but it was added in PR #20380. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -0,0 +1,31 @@ | |||
| - name: abusech.yaraify | |||
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/data_stream/yaraify/fields/fields.yml:1
The new yaraify data stream ships no fields/ecs.yml, so every ECS field its pipeline writes is undeclared; add an ecs.yml listing them with external: ecs.
Details
packages/ti_abusech/data_stream/yaraify/fields/ contains only base-fields.yml, beats.yml, fields.yml and is-ioc-transform-source-true.yml. Every other data stream in this package (url, malware, malwarebazaar, threatfox, sslblacklist, ja3_fingerprints) also has an ecs.yml, and the companion transform elasticsearch/transform/latest_yaraify/fields/ecs.yml declares exactly the ECS fields that are missing here.
The yaraify pipeline populates ecs.version, event.kind, event.category, event.type, event.id, event.original, error.message, tags, related.user, related.hash, rule.name, user.name, threat.indicator.name, threat.indicator.description, threat.indicator.type, threat.indicator.first_seen, threat.indicator.reference, threat.indicator.marking.tlp and threat.indicator.file.hash.md5 -- none of which are declared for the data stream.
The effect is already visible in the generated documentation committed in this PR: the 'YARAify -> Exported fields' table in packages/ti_abusech/docs/README.md (lines 406-427) lists only @timestamp, the abusech.yaraify.* custom fields, data_stream., event.dataset, event.module, input.type, labels.is_ioc_transform_source and threat.feed.. Compare it with the 'URL' table immediately above, which documents threat.indicator.first_seen, threat.indicator.last_seen and the rest. Users of the new data stream get no field reference for the ECS fields the pipeline actually produces, and the source index mapping falls back to dynamic mapping instead of the ECS definitions the transform destination index uses.
Recommendation:
Add packages/ti_abusech/data_stream/yaraify/fields/ecs.yml mirroring the declarations already written for the transform in elasticsearch/transform/latest_yaraify/fields/ecs.yml:
- external: ecs
name: ecs.version
- external: ecs
name: error.message
- external: ecs
name: tags
- external: ecs
name: related.hash
- external: ecs
name: related.user
- external: ecs
name: event.kind
- external: ecs
name: event.category
- external: ecs
name: event.type
- external: ecs
name: event.id
- external: ecs
name: event.original
- external: ecs
name: rule.name
- external: ecs
name: user.name
- external: ecs
name: threat.indicator.name
- external: ecs
name: threat.indicator.description
- external: ecs
name: threat.indicator.type
- external: ecs
name: threat.indicator.first_seen
- external: ecs
name: threat.indicator.reference
- external: ecs
name: threat.indicator.marking.tlp
- external: ecs
name: threat.indicator.file.hash.md5Then regenerate docs/README.md with elastic-package build.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| field: threat.indicator.type | ||
| tag: set_threat_indicator_type_file | ||
| value: file | ||
| if: ctx.threat?.indicator?.file?.hash?.md5 != null |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/data_stream/yaraify/elasticsearch/ingest_pipeline/default.yml:261
threat.indicator.type is only set when yarahub_reference_md5 is present, so YARA rules without a reference hash are indexed with no indicator type; add an 'unknown' fallback like the url pipeline has.
Details
set_threat_indicator_type_file is the only processor that assigns threat.indicator.type, and it is guarded by if: ctx.threat?.indicator?.file?.hash?.md5 != null. Rules that carry no yarahub_reference_md5 therefore end up with no threat.indicator.type at all.
This is not hypothetical -- it is visible in the fixtures committed by this PR. In _dev/test/pipeline/test-yaraify-sample.log-expected.json documents 2 and 3 (win_othermw, win_samplemw) and in test-yaraify-community-sample.log-expected.json document 2, the threat.indicator object contains description, first_seen, marking.tlp and name but no type. The same gap is present in the committed sample_event.json.
threat.indicator.type is the field the Security threat-intel UI and indicator-match rules use to classify an indicator, and it is declared in the latest_yaraify transform's ecs.yml, so the destination index expects it. The url data stream pipeline in this same package already handles this with a trailing set_threat_indicator_type_ac7917ab processor that defaults the value to unknown.
Recommendation:
Add a fallback after the existing set_threat_indicator_type_file processor, matching the pattern already used in data_stream/url/elasticsearch/ingest_pipeline/default.yml:
- set:
field: threat.indicator.type
tag: set_threat_indicator_type_default
value: unknown
if: ctx.threat?.indicator?.type == nullRegenerate the affected *-expected.json fixtures and sample_event.json afterwards.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - name: username | ||
| - name: password | ||
| vars: | ||
| - name: api_type |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/manifest.yml:74
Moving api_type/username/password from the malwarebazaar data stream to input-level vars silently resets existing 4.2.x Commercial MalwareBazaar policies to Community; record it as a breaking change and document the migration.
Details
This PR adds api_type, username and password as policy-template input vars in the root manifest, and the same three vars are deleted from data_stream/malwarebazaar/manifest.yml (removed lines 14-42 of that file). Those data-stream vars were introduced in 4.2.0, which already shipped Commercial API support for MalwareBazaar.
On upgrade, a policy that stored api_type: commercial plus username/password at the MalwareBazaar data-stream level loses those values -- Fleet drops vars that no longer exist on the stream, and the newly added input-level api_type resolves to its community default with empty username/password. The MalwareBazaar stream therefore reverts from the Commercial API to the Community API without the user changing anything.
The changelog for 4.4.0 records only two enhancement entries (URLhaus Commercial API, new YARAify data stream). Neither the variable relocation nor its upgrade impact is mentioned anywhere in changelog.yml or in _dev/build/docs/README.md.
Recommendation:
Record the relocation explicitly in packages/ti_abusech/changelog.yml and add a migration note to _dev/build/docs/README.md:
- version: "4.4.0"
changes:
- description: Move `API Type`, `Username` and `Password` from the MalwareBazaar data stream to integration-level settings. Existing policies using the MalwareBazaar Commercial API must reconfigure these settings after upgrading.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380
- description: Add Commercial API support for the URLhaus data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380
- description: Add YARAify data stream with Community and Commercial API support.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| conditions: | ||
| kibana: | ||
| version: "^8.19.0 || ^9.1.0" | ||
| version: "^9.1.0" |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/ti_abusech/manifest.yml:12
conditions.kibana.version is narrowed from '^8.19.0 || ^9.1.0' to '^9.1.0', dropping 8.19 support with no changelog entry; restore the 8.19 range or record the drop as a breaking change.
Details
The constraint changes from ^8.19.0 || ^9.1.0 to ^9.1.0 in the same commit that bumps format_version from 3.3.2 to 3.4.2. That makes 4.4.0 uninstallable on 8.19 stacks, which will remain on 4.2.0 and therefore never receive the URLhaus Commercial API support or the YARAify data stream.
Nothing added in this PR appears to require Kibana 9.1: the new input-level required_vars block, the select-typed api_type var, secret: true password vars and the latest transform are all supported by the previously declared 8.19 floor. The changelog entries for 4.4.0 are both typed enhancement and make no mention of the narrowed compatibility range.
Recommendation:
If no feature in this PR requires 9.1, restore the previous range:
conditions:
kibana:
version: "^8.19.0 || ^9.1.0"If the narrowing is intentional, name the feature that requires it and record it in changelog.yml:
- description: Drop support for Kibana 8.19; the integration now requires 9.1.0 or later.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| # Bump this version to delete, reinstall, and restart the transform during package. | ||
| # Version bump is needed if there is any code change in transform. | ||
| fleet_transform_version: 0.2.0 | ||
| settings: |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/ti_abusech/elasticsearch/transform/latest_yaraify/transform.yml:43
The new latest_yaraify transform omits settings.unattended and does not exclude cold/frozen tiers in its source query; add both.
Details
settings contains only num_failure_retries: -1. Without unattended: true the transform does not auto-recover from transient failures such as a temporarily unavailable destination index, and it has to be restarted manually.
The source.query also filters only on must_not: exists: error.message. It does not exclude the data_frozen and data_cold tiers, so once the source data stream ages into those tiers the transform scans expensive storage on every 30s run.
Recommendation:
source:
index:
- "logs-ti_abusech.yaraify-*"
query:
bool:
must_not:
- exists:
field: error.message
- terms:
_tier:
- data_frozen
- data_coldand:
settings:
unattended: true
num_failure_retries: -1🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| tag: set_event_kind | ||
| value: enrichment | ||
| - set: | ||
| field: event.category |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/ti_abusech/data_stream/yaraify/elasticsearch/ingest_pipeline/default.yml:59
event.category and event.type are ECS array fields but are written with set in the new yaraify pipeline; use append with allow_duplicates: false.
Details
set_event_category and set_event_type (lines 58-65) assign event.category and event.type with a set processor and a list literal. Both are ECS array fields, and the pipeline standard is to build them with append so that a value added by an earlier stage or by a custom pipeline is not overwritten. Output is identical today because nothing else writes these fields in this brand-new data stream, so this is a convention fix rather than a behavioural bug.
Recommendation:
- append:
field: event.category
tag: append_event_category
value: threat
allow_duplicates: false
- append:
field: event.type
tag: append_event_type
value: indicator
allow_duplicates: false🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| description: >- | ||
| abuse.ch Auth Key for the Community API. Required when API Type is Community API. See [documentation](https://abuse.ch/blog/community-first/) for details. | ||
| abuse.ch Auth Key for the Community API. Required when API Type is Community API for data streams that use the Community API (Threat Fox, Malware Bazaar, URLhaus, YARAify). Not used when API Type is Commercial API. See [documentation](https://abuse.ch/blog/community-first/) for details. | ||
| - name: username |
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/ti_abusech/manifest.yml:98
Moving api_type/username/password from the malwarebazaar stream vars up to the cel input vars discards the values already stored in 4.2.0 policies; keep them reachable or ship an explicit migration note so existing Commercial API users are not silently reverted to Community.
Details
Version 4.2.0 (currently on main) defined api_type, username and password inside packages/ti_abusech/data_stream/malwarebazaar/manifest.yml under streams[0].vars. This PR deletes all three from that file (the malwarebazaar manifest hunk is pure deletion) and re-declares them here at policy_templates[].inputs[].vars. Fleet preserves package-policy var values by name within the same scope; a var that disappears from the data stream scope is dropped on upgrade, and the newly introduced input-level username/password have no default. So an existing policy that was configured for the MalwareBazaar Commercial API loses its stored username and its secret-referenced password on upgrade to 4.4.0, and api_type falls back to its default 'community'. Combined with the new required_vars block on lines 63-72, the upgraded policy also fails the 'commercial' requirement group, so the user has to reconfigure before collection resumes. Neither changelog entry for 4.4.0 records this.
Recommendation:
Either keep the three vars addressable at the data stream level while the input-level copies are introduced, or make the relocation explicit in the changelog so Fleet users know they must re-enter credentials:
# packages/ti_abusech/changelog.yml
- version: "4.4.0"
changes:
- description: |
The `API Type`, `Username (Commercial)` and `Password (Commercial)` settings moved from the
MalwareBazaar data stream to the shared `Collect abuse.ch logs via API` input so URLhaus and
YARAify can use them. Existing policies that used the MalwareBazaar Commercial API must set
**API Type**, **Username** and **Password** again at the integration level after upgrading.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -0,0 +1,31 @@ | |||
| - name: abusech.yaraify | |||
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/ti_abusech/data_stream/yaraify/fields/fields.yml:1
The new yaraify data stream ships no fields/ecs.yml, so threat.indicator.first_seen and the other ECS fields the pipeline writes are left to dynamic mapping while latest_yaraify maps them explicitly; add data_stream/yaraify/fields/ecs.yml (and agent.yml) to match the sibling data streams.
Details
packages/ti_abusech/data_stream/yaraify/fields/ contains only base-fields.yml, beats.yml, fields.yml and is-ioc-transform-source-true.yml. Every other data stream in this package ships fields/ecs.yml and fields/agent.yml, and this PR's own elasticsearch/transform/latest_yaraify/fields/ecs.yml declares threat.indicator.first_seen, threat.indicator.type, threat.indicator.marking.tlp, user.name, rule.name, related.hash, related.user, event.id and event.original as external: ecs. The result is that the source data stream index and the transform destination index mapped from the same documents get their mappings from different sources for the same field names. data_stream/url/fields/ecs.yml deliberately declares exactly the threat.indicator date fields (first_seen/last_seen/modified_at) — that file exists because of the 2.3.1 bugfix 'Fix ECS date mapping on threat fields', which is the same date field the yaraify pipeline now populates at lines 122-134 and 176-190 without any declaration. fields/agent.yml (cloud.image.id, host.containerized, host.os.build, host.os.codename) is also absent, and those fields appear in documents produced by the Docker-based system tests.
Recommendation:
Add the missing field files to the data stream, mirroring the transform's declarations:
# packages/ti_abusech/data_stream/yaraify/fields/ecs.yml
- external: ecs
name: threat.indicator.first_seen
- external: ecs
name: threat.indicator.type
- external: ecs
name: threat.indicator.marking.tlp
- external: ecs
name: threat.indicator.name
- external: ecs
name: threat.indicator.description
- external: ecs
name: threat.indicator.reference
- external: ecs
name: threat.indicator.file.hash.md5
- external: ecs
name: rule.name
- external: ecs
name: user.name
- external: ecs
name: related.user
- external: ecs
name: related.hash
- external: ecs
name: event.idand copy packages/ti_abusech/data_stream/url/fields/agent.yml to packages/ti_abusech/data_stream/yaraify/fields/agent.yml.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| field: threat.indicator.type | ||
| tag: set_threat_indicator_type_file | ||
| value: file | ||
| if: ctx.threat?.indicator?.file?.hash?.md5 != null |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/data_stream/yaraify/elasticsearch/ingest_pipeline/default.yml:261
threat.indicator.type is only set when a rule carries yarahub_reference_md5, so YARAify rules without a reference hash are indexed with no indicator type at all; add an 'unknown' fallback like the url pipeline has.
Details
This set processor is the only thing in the pipeline that assigns threat.indicator.type, and it is guarded by ctx.threat?.indicator?.file?.hash?.md5 != null. Both pipeline test fixtures confirm the gap: in _dev/test/pipeline/test-yaraify-community-sample.log-expected.json and test-yaraify-sample.log-expected.json the second and third documents (win_othermw, win_samplemw — no yarahub_reference_md5) have no threat.indicator.type, and the committed sample_event.json has none either. Every other data stream in the package always ends up with a value; data_stream/url/elasticsearch/ingest_pipeline/default.yml line 228 has an explicit set threat.indicator.type: unknown if ctx.threat?.indicator?.type == null fallback for exactly this reason. Indicators with no threat.indicator.type are excluded from the Threat Intelligence view and from the 'Indicator Type' controls used on the abuse.ch dashboards.
Recommendation:
Add the same fallback the url pipeline uses, after the file-hash branch:
- set:
field: threat.indicator.type
tag: set_threat_indicator_type_file
value: file
if: ctx.threat?.indicator?.file?.hash?.md5 != null
- set:
field: threat.indicator.type
tag: set_threat_indicator_type_unknown
value: unknown
if: ctx.threat?.indicator?.type == null🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -1,4 +1,12 @@ | |||
| # newer versions go on top | |||
| - version: "4.4.0" | |||
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/changelog.yml:2
4.4.0 is recorded as two enhancements, but it also narrows conditions.kibana.version from '^8.19.0 || ^9.1.0' to '^9.1.0'; add a breaking-change entry so 8.19 users know the package no longer installs for them.
Details
packages/ti_abusech/manifest.yml changes conditions.kibana.version from '^8.19.0 || ^9.1.0' to '^9.1.0' (needed for the new required_vars block, which requires package-spec 3.4.0 and Kibana 9.1). Dropping a supported stack range is user-visible: an 8.19 deployment running 4.2.0 can no longer upgrade and the package disappears from its integrations list. The package has consistently used type: breaking-change for changes of this class (3.0.0 auth-key requirement, 4.0.0 URL endpoint switch), but both 4.4.0 entries are type: enhancement. Separately, main is currently at 4.2.0, so this bump skips 4.3.0 — worth confirming that is intentional rather than a stale rebase.
Recommendation:
Record the dropped 8.x support explicitly:
- version: "4.4.0"
changes:
- description: |
Kibana constraint narrowed to `^9.1.0`. The package now uses `required_vars`, which requires
Kibana 9.1.0 or newer; Kibana 8.19 deployments must stay on 4.2.0.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380
- description: Add Commercial API support for the URLhaus data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380
- description: Add YARAify data stream with Community and Commercial API support.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| type: integer | ||
| title: Page Size | ||
| multi: false | ||
| required: false |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/ti_abusech/data_stream/url/manifest.yml:53
page_size is declared required: false but cel.yml.hbs interpolates it into state unconditionally, so clearing the field renders page_size: (null) and breaks string(state.page_size) in the Commercial branch; mark it required like initial_interval.
Details
data_stream/url/agent/stream/cel.yml.hbs emits page_size: {{page_size}} outside any {{#if}} guard, and the Commercial branch evaluates "page_size": [string(state.page_size)]. If a user clears this optional field in Fleet the template renders page_size: with no value, state.page_size is null, and string() has no matching overload, so the whole CEL program fails to evaluate rather than falling back to the default. initial_interval, which is read the same way, is correctly declared required: true. The new yaraify data stream manifest has the same required: false on page_size.
Recommendation:
Make the value non-clearable, matching initial_interval:
- name: page_size
type: integer
title: Page Size
multi: false
required: true
show_user: false
default: 100
description: Number of malware URLs to request per page when using the Commercial API.Alternatively guard the template so an empty value is simply omitted from state:
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -0,0 +1,31 @@ | |||
| - name: abusech.yaraify | |||
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/data_stream/yaraify/fields/fields.yml:1
The new yaraify data stream has no fields/ecs.yml, so every ECS field its pipeline writes is undeclared; add an ecs.yml listing them with external: ecs.
Details
packages/ti_abusech/data_stream/yaraify/fields/ contains only base-fields.yml, beats.yml, fields.yml and is-ioc-transform-source-true.yml. The new ingest pipeline populates a large set of ECS fields that are declared nowhere: threat.indicator.name, threat.indicator.description, threat.indicator.first_seen (date), threat.indicator.type, threat.indicator.reference, threat.indicator.marking.tlp, threat.indicator.file.hash.md5, user.name, rule.name, related.user, related.hash, event.id, event.kind, event.category, event.type, event.original, ecs.version, error.message and tags. Two pieces of evidence that this is an oversight rather than a choice: (a) every sibling data stream in this package (e.g. data_stream/url/fields/) ships a fields/ecs.yml, and (b) the transform added in this same PR, elasticsearch/transform/latest_yaraify/fields/ecs.yml, declares exactly this field list. The consequence is visible in the generated docs: the '#### YARAify' exported-fields table in packages/ti_abusech/docs/README.md lists only abusech.yaraify., data_stream., event.dataset/module, input.type, labels.* and threat.feed.* - no threat.indicator.* rows at all, while the URL and ThreatFox tables include threat.indicator.first_seen/last_seen. Date-typed fields such as threat.indicator.first_seen therefore have no package-provided mapping.
Recommendation:
Add packages/ti_abusech/data_stream/yaraify/fields/ecs.yml mirroring the transform's list:
- external: ecs
name: ecs.version
- external: ecs
name: error.message
- external: ecs
name: tags
- external: ecs
name: related.hash
- external: ecs
name: related.user
- external: ecs
name: event.kind
- external: ecs
name: event.category
- external: ecs
name: event.type
- external: ecs
name: event.id
- external: ecs
name: event.original
- external: ecs
name: rule.name
- external: ecs
name: user.name
- external: ecs
name: threat.indicator.name
- external: ecs
name: threat.indicator.description
- external: ecs
name: threat.indicator.type
- external: ecs
name: threat.indicator.first_seen
- external: ecs
name: threat.indicator.reference
- external: ecs
name: threat.indicator.marking.tlp
- external: ecs
name: threat.indicator.file.hash.md5Then regenerate the docs so the YARAify exported-fields table matches the other data streams.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - name: api_type | ||
| value: community | ||
| - name: auth_key | ||
| commercial: |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/ti_abusech/manifest.yml:68
The commercial required_vars group omits auth_key, but the threatfox and malware data streams are Community-only and enabled by default, so a Commercial install collects nothing from them; add auth_key to the commercial group.
Details
api_type is a single package-level var shared by all data streams, but only url, malwarebazaar and yaraify branch on it. data_stream/threatfox/agent/stream/cel.yml.hbs and data_stream/malware/agent/stream/cel.yml.hbs contain no api_type handling at all - they always call the Community endpoint and send Auth-Key: state.auth_key. Both of those data streams have enabled: true in their manifests. With the required_vars groups as written, a user who selects 'Commercial API' is only prompted for username and password; auth_key is never required and the auth_key description on line 97 tells them it is 'Not used when API Type is Commercial API'. The default policy for a Commercial install therefore ships threatfox and malware with no Auth-Key and their requests are rejected by abuse.ch.
Recommendation:
Require auth_key in both groups, since the Community-only data streams need it regardless of the selected API type:
required_vars:
community:
- name: api_type
value: community
- name: auth_key
commercial:
- name: api_type
value: commercial
- name: auth_key
- name: username
- name: passwordand correct the auth_key description so it no longer says the key is unused for Commercial:
description: >-
abuse.ch Auth Key for the Community API. Always required for the ThreatFox and Malware data streams, which only support the Community API,
and for URLhaus, MalwareBazaar and YARAify when API Type is Community API. See [documentation](https://abuse.ch/blog/community-first/) for details.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| @@ -1,4 +1,12 @@ | |||
| # newer versions go on top | |||
| - version: "4.4.0" | |||
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/ti_abusech/changelog.yml:2
Moving malwarebazaar's api_type/username/password from data-stream vars to package-level input vars drops the values existing 4.2.x commercial users configured; record it as a breaking change and document the re-entry step.
Details
data_stream/malwarebazaar/manifest.yml deletes the api_type, username and password stream vars that 4.2.0 introduced, and packages/ti_abusech/manifest.yml re-adds them as input-level vars on the policy template. Fleet upgrades package policies by matching vars within their own scope, so a stream var that no longer exists in the new manifest is dropped and the new input-level var is initialised from its default (api_type: community). Anyone who configured MalwareBazaar Commercial collection under 4.2.x silently reverts to the Community path with no credentials after upgrading to 4.4.0. The 4.4.0 changelog records only two enhancement entries and says nothing about this relocation. The same release also narrows conditions.kibana.version from ^8.19.0 || ^9.1.0 to ^9.1.0, which blocks the upgrade for 8.x stacks and is likewise unrecorded.
Recommendation:
Add explicit entries so the relocation and the support drop are visible in the release notes, and mention the re-entry step in the README upgrade guidance:
- version: "4.4.0"
changes:
- description: |
Move `API Type`, `Username` and `Password` from the MalwareBazaar data stream to package-level settings.
Policies that configured MalwareBazaar Commercial collection in 4.2.x must re-enter these values after upgrading.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380
- description: Drop support for Kibana 8.x; the integration now requires Kibana 9.1.0 or later.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20380
- description: Add Commercial API support for the URLhaus data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380
- description: Add YARAify data stream with Community and Commercial API support.
type: enhancement
link: https://github.com/elastic/integrations/pull/20380🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| type: select | ||
| title: API Type | ||
| description: >- | ||
| Select the abuse.ch API to use for ThreatFox, URLhaus, MalwareBazaar, and YARAify. Community API uses an Auth Key. |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/ti_abusech/manifest.yml:78
The API Type description lists ThreatFox as a data stream the setting applies to, but ThreatFox has no Commercial support in this PR; drop it from the list.
Details
The description reads 'Select the abuse.ch API to use for ThreatFox, URLhaus, MalwareBazaar, and YARAify.' data_stream/threatfox/agent/stream/cel.yml.hbs has no api_type in its state block and no commercial branch in its CEL program - it always POSTs to the Community ThreatFox endpoint with the Auth-Key header. Selecting 'Commercial API' has no effect on ThreatFox other than removing the auth_key prompt (see finding 2). The auth_key description on line 97 repeats the same list and additionally omits the malware data stream, which also sends Auth-Key.
Recommendation:
List only the data streams that actually branch on api_type:
- name: api_type
type: select
title: API Type
description: >-
Select the abuse.ch API to use for URLhaus, MalwareBazaar, and YARAify. Community API uses an Auth Key.
Commercial API uses Spamhaus credentials and JWT authentication. The ThreatFox, Malware, SSL Blacklist and
JA3 Fingerprints data streams always use the Community API and always require the Auth Key.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| - rename: | ||
| field: abusech.yaraify.author | ||
| target_field: user.name |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/ti_abusech/data_stream/yaraify/elasticsearch/ingest_pipeline/default.yml:187
The YARAify pipeline maps the rule author only to user.name, while every other data stream in this package maps the submitter to threat.indicator.provider; also copy author to threat.indicator.provider.
Details
url, malwarebazaar and threatfox all rename their submitter field to threat.indicator.provider (for example rename_abusech_url_reporter_to_threat_indicator_provider_7c166e1f in data_stream/url/elasticsearch/ingest_pipeline/default.yml:153). The YARAify pipeline instead renames abusech.yaraify.author to user.name and never sets threat.indicator.provider, so YARAify indicators are the only ones in this package with no provider attribution in the Kibana threat intel UI or in the latest_yaraify destination index. user.name is also a poor fit for the value: the samples carry display names such as 'Example Author' rather than logins.
Recommendation:
Keep the user.name rename if it is wanted, and add a provider copy next to it:
- rename:
field: abusech.yaraify.author
target_field: user.name
tag: rename_author_to_user_name
ignore_missing: true
- set:
field: threat.indicator.provider
tag: set_threat_indicator_provider_from_author
copy_from: user.name
if: ctx.user?.name != nullRemember to add threat.indicator.provider to the new data stream fields/ecs.yml (finding 1) and to elasticsearch/transform/latest_yaraify/fields/ecs.yml.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ).as(since, | ||
| request( | ||
| "GET", | ||
| state.url.trim_right("/") + "/urlhaus/v1/urls?" + { |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/ti_abusech/data_stream/url/agent/stream/cel.yml.hbs:97
Selecting Commercial API while leaving the data stream URL at its Community default builds a nonsense endpoint; add a guard like the existing deprecated-URL check.
Details
api_type is a package-level setting while url stays per data stream, and the url data stream keeps its Community default https://urlhaus-api.abuse.ch/v1/urls/recent/. Switching the package to Commercial without also editing each data stream's URL makes the program request https://urlhaus-api.abuse.ch/v1/urls/recent/v1/login and .../urls/recent/urlhaus/v1/urls. The user only sees a generic non-200 error message. The community branch of this same program already guards a comparable misconfiguration (the deprecated /downloads/json endpoint) with an explicit configuration_error, and the commercial branch already guards missing username/password, so a matching check would be consistent. The yaraify template has the same shape.
Recommendation:
Add a configuration guard at the top of the commercial branch, alongside the existing username/password check:
state.?api_type.orValue("community") == "commercial" ?
(
state.url.contains("urlhaus-api.abuse.ch") ?
{
"events": {
"error": {
"code": "400",
"id": "configuration_error",
"message": "api_type is 'commercial' but the URL is still set to the Community endpoint. Set the URL to the commercial API base URL, for example https://api.spamhaus.com.",
},
},
"want_more": false,
}
: !has(state.username) || !has(state.password) ?
...
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
🚀 Benchmarks reportTo see the full report comment with |
efd6
left a comment
efd6
left a comment
There was a problem hiding this comment.
Please go over and address or respond to the Vera comments.
| ) | ||
| ) | ||
| : | ||
| state.url.contains("/downloads/json") ? |
| ).as(since, | ||
| request( | ||
| "GET", | ||
| state.url.trim_right("/") + "/urlhaus/v1/urls?" + { |
There was a problem hiding this comment.
If you address the comments for the url data stream, also do it here.
|
/test |
kcreddy
left a comment
kcreddy
left a comment
There was a problem hiding this comment.
Correct the PR description and commit message
moxarth-rathod
changed the title
…nsform.yml Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>
|
🟢 No issues across the latest commits fc1ae98. Review summaryIssues found across earlier commits d9cb765 — 3 high, 2 medium, 1 low
Issues found across earlier commits 0181437 — 2 high, 2 medium, 1 low
Issues found across earlier commits 5f00644 — 1 high, 2 medium, 3 low
Issues found across earlier commits 98f3fb2 — 4 high, 6 medium
Issues found across earlier commits 2f1b703 — 2 high, 7 medium, 1 low
Issues found across earlier commits 21f540c — 3 high, 9 medium, 1 low
🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills
|
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
|
|
Tick the box to add this pull request to the merge queue (same as
|
|
Package ti_abusech - 5.1.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/5.1.0/ |
The YARAify change re-exported the ti_abusech dashboards with useCurrentFilters: false on every link, so this PR's changes to the package had become pure key reordering plus a no-op version bump. Restore the package to main.
…ing on dashboard links panels (#21175) Links panels default to useCurrentFilters: true, which forwards app-state filters from the source dashboard to the destination. Kibana then merges those with the destination's saved filters. When each dashboard carries a dashboard-level dataset filter (for example data_stream.dataset), navigation via links combines conflicting filters and panels show no data. Set useCurrentFilters: false on every dashboardLink in the affected dashboards. Pinned filters are still forwarded, so analysts can opt in by pinning a filter before navigating. For axonius, jupiter_one, and m365_defender, useCurrentDateRange is set to true, matching Kibana's default so time-range behaviour is unchanged. For jamf_pro, the Overview link is set to useCurrentDateRange: false to match the four sibling links in the same panel, which already had that value. ti_abusech was originally included but is no longer needed: #20380 re-exported its dashboards with useCurrentFilters: false already set, so its changes here reduced to key reordering with no effect.
Proposed commit message
Checklist
changelog.ymlfile.How to test this PR locally
Related issues