[XM Cyber] Migrate VRM data streams to the vrmReport API and add vulnerability_instance data stream - #21162
[XM Cyber] Migrate VRM data streams to the vrmReport API and add vulnerability_instance data stream#21162muskan-agarwal26 wants to merge 5 commits into
Conversation
1. Added vulnerability instance datastream development. 2. Updated all datastream dashboard to include link of new datastream dashboard.
Elastic Docs Style Checker (Vale)Summary: 2 warnings, 1 suggestion found
|
| File | Line | Rule | Message |
|---|---|---|---|
| packages/xm_cyber/data_stream/vulnerability_instance/fields/fields.yml | 19 | Elastic.QuotesPunctuation | Place punctuation inside closing quotation marks. |
| packages/xm_cyber/data_stream/vulnerability_instance/manifest.yml | 34 | Elastic.Latinisms | Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'. |
💡 Suggestions (1): Optional style improvements. Apply when helpful.
| File | Line | Rule | Message |
|---|---|---|---|
| packages/xm_cyber/data_stream/vulnerability_instance/fields/fields.yml | 19 | Elastic.WordChoice | Consider using 'can, might' instead of 'May', unless the term is in the UI. |
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
🚀 Benchmarks reportTo see the full report comment with |
muskan-agarwal26
added
enhancement
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| - set: | ||
| tag: set_ecs_version_8c2c9474 | ||
| field: ecs.version | ||
| value: 9.5.0 |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/xm_cyber/data_stream/vulnerability_instance/elasticsearch/ingest_pipeline/default.yml:7
vulnerability_instance pipeline sets ecs.version 9.5.0 but build.yml pins git@v9.4.0 (siblings use 9.4.0) — change it to 9.4.0 so the reported ECS version matches the version the mappings are built against.
Details
The new vulnerability_instance pipeline hardcodes ecs.version: 9.5.0, but the package's _dev/build/build.yml pins the ECS reference at git@v9.4.0, and the three sibling pipelines changed in this same PR (device, product, vulnerability) all set ecs.version: 9.4.0. The ecs.version value written into every document must match the ECS version the field mappings were actually generated from (the build.yml pin); a mismatch mislabels the documents and is the exact pipeline/build.yml inconsistency flagged as HIGH by the review rubric. vulnerability_instance is not an entity data stream (its pipeline sets event.kind: event, not asset, and no entity.attributes.*/entity.lifecycle.*/entity.relationships.* fields are declared), so it does not require the 9.5.0 pin — it should align with the rest of the package at 9.4.0. (Bumping build.yml to 9.5.0 instead would break the other three pipelines that stay at 9.4.0, so aligning this one down is the correct minimal fix.)
Recommendation:
Set the new pipeline's ECS version to match build.yml (git@v9.4.0) and its sibling pipelines:
- set:
tag: set_ecs_version_8c2c9474
field: ecs.version
value: 9.4.0| value: 9.5.0 | |
| value: 9.4.0 |
🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
✅ All changelog entries have the correct PR link. |
| dependencies: | ||
| ecs: | ||
| reference: 'git@v9.4.0' | ||
| reference: 'git@v9.5.0' |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/xm_cyber/_dev/build/build.yml:3
The build.yml ECS pin bump to git@v9.5.0 now mismatches the risk_score pipeline, which still sets ecs.version 9.4.0 — bump risk_score's set_ecs_version processor (and its regenerated sample_event) to 9.5.0 so the reported version matches the mappings built into the package.
Details
This delta bumps the package ECS pin to git@v9.5.0 and raises ecs.version to 9.5.0 in every data stream pipeline except risk_score. data_stream/risk_score/elasticsearch/ingest_pipeline/default.yml still sets field: ecs.version / value: 9.4.0, and data_stream/risk_score/sample_event.json still reports "version": "9.4.0". Because build.yml is the single pin that determines which ECS field definitions are imported for the whole package, risk_score documents will now advertise ecs.version 9.4.0 while actually being mapped against ECS 9.5.0 — an internal inconsistency the reviewer skill flags as HIGH (pipeline ecs.version out of sync with the build.yml pin). The prior review finding (2d86bd985d377cbd) recommended aligning on 9.4.0; that recommendation is superseded here because the package contains an entity data stream (entity_inventory sets event.kind: asset), which requires the git@v9.5.0 pin, so 9.5.0 is the correct target and risk_score is the one stream left behind. The risk_score pipeline is not itself a changed file, so this is reported on the build.yml line that introduces the mismatch.
Recommendation:
Bring the risk_score pipeline in line with the new git@v9.5.0 pin (matching the other six pipelines), then regenerate its sample_event via system tests:
- set:
field: ecs.version
tag: set_ecs_version_8c2c9474
value: 9.5.0🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Review summaryIssues found across the latest commits ed634b3 — 1 high
Issues found across earlier commits 725bf43 — 1 high
🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills
|
💚 Build Succeeded
History
|
1 participant
Type of change
Proposed commit message
vulnerability,device,productnow call/api/v2/vrm/public/vrmReport/{vulnerabilities,devices,products}and paginate with thetop-level
nextPageLinkinstead ofmeta.next.nextPageLinkis an absolute URL whosescheme/host can differ from the configured base URL (both
http://andhttps://have beenobserved), so the input parses it and reuses only its path and query on top of the configured URL.
product:productOperatingSystemswas renamed by the API toproductOperatingSystem;field, pipeline output and Product dashboard updated accordingly.
device: the API no longer returnsisChokePointor the per-deviceapps[]array (that datamoved to
vulnerabilityInstances). Removedxm_cyber.device.is_choke_point, allxm_cyber.device.apps.*fields and their pipeline processors. Device dashboard KPI now useschoke_point_level.vulnerability: the API no longer returnsstatus,exploitKitExist,isExploitedInTheWildor
techniqueId(the feed only contains open CVEs). Removed those fields and the dashboardpanels that depended on them (Status filter/pie, exploited KPI now based on
in_cisa_kev).Added the nine new advisory URL fields (
nvd_url,cve_org_url,msrc_url,red_hat_url,amazon_linux_url,debian_url,ubuntu_url,suse_url,oracle_url). Missing CVSS scoresnow arrive as
0with an empty vector instead ofnull; the pipeline drops the score when thevector is empty so
vulnerability.score.baseis not set to0.latest_vulnerabilitytransform fields synced and
fleet_transform_versionbumped to 0.2.0.vulnerability_instancedata stream: one event per device from/api/v2/vrm/public/vrmReport/vulnerabilityInstances, withproduct_versions[](vendor,name, version, file paths, active CVEs with safe version, closed CVE ids). Maps
host.id,device.id,vulnerability.enumeration. Includes ILM policy, pipeline/system tests and a new[Logs XM Cyber] Vulnerability Instancesdashboard; existing dashboards link to it.Checklist
changelog.ymlfile.How to test this PR locally
Related issues
Screenshots