Skip to content

[stormshield] Add thematic dashboards and rework the Overview - #20088

Closed
arenard wants to merge 4 commits into
elastic:mainfrom
arenard:stormshield-dashboards
Closed

[stormshield] Add thematic dashboards and rework the Overview#20088
arenard wants to merge 4 commits into
elastic:mainfrom
arenard:stormshield-dashboards

Conversation

@arenard

@arenard arenard commented Jul 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

Add three thematic dashboards keyed on the log families the pipeline now categorizes: Network Traffic (connection/filter events, verdicts, endpoints, rules, byte volumes), Web Activity (domains, HTTP methods and return codes, site categories, volumes) and Users & VPN (authentication, administration sessions and IPsec/SSL VPN events with their outcomes, users, directories, groups and peers). Every dashboard scopes its data with dashboard-level data_stream.dataset and stormshield.logtype filters, keeps panel queries empty, and drops the log-type control that its own filter would contradict.

Add a links panel at the top of the Overview pointing to the four other dashboards, extend it with an appliance health row covering the periodic statistics families (scoped with panel-level queries, since the Overview carries no log-type filter), title its event-count metric panel, and replace the terms aggregation on event.duration with a median over time. Fix the dashboard controls, which carried no data view reference and were silently dropped by Kibana at render time. Rename all dashboards to the [Stormshield SNS] title prefix so future Stormshield product integrations can share the vendor namespace unambiguously. Replace the outdated Overview screenshot with captures of the five dashboards rendered on a live stack.

Design notes

  • The dashboard count and per-dashboard density are calibrated on comparable packages (checkpoint ships four dashboards of 7-14 panels): five dashboards total, none under 8 panels.
  • Collapsible dashboard sections were considered and set aside: packages that use them (anthropic) require Kibana ^8.19.0 || ^9.1.0, while this package supports ^8.11.4. The appliance health row of the Overview is the natural candidate for a collapsed section once the version floor moves.
  • The links panel follows the structure used by other packages in the repository, with dashboard references resolved through saved-object references.
  • Dashboard saved-object IDs of the two existing dashboards are unchanged, so upgrades replace them in place.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • elastic-package check passes and the package installs into Kibana 9.4.3: all five dashboards import, the Overview resolves its four dashboard links, and rendering shows no dropped-panel warnings
  • The manifest screenshots are captures of the five dashboards rendered by Kibana 9.4.3 over fixture-derived events
  • Every aggregated field is aggregatable and declared in the package field files; all panel-level queries are empty and scoping is done by dashboard-level filters
  • Existing dashboard saved-object IDs are preserved

How to test this PR locally

cd packages/stormshield
elastic-package check
elastic-package stack up -d
elastic-package install

Related issues

@arenard
arenard force-pushed the stormshield-dashboards branch from a2b2382 to 566d6e0 Compare July 10, 2026 17:25
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@arenard
arenard force-pushed the stormshield-dashboards branch 3 times, most recently from 1b6796f to 06384f6 Compare July 10, 2026 18:00
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@andrewkroh andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:stormshield StormShield SNS dashboard Relates to a Kibana dashboard bug, enhancement, or modification. labels Jul 10, 2026
@arenard
arenard force-pushed the stormshield-dashboards branch 2 times, most recently from 5568233 to 187dfca Compare July 14, 2026 14:13
This was referenced Jul 14, 2026
Closed
Merged
@arenard
arenard force-pushed the stormshield-dashboards branch 3 times, most recently from 53ed7d6 to 79181ab Compare July 17, 2026 15:38
@arenard
[stormshield] Add outcome, identity and HTTP mappings and extend the …
cb25778 
…metadata allow-list

Derive `event.outcome` on the auth, server and xvpn families from the vendor `error` return code (`0` maps to success, other codes to failure). The official field documentation lists exactly these three families for `error`; on vpn logs the same key is a severity level and is left untouched. The vendor documentation pairs `0`-first numeric examples with `Success`-first display labels without stating the convention outright, so the mapping follows the strong implication of the field tables.

Map `domain` to `user.domain` and `usergroup` to `user.group.name`, splitting the comma-separated multi-valued form that SNS 5.x emits. On web logs, map `op` to `http.request.method` and `result` to `http.response.status_code`; on other families these keys carry protocol-specific codes (FTP, Modbus) and are not promoted. Map the module name in `service` to `process.name` on the system, sandboxing and routing families only — on pvm logs the same key names the vulnerable product detected on a scanned host. Set `observer.product`. All mappings are copies; the vendor keys remain available under `stormshield.metadata`.

Extend the `stormshield.metadata` allow-list with the documented vendor keys that previously leaked as undeclared `stormshield.*` fields: host and IP reputation (`srchostrep`, `dsthostrep`, `srciprep`, `dstiprep`), alarm packet captures (`pktlen`, `pktdump`, `pktdumplen`), Ethernet protocol (`etherproto`), SSO agent (`agentid`), IKE peer name (`peername`), and the industrial-protocol plugin fields (`unitid`/`unit_id`, `cipservicecode`, `cipclassid`, `requestmode`, `responsemode`, `UI`, `error_class`, `error_code`, `format`).

The NAT reading of the pipeline was verified against the official field tables while preparing this change: `origdst`/`origdstport` are documented as the destination before translation and map to `destination.ip`/`destination.port`; `dst` is the translated destination and maps to `destination.nat.ip` when it differs. A connection sample with the documented example values now locks this in the pipeline tests, alongside new samples for the alarm reputation and packet-capture fields, a Modbus plugin line (whose `result` stays under metadata and does not produce `http.*`), a server line, a system line, and multi- and single-valued `usergroup` forms.
@arenard
arenard force-pushed the stormshield-dashboards branch from 8eab066 to 5d10b00 Compare July 20, 2026 08:04
@arenard

arenard commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

The changelog entries linking #20085 belong to the 1.7.0 release and show up in this diff only because the branch is stacked on that PR. The 1.8.0 entries introduced here all link this PR. This check will resolve once #20085 merges and this branch is updated on top of main.

@arenard
arenard force-pushed the stormshield-dashboards branch 5 times, most recently from 8288545 to fb93486 Compare July 20, 2026 10:29
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jul 20, 2026

Copy link
Copy Markdown

⏳ Build in-progress, with failures

Failed CI Steps

History

@arenard
arenard force-pushed the stormshield-dashboards branch from fb93486 to b9fcf25 Compare July 20, 2026 10:43
@arenard
[stormshield] Type the detection-relevant vendor fields
78b1112 
Declare typed stormshield.* fields for the documented vendor keys that
carry detection value instead of routing them to the flattened metadata
object: the alarm context (class, classification, repeat, sensible,
target, icmptype, icmpcode) and packet capture (pktlen, pktdumplen,
pktdump), the IP and host reputation keys (srciprep, dstiprep,
srchostrep, dsthostrep), the IKE negotiation keys of IPsec VPN logs
(phase, side, ikev, spi_in, spi_out, cookie_i, cookie_r, localnet,
remotenet, remoteid), the authentication keys (method, totp, confid,
agentid), the administration session identifier (sessionid), the filter
slot (slotlevel, slotname), the Ethernet protocol (etherproto), the
industrial and REST protocol identifiers (unitid, cipservicecode,
cipclassid, requestmode, responsemode, format, error_class, error_code),
the web filtering keys (cat_site, urlruleid, contentpolicy), the
antivirus, antispam and sandboxing verdicts (virus, spamlevel, filename,
filetype, sandboxing, sandboxinglevel) and the application
identification keys (clientappid, serverappid, groupid).

Numeric keys are converted to long so they stay aggregatable, the
comma-separated IP reputation keys are split into arrays and the curly
braces of the vendor cat_site format are removed. spamlevel stays
keyword because the documented values include X and ?, and
contentpolicy keeps its zero-padded form. The field descriptions follow
the wording of the official audit log documentation, cross-checked
against the v4 and v5 field tables; the keys that v5 no longer
documents are marked as such. The error key stays under
stormshield.metadata where the outcome derivation reads it. Most of the
moved keys reached the flattened object in the 1.6.0 and 1.7.0
releases, so relocating them now, before those releases see wide
adoption, keeps the migration cost minimal.

Add pipeline test samples derived from the official documentation
examples for ICMP alarms, for the NTP and S7 protocol inspections, and
for the antivirus, antispam and sandboxing verdicts on an SMTP flow.
@arenard
arenard force-pushed the stormshield-dashboards branch from b9fcf25 to 68d774a Compare July 20, 2026 10:47
arenard added 2 commits July 20, 2026 13:05
@arenard
[stormshield] Type the appliance health counters
0669795 
Type the counters of the periodic statistics logs as aggregatable
numbers instead of flattened strings. The scalar counters of the filter
statistics log (accepted and blocked totals, memory and state gauges)
move to long fields under stormshield.filterstat, keeping the names the
vendor writes. The CPU counters of the monitor log (time allocated to
user processes, kernel time and system disruptions, as the official
documentation describes them) move to stormshield.monitor.cpu, and the
security level of the monitor log becomes the long field
stormshield.security. Counters whose value does not parse as a number
fall back to the flattened metadata object unchanged, and the composite
in/out pairs stay under metadata where the existing byte extraction
reads them.

These fields feed appliance health dashboards and alerting on CPU
time, memory gauges and blocked packet rates, which the flattened
object could not serve. The per-method authentication counters of the
authstat log stay under metadata: the official field tables do not
document them, and the per-interface structures of the monitor log
stay as well, pending a document layout that supports per-interface
aggregation.
@arenard
[stormshield] Add thematic dashboards and rework the Overview
badc535 
Add three thematic dashboards keyed on the log families the pipeline now categorizes: Network Traffic (connection/filter events, verdicts, endpoints, rules, byte volumes), Web Activity (domains, HTTP methods and return codes, site categories, volumes) and Users & VPN (authentication, administration sessions and IPsec/SSL VPN events with their outcomes, users, directories, groups and peers). Every dashboard scopes its data with dashboard-level `data_stream.dataset` and `stormshield.logtype` filters, keeps panel queries empty, and drops the log-type control that its own filter would contradict.

Add a links panel at the top of the Overview pointing to the four other dashboards, extend it with an appliance health row covering the periodic statistics families (scoped with panel-level queries, since the Overview carries no log-type filter), title its event-count metric panel, and replace the terms aggregation on `event.duration` with a median over time. Fix the dashboard controls, which carried no data view reference and were silently dropped by Kibana at render time. Rename all dashboards to the `[Stormshield SNS]` title prefix so future Stormshield product integrations can share the vendor namespace unambiguously. Replace the outdated Overview screenshot with captures of the five dashboards rendered on a live stack.
@arenard
arenard force-pushed the stormshield-dashboards branch from 68d774a to badc535 Compare July 20, 2026 11:05
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Changelog link mismatch — expected https://github.com/elastic/integrations/pull/20088 in the following file(s):

  • packages/stormshield/changelog.yml

Tip

If expected, add the changelog-link-check:skip label to skip this check. Or, if an issue link was intended, use .../issues/<n> instead.

View Buildkite build
@arenard

@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown

💔 Build Failed

Failed CI Steps

History

@botelastic

botelastic Bot commented Aug 19, 2026

Copy link
Copy Markdown

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic Bot added the Stalled label Aug 19, 2026
@botelastic

botelastic Bot commented Sep 18, 2026

Copy link
Copy Markdown

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

@botelastic botelastic Bot closed this Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:stormshield StormShield SNS Stalled

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@arenard @andrewkroh