Skip to content

Add arkime capture statistics to the packet capture statistics dashboard #703

Closed

Description

@mmguero

Malcolm's Packet Capture Statistics dashboards shows information from Zeek and Suricata about packet loss, numbers of packets, missed ACKs, etc.

A user has requested to add to that dashboard, if possible:

  • Statistics from Arkime capture
    • I don't know where this is stored or if it's available to us, but I imagine it must be somewhere as I think Arkime has a corresponding visualization in its stats page
  • Statistics from the NIC itself (/proc-level)
    • netdev-json.sh is already used by fluent-bit on hedgehog and in the ISO-installed version of Malcolm to send NIC-level capture information, we could probably integrate it into the capture container (?) as well, but we'd have to hook up the plumbing to get it sent into the log pipeline which can be a bit of a trick (in Docker mode, at least) due to the network: host mode on the capture container. Probably need to make it just output to a file or something and have filebeat pick it up.
    • NOTE: moving this second part to another issue to track separately

Activity

  1. added this to the z.staging milestone on Jul 2, 2025
  2. added
    captureRelating to pcap-capture container
    arkimeRelating to Malcolm's use of Arkime
    dashboardsRelating to Malcolm's OpenSearch Dashboards interface
    on Jul 2, 2025
  3. added theissue type on Jul 2, 2025
  4. moved this to Todo (design) in Malcolmon Jul 2, 2025
  5. modified the milestones: z.staging, v25.07.0 on Jul 9, 2025
  6. modified the milestones: v25.07.0, v25.08.0 on Jul 24, 2025
  7. modified the milestones: v25.08.1, v25.09.0 on Aug 18, 2025
  8. scott-jeffery commented on Sep 3, 2025

    @scott-jeffery
    Collaborator

    I've added a PR for the Arkime portion of this Issue

    Arkime updates the arkime_stats_v30 index with internally calculated capture statistics. The method used to determine capture statistics depends on the Arkime reader used. For example: the tpacketv3 reader determines capture statistics via a getsockopt call. These statistics are updated in a single cumulative record for each Arkime instance. If Malcolm is run along with a Hedgehog sensor there will be two records in the arkime_stats_v30 index, one for Arkime running under Malcolm and one for Arkime running under Hedgehog. (In reality there will be three records as arkime keeps an initial "dummy" record which we filter out when displayed)

    The update timing for each arkime_stats_v30 record varies depending how Arkime is run. Arkime starts a separate thread with several timers that may or may not cause a record update on each pass. Typically Arkime running under the Malcolm profile will update the capture statistics about every 10 minutes while Arkime run under Hedgehog can be as often as every two seconds.

    A new visualization pane has been added to the "Packet Capture Statistics" Malcolm dashboard. This required a new Index Pattern (arkime_stats_template.json) to make the index fields available as visualization elements. Since Arkime updates a single cumulative record, rather than adding time-series records like Zeek and Suricata do, the shared-object-creation.sh shell script was updated to handle time-series index patterns as well as non-time-series index patterns with a "null" timeFieldName.

    Further work could be done to better understand the Arkime update timing and reader dependent capture data sources. This PR marks the first step in displaying the capture data that Arkime makes available in an index.

  9. mmguero commented on Sep 4, 2025

    @mmguero
    CollaboratorAuthor

    @scott-jeffery I've merged your PR into idaholab/staging. I made one minor change (mmguero-dev/Malcolm@ffbaa06; ignore the other formatting junk in config.py) to the dashboard: rather than explicitly defining the arkime stats index pattern in the output of the dashboard JSON, I removed it (and adjusted the dashboard export API to remove it going forward) as the index template itself is enough: Arkime can create the index pattern and the template will let dashboards know what to do about it.

  10. moved this from Todo (design) to In Progress in Malcolmon Sep 4, 2025
  11. added a commit that references this issue on Sep 16, 2025
  12. self-assigned this
    on Sep 16, 2025
  13. changed the title [-]Add arkime and NIC info to the packet capture statistics dashboard[/-] [+]Add arkime capture statistics to the packet capture statistics dashboard[/+] on Sep 16, 2025
  14. moved this from In Progress to Testing in Malcolmon Sep 16, 2025
  15. removed their assignment
    on Sep 16, 2025
  16. added a commit that references this issue on Sep 22, 2025
  17. moved this from Testing to Done in Malcolmon Sep 23, 2025
  18. moved this from Done to Released in Malcolmon Sep 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arkimeRelating to Malcolm's use of ArkimecaptureRelating to pcap-capture containerdashboardsRelating to Malcolm's OpenSearch Dashboards interfaceenhancementNew feature or request

Fields

No fields configured for Feature.

Projects

  • Status
    Released

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions