You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add arkime capture statistics to the packet capture statistics dashboard #703
Malcolm's Packet Capture Statistics dashboards shows information from Zeek and Suricata about packet loss, numbers of packets, missed ACKs, etc.
A user has requested to add to that dashboard, if possible:
Statistics from Arkime capture
I don't know where this is stored or if it's available to us, but I imagine it must be somewhere as I think Arkime has a corresponding visualization in its stats page
Statistics from the NIC itself (/proc-level)
netdev-json.sh is already used by fluent-bit on hedgehog and in the ISO-installed version of Malcolm to send NIC-level capture information, we could probably integrate it into the capture container (?) as well, but we'd have to hook up the plumbing to get it sent into the log pipeline which can be a bit of a trick (in Docker mode, at least) due to the network: host mode on the capture container. Probably need to make it just output to a file or something and have filebeat pick it up.
NOTE: moving this second part to another issue to track separately
I've added a PR for the Arkime portion of this Issue
Arkime updates the arkime_stats_v30 index with internally calculated capture statistics. The method used to determine capture statistics depends on the Arkime reader used. For example: the tpacketv3 reader determines capture statistics via a getsockopt call. These statistics are updated in a single cumulative record for each Arkime instance. If Malcolm is run along with a Hedgehog sensor there will be two records in the arkime_stats_v30 index, one for Arkime running under Malcolm and one for Arkime running under Hedgehog. (In reality there will be three records as arkime keeps an initial "dummy" record which we filter out when displayed)
The update timing for each arkime_stats_v30 record varies depending how Arkime is run. Arkime starts a separate thread with several timers that may or may not cause a record update on each pass. Typically Arkime running under the Malcolm profile will update the capture statistics about every 10 minutes while Arkime run under Hedgehog can be as often as every two seconds.
A new visualization pane has been added to the "Packet Capture Statistics" Malcolm dashboard. This required a new Index Pattern (arkime_stats_template.json) to make the index fields available as visualization elements. Since Arkime updates a single cumulative record, rather than adding time-series records like Zeek and Suricata do, the shared-object-creation.sh shell script was updated to handle time-series index patterns as well as non-time-series index patterns with a "null" timeFieldName.
Further work could be done to better understand the Arkime update timing and reader dependent capture data sources. This PR marks the first step in displaying the capture data that Arkime makes available in an index.
@scott-jeffery I've merged your PR into idaholab/staging. I made one minor change (mmguero-dev/Malcolm@ffbaa06; ignore the other formatting junk in config.py) to the dashboard: rather than explicitly defining the arkime stats index pattern in the output of the dashboard JSON, I removed it (and adjusted the dashboard export API to remove it going forward) as the index template itself is enough: Arkime can create the index pattern and the template will let dashboards know what to do about it.
changed the title [-]Add arkime and NIC info to the packet capture statistics dashboard[/-][+]Add arkime capture statistics to the packet capture statistics dashboard[/+]on Sep 16, 2025
Activity
scott-jeffery commentedon Sep 3, 2025
I've added a PR for the Arkime portion of this Issue
Arkime updates the arkime_stats_v30 index with internally calculated capture statistics. The method used to determine capture statistics depends on the Arkime reader used. For example: the tpacketv3 reader determines capture statistics via a getsockopt call. These statistics are updated in a single cumulative record for each Arkime instance. If Malcolm is run along with a Hedgehog sensor there will be two records in the arkime_stats_v30 index, one for Arkime running under Malcolm and one for Arkime running under Hedgehog. (In reality there will be three records as arkime keeps an initial "dummy" record which we filter out when displayed)
The update timing for each arkime_stats_v30 record varies depending how Arkime is run. Arkime starts a separate thread with several timers that may or may not cause a record update on each pass. Typically Arkime running under the Malcolm profile will update the capture statistics about every 10 minutes while Arkime run under Hedgehog can be as often as every two seconds.
A new visualization pane has been added to the "Packet Capture Statistics" Malcolm dashboard. This required a new Index Pattern (arkime_stats_template.json) to make the index fields available as visualization elements. Since Arkime updates a single cumulative record, rather than adding time-series records like Zeek and Suricata do, the shared-object-creation.sh shell script was updated to handle time-series index patterns as well as non-time-series index patterns with a "null" timeFieldName.
Further work could be done to better understand the Arkime update timing and reader dependent capture data sources. This PR marks the first step in displaying the capture data that Arkime makes available in an index.
mmguero commentedon Sep 4, 2025
@scott-jeffery I've merged your PR into idaholab/staging. I made one minor change (mmguero-dev/Malcolm@ffbaa06; ignore the other formatting junk in
config.py) to the dashboard: rather than explicitly defining the arkime stats index pattern in the output of the dashboard JSON, I removed it (and adjusted the dashboard export API to remove it going forward) as the index template itself is enough: Arkime can create the index pattern and the template will let dashboards know what to do about it.for cisagov#703, record stats for network interfaces periodically
[-]Add arkime and NIC info to the packet capture statistics dashboard[/-][+]Add arkime capture statistics to the packet capture statistics dashboard[/+]tweaks to dashbaord for cisagov#703, add arkime capture stats to the …