Repository navigation
Arkime_Capture_Statistics - #640
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is the Arkime part of Issue #703
Arkime updates the arkime_stats_v30 index with internally calculated capture statistics. The method used to determine capture statistics depends on the Arkime reader used. For example: the tpacketv3 reader determines capture statistics via a getsockopt call. These statistics are updated in a single cumulative record for each Arkime instance. If Malcolm is run along with a Hedgehog sensor there will be two records in the arkime_stats_v30 index, one for Arkime running under Malcolm and one for Arkime running under Hedgehog. (In reality there will be three records as arkime keeps an initial "dummy" record which we filter out when displayed)
The update interval for each arkime_stats_v30 record varies depending how Arkime is run. Arkime starts a separate thread with several timers that may or may not cause a record update on each pass. Typically Arkime running under the Malcolm profile will update the capture statistics about every 10 minutes while Arkime run under Hedgehog can be as often as every two seconds.
A new visualization pane has been added to the "Packet Capture Statistics" Malcolm dashboard. This required a new Index Pattern (arkime_stats_template.json) to make the index fields available as visualization elements. Since Arkime updates a single cumulative record, rather than adding time-series records like Zeek and Suricata do, the shared-object-creation.sh shell script was updated to handle time-series index patterns as well as non-time-series index patterns with a "null" timeFieldName.
Further work could be done to better understand the Arkime update timing and reader dependent capture data sources. This PR marks the first step in displaying the capture data that Arkime makes available in an index.