Skip to content

Double imports when restarting Malcolm #588

Closed

Description

@KchChr

When restarting Malcom (./scripts/stop ./scripts/start) or rebooting the VM, Malcolm reimports all files in the pcap/processed folder and they appear double in the arkime - files view.

To reproduce: restart Malcolm. This behaviour appears everytime Malcolm restarts or the VM restarts.

I was not able to find an „expect behaviour“.

Malcolm version 24.03.0

I am running Malcolm by cloning the Git Repo on a Debian 12 VM airgapped

Activity

  1. mmguero commented on Feb 14, 2025

    @mmguero
    Collaborator

    Hm, there's supposed to be a check against the arkime files database that prevents that from happening, I'll have to look into that. In the meantime, try this:

    Set those values to true and restart Malcolm and see how that goes.

  2. KchChr commented on Feb 14, 2025

    @KchChr
    Author

    This worked. I assume this is a workaround?

    Thank you for the fast answer.
    Have a nice weekend!

  3. mmguero commented on Feb 14, 2025

    @mmguero
    Collaborator

    Yes, this is a workaround, but like I said it should have prevented that with the files database check I mentioned, so I'll leave this bug open to try to figure that out.

  4. dev-cti commented on Oct 9, 2025

    @dev-cti

    After some investigations, we found the root cause of the bug causing multiple imports after a Malcolm restart.

    Reproduce the bug

    • Set the PCAP_PIPELINE_VERBOSITY to -vvv to get logs
    • Upload or copy a new PCAP file in upload directory, wait for analysis completion and insertion into Arkime's OS file indexes
    • touch the same PCAP file in processed directory, this triggers an filesystem event handled by pcap_watcher.py
    • The file is re-analyzed and not seen as duplicate

    Root Cause

    While processing a new file (in pcap_processor.py) arkimeCaptureFileWorker invokes Arkime with --node parameter with [nodeName]-upload when the capture file is uploaded/copied in upload directory :

    # if this is an uploaded PCAP (not captured "live"")
    #   append -upload to the node name used (which originates from PCAP_NODE_NAME)
    tmpNodeName = fileInfo[FILE_INFO_DICT_NODE] if (FILE_INFO_DICT_NODE in fileInfo) else nodeName
    if tmpNodeName and (
    	(not (FILE_INFO_DICT_LIVE in fileInfo)) or (not fileInfo[FILE_INFO_DICT_LIVE])
    ):
    	tmpNodeName = tmpNodeName + '-upload'

    The pcap-publisher (pcap-watcher.py) in pcap-monitor service checks if the filename is a duplicate in EventWatcher's processFile :

    # check with Arkime's files index in OpenSearch and make sure it's not a duplicate
    fileIsDuplicate = False
    if self.useOpenSearch:
    	s = (
    		SearchClass(using=self.openSearchClient, index=ARKIME_FILES_INDEX)
    		.filter("term", node=args.nodeName) # [1]
    		.query("wildcard", name=f"*{os.path.sep}{relativePath}")
    	)
    	response = s.execute()
    	for hit in response:
    		fileInfo = hit.to_dict()
    		if (ARKIME_FILE_SIZE_FIELD in fileInfo) and (fileInfo[ARKIME_FILE_SIZE_FIELD] == fileSize):
    			fileIsDuplicate = True
    			break
    
    if fileIsDuplicate:
    	# this is duplicate file (it's been processed before) so ignore it
    	self.logger.info(f"{scriptName}:\t📋\t{pathname}")

    The duplicate search always fails because the case where -upload is appened to nodeName is never checked in the query filter in [1].

    Fix

    A working fix is to change the filter term to regexp to match nodeName.* :

    s = (
    	SearchClass(using=self.openSearchClient, index=ARKIME_FILES_INDEX)
    	 .filter("regexp", node=f"{args.nodeName}.*")
    	.query("wildcard", name=f"*{os.path.sep}{relativePath}")
    )
  5. mmguero commented on Oct 9, 2025

    @mmguero
    Collaborator

    thanks for the thorough analysis, i will take a look at this.

  6. moved this to Todo (investigate) in Malcolmon Oct 9, 2025
  7. added this to the v25.10.0 milestone on Oct 9, 2025
  8. added theissue type on Oct 9, 2025
  9. added
    uploadRelating to PCAP and/or Zeek log ingestion
    on Oct 9, 2025
  10. self-assigned this
    on Oct 13, 2025
  11. moved this from Todo (investigate) to Testing in Malcolmon Oct 13, 2025
  12. added a commit that references this issue on Oct 13, 2025
  13. moved this from Testing to Done in Malcolmon Oct 13, 2025
  14. moved this from Done to Released in Malcolmon Nov 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinguploadRelating to PCAP and/or Zeek log ingestion

Type

Fields

Frequency

None yet

Projects

  • Status
    Released

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions