You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When restarting Malcom (./scripts/stop ./scripts/start) or rebooting the VM, Malcolm reimports all files in the pcap/processed folder and they appear double in the arkime - files view.
To reproduce: restart Malcolm. This behaviour appears everytime Malcolm restarts or the VM restarts.
I was not able to find an „expect behaviour“.
Malcolm version 24.03.0
I am running Malcolm by cloning the Git Repo on a Debian 12 VM airgapped
Hm, there's supposed to be a check against the arkime files database that prevents that from happening, I'll have to look into that. In the meantime, try this:
Yes, this is a workaround, but like I said it should have prevented that with the files database check I mentioned, so I'll leave this bug open to try to figure that out.
After some investigations, we found the root cause of the bug causing multiple imports after a Malcolm restart.
Reproduce the bug
Set the PCAP_PIPELINE_VERBOSITY to -vvv to get logs
Upload or copy a new PCAP file in upload directory, wait for analysis completion and insertion into Arkime's OS file indexes
touch the same PCAP file in processed directory, this triggers an filesystem event handled by pcap_watcher.py
The file is re-analyzed and not seen as duplicate
Root Cause
While processing a new file (in pcap_processor.py) arkimeCaptureFileWorker invokes Arkime with --node parameter with [nodeName]-upload when the capture file is uploaded/copied in upload directory :
# if this is an uploaded PCAP (not captured "live"")# append -upload to the node name used (which originates from PCAP_NODE_NAME)tmpNodeName=fileInfo[FILE_INFO_DICT_NODE] if (FILE_INFO_DICT_NODEinfileInfo) elsenodeNameiftmpNodeNameand (
(not (FILE_INFO_DICT_LIVEinfileInfo)) or (notfileInfo[FILE_INFO_DICT_LIVE])
):
tmpNodeName=tmpNodeName+'-upload'
The pcap-publisher (pcap-watcher.py) in pcap-monitor service checks if the filename is a duplicate in EventWatcher's processFile :
# check with Arkime's files index in OpenSearch and make sure it's not a duplicatefileIsDuplicate=Falseifself.useOpenSearch:
s= (
SearchClass(using=self.openSearchClient, index=ARKIME_FILES_INDEX)
.filter("term", node=args.nodeName) # [1]
.query("wildcard", name=f"*{os.path.sep}{relativePath}")
)
response=s.execute()
forhitinresponse:
fileInfo=hit.to_dict()
if (ARKIME_FILE_SIZE_FIELDinfileInfo) and (fileInfo[ARKIME_FILE_SIZE_FIELD] ==fileSize):
fileIsDuplicate=TruebreakiffileIsDuplicate:
# this is duplicate file (it's been processed before) so ignore itself.logger.info(f"{scriptName}:\t📋\t{pathname}")
The duplicate search always fails because the case where -upload is appened to nodeName is never checked in the query filter in [1].
Fix
A working fix is to change the filter term to regexp to match nodeName.* :
Activity
mmguero commentedon Feb 14, 2025
Hm, there's supposed to be a check against the arkime files database that prevents that from happening, I'll have to look into that. In the meantime, try this:
PCAP_PIPELINE_IGNORE_PREEXISTINGin config/upload-common.envEXTRACTED_FILE_IGNORE_EXISTINGin config/zeek.envSet those values to
trueand restart Malcolm and see how that goes.KchChr commentedon Feb 14, 2025
This worked. I assume this is a workaround?
Thank you for the fast answer.
Have a nice weekend!
mmguero commentedon Feb 14, 2025
Yes, this is a workaround, but like I said it should have prevented that with the files database check I mentioned, so I'll leave this bug open to try to figure that out.
dev-cti commentedon Oct 9, 2025
After some investigations, we found the root cause of the bug causing multiple imports after a Malcolm restart.
Reproduce the bug
PCAP_PIPELINE_VERBOSITYto-vvvto get logsuploaddirectory, wait for analysis completion and insertion into Arkime's OS file indexestouchthe same PCAP file inprocesseddirectory, this triggers an filesystem event handled bypcap_watcher.pyRoot Cause
While processing a new file (in
pcap_processor.py)arkimeCaptureFileWorkerinvokes Arkime with--nodeparameter with[nodeName]-uploadwhen the capture file is uploaded/copied inuploaddirectory :The
pcap-publisher(pcap-watcher.py) inpcap-monitorservice checks if the filename is a duplicate inEventWatcher'sprocessFile:The duplicate search always fails because the case where
-uploadis appened tonodeNameis never checked in the query filter in [1].Fix
A working fix is to change the filter term to regexp to match
nodeName.*:mmguero commentedon Oct 9, 2025
thanks for the thorough analysis, i will take a look at this.
cisagov#588, fix double-imports of PCAPs on Malcolm restart (thanks @…