Repository navigation
Keep PCAP duplicate checks enabled when startup health waiting is disabled - #1146
Conversation
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
|
Thanks for this. The diagnosis is right: with One thing needs handling before this goes in. In Before this change, that gap only mattered with waiting enabled, and only if the cluster went bad after startup, since startup waited for the files index to reach yellow. With waiting disabled, the lookup now runs as soon as the cluster answers So the change trades republishing duplicates for losing captures during warm-up, and losing a capture is the worse outcome. I'd suggest failing open: if the lookup fails, log it and publish anyway, which is what happened before this PR. fileIsDuplicate = False
if self.useOpenSearch:
try:
s = (
SearchClass(using=self.openSearchClient, index=ARKIME_FILES_INDEX)
.filter("regexp", node=fr"{args.nodeName}(-upload)?")
.query("wildcard", name=f"*{os.path.sep}{relativePath}")
)
response = s.execute()
for hit in response:
fileInfo = hit.to_dict()
if (ARKIME_FILE_SIZE_FIELD in fileInfo) and (fileInfo[ARKIME_FILE_SIZE_FIELD] == fileSize):
fileIsDuplicate = True
break
except Exception as e:
# an unreachable cluster or a missing files index shouldn't cost us the capture
self.logger.warning(
f"{scriptName}:\t{args.opensearchMode} duplicate check failed for {pathname}, processing it anyway: {e}"
)Your fixture covers this with one more test. It fails on @pytest.mark.parametrize('wait', [False, True])
def test_failed_duplicate_lookup_still_publishes(watcher_factory, pcap_path, wait):
"""A lookup error (cluster not ready, files index missing) must not drop the capture."""
state = watcher_factory(wait=wait)
state.query.execute.side_effect = ConnectionError('index not ready')
state.watcher.processFile(str(pcap_path))
state.publisher.send_string.assert_called_once()It's parametrized over both wait modes because the gap also exists with waiting enabled, if the cluster degrades after startup. Could you add both to this PR? |
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
|
Thanks @mmguero, appreciate the review and merge! |
Summary
Fixes #1145.
Enable the existing duplicate lookup after a successful database connection when the optional startup health wait is disabled. Previously the skipped loop left
healthy=False, permanently disabling the lookup and allowing matching captures to be republished.The normal health-waiting path, no-database behavior, size matching and publisher payload are unchanged. This is separate from the node-name matching issue in #588 and the current threat-feed PRs.
Testing
python -m pytest -q shared/tests/test_pcap_duplicate_checks.py11 tests passed. Five fail against unchanged main and six controls pass. Coverage includes both wait settings, duplicate suppression, new captures, size mismatches, unavailable/unconfigured database paths and invalid files. PCAP fixtures are small generated headers recognized by the real libmagic library. Database clients, query responses and ZeroMQ contexts are mocked; no network listener or external service is opened.
Tested on macOS with Python 3.12. A checksum-verified Homebrew libmagic bottle was unpacked into the local test workspace only. No dependencies, system configuration, running Malcolm deployment, Zeek/Arkime process, workflow or container image was changed. Full deployment testing was not run.
New-test formatting, scoped syntax/lint checks and
git diff --checkpass. No dependency or workflow changes.Thanks for taking a look.
Review follow-up
Duplicate lookup now fails open on an unavailable cluster, missing index or lookup/iteration error. The capture is published and a warning is logged; later successful lookups still suppress duplicates. Eight new cases cover both startup-wait modes, three failure types and subsequent recovery. All 19 targeted tests pass, compared with 8 failing and 11 passing before this correction. Black and
git diff --checkpass. Scoped Flake8 reports the same 13 pre-existing unused-global diagnostics before and after; no new findings were introduced. All database/query/publisher boundaries remain mocked, with real PCAP file recognition. No capture, cloud endpoint or deployment was contacted.