Skip to content

Keep PCAP duplicate checks enabled when startup health waiting is disabled - #1146

Merged
mmguero merged 2 commits into
cisagov:sylvesterkaczmarek-mergesfrom
sylvesterkaczmarek:fix/pcap-dedup-without-health-wait-20261006
Oct 6, 2026
Merged

mmguero merged 2 commits into
cisagov:sylvesterkaczmarek-mergesfrom
sylvesterkaczmarek:fix/pcap-dedup-without-health-wait-20261006

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1145.

Enable the existing duplicate lookup after a successful database connection when the optional startup health wait is disabled. Previously the skipped loop left healthy=False, permanently disabling the lookup and allowing matching captures to be republished.

The normal health-waiting path, no-database behavior, size matching and publisher payload are unchanged. This is separate from the node-name matching issue in #588 and the current threat-feed PRs.

Testing

python -m pytest -q shared/tests/test_pcap_duplicate_checks.py

11 tests passed. Five fail against unchanged main and six controls pass. Coverage includes both wait settings, duplicate suppression, new captures, size mismatches, unavailable/unconfigured database paths and invalid files. PCAP fixtures are small generated headers recognized by the real libmagic library. Database clients, query responses and ZeroMQ contexts are mocked; no network listener or external service is opened.

Tested on macOS with Python 3.12. A checksum-verified Homebrew libmagic bottle was unpacked into the local test workspace only. No dependencies, system configuration, running Malcolm deployment, Zeek/Arkime process, workflow or container image was changed. Full deployment testing was not run.

New-test formatting, scoped syntax/lint checks and git diff --check pass. No dependency or workflow changes.

Thanks for taking a look.

Review follow-up

Duplicate lookup now fails open on an unavailable cluster, missing index or lookup/iteration error. The capture is published and a warning is logged; later successful lookups still suppress duplicates. Eight new cases cover both startup-wait modes, three failure types and subsequent recovery. All 19 targeted tests pass, compared with 8 failing and 11 passing before this correction. Black and git diff --check pass. Scoped Flake8 reports the same 13 pre-existing unused-global diagnostics before and after; no new findings were introduced. All database/query/publisher boundaries remain mocked, with real PCAP file recognition. No capture, cloud endpoint or deployment was contacted.

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@mmguero

mmguero commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this. The diagnosis is right: with --opensearch-wait false, healthy is never set, so the duplicate lookup never runs, and the fix restores it for that mode.

One thing needs handling before this goes in. In processFile, the lookup isn't protected: s.execute() runs outside any try, and the only except in that method is the zmq.Again around send_string. If the query raises, the exception leaves processFile before the file is published, and that capture is never processed.

Before this change, that gap only mattered with waiting enabled, and only if the cluster went bad after startup, since startup waited for the files index to reach yellow. With waiting disabled, the lookup now runs as soon as the cluster answers cluster.health(wait_for_status='red'), and that doesn't mean the Arkime files index exists yet. On a fresh or slow deployment, a capture that arrives before Arkime creates its index makes execute() raise an index-not-found error, and the capture is dropped. On main, that capture would have been published, because no lookup ran.

So the change trades republishing duplicates for losing captures during warm-up, and losing a capture is the worse outcome. I'd suggest failing open: if the lookup fails, log it and publish anyway, which is what happened before this PR.

                fileIsDuplicate = False
                if self.useOpenSearch:
                    try:
                        s = (
                            SearchClass(using=self.openSearchClient, index=ARKIME_FILES_INDEX)
                            .filter("regexp", node=fr"{args.nodeName}(-upload)?")
                            .query("wildcard", name=f"*{os.path.sep}{relativePath}")
                        )
                        response = s.execute()
                        for hit in response:
                            fileInfo = hit.to_dict()
                            if (ARKIME_FILE_SIZE_FIELD in fileInfo) and (fileInfo[ARKIME_FILE_SIZE_FIELD] == fileSize):
                                fileIsDuplicate = True
                                break
                    except Exception as e:
                        # an unreachable cluster or a missing files index shouldn't cost us the capture
                        self.logger.warning(
                            f"{scriptName}:\t{args.opensearchMode} duplicate check failed for {pathname}, processing it anyway: {e}"
                        )

Your fixture covers this with one more test. It fails on main and on this PR as it stands, and passes with the try:

@pytest.mark.parametrize('wait', [False, True])
def test_failed_duplicate_lookup_still_publishes(watcher_factory, pcap_path, wait):
    """A lookup error (cluster not ready, files index missing) must not drop the capture."""
    state = watcher_factory(wait=wait)
    state.query.execute.side_effect = ConnectionError('index not ready')
    state.watcher.processFile(str(pcap_path))
    state.publisher.send_string.assert_called_once()

It's parametrized over both wait modes because the gap also exists with waiting enabled, if the cluster degrades after startup. Could you add both to this PR?

@mmguero mmguero self-assigned this Oct 6, 2026
@mmguero mmguero added upload Relating to PCAP and/or Zeek log ingestion capture Relating to pcap-capture container labels Oct 6, 2026
@mmguero mmguero added this to the v26.10.0 milestone Oct 6, 2026
@mmguero
mmguero changed the base branch from main to sylvesterkaczmarek-merges October 6, 2026 21:15
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Thanks @mmguero. Fixed in b4b3f65: failed duplicate lookups now log a warning and publish the capture. Both wait modes and recovery after a failed lookup are covered; all 19 tests pass, with eight new cases failing before the fix.

@mmguero
mmguero merged commit 9a8b8fb into cisagov:sylvesterkaczmarek-merges Oct 6, 2026
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Thanks @mmguero, appreciate the review and merge!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capture Relating to pcap-capture container upload Relating to PCAP and/or Zeek log ingestion

Projects

Status: Released

Development

Successfully merging this pull request may close these issues.

Disabling the startup health wait disables PCAP duplicate detection

2 participants