Skip to content

Remove partial output when archive extraction fails - #1239

Merged
mmguero merged 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/remove-partial-failed-extractions-20261015
Oct 15, 2026
Merged

mmguero merged 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/remove-partial-failed-extractions-20261015

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor

Description
safe_extract removes its destination only for ArchiveBombError. Truncated compression streams, a bad ZIP checksum, or a missing source can leave an empty or partially populated directory even though extraction fails. The Filebeat caller proceeds to discover extracted .log files after the command, so partial output can be mistaken for completed input.

Clean up the newly created destination for ordinary extraction exceptions as well, and include MIME detection in the guarded operation. Directory creation remains outside the try block: a pre-existing destination is never removed. The original exception is re-raised and source archives are preserved. Successful extraction and configured archive-bomb limits keep their existing behavior.

Add libarchive-c==5.3 to shared test dependencies, matching the existing Filebeat runtime requirement. There is no new production dependency.

Related issue(s)
New reproducible regression found on current contributor-prs. No existing tracking issue or overlapping open PR was found. The existing relative-path and zstd extraction fixes on that branch are retained.

Type of change

  • Bug fix

How has this been tested?
python -m pytest -q shared/tests/test_safe_extract_failure_cleanup.py

10 cases pass on macOS and Python 3.12 using the actual command-line extractor, libarchive-c 5.3 and libmagic. Five cases fail on unchanged contributor-prs. Tests create real gzip, bzip2, xz, tar and ZIP fixtures in temporary directories and cover truncated streams, invalid ZIP CRC, a missing source, existing-destination preservation, size-limit cleanup, successful extraction, and unchanged source bytes.

A libmagic library was staged locally from an official Homebrew bottle for testing; it is not included in the PR. Black (120-column, skip string normalization), Ruff correctness checks, Python compilation and git diff --check pass. No full Filebeat container or live ingestion deployment was run.

Documentation

  • No documentation update is needed; extraction errors still propagate and there are no new user-facing options.

Was AI used in writing the code, tests, or description for this pull request?

  • Yes

OpenAI ChatGPT generated the patch, tests and this description. Validation included inspecting the small production diff, checking the current contributor branch and existing contributions, reproducing failures before the change, and running real archive extraction tests afterward. Maintainer review is still required.

@mmguero
mmguero merged commit 51086bb into cisagov:contributor-prs Oct 15, 2026
@mmguero mmguero self-assigned this Oct 15, 2026
@mmguero mmguero added the upload Relating to PCAP and/or Zeek log ingestion label Oct 15, 2026
@mmguero mmguero added this to the v26.10.0 milestone Oct 15, 2026
@mmguero mmguero added this to Malcolm Oct 15, 2026
@github-project-automation github-project-automation Bot moved this to Done in Malcolm Oct 15, 2026
mmguero added a commit to mmguero-dev/Malcolm that referenced this pull request Oct 15, 2026
filebeat-process-zeek-folder.sh ran the find over the extraction
directory regardless of safe-extract.py's exit status, so any logs
written before a failure got symlinked into current/ and ingested.
Check the exit status, log the failure to stderr, and exit the
per-file subshell so the rest of the batch continues.

Pairs with cisagov#1239, which removes partial output on failed extraction.

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
@mmguero mmguero mentioned this pull request Oct 29, 2026
@mmguero mmguero moved this from Done to Released in Malcolm Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

upload Relating to PCAP and/or Zeek log ingestion

Projects

Status: Released

Development

Successfully merging this pull request may close these issues.

2 participants