Skip to content

Add bounded PCAP-over-IP client ingestion - #1256

Open
sylvesterkaczmarek wants to merge 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:feat/464-pcap-over-ip-client-20261016
Open

sylvesterkaczmarek wants to merge 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:feat/464-pcap-over-ip-client-20261016

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Maintenance update on 6 November 2026

Refreshed onto contributor-prs at 7046399b after the integration branch history changed. This PR now contains only its three-file topic change in one commit.

Current validation: python -m pytest -q shared/tests/test_pcap_over_ip.py reports 21 passed, including real localhost TCP ingestion. Python compilation, focused Ruff error checks and git diff --check pass. A live Malcolm/PolarProxy deployment remains untested.

The earlier validation counts below record the original branch state. ChatGPT assisted with this refresh and validation.


Summary

Refs #464.

Add an opt-in PCAP-over-IP client that imports a complete, EOF-terminated classic PCAP stream from a remote TCP producer into Malcolm's existing pcap/upload pipeline.

# On the producer:
nc -l 57012 < sniffed.pcap

# On the Malcolm host:
python3 scripts/pcap_over_ip.py --host 127.0.0.1 --port 57012
  • Outbound client connection only; no new listener, container or exposed port.
  • Verify PCAP magic, endian/version, snaplen, each packet header, payload length and timestamp precision. Reject PCAPNG, malformed, truncated and empty captures.
  • Stream in 64 KiB chunks with a 1 GiB default size cap, configurable maximum and separate connection/idle timeouts.
  • Stage outside the watched upload directory; after a complete stream, publish with an atomic hard link and a collision-resistant name, never overwriting an upload or leaving a partial PCAP on failure.
  • Document the usage and the security caveat: plain PCAP-over-TCP is not encrypted or authenticated and should be used only over trusted networks or protected tunnels.

Validation

  • python -m pytest -q shared/tests/test_pcap_over_ip.py: 21 passed, including localhost TCP sender/subprocess coverage, chunked delivery, all four classic PCAP magic formats, truncated packets, invalid headers, size limits, collision handling and partial-file cleanup.
  • python -m pytest -q shared/tests: 652 passed, 6 skipped, 31 subtests passed.
  • Python compilation, Ruff, Black, git diff --check: passed.

This is a client-only, bounded, single-stream implementation of #464. Continuous stream rotation, server mode and container-wide real-time ingestion remain out of scope. A live Malcolm/PolarProxy deployment has not been tested; the real localhost TCP stream and atomic upload publication have been tested.

AI disclosure

ChatGPT assisted with the implementation, tests, documentation and PR text. Validation was run against the current contributor-prs branch.

@mmguero mmguero added this to Malcolm Oct 19, 2026
@mmguero mmguero moved this to Review in Malcolm Oct 19, 2026
@mmguero mmguero added the upload Relating to PCAP and/or Zeek log ingestion label Oct 19, 2026
Refresh the focused change onto the current contributor-prs history.
Validated the PCAP-over-IP regression suite (21 passed).
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the feat/464-pcap-over-ip-client-20261016 branch from 5c515be to 2fffa78 Compare November 6, 2026 07:38
@mmguero mmguero added this to the z.staging milestone Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

upload Relating to PCAP and/or Zeek log ingestion

Projects

Status: Review

Development

Successfully merging this pull request may close these issues.

2 participants