Skip to content

Simplify filesystem watcher event tracking and preserve queue semantics - #1250

Open
sylvesterkaczmarek wants to merge 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/765-refactor-watchdog-file-event-tracking-20261016
Open

sylvesterkaczmarek wants to merge 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/765-refactor-watchdog-file-event-tracking-20261016

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Maintenance update on 6 November 2026

Refreshed onto contributor-prs at 7046399b after the integration branch history changed. This PR now contains its 2-file topic change in one commit.

python -m pytest -q shared/tests/test_watch_common_file_events.py shared/tests/test_watch_common_read_events.py shared/tests/test_watch_moved_paths.py: 40 passed. Python compilation and Ruff pass. Git whitespace checks pass.

Preserved the newer upstream read-only-close cancellation and separate accounting for overlapping opens. Destinationless moves now retire tracked/staged source entries; four new regressions failed before the correction and pass afterward. Assertions in the topic tests now run outside dictionary contexts that could suppress assertion failures. No live Malcolm deployment was run.

Earlier validation counts and complexity measurements below record the original branch state. No fresh Radon score is claimed. ChatGPT assisted with this refresh, corrections and validation.


Description

Refactor the filesystem watcher event handler in shared/bin/watch_common.py to separate event history coalescing, staging/promotion, priority ordering, deletion, and move handling from event receipt. The original FileOperationEventHandler.on_any_event scored E (40) in Radon; the revised handler scores B (6). All extracted helpers score A or B. This reduces complexity without altering how watchers schedule completed PCAP and other upload files.

  • Preserve metadata-only modification suppression so unchanged file sizes don't delay processing.
  • Keep an initial opened or modified event staged until a real content change or a close/create/move event promotes it.
  • Preserve immediate processing priority for close events, consecutive event coalescing, deletion cleanup, and move handling across watched subdirectories.
  • Ignore moved events with no destination instead of inserting a non-path queue key; continue handling disappearing/inaccessible files without taking down the observer.
  • Remove unused imports and add focused regression coverage using real watchdog event classes and synthetic local files.

Related issue(s)

Refs #765, specifically the outstanding high-complexity FileOperationEventHandler.on_any_event entry. This PR does not complete the broader Python complexity review.

Type of change

  • Bug fix (missing moved-event destination handling)
  • New feature or enhancement
  • Documentation
  • Other: maintainability refactor

How has this been tested?

python -m pytest -q shared/tests/test_watch_common_file_events.py
python -m pytest -q shared/tests
python -m radon cc -s shared/bin/watch_common.py
python -m ruff check --select E4,E7,E9,F shared/bin/watch_common.py shared/tests/test_watch_common_file_events.py

16 focused tests passed on file create, open, modify, close, delete, and moved-event paths (including cross-directory moves, staging, unchanged sizes, coalescing, and missing destinations). The full shared/tests suite passed: 647 tests passed, 6 skipped, 31 subtests passed with Python 3.12 and current upstream test dependencies. Python compilation, targeted Ruff checks and git diff --check passed. The functionality was validated offline with real watchdog event classes and temporary files; no live Malcolm/Zeek deployment was exercised.

Documentation

  • No user-facing options change; internal helper docstrings updated.

Was AI used in writing the code, tests, or description for this pull request?

  • No
  • Yes (please describe below)

ChatGPT assisted with the refactor, the regression tests and the PR description. The code was reviewed against the existing watcher semantics, and the stated offline regression and static checks were executed.

Additional context

Based on the current contributor-prs branch. Independent of existing open PRs and does not introduce dependencies.

@mmguero mmguero self-assigned this Oct 19, 2026
@mmguero mmguero added this to Malcolm Oct 19, 2026
@mmguero mmguero moved this to Review in Malcolm Oct 19, 2026
@mmguero mmguero added the upload Relating to PCAP and/or Zeek log ingestion label Oct 19, 2026
Refresh the focused change onto contributor-prs at 7046399.

`python -m pytest -q shared/tests/test_watch_common_file_events.py shared/tests/test_watch_common_read_events.py shared/tests/test_watch_moved_paths.py`: **40 passed**. Python compilation and Ruff pass.

Preserved the newer upstream read-only-close cancellation and separate accounting for overlapping opens. Destinationless moves now retire tracked/staged source entries; four new regressions failed before the correction and pass afterward. Assertions in the topic tests now run outside dictionary contexts that could suppress assertion failures. No live Malcolm deployment was run.
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/765-refactor-watchdog-file-event-tracking-20261016 branch from 1c2679d to e432af5 Compare November 6, 2026 07:48
@mmguero mmguero added this to the z.staging milestone Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

upload Relating to PCAP and/or Zeek log ingestion

Projects

Status: Review

Development

Successfully merging this pull request may close these issues.

2 participants