Skip to content

Keep Filebeat cleanup locks on a stable inode - #1230

Closed
sylvesterkaczmarek wants to merge 3 commits into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/filebeat-preserve-persistent-cleanup-lock-20261013
Closed

sylvesterkaczmarek wants to merge 3 commits into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/filebeat-preserve-persistent-cleanup-lock-20261013

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor

Problem

Filebeat's cleanup job uses flock to prevent concurrent pruning, but the
entrypoint unlinks the lock file in a finally block. That also executes when
the lock cannot be acquired. A waiting process can therefore remove the path
while another process still holds the old inode, allowing a third instance
to create a new file and run cleanup concurrently.

Fix

Keep the lock file at a stable path and inode. Open it in append mode,
attempt the existing nonblocking exclusive flock, and rely on closing the
file descriptor to release the lock. Never unlink the lock file on success,
contention, or cleanup failure.

The file is intentionally persistent. flock releases automatically when the
process exits, including after a crash, so this does not leave a stale lock.

Tests

Four regression tests cover lock contention, repeated invocations using the
same inode, exceptions and retries, and nested concurrent cleanup attempts.

All four failed against unmodified main (three assertion failures and one
error). All four pass with this change.

  • Python unittest discovery: 4 passed.
  • Ruff selected correctness rules, Python compilation and git diff --check
    passed.

Tests extract and execute the actual production entrypoint with a temporary
lock path, so they do not require Malcolm's container-only dependencies or
delete real log files. End-to-end Filebeat ingestion was not run.

@mmguero
mmguero changed the base branch from main to contributor-prs October 13, 2026 15:53
@mmguero

mmguero commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

Thanks, but this is a duplicate of #1160, which fixed the same lock-unlink race in the same way (append mode, no finally unlink, lock released when the descriptor closes) and has already been merged to contributor-prs along with its test in shared/tests/test_filebeat_cleanup_lock.py.

This diff is based on an older version of clean-processed-folder.py (index 2ce8a8994), from before that fix landed. Before opening new PRs, please rebase on the current contributor-prs and check for existing PRs against the same file or issue. That would have caught this one and #1186.

@mmguero mmguero closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants