Repository navigation
Keep Filebeat cleanup locks on a stable inode - #1230
Closed
sylvesterkaczmarek wants to merge 3 commits into
Closed
sylvesterkaczmarek wants to merge 3 commits into
sylvesterkaczmarek wants to merge 3 commits into
Conversation
Collaborator
|
Thanks, but this is a duplicate of #1160, which fixed the same lock-unlink race in the same way (append mode, no This diff is based on an older version of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Filebeat's cleanup job uses flock to prevent concurrent pruning, but the
entrypoint unlinks the lock file in a finally block. That also executes when
the lock cannot be acquired. A waiting process can therefore remove the path
while another process still holds the old inode, allowing a third instance
to create a new file and run cleanup concurrently.
Fix
Keep the lock file at a stable path and inode. Open it in append mode,
attempt the existing nonblocking exclusive flock, and rely on closing the
file descriptor to release the lock. Never unlink the lock file on success,
contention, or cleanup failure.
The file is intentionally persistent. flock releases automatically when the
process exits, including after a crash, so this does not leave a stale lock.
Tests
Four regression tests cover lock contention, repeated invocations using the
same inode, exceptions and retries, and nested concurrent cleanup attempts.
All four failed against unmodified main (three assertion failures and one
error). All four pass with this change.
passed.
Tests extract and execute the actual production entrypoint with a temporary
lock path, so they do not require Malcolm's container-only dependencies or
delete real log files. End-to-end Filebeat ingestion was not run.