Repository navigation
[Proposal] : Better handling for *bad* worker nodes. #1615
Description
Problem :
In case we have a bad worker having environment issues e.g disk issues and all tasks being assigned to that worker starts failing, If configured with FillCapacityWorkerSelectStrategy, Overlord keeps assigning new tasks to the bad worker since it is the most likely the one with the least used capacity.
Solution :
Approach 1 : Add periodic sanity check to the workers.
- while starting up a worker will fork and run a sanity check task and announce itself if the task succeeds. If it fails it will retry it n (configurable) times and exits if the task fails repeatedly.
- worker will periodically run sanity checks on its own by forking a small test-task
- If the dummy task fails repeatedly, the worker disables itself and overlord will not assign any more tasks to this.
Approach 2 : Make overlord smart to identify faulty workers and blacklist them.
- maintain a count of failed tasks, and blacklist a worker when it exceeds a configurable threshold. the counter will need to be reset when a task succeeds on a worker.
- send a small NoOp dummy task to the worker before blacklisting it in order to distinguish between issues in task itself or worker environment.
- Optionally also verify all the newly added workers too by assigning a dummy task.
- Autoscaling should be able to replace the blacklisted worker.
I like Approach 1 and I think its better to have the worker do sanity checks on itself instead of overlord asking workers to do it.
Thoughts ? Suggestions?
Activity
[-]make overlord smart to identify faulty workers and blacklist them[/-][+][Proposal] : Better handling for *bad* worker nodes. [/+]drcrallen commentedon Aug 11, 2015
I agree with number 1, it fits in better with my Tiering proposal for tasks as well.
gianm commentedon Aug 11, 2015
Just to be different, I like 2 because it would catch more problems. A worker that is broken in some other way than just all tasks immediately failing won't be caught by method 1. But either one is better than nothing so I am not picky.
drcrallen commentedon Aug 11, 2015
IMHO the worker should be responsible for reporting its own status. And different types of workers would have different criteria for "healthy".
Obviously the overlord should have some level of "Can't even reach the middle manager" case handling.
himanshug commentedon Aug 13, 2015
I like both. But, I would go for 1 because it is more self-contained within the worker and simpler to implement. Also, It will be great to increase the scope of 1 a little bit to track count of consecutive failures (for any tasks assigned to it and not just the sanity task, and disable itself if there were too many consecutive failures.
In future, We can evolve towards 2 if/when we notice cases not caught by 1.
vogievetsky commentedon Jun 9, 2019
What is the status of this within the upcoming 0.15.0 release?
himanshug commentedon Jun 16, 2019
sort of handled by #3643 and subsequent related changes.
github-actions commentedon May 15, 2023
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
github-actions commentedon Jun 12, 2023
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.