server: fix deadlock in load_models() when erasing a finished download - #25358
Conversation
The download monitoring thread acquires the models mutex on its way out, but load_models() joined it from the erase loop while holding that mutex. Join it outside the lock via threads_to_join like the other monitoring threads.
c333552 to
24b4814
Compare
|
it's a bit surprise that python script never trigger a timeout though, I'm still a bit skeptical if this is truly a problem in C++ code |
|
Python can't time out: Proof it hangs in C++: in the CI log, It's the erase loop joining the DOWNLOADED monitoring thread while holding the models mutex, while that thread waits on the same mutex to clean up stopping_models -> deadlock. Add a 1s sleep before that thread's final lock_guard and it reproduces 100%. |
|
He inserted a sleep to get a deterministic reproduction, and ggerganov had killed the C++ process to unblock the Python process. We should re-run the CI a few times to be sure: |
|
hmm ok that makes sense. probably |
A hung server now fails the test after 10 minutes instead of stalling the CI job for hours. Explicit timeouts are unchanged.
ggml-org#25358) * server: fix deadlock in load_models() when erasing a finished download The download monitoring thread acquires the models mutex on its way out, but load_models() joined it from the erase loop while holding that mutex. Join it outside the lock via threads_to_join like the other monitoring threads. * server: add default timeout to test requests A hung server now fails the test after 10 minutes instead of stalling the CI job for hours. Explicit timeouts are unchanged.
ggml-org#25358) * server: fix deadlock in load_models() when erasing a finished download The download monitoring thread acquires the models mutex on its way out, but load_models() joined it from the erase loop while holding that mutex. Join it outside the lock via threads_to_join like the other monitoring threads. * server: add default timeout to test requests A hung server now fails the test after 10 minutes instead of stalling the CI job for hours. Explicit timeouts are unchanged.
2 participants
Overview
Fix models download deadlock / test on real CI
Additional information
Requirements