This repository was archived by the owner on Jul 3, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Jul 3, 2023. It is now read-only.
Force pipeline termination from pipeline manager #397
Description
The runner handles the /shutdown command by first calling controller.stop() and then sending a termination message to the actix dispatch loop. Both operations can be slow or even get stuck in some cases. We will need a way to enforce eventual termination from the pipeline manager by killing the process/pod after a timeout.
Activity
lalithsuresh commentedon May 4, 2023
@ryzhyk I get the operations being slow, but do we know when/why they get stuck? Because that would be a bug.
ryzhyk commentedon May 4, 2023
controller.stop()waits for any in-progress iteration of the circuit to complete. Which should normally be fast, but can take a while if there's a perf issue in the circuit or we're processing a very large input batch. Another possible reason is buggy connectors that refuse to shutdown instantly or at all.