Repository navigation
Support retrying on failures for non-realtime ingestion #5428
Description
An indexing task might fail due to various reasons, and one of them is node failures. If an indexing task fails due to node failures, it might be completed by reissuing the failed task on another node. In this case, it would be better if the overlord can resubmit failed tasks automatically without manual handling.
I'm thinking to introduce a new taskContext, called retryCount, to specify how many times each task should be retried on failures.
Activity
nishantmonu51 commentedon Feb 27, 2018
[-]Support retrying on failures for batch ingestion[/-][+]Support retrying on failures for non-realtime ingestion[/+]jihoonson commentedon Feb 27, 2018
@nishantmonu51 thanks for your comment!
Good point. I'll add it.
I'm thinking to
retryCountto taskStatus, so users can check how many times a task failed before succeeding via overlord's taskStatus API (http://<OVERLORD_IP>:<port>/druid/indexer/v1/task/{taskId}/status). Does it make sense?Makes sense. I changed the title to
Support retrying on failures for non-realtime ingestionbecause supervisors or tranquilities are responsible for retrying in realtime ingestion.gianm commentedon Feb 27, 2018
I this this is a good direction to go -- it's nicer for Druid to take on retrying rather than to push the burden to users. However we need to try to do it right, and the main thing there imo is ensuring idempotency.
It is possible for tasks to have side effects and yet still fail (for example: it publishes and then immediately crashes before it can do final acts like uploading its log files). This means when retrying we have to verify that tasks are idempotent. Otherwise - we will potentially load the same data twice. With Kafka based ingestion, we are OK because of the metadata check (verifying offsets when segments are published). With batch indexing, in overwrite mode we are OK too (since overwriting is idempotent). But in append mode, extra work is needed.
jihoonson commentedon Feb 27, 2018
@gianm that's a good point. Maybe we need finer-granular task status tracking to avoid extra work.
gianm commentedon Feb 27, 2018
@jihoonson I don’t worry too much about extra work, as long as the task does the correct thing. What I do really worry about is the possibilty of appending the same data twice due to retrying after some data had been published.
jihoonson commentedon Feb 27, 2018
@gianm ah yeah, it's more important to make sure there is no duplicated work.
nishantmonu51 commentedon Mar 1, 2018
@jihoonson: Adding retryCount to the task status sounds good.
nishantmonu51 commentedon Mar 1, 2018
One way to achieve idempotency and avoid duplicate data could be to add a publisher field to the metadata storage when publishing segments.
For tasks it can be the taskID, when the task is retried, the retry attempts can check whether there are any segments already published with the same task id or not. This has following assumptions though -
github-actions commentedon Jun 29, 2023
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
github-actions commentedon Jul 28, 2023
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.