Repository navigation
Incomplete numbered shard specs partition when index task fails and retries #11322
Closed as not planned
Description
Affected Version
0.21.1
Description
Loaded a large dataset using the native parallel index. The load ran for 4 hrs, during that some tasks failed and retried.
Noticed that the numbered shards generated for the interval were not continuous and were missing in between overall max(partition_num) was 2289, however, the total shard count was 2263.
On a quick look it seems that on a subtask failure, it may happen that some partition numbers were allocated to a task and are never pushed, on a retry, the new task gets newer partition numbers.
The overall result was that the ingestion completed successfully with retries and the coordinator loaded the data.
The broker segment timeline was incomplete due to missing partition numbers and the data was never queryable.
Activity
abhishekagarwal87 commentedon Jun 2, 2021
It should get fixed by #11189
mr000al commentedon Jun 21, 2021
We're facing the same issue - has anyone found a temporary work around on this?
github-actions commentedon Nov 6, 2023
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
github-actions commentedon Dec 4, 2023
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.