Repository navigation
getCompactionTaskCapacity is not worker category aware. #15847
Description
Description
The getCompactionTaskCapacity function which is used to ascertain that the Druid cluster has enough task slots before the coordinator schedules additional compaction tasks doesn't take into consideration overlord dynamic config. The overlord dynamic config can prevent compaction tasks from running on specific categories of workers. This way, the compaction task capacity is incorrectly overestimated.
For example, I have two worker categories, compaction-category with a total of 600 task slots and another ingestion-category with 2000 slots(high number because of multiple ingestion task replicas).
Using the overlord dynamic config,
-
compaction-categoryis configured to run the following task types,- kill
- compact
- single_phase_sub_task
- partial_dimension_cardinality
- partial_index_generate
- partial_index_generic_merge
-
ingestion-categoryis configured to run,- index_kafka
Now, getCompactionTaskCapacity would return 2600 as the total capacity, which is inaccurate since only 600 slots are actually available for compaction tasks. While this might not pose a problem in a healthy cluster, it becomes critical during compaction task failures. The oversight leads to the coordinator creating excessive compaction tasks, resulting in contention on compaction slots and slowing down all compaction tasks. This creates a feedback loop where the increasing number of compaction tasks exacerbates contention, ultimately overwhelming the overlord with too many tasks to handle
Affected Version
Saw this on Druid 25. Is also present in master.
Activity
m-ghazanfar commentedon Feb 16, 2024
My overlord dynamic config looks like this,
overlord dynamic config
{ "type": "equalDistributionWithCategorySpec", "workerCategorySpec": { "categoryMap": { "kill": { "defaultCategory": "abc_category", "categoryAffinity": {} }, "compact": { "defaultCategory": "abc_category", "categoryAffinity": {} }, "single_phase_sub_task": { "defaultCategory": "abc_category", "categoryAffinity": {} }, "partial_dimension_cardinality": { "defaultCategory": "abc_category", "categoryAffinity": {} }, "partial_index_generate": { "defaultCategory": "abc_category", "categoryAffinity": {} }, "partial_index_generic_merge": { "defaultCategory": "abc_category", "categoryAffinity": {} }, "partial_range_index_generate": { "defaultCategory": "abc_category", "categoryAffinity": {} }, "partial_dimension_distribution": { "defaultCategory": "abc_category", "categoryAffinity": {} }, "index_kafka": { "defaultCategory": "xxx_category", "categoryAffinity": { "datasource_2": "yyy_category", "datasource_3": "zzz_category" } } }, "strong": true } }m-ghazanfar commentedon Feb 16, 2024
We worked around this by explicitly setting the
maxCompactionTaskSlotsto the total available worker slots on the compaction tier.(Actually, we set it a little higher than that to improve middle manager utilisation)See Update capacity for compaction tasks
Example,