Repository navigation
Parallel consumer doesn't process any more records due to incorrect management of work #547
Description
Parallel Consumer threads pauses polling messages from topic/partitions(s) and hence doesn't invoke user function, due to the "leak" in the counter of non-existent work when partition state changes. (Rebalancing, offline etc).
Confluent library version: 0.5.2.4
Possible (Observed) root cause -
The problem appears to be stemming from not cleaning up the variable WorkManager.java#numberRecordsOutForProcessing, when the partition is revoked (partition epoch changes). The effect is gradual, until the numberOfRecordsOutForProcessing hits the Max value formulated via ParallelConsumerOptions.
Due to the above, the consumer stops polling/processing messages which causes consumer lag to build up.
Temporary Work Around: Restarting the process.
Activity
Fixes confluentinc#547 - Cleanup WorkManager#numberRecordsOutForProce…
[-]Parallel consumer doesn't process more records due to incorrect management of work[/-][+]Parallel consumer doesn't process any more records due to incorrect management of work[/+]RaviKalasapur commentedon Feb 14, 2023
My honest attempt to add a test case which fails and a possible fix too. :)
bscully-ot commentedon Feb 14, 2023
cc Roman Kolesnev (@rkolesnev)
Fixes confluentinc#547 - Cleanup WorkManager#numberRecordsOutForProce…
Fixes confluentinc#547 - Cleanup WorkManager#numberRecordsOutForProce…
Fixes confluentinc#547 - Cleanup WorkManager#numberRecordsOutForProce…
Fixes confluentinc#547 - Cleanup WorkManager#numberRecordsOutForProce…
Fixes confluentinc#547 - Cleanup WorkManager#numberRecordsOutForProce…
fix(core) #155: stop the "Max loading factor steps reached" WARN on e…
fix(core) #155: stop the "Max loading factor steps reached" WARN on e…