Repository navigation
Provide option for max retires, and a call back when reached (potential DLQ) #196
Description
Atm, user has to implement the max retry functionality themselves, as documented here: retries docs branch
This should be part of the system, as an option for a max retry count. When the count is reached, a callback should be executed, providing the user the WorkContainer and the chance to do something with it - maybe produce it do a different topic (DLQ) or simply log it.
Reacted by Clément MATHIEU, Daniele Ascione, Jakob Fri and Ruslan Taghiyev
Activity
sarwarbhuiyan commentedon Feb 15, 2022
Following up on the conversation we were having on this issue, there would be an approach here so as not to require the lambda signature to change to pass in WorkContainer vs ConsumerRecord.
Since the ConsumerRecord has a headers() object, we can increment a retry count as a known header in that in the framework. That way, the user could have access to the retryCount from the lambda and skip the exception (we could just do a docs section on that) if they want. Or the framework could compare a configured max_retries and check the retry_count header and trap out of continuing to process that record.
This is very similar to how Spring utilizes ConsumerRecord headers to pass a serialization exception so as to be able to handle a poison pill without causing an exception to be thrown.
astubbs commentedon Feb 15, 2022
Draft implementation of this in #197 - user can easily implement max retries and DLQ upon exceeded
astubbs commentedon May 17, 2022
FYI see
PollContextobject passed in to lambda to get failure counts of records.astubbs commentedon Jul 27, 2022
DLQ tracked here:
astubbs commentedon Aug 5, 2026
Tracked in the maintained fork as astubbs#141, which carries a summary of this
issue and its current status there.
The Confluent-hosted version of this project is no longer maintained - its README says so.
Parallel Consumer itself continues in my fork: https://github.com/astubbs/parallel-consumer
Nothing in this repository is closed or changed by this comment.
docs(inflight): record the poisoned transaction that is never aborted…