Repository navigation
confluentinc#203: Bug: ConsumerOffsetCommitter goes into failure state after broker downtime #248
Description
Mirror of confluentinc/parallel-consumer#203
Upstream title (verbatim, 2023-07-07):Bug: ConsumerOffsetCommitter goes into failure state after broker downtime
Opened by nioertel, 2022-03-01 · 3 comments upstream
Closed upstream 2023-07-07 as completed by an administrative sweep, while labelled
wait for info/cannot-reproduce. Not a resolution.
Summarised, not copied. Discussion belongs here - the Confluent-hosted repository is no longer maintained.
Summary
After a broker restart, ConsumerOffsetCommitter appears to retry offset commits endlessly without
success.
Reported setup: 4 JVMs consuming one 24-partition topic. Processing healthy, broker killed, records
already polled but not yet processed sitting in memory, broker restarted. Processing resumes, but
half the instances run at very low throughput and log continuously:
[ERROR] i.c.p.i.ConsumerOffsetCommitter - Error committing offsets
org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a
retriable exception. You should retry committing the latest consumed offsets.
Caused by: org.apache.kafka.common.errors.DisconnectException
The reporter's hypothesis: the instance is trying to process cached records from partitions that
have since been reassigned to another instance, so the commits cannot succeed.
Upstream discussion
Three diagnostic questions were asked - full stack trace; how long recovery was allowed after the
broker returned, the partition replication factor, and whether a new leader was elected while the
broker was down; and whether the instance was still processing and polling while retrying. None
were answered, hence wait for info / cannot-reproduce.
Fork status
Fixed and closed, shipped in v0.6.0.0. #470 closed the silent-loss path this report exposed: under PERIODIC_CONSUMER_ASYNCHRONOUS an offset is recorded as committed only when the broker acknowledges it, so a failed or dropped acknowledgement leaves it dirty to be re-sent. The throughput collapse after a broker restart did not reproduce against the current code, and the fork has since fixed a large amount in this area, so it is most likely fixed by one of those changes; a recurrence on 0.6.0.0 or later is a new issue. The comment below carries the account.
Suggested next step
This is the one closure in the sweep that was arguably defensible on the merits, since the reporter
went quiet - though "completed" is still the wrong reason for it. Before doing anything, it is worth
trying to reproduce, because the fork has the tool the upstream investigation was building:
Upstream PR #345 ("fixes #203: Broker
disconnect testing and ChaosBroker", branch astubbs/bugs/broker-disconnect, head bc620f3b8) was
closed unmerged in the 2023-06-15 sweep. Commit still reachable. That harness is what would settle
whether the async-commit path still misbehaves under a broker bounce.
Mirrored because the upstream issue was closed by an administrative sweep on 2023-07-07, not by a fix or a decision: all 28 issues closed that day carried the single comment "Closing Issue" and were marked completed. A separate sweep three weeks earlier (2023-06-15) closed 35 unmerged PRs with "Closing - Stale.", several of which implemented these issues. Mapping lives in src/docs/development/upstream-map.yaml.
Activity
fix(core) #248: an async commit counts when the broker answers, not w…
fix(core) #248: the partition remembers its commit offer WHOLE, so su…
fix(core) #248: a permanent async commit failure is an ERROR, because…
fix(core) astubbs#248: an async commit counts as committed when the b…
astubbs commentedon Sep 17, 2026
@nioertel - the fork's fix for what this report exposed has shipped, and the issue is closed with it.
Part of this is now fixed, in #470, and part of it is not. Both halves are worth stating, because the part that is fixed is not the part the original report was mainly about.
What is fixed: the async commit path no longer records an offset as committed at the moment the request is SENT.
The
todo keep work in limbo until async response is received?marker this issue quotes verbatim is deleted by that PR, and what stood behind it is now implemented. UnderPERIODIC_CONSUMER_ASYNCHRONOUS- the shipped default commit mode -commitAsyncreturns as soon as the request is handed to the client, and PC was marking the partition clean on the next line. The consequence was worse than a missing retry: marking clean ends the story, socollectCommitDataForDirtyPartitions()returned empty from then on and no furthercommitAsyncwas ever issued for those offsets. A callback then arriving with theRetriableCommitFailedException@nioertel reported had nothing left to retry - "retriable" described a retry nothing would perform - and a callback that never arrived was indistinguishable from success. The broker's committed offset stayed behind PC's belief, silently and permanently, so whoever owned the partition next resumed from a position the broker never recorded.Now an offset is recorded as committed only when the broker acknowledges it. A failed or dropped acknowledgement leaves the offsets dirty, and the next commit cycle re-sends them - the same handling the synchronous path's rejections already get: logged, not fatal, still owed. The failure line is one WARN saying exactly that, rather than the previous ERROR that promised a re-commit it could not always deliver.
The reported symptom itself.
The actual symptom @nioertel reported was a throughput collapse with endless retries after the broker restart. It did not reproduce against the current code, and the fork has since fixed a large amount in exactly this area - the commit path above, the rebalance-commit deadlock and the rest of the confluentinc#857 family, and the intake stall - so it is most likely fixed by one of them. What cannot be said is which.
The report is several years old, the code it was made against is gone, and the symptom does not reproduce, so this mirror is closed rather than held open for a reproduction nobody can run. If it recurs on 0.6.0.0 or later, a new issue with the consumer configuration and how long the retry storm lasted relative to the outage is the right place for it.
A note on this issue's history, since it may look stranger than it is: it was closed upstream as completed in a July 2023 administrative sweep while it was still labelled
wait for info, with three diagnostic questions unanswered. That closure carried no fix and no triage. This fork mirrors it because it was never actually resolved.Shipped in v0.6.0.0.