Repository navigation
Parallel Consumer keeps committing the old offset after OFFSET_OUT_OF_RANGE in auto.offset.reset = latest #409
Description
Hi Antony Stubbs (@astubbs),
Thanks for this great library. Recently, I have been consistently seeing the consumer lag on some partitions of the topic when offset become out of range for that partition. This happens even with latest PC version - 0.5.2.2. I'm using KEY ordering.
This might be related to #352. Sample logs below.
- Last committed offset is 86321 and it's no longer available on broker due to retention policy.
- And it gets reset to latest offset - 88617
12-09-2022 11:48:42.904 TraceId/SpanId: / [pc-broker-poll] INFO org.apache.kafka.clients.consumer.internals.Fetcher.handleOffsetOutOfRange - [Consumer clientId=my-topic-consumer-client, groupId=my-topic-consumer] Fetch position FetchPosition{offset=86321, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[localhost:6668 (id: 1 rack: null)], epoch=596}} is out of range for partition my-topic-4, resetting offset 12-09-2022 11:48:43.148 TraceId/SpanId: / [pc-broker-poll] INFO org.apache.kafka.clients.consumer.internals.SubscriptionState.maybeSeekUnvalidated - [Consumer clientId=my-topic-consumer-client, groupId=my-topic-consumer] Resetting offset for partition my-topic-4 to position FetchPosition{offset=88617, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[localhost:6668 (id: 1 rack: null)], epoch=596}}. ..
New message at offset 88618 get consumed by the consumer
12-09-2022 11:49:32.550 TraceId/SpanId: c65c59c9b8c75d88/b26a1c4eea95894d [pc-pool-3-thread-1] INFO my.test.kafka.consumer.MyConsumer.consume - event=PROCESSING_COMPLETED, Event details: [id=cfd1d7e0-324d-11ed-8e3e-99871a3f782d, name=Event started, partition=4, offset=88618] 12-09-2022 11:49:32.550 TraceId/SpanId: a49e9995c6147711/d466506b2f29b224 [pc-pool-3-thread-1] INFO my.test.kafka.consumer.MyConsumer.consume - event=PROCESSING_COMPLETED, Event details: [id=cbabd4e0-324d-11ed-8e3e-99871a3f782d, name=Event completed, partition=4, offset=88617] 12-09-2022 11:49:32.781 TraceId/SpanId: c2a3cfd96ea76036/37104cfb0345e5b1 [pc-pool-3-thread-3] INFO my.test.kafka.consumer.MyConsumer.consume - event=PROCESSING_COMPLETED, Event details: [id=cff676e0-324d-11ed-8e3e-99871a3f782d, name=Other event created, partition=4, offset=88619] ..
But, old offset 86321 was being repeatedly committed. Hence, the consumer lag happens and it's continuously growing.
12-09-2022 11:49:35.552 TraceId/SpanId: / [pc-broker-poll] DEBUG io.confluent.parallelconsumer.internal.AbstractOffsetCommitter.retrieveOffsetsAndCommit - Commit starting - find completed work to commit offsets 12-09-2022 11:49:35.552 TraceId/SpanId: / [pc-broker-poll] DEBUG io.confluent.parallelconsumer.internal.AbstractOffsetCommitter.retrieveOffsetsAndCommit - Will commit offsets for 5 partition(s): {my-topic-4=OffsetAndMetadata{offset=86321, leaderEpoch=null, metadata='bgAKAAMACgACAAsBDgACB8Y='}, my-topic-3=OffsetAndMetadata{offset=89148, leaderEpoch=null, metadata='bAAJfgEA'}, my-topic-2=OffsetAndMetadata{offset=88725, leaderEpoch=null, metadata='bAATPIAGAA=='}, my-topic-1=OffsetAndMetadata{offset=107886, leaderEpoch=null, metadata='bAAGIAA='}, my-topic-0=OffsetAndMetadata{offset=89200, leaderEpoch=null, metadata='bgALAAE='}} 12-09-2022 11:49:35.552 TraceId/SpanId: / [pc-broker-poll] DEBUG io.confluent.parallelconsumer.internal.AbstractOffsetCommitter.retrieveOffsetsAndCommit - Begin commit ..
- Is it possible that PC is not aware of "auto.offset.reset=latest" event and highest committable offset somehow remains 86321 and that's why it gets committed with every commit loop?
- Is it possible for PC to be able to recover from this situation just like normal consumer does and be aware of "auto.offset.reset=latest" and commit the offsets of new messages as it consumes, avoiding consumer lag.
- Under normal circumstances, are there any possible exceptional scenarios where we can run into consumer lag and PC is not able to recover from it hence the lag continue to grow? E.g. what happens if it somehow failed to commit an offset, how can PC recover from this without restarting the consumer?
Activity
astubbs commentedon Sep 15, 2022
Yup, you got it in one! Nice find! The WorkManager needs to be informed from the consumer of the issue, and update the state accordingly. Want to take a stab at a PR?
Is your work around for this, restating PC?
PC will keep trying to do the commit every commit frequency - i.e. it will retry. I'm not aware of any state the underlaying consumer could get into, which would prevent a retry from happening.
However, as you've identified - if the partition's state has changed on the broker (through retention or compaction) - WorkManager must be made aware.
leonoah86 commentedon Sep 18, 2022
Hi Antony Stubbs (@astubbs), thanks for looking into this and confirming it's a verified bug on "offset out of range" scenarios.
As for PR, I don't feel like I'm ready for it at this point. Will definitely try later if I can.
No, restarting PC didn't help at all. I had to manually reset the offset on topic to clear the CG lag. The reason why I asked that specific question was that I could be wrong, but have a suspicion that CG lag might have started happening before "offset out of range" issue comes in.
Could you help confirm if my understanding below is correct?
Also, please help me understand how PC would behave in below hypothetical scenario. Please assume, they all belong to same key and we are using KEY ordering.
1,2,3,4(unknown),5,6,7,8,9,10,..N,...Where,
Do you think this is a valid scenario. If yes, how will PC recover from such situation without causing the CG lag?
Can I expect that the fix for this verified bug will also take care of such consumer problem?
Please share if you think there's any other possible edge case scenarios where highest committable offset somehow stuck or remains the same.
leonoah86 commentedon Sep 27, 2022
Hi Antony Stubbs (@astubbs), appreciate if you could share ETA on when the fix for this bug could be released and if you could answer some of my questions in previous comment? This will really help with my investigation on future CG lag issues. Thanks!
astubbs commentedon Sep 27, 2022
Hi again! Just finished prepping for Confluent's Current conference next week, then there's these issues in the merge queue for this week:
Then I'll look at patching this, I consider this the next highest priority for PC.
astubbs commentedon Sep 30, 2022
Can you help me understand in your example - is record 86321 effectively disappearing underneath PC, so it's unable to successfully complete processing of it?
START: fixes #409: Truncate state on offset high reset
astubbs commentedon Sep 30, 2022
Hi leonoah86 , take a look at the direction I'm heading in:
I thought this wold only need to be done on startup, but it probably needs to be checked continuously. Take a look and let me know if you think that will cover your situations.
astubbs commentedon Sep 30, 2022
Yeah that's right - it can continue, because it's just processing what the broker sends it, there's no problem there.
Yes, it skips what's decoded from the metadata as completed previously. On startup, it loads the metadata - the metadata isn't kept in sync with what's on the partition by the broker.
The offset reset is done locally by the client, not on the broker. Broker just responds with the >= offset from the committed offset.
yeah...
PC will just continue retrying 4 until either it succeeds, or retries expires or user function skips it. See the offset storage FAQ part of the readme.
Other keys in the partition will continue to be processed in parallel.
Yes, by default every second. It's configurable.
Only records with different keys.
yes
as far as I can tell, the bug report us unrelated to the scenario you describe. Please clarify if I got that wrong..
leonoah86 commentedon Oct 1, 2022
Hi Antony Stubbs (@astubbs), thanks for your response and looking into this.
No, it's available when it was being processed and later it was deleted by retention policy.
To simulate in my local, I used a topic with smallest possible retention time and size. Started the consumer which is slow and intermittently trying to interrupt the consume thread. And then I kept pumping the messages faster than what consumer can process. After a while, I started another consumer to trigger the rebalance. Then this "off out of range" error started showing up.
I'm still trying to figure out what caused the CG lag before we eventually ran into "offset out of range" scenario. I suspect it's the scenario where consumer thread somehow got interrupted and caused the highest committable offset stuck. We had the similar issue in 2 other different environments. I have seen evidences in Splunk, where processing of some records in the partition in question got interrupted (i.e. didn't print "PROCESSING_COMPELTED") and were not retried.
Please note that my consumer implementation skips the record on failure. So, retry may not be applicable for scenario. Can you confirm if PC guarantee that in scenarios where consumer thread got interrupted, such records will remain in the queue and will be retried later?
astubbs commentedon Oct 1, 2022
Yes unless there's explicit success, things will always get retried.
If you have the time - writing an integration test with the scenario you describe would be awesome. But I think I've figured it out. Will try finish the draft on the flight to Austin today for Current :)
astubbs commentedon Oct 1, 2022
Regarding retries - try with the version released yesterday 0.5.2.3 - there were some fixes there related to retrying.
leonoah86 commentedon Oct 2, 2022
Thanks Antony Stubbs (@astubbs). Will try to replicate the scenario I described using latest version and update.
Regarding fixes for truncating state on offset high reset, with normal Kafka consumer, sometimes, we manually reset the offset of CG to an earlier offset to force replay and reprocess old messages.
3 remaining items
astubbs commentedon Oct 4, 2022
I haven't tested it yet, but if you want to try it - you can check out that branch and give it a go.
leonoah86 commentedon Oct 4, 2022
Not really, in the scenario where thread invoking user function get stuck or interrupted, I don't think there's anything we can do from user function. I think PC could do something about it. Right now, it prints the warning message if the messages remain in queue for more than 10s. I'm wondering if PC should invoke the user function with timeout and either retry or drop the message from queue (skip), but not everyone may want to retry or skip, configurable behavior? if it's taking longer than a certain threshold so that the lag doesn't happen when this happened. What do you think?
In normal kafka client, we have max.poll.interval.ms which is 5 minutes by default. If the processing of the messages in a batch take longer than 5 minutes, rebalance will occurs. Unfinished offsets will be reprocessed again on next poll. This is how it can recover from such scenario. So, I thought it would be nice if PC could do something similar with timeout.
Meanwhile, let me check out your branch and give it a go.
astubbs commentedon Oct 6, 2022
FYI that code as pushed def doesn't work - sorry if you spent some time on it. I have a tested branch now that's looking good. Just finishing some refactoring. Will push it up soon.
issue #409 captured in test and fixed
astubbs commentedon Oct 11, 2022
ok I got what you were saying the other week. That would be a completely separate issue / feature. It should also only /ever/ happen if your user function continues to fail processing. In which case the user should choose to skip it or not - not PC. Anything else is a bug in PC, which should be fixed. (i.e. this).
I guess you could argue the feature could be a "fail safe" for bugs in PC. But you could also do the same by monitoring consumer lag like any consumer application, and if lag gets too big - try kill the app.
If you want to pursue this feature - please open a new issue and we can discuss it there, I'd like to keep this one focused.
I've fixed the issue, and am pushing up the clean brach today.
fixes #409: Adds support for compacted topics and commit offset reset…
astubbs commentedon Oct 12, 2022
Hi leonoah86 this is merged now. Can you please test the snapshot version?
leonoah86 commentedon Oct 13, 2022
Hi Antony Stubbs (@astubbs), thanks for your responses and working on the fix.
yes. I will open a new issue.
Currently, I'm a bit busy with BAU works. Will test it in a few days and update.
refactor #409: Clarify truncation code
refactor #409: Clarify truncation code - tests passing
refactor #409: Clarify truncation code - tests passing
refactor #409: Clarify truncation and reset code (#445)
leonoah86 commentedon Nov 7, 2022
Hi Antony Stubbs (@astubbs), sorry for the delay. I ran into some issue building the snapshot version behind proxy, using latest master branch and downloading the latest snapshot version into company workspace where I have test setup and everything.
Could you release the new version if you are good with the changes? Even if I do test, it will just be a random test specific to my use case. I can always file a new bug against the release version or update here if it's not working as expected.
leonoah86 commentedon Nov 7, 2022
Please ignore. I just saw you have already released the new version 4 days ago. Will try that out. Thanks!