Repository navigation
Question regarding the key ordering #415
Description
For key ordering scenario demonstrated in 10.4. Ordered by Key, if 5E in partition one can never be processed successfully, does it cause the committed offset to stick to 5E and eventually exceed the OffsetMap size limit?
Activity
astubbs commentedon Sep 23, 2022
That's correct - the committed offset would "stick" to 5, with the metadata payload containing all the per msg ack's beyond 5.
No, as if the payload size hits 75% or more of the limit (4kB), the back pressure system kicks in, and no more records will be taken for processing, until it drops below 75% again. Instead it will keep retrying existing records.
However, if the only record to continually fail is 5, and all others succeed, let's say offset 6-50,000, then the meta data payload is only ~2 shorts (1 and (50,000-6=) 49,994), as it will use run length encoding. So it's very efficient.
Does that answer your question?
This is described a bit in the readme, under 0.3 features: https://github.com/confluentinc/parallel-consumer#features-3
And lightly under https://github.com/confluentinc/parallel-consumer#storage-notes , and in the talk. However - I will make this more apparent in the readme - thanks for asking!