confluentinc#40: Feature: Transactional mode for individual consume producer loop instead of periodic batch #232

Mirror of confluentinc/parallel-consumer#40
Upstream title (verbatim, 2023-07-07): Feature: Transactional mode for individual consume producer loop instead of periodic batch
Opened by astubbs, 2020-11-30 · no upstream discussion
Closed upstream 2023-07-07 as completed by an administrative sweep. Not a resolution.
Summarised, not copied. Discussion belongs here - the Confluent-hosted repository is no longer maintained.

Summary

Transactional mode currently commits in batches of records on a periodic cycle. Add a mode that
opens and commits a transaction per individual consume-produce flow, for users who want finer-grained
transactional boundaries and will pay the throughput cost to get them.

Fork status

Not implemented - confirmed by reading the code. ParallelConsumerOptions.CommitMode offers
exactly three values, all periodic:

• PERIODIC_TRANSACTIONAL_PRODUCER
• PERIODIC_CONSUMER_SYNC
• PERIODIC_CONSUMER_ASYNCHRONOUS

There is no per-record transactional mode.

Worth a hard look at cost before building: a Kafka transaction per record means a beginTransaction
and commitTransaction round trip per record, which collapses PC's throughput advantage to roughly
that of a single-threaded consumer. It may still be the right option for low-volume,
high-value streams, but it should be documented as the slowest mode by a wide margin, and it
interacts directly with the produce lock in ProducerManager.

_{Mirrored because the upstream issue was closed by an administrative sweep on 2023-07-07, not by a fix or a decision: all 28 issues closed that day carried the single comment "Closing Issue" and were marked completed. A separate sweep three weeks earlier (2023-06-15) closed 35 unmerged PRs with "Closing - Stale.", several of which implemented these issues. Mapping lives in src/docs/development/upstream-map.yaml.}