Skip to content

Ensure all PC API's are thread safe #186

Open

Description

Currently PC is multi threaded but underlying Kafka consumer is not. So a scenario such as

  1. thread one subscribes to topic one
  2. thread two subscribes to topic two

will fail since the underlying kafka consumer will reject the second subscribe, due to the consumer's thread safety checks

Relates to: Enable changing of topic subscription before or after PC has started #187

Fixed by:

Activity

  1. changed the title [-]Provide complete Parallel consumer multithreading[/-] [+]Ensure all PC API's are thread safe[/+] on Feb 3, 2022
  2. astubbs commented on Aug 5, 2026

    @astubbs
    Contributor

    Tracked in the maintained fork as astubbs#139, which carries a summary of this
    issue and its current status there.

    The Confluent-hosted version of this project is no longer maintained - its README says so.
    Parallel Consumer itself continues in my fork: https://github.com/astubbs/parallel-consumer

    Nothing in this repository is closed or changed by this comment.

  3. astubbs commented on Sep 18, 2026

    @astubbs
    Contributor

    Peter Fowler (@fowlerp-qlik) - a status from the maintained fork, where this is tracked as astubbs#139 and held as a 1.0 blocker:

    • Not implemented. The "Fixed by: feature #520: Thread safe User API exposure of ALL Consumer APIs #346" note names the intended fix, not a landed one: feature #520: Thread safe User API exposure of ALL Consumer APIs #346 was a draft, gated behind two other PRs, and all three were closed unmerged in the 2023-06-15 sweep.
    • The fork has now audited the public surface method by method, against its v0.6.0.0 release. Unsafe off the constructing thread: pauseIfRunning / resumeIfPaused and the close overloads share a check-then-set on the run state with no lock, so a pause racing a close can resurrect a closing consumer; subscribe is safe before the first poll and unsafe after; setLongPollTimeout writes a static. Safe: loop-end callback registration (fixed), getFailureCause, workRemaining, getPausedPartitionSize, the commit and wake-up requests. One internal class that claimed thread safety in its javadoc while reading under no lock has been fixed.
    • What clears the blocker on the fork: a per-method javadoc contract saying which threads may call what, an atomic run-state transition, and a concurrent test against a live instance. Exposing every Consumer API safely, the full ambition of feature #520: Thread safe User API exposure of ALL Consumer APIs #346, is a separate feature.

    No date; the audit is the part that makes it actionable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions