Skip to content

Refactor: Consider a shared nothing architecture, to reduce thread complexity #200

Open

Description

ATM there's quite a bit of work in managing SHARED state between threads/sub systems. Although this is working well, it's complex. A shared nothing architecture would mean there are no sync issues, however work would need to be done to implement async request response, or maybe just pushing messages between systems. This might actually not be that hard to implement. It's basically just the Actor Mailbox pattern. We already use mail boxes really, but they're not used in clear boundaries between acts and state systems. New mail boxes would replace the existing concurrent collections that we already use and rely on, but in a more structured and clearly delineated fashion.

There are three thread actors atm - the controller, the poller, and the workers. Main sync issues occur because the poller and the controller share access to the WorkManager, PartitionMonitor and ShardManager. This would change it so that only the controller has access to these, and the poller instead pushes messages to it's inbox - partition assignment changes and new polled messages.

Poller would try to make sure controllers Records inbox always has enough queued. And it would push assignment messages to it, and it would be ok that the controller process these messages only at the start of it's cycle. As it would simply drop results from records that aren't assigned anymore, and async remove from inbound queues stale records before they're processed.

The ultimate simplification would be to eliminate the seperate poller thread. I think the main reason there's a seperate thread is because of blocking polls to the broker when there's no data in the partitions. It will wait up until some max. However, two options:
a) go with the interrupting model and interrupt a blocking poll if work arrives for the controller to do.
b) don't block when polling?

Activity

  1. astubbs commented on May 6, 2022

    @astubbs
    ContributorAuthor
  2. added 2 commits that reference this issue on Sep 29, 2022
  3. astubbs commented on Aug 5, 2026

    @astubbs
    ContributorAuthor

    Tracked in the maintained fork as astubbs#142, which carries a summary of this
    issue and its current status there.

    The Confluent-hosted version of this project is no longer maintained - its README says so.
    Parallel Consumer itself continues in my fork: https://github.com/astubbs/parallel-consumer

    Nothing in this repository is closed or changed by this comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions