You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Refactor: Consider a shared nothing architecture, to reduce thread complexity #200
ATM there's quite a bit of work in managing SHARED state between threads/sub systems. Although this is working well, it's complex. A shared nothing architecture would mean there are no sync issues, however work would need to be done to implement async request response, or maybe just pushing messages between systems. This might actually not be that hard to implement. It's basically just the Actor Mailbox pattern. We already use mail boxes really, but they're not used in clear boundaries between acts and state systems. New mail boxes would replace the existing concurrent collections that we already use and rely on, but in a more structured and clearly delineated fashion.
There are three thread actors atm - the controller, the poller, and the workers. Main sync issues occur because the poller and the controller share access to the WorkManager, PartitionMonitor and ShardManager. This would change it so that only the controller has access to these, and the poller instead pushes messages to it's inbox - partition assignment changes and new polled messages.
Poller would try to make sure controllers Records inbox always has enough queued. And it would push assignment messages to it, and it would be ok that the controller process these messages only at the start of it's cycle. As it would simply drop results from records that aren't assigned anymore, and async remove from inbound queues stale records before they're processed.
The ultimate simplification would be to eliminate the seperate poller thread. I think the main reason there's a seperate thread is because of blocking polls to the broker when there's no data in the partitions. It will wait up until some max. However, two options:
a) go with the interrupting model and interrupt a blocking poll if work arrives for the controller to do.
b) don't block when polling?
Activity
fix: Concurrency and State improvements (#190)
fix: Concurrency and State improvements (#190)
fix: Concurrency and State improvements (#190)
astubbs commentedon May 6, 2022
Drafted in
START: fixes #419 NoSuchElementException during race condition in Par…
fixes #419 NoSuchElementException during race condition in PartitionS…
docs(refactoring): enumerate abandoned branches + prior drafts as a r…
docs: add refactoring backlog (deferred internal refactors) (#67)
docs(refactoring): flag OffsetMapCodecManager caching as not thread-s…
docs: qualify bare upstream issue refs now that fork mirrors exist
astubbs commentedon Aug 5, 2026
Tracked in the maintained fork as astubbs#142, which carries a summary of this
issue and its current status there.
The Confluent-hosted version of this project is no longer maintained - its README says so.
Parallel Consumer itself continues in my fork: https://github.com/astubbs/parallel-consumer
Nothing in this repository is closed or changed by this comment.
docs+ci: mirror upstream tracker, and make issue references name thei…
docs+ci: mirror upstream tracker, and make issue references name thei…
docs+ci: mirror upstream tracker, and make issue references name thei…
docs+ci: mirror upstream tracker, and make issue references name thei…
docs(build) #180: qualify every issue reference, and drop a dead work…
docs(upstream): register the actor-IPC and thread-model branch famili…
docs(ideation): actor-collection revival - six ranked, evidence-verif…
docs(plan) #119: correct cluster 2 - the two commit predicates never …
docs(solutions) #119: record why two threads share one consumer, and …
docs(solutions) #119: a mirror of another component's state is a cont…
docs(inflight): no shipped static analysis sees the torn-read family,…
docs(inflight): the OTHER field on the commit path is unfenced too, a…
fix(core): the dirty flag crossed threads unfenced - one volatile, me…
refactor(core) confluentinc#857: derive the back-pressure pause from …