You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
That PR provides the low level building blocks of Kafka ingestion, but since each task has a finite lifecycle, it is expected that a process will be supervising existing tasks and creating new ones when needed. @dclim is working on this process in https://github.com/dclim/druid/tree/kafka-supervisor.
The current thinking is that this process will run in a thread on the overlord, and will be responsible for starting, killing, and communicating with tasks when necessary. It will also talk to Kafka brokers to understand what offsets exist for the target topic.
There will also need to be a system that this supervisor can slot into- e.g. a SupervisorManager on the overlord.
Activity