Skip to content

Supervisor for KafkaIndexTask #2635

Closed
Closed

Description

@gianm

See #2200.

That PR provides the low level building blocks of Kafka ingestion, but since each task has a finite lifecycle, it is expected that a process will be supervising existing tasks and creating new ones when needed. @dclim is working on this process in https://github.com/dclim/druid/tree/kafka-supervisor.

The current thinking is that this process will run in a thread on the overlord, and will be responsible for starting, killing, and communicating with tasks when necessary. It will also talk to Kafka brokers to understand what offsets exist for the target topic.

There will also need to be a system that this supervisor can slot into- e.g. a SupervisorManager on the overlord.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions