Repository navigation
Druid Coordinator stop assign segments after the zk restart #17807
Description
Affected Version
Druid 25.0
Zookeeper version: 3.5.10
Description
- Zookeeper restart due to the node crash in 05:50 UTC
- Coordinator log
2025-03-17T05:50:23,080 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am no longer the leader... 2025-03-17T05:50:24,370 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow! Starting coordination in [PT30S]. - There are no new segment assignments, but the following log is present with no error logs:
Polled and found 201 rule(s) for 193 datasource(s).
I have attempted the following approaches to recover from the issue:
- Restarted the coordinator leader and follower, but this did not help.
- Restarted the Zookeeper follower, but this did not help.
- Restarted the Zookeeper leader, which resolved the issue.
Activity
gianm commentedon Apr 1, 2025
It seems like some kind of issue with leader election. It's unfortunately hard to debug from just this provided information. It would possibly be helpful to see more logs. If this ever happens again, it's also possibly helpful to see a thread dump of the apparently-stuck Coordinator before restarting it.