Fix service discovery bug in `kubernetes-extensions` by capistrant · Pull Request #19139 · apache/druid

capistrant · 2026-03-11T22:14:31Z

Description

Bug Report

The k8s service discovery is not removing discovered nodes whose pods still exist with service announcement labels, but the underlying services are actually unhealthy.

For example, if a broker container is killed but the pod that manages it remains in the namespace with announcement labels, all druid services will maintain this service in their discovered services cache. This leads to queries being routed to a broker that cannot possibly execute the request. If this pod remains in an announced but unhealthy state for any meaningful period of time, the cluster functionality can be severely compromised.

Desired behavior in the above example would be that the broker is removed from discovered services caches, at least until the underlying container for the pod is restarted and the pod is healthy again.

Fix Details

My proposed fix starts using a pods readiness flag in the discovery logic. If a pod is not ready, the underlying services will not be added to service discovery caches they are not in and will be removed from any caches that they were in. These services can be added back once they have a MODIFIED or ADDED event in addition to being ready again.

Fix Risks

The biggest risk I see is that this new reliance on readiness probe introduces an expectation that this probe is accurate and stable. I try to call out in documentation that this needs to be considered when defining the readiness probe for a pod as a way to mitigate unexpected changes for users. This could be included in a release note as well to tip off any users of the extension.

Alternatives

Using Pod Phase is another option for this. This feels like a more permissive check than readiness in that it may be more lax with removing a service from discovered hosts than readiness.

The idea would be to treat Running and Unknown pod phase states as discoverable. Unknown is there to avoid k8s control plane communication issues / transient problems from removing hosts from discovered lists. One downside here based on k8s documentation (linked above) is that a pod will be in running phase if a container is in the process of starting or restarting. This technically means that services who are not actually ready to be in the cluster could still be discovered if they have the labels and are in a transient state trying to start. The fear here being that a service gets into a restart loop without changing phase, so it remains discoverable but is not going to become healthy in the near future (or ever).

Release note

TBD

Key changed/added classes in this PR

DefaultK8sApiClient
BaseNodeRoleWatcher
WatchResult

This PR has:

…very

capistrant · 2026-03-11T22:32:37Z

marking this as draft while I evaluate a competing approach that uses pod phase instead of readiness

capistrant · 2026-03-18T16:04:02Z

marking this as draft while I evaluate a competing approach that uses pod phase instead of readiness

I think that I prefer readiness to pod phase.

being ready means that you are ready to announce yourself to the cluster. If a readiness probe is properly configured using our recommendations, if the probe fails enough to cross threshold of being not ready, the service is no longer ready to be announced and should stop getting traffic.

also in protecting against the worst case of a pod having announce labels but the container having been killed without proper shutdown + not being rescheduled, readiness will properly handle undiscovering such a service nearly immediately upon crash.

gianm · 2026-03-18T17:22:53Z

+                      effectiveType = WatchResult.NOT_READY;
+                    } else if (WatchResult.ADDED.equals(item.type)) {
+                      // Pod is not ready yet (e.g., still starting up). Skip this event entirely.
+                      // It will appear via a MODIFIED event that remaps to ADDED for dicovery, once it becomes ready.


discovery (spelling)

gianm · 2026-03-18T17:26:47Z

@@ -26,9 +26,18 @@
 public interface WatchResult


This class used to be a relatively thin layer over k8s watches, now there's remapping and synthetic events going on. The javadoc of this class should make clear that it's not meant to be a thin layer over k8s watches, it's meant to be aligned with the needs of service discovery.

gianm · 2026-03-18T17:46:43Z

@@ -273,6 +273,11 @@ private void keepWatching(String labelSelector, String resourceVersion)
                  case WatchResult.DELETED:
                    baseNodeRoleWatcher.childRemoved(item.object.getNode());


Can DELETED fire after NOT_READY (or even before ADDED, if a pod is deleted before it ever becomes ready)? I think the logging in those cases will end up noisy, since childRemoved complains loudly if the service doesn't exist. You added a skipIfUnknown that is used when NOT_READY fires, but I wonder if now we should always act like that.

yes, these could certainly happen. thank you for calling it out

…ix (apache apache#19139) to druid 30 (#469) * OBSDATA-13995 Wire TCP connect timeout into Netty HTTP client The Netty ClientBootstrap had no connectTimeoutMillis set, so TCP SYN to an unreachable host would hang for the OS default (~2 min on Linux). The existing druid.broker.http.clientConnectTimeout config (default 500ms) was only applied to the Jetty client used by the Router, not the Netty client used by the Broker. Thread connectTimeout through HttpClientConfig into the Netty bootstrap via bootstrap.setOption("connectTimeoutMillis", ...). Queries to unreachable Historicals now fail in 500ms instead of hanging for minutes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix service discovery bug in `kubernetes-extensions` (apache#19139) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>

… broker TCP connect timeout (#450) * OBSDATA-13995 Wire TCP connect timeout into Netty HTTP client The Netty ClientBootstrap had no connectTimeoutMillis set, so TCP SYN to an unreachable host would hang for the OS default (~2 min on Linux). The existing druid.broker.http.clientConnectTimeout config (default 500ms) was only applied to the Jetty client used by the Router, not the Netty client used by the Broker. Thread connectTimeout through HttpClientConfig into the Netty bootstrap via bootstrap.setOption("connectTimeoutMillis", ...). Queries to unreachable Historicals now fail in 500ms instead of hanging for minutes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix service discovery bug in `kubernetes-extensions` (apache#19139) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>

…oker connect timeout (#474) Scoped revert of the kubernetes-extensions / service-discovery half of 455f3e5 (OBSDATA-13995 Backport connect timeout (#450) and service discovery fix (apache apache#19139) to druid 30 (#469)). The broker TCP connect-timeout fix (HttpClientConfig.java, HttpClientInit.java, HttpClientModule.java, DruidHttpClientConfig.java) is untouched and remains in place - it is unrelated and unaffected by this incident. The isPodReady() filter added to DefaultK8sApiClient.listPods()/ watchPods() requires k8s to already report a pod's container ready before that pod can be treated as self-discovered. But a pod's readiness/startup probe (still pointed at /status/selfDiscovered on every environment using this image) only passes once self-discovery succeeds - a circular dependency. This crash-restart-looped every overlord pod on druid-devel-44409 (running druid-34, which carries the identical filter) after it picked up an image containing this filter; druid-30 is exposed to the same risk on its next fresh pod rollout. The real fix (a new /status/ready endpoint backed by a purely local ServiceAnnouncementState flag, breaking the circularity) is up as PR #472 on this branch's line, but the filter and the fix must ship in the same image tag - landing them in different tags is what caused the incident, and leaving the filter in now blocks other in-flight work on this branch. Reverting here to unblock; the filter will be re-added together with PR #472 once both are ready to ship as a single image. This reverts the isPodReady() filter and its tests to their pre-455f3e5715 state; docs/development/extensions-core/kubernetes.md and website/.spelling are reverted alongside since both are attributable solely to the service-discovery change.

…oker connect timeout (#473) Scoped revert of the kubernetes-extensions / service-discovery half of b0f841b ([OBSDATA-13995] Backport service discovery fix (apache#19139) and add broker TCP connect timeout (#450)). The broker TCP connect-timeout fix (HttpClientConfig.java, HttpClientInit.java, HttpClientModule.java) is untouched and remains in place - it is unrelated and unaffected by this incident. The isPodReady() filter added to DefaultK8sApiClient.listPods()/ watchPods() requires k8s to already report a pod's container ready before that pod can be treated as self-discovered. But a pod's readiness/startup probe (still pointed at /status/selfDiscovered on every environment using this image) only passes once self-discovery succeeds - a circular dependency. This crash-restart-looped every overlord pod on druid-devel-44409 after it picked up an image containing this filter. The real fix (a new /status/ready endpoint backed by a purely local ServiceAnnouncementState flag, breaking the circularity) is up as PR #471 on this branch's line, but the filter and the fix must ship in the same image tag - landing them in different tags is what caused the incident, and leaving the filter in now blocks other in-flight work on this branch. Reverting here to unblock; the filter will be re-added together with PR #471 once both are ready to ship as a single image. This reverts the isPodReady() filter and its tests to their pre-b0f841bfb3 state; docs/development/extensions-core/kubernetes.md and website/.spelling are reverted alongside since both are attributable solely to the service-discovery change.

Backport of upstream commit ee614dc (apache#19148), which predates and is required by the service-discovery isPodReady() filter backported in b0f841b ([OBSDATA-13995] Backport service discovery fix (apache#19139)...). Without this endpoint, a pod's readiness/startup probe pointed at /status/selfDiscovered deadlocks: k8s only marks a container ready after its probe passes, but the k8s-discovery isPodReady() filter requires k8s to already report the container ready before self-discovery (and therefore /status/selfDiscovered) can succeed. This surfaced as a crash-restart loop on druid-devel-44409 overlord pods immediately after rolling to an image containing the isPodReady() filter. This commit adds ServiceAnnouncementState, a purely local flag flipped by the node's own announce()/unannounce() lifecycle handlers, and exposes it via a new /status/ready endpoint that upstream now recommends as the probe target instead of /status/selfDiscovered (see the docs update in apache#19139). Wiring probes to /status/ready is a separate, follow-up change to the helm chart / operator defaults. (cherry picked from commit ee614dc)

…#495) * [OBSDATA-13995] Backport /status/ready endpoint (apache apache#19148) Backport of upstream commit ee614dc (apache#19148), which predates and is required by the service-discovery isPodReady() filter backported in b0f841b ([OBSDATA-13995] Backport service discovery fix (apache#19139)...). Without this endpoint, a pod's readiness/startup probe pointed at /status/selfDiscovered deadlocks: k8s only marks a container ready after its probe passes, but the k8s-discovery isPodReady() filter requires k8s to already report the container ready before self-discovery (and therefore /status/selfDiscovered) can succeed. This surfaced as a crash-restart loop on druid-devel-44409 overlord pods immediately after rolling to an image containing the isPodReady() filter. This commit adds ServiceAnnouncementState, a purely local flag flipped by the node's own announce()/unannounce() lifecycle handlers, and exposes it via a new /status/ready endpoint that upstream now recommends as the probe target instead of /status/selfDiscovered (see the docs update in apache#19139). Wiring probes to /status/ready is a separate, follow-up change to the helm chart / operator defaults. (cherry picked from commit ee614dc) * Fix embedded-tests MinIO image pull failure by switching to quay.io minio/minio was removed from Docker Hub (INC-13385), breaking MinIOStorageResource-based embedded tests. Point at the equivalent quay.io/minio/minio tag, matching the fix already adopted elsewhere (e.g. confluentinc/flink-control-plane#13593). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "Fix embedded-tests MinIO image pull failure by switching to quay.io" This reverts commit f797bdd. * ci: switch MinIO test image from Docker Hub to Quay * Drop ServerSideEncryptingAmazonS3MultipartUploadTest cherry-pick This upstream test (from apache#20331) targets the SDK v2-style ServerSideEncryptingAmazonS3/S3Utils API that isn't present on this 34.0.0-confluent fork yet (still on AWS SDK v1), so it doesn't compile. Keeping just the MinIO image fix for MinIOStorageResource.java and S3StorageConnectorTest.java, which is what's needed to unblock PR #495. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Gian Merlino <gianmerlino@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>

* Restarted the watch after stream reset * Addressed review comments * Add unit test and addresses styling issues * Fix service discovery bug in `kubernetes-extensions` (#19139) * Address review comments. Remove hard dependency of okhttp3 internals * fix: remove duplicate JUnit4 tests reintroduced by rebase Rebasing onto apache/druid master silently duplicated three childRemoved(skipIfUnknown) tests: the JUnit5 versions already in this branch's history merged alongside stale JUnit4 copies from before the junit5-migration commit, instead of conflicting. * Handle Bookmark event Address review comments Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com> --------- Signed-off-by: Vivek Dhiman <approach2vivek@gmail.com> Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>

…9139) on Druid 34 (#504) * [OBSDATA-15475] Re-land service-discovery isPodReady() filter (apache#19139) Reverts 5568688 (#473). The /status/ready endpoint (apache#19148, #495) is now in this branch's ancestry, so any tag containing the filter also contains the endpoint. Do not roll a tag built from this to a cluster that has not completed its Phase B probe-path flip (OBSDATA-15474); the filter plus probes still on /status/selfDiscovered reproduces the 2026-07-20 devel-44409 overlord crash-loop. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Reduce log noise for k8s NodeRoleWatcher (apache#19077) (cherry picked from commit 010be96) * [OBSDATA-15475] Address review: Ready condition, BOOKMARK rv, relist backoff, tests - isPodReady requires the pod Ready condition plus all containers ready, so node-lost pods (stale container statuses) and not-ready sidecars are dropped. - Port the BOOKMARK resourceVersion fix from apache#19056. - Back off watcherErrorRetryWaitMS before relisting on a null watch item. - Add watchPods tests against a mock watch stream and expand isPodReady tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [OBSDATA-15475] Apply terminatingStateCheckDuration to watch path The MODIFIED->ADDED remap re-added stuck-terminating pods that listPods had excluded. Share the termination rule between list and watch: past the check duration, MODIFIED maps to NOT_READY and ADDED is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>

capistrant added 8 commits March 11, 2026 16:16

Potential fix

dbe34a8

self review iteration

e539452

minor fixes

cb71279

clarify why remapping modified to added is safe

fcce511

Add more info about how readiness probes come into play for k8s disco…

474c33f

…very

more comment and log cleanup

23af381

Fix doc spelling

1340130

refactor to make reasoning about new behavior more easy

23f557e

capistrant added the Bug label Mar 11, 2026

github-actions Bot added the Area - Documentation label Mar 11, 2026

capistrant added the Kubernetes label Mar 11, 2026

capistrant marked this pull request as draft March 11, 2026 22:32

capistrant added 7 commits March 13, 2026 13:32

Merge branch 'master' into k8s-discovery-bug

27cefc1

Merge branch 'master' into k8s-discovery-bug

f8f057b

change example in docs to new readiness probe

106fe38

Fix tests file

5ca2069

Merge branch 'master' into k8s-discovery-bug

d2f7c1b

Some doc sharpening

0a9b72f

add some common k8s language to spelling

4e1db73

capistrant marked this pull request as ready for review March 17, 2026 22:31

capistrant added the Release Notes label Mar 17, 2026

capistrant added this to the 37.0.0 milestone Mar 17, 2026

gianm reviewed Mar 18, 2026

View reviewed changes

Address review comments on log noiseness and javadocs

818bffc

gianm approved these changes Mar 18, 2026

View reviewed changes

gianm merged commit 15e98ba into apache:master Mar 18, 2026
37 checks passed

airlock-confluentinc Bot pushed a commit to confluentinc/druid that referenced this pull request Jul 1, 2026

Fix service discovery bug in kubernetes-extensions (apache#19139)

6ccac59

saithal-confluent mentioned this pull request Jul 1, 2026

[OBSDATA-13995] Backport service discovery fix (#19139) and add broker TCP connect timeout confluentinc/druid#450

Merged

3 tasks done

pagrawal10 mentioned this pull request Jul 2, 2026

Fix service discovery bug in kubernetes-extensions (#19139) confluentinc/druid#468

Closed

10 tasks

saithal-confluent mentioned this pull request Jul 2, 2026

OBSDATA-13995 Backport connect timeout (#450) and service discovery fix (apache #19139) to druid 30 confluentinc/druid#469

Merged

This was referenced Jul 20, 2026

[OBSDATA-13995] Revert service-discovery isPodReady() filter (keep connect-timeout fix) confluentinc/druid#473

Merged

[OBSDATA-13995] Revert service-discovery isPodReady() filter (keep connect-timeout fix) confluentinc/druid#474

Merged

vivek807 pushed a commit to deep-bi/druid that referenced this pull request Sep 1, 2026

Fix service discovery bug in kubernetes-extensions (apache#19139)

b0964d3

vivek807 pushed a commit to deep-bi/druid that referenced this pull request Sep 1, 2026

Fix service discovery bug in kubernetes-extensions (apache#19139)

39a7071

vivek807 pushed a commit to deep-bi/druid that referenced this pull request Sep 1, 2026

Fix service discovery bug in kubernetes-extensions (apache#19139)

40c121e

vivek807 pushed a commit to deep-bi/druid that referenced this pull request Oct 8, 2026

Fix service discovery bug in kubernetes-extensions (apache#19139)

8635609

saithal-confluent mentioned this pull request Oct 15, 2026

[OBSDATA-15475] Re-land isPodReady service-discovery filter (apache/druid#19139) on Druid 34 confluentinc/druid#504

Merged

3 tasks done

Navigation Menu

Repository navigation

Fix service discovery bug in `kubernetes-extensions` - #19139

Fix service discovery bug in `kubernetes-extensions`#19139
gianm merged 16 commits into
apache:masterfrom
capistrant:k8s-discovery-bug

capistrant commented Mar 11, 2026 •

edited

Loading

Uh oh!

capistrant commented Mar 11, 2026

Uh oh!

capistrant commented Mar 18, 2026

Uh oh!

gianm Mar 18, 2026

Uh oh!

gianm Mar 18, 2026

Uh oh!

gianm Mar 18, 2026

Uh oh!

capistrant Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -273,6 +273,11 @@ private void keepWatching(String labelSelector, String resourceVersion)
		case WatchResult.DELETED:
		baseNodeRoleWatcher.childRemoved(item.object.getNode());

Conversation

capistrant commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Bug Report

Fix Details

Fix Risks

Alternatives

Release note

Key changed/added classes in this PR

Uh oh!

capistrant commented Mar 11, 2026

Uh oh!

capistrant commented Mar 18, 2026

Uh oh!

gianm Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

gianm Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

gianm Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

capistrant Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

capistrant commented Mar 11, 2026 •

edited

Loading