Repository navigation
Add /status/ready endpoint for service health. - #19148
Merged
Merged
Conversation
Currently the most natural endpoint to use for service health (e.g. if adding Druid services to a load balancer) is /status/health. However, this does not play nicely with graceful shutdown mechanisms. When druid.server.http.unannouncePropagationDelay is used, there is a delay between unannounce and server shutdown, which allows Druid's internal service discovery to stop sending traffic to a service before it shuts down its server. However, /status/health continues to return OK until the server is shut down, so external load balancers cannot take advantage of this. This patch adds /status/ready, an endpoint that is tied to announcement. It allows external load balancers to take advantage of this graceful shutdown mechanism.
kfaraz
approved these changes
Mar 13, 2026
vivek807
pushed a commit
to deep-bi/druid
that referenced
this pull request
Sep 1, 2026
Currently the most natural endpoint to use for service health (e.g. if adding Druid services to a load balancer) is /status/health. However, this does not play nicely with graceful shutdown mechanisms. When druid.server.http.unannouncePropagationDelay is used, there is a delay between unannounce and server shutdown, which allows Druid's internal service discovery to stop sending traffic to a service before it shuts down its server. However, /status/health continues to return OK until the server is shut down, so external load balancers cannot take advantage of this. This patch adds /status/ready, an endpoint that is tied to announcement. It allows external load balancers to take advantage of this graceful shutdown mechanism.
airlock-confluentinc Bot
pushed a commit
to confluentinc/druid
that referenced
this pull request
Sep 18, 2026
Backport of upstream commit ee614dc (apache#19148), which predates and is required by the service-discovery isPodReady() filter backported in b0f841b ([OBSDATA-13995] Backport service discovery fix (apache#19139)...). Without this endpoint, a pod's readiness/startup probe pointed at /status/selfDiscovered deadlocks: k8s only marks a container ready after its probe passes, but the k8s-discovery isPodReady() filter requires k8s to already report the container ready before self-discovery (and therefore /status/selfDiscovered) can succeed. This surfaced as a crash-restart loop on druid-devel-44409 overlord pods immediately after rolling to an image containing the isPodReady() filter. This commit adds ServiceAnnouncementState, a purely local flag flipped by the node's own announce()/unannounce() lifecycle handlers, and exposes it via a new /status/ready endpoint that upstream now recommends as the probe target instead of /status/selfDiscovered (see the docs update in apache#19139). Wiring probes to /status/ready is a separate, follow-up change to the helm chart / operator defaults. (cherry picked from commit ee614dc)
saithal-confluent
added a commit
to confluentinc/druid
that referenced
this pull request
Sep 18, 2026
…#495) * [OBSDATA-13995] Backport /status/ready endpoint (apache apache#19148) Backport of upstream commit ee614dc (apache#19148), which predates and is required by the service-discovery isPodReady() filter backported in b0f841b ([OBSDATA-13995] Backport service discovery fix (apache#19139)...). Without this endpoint, a pod's readiness/startup probe pointed at /status/selfDiscovered deadlocks: k8s only marks a container ready after its probe passes, but the k8s-discovery isPodReady() filter requires k8s to already report the container ready before self-discovery (and therefore /status/selfDiscovered) can succeed. This surfaced as a crash-restart loop on druid-devel-44409 overlord pods immediately after rolling to an image containing the isPodReady() filter. This commit adds ServiceAnnouncementState, a purely local flag flipped by the node's own announce()/unannounce() lifecycle handlers, and exposes it via a new /status/ready endpoint that upstream now recommends as the probe target instead of /status/selfDiscovered (see the docs update in apache#19139). Wiring probes to /status/ready is a separate, follow-up change to the helm chart / operator defaults. (cherry picked from commit ee614dc) * Fix embedded-tests MinIO image pull failure by switching to quay.io minio/minio was removed from Docker Hub (INC-13385), breaking MinIOStorageResource-based embedded tests. Point at the equivalent quay.io/minio/minio tag, matching the fix already adopted elsewhere (e.g. confluentinc/flink-control-plane#13593). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "Fix embedded-tests MinIO image pull failure by switching to quay.io" This reverts commit f797bdd. * ci: switch MinIO test image from Docker Hub to Quay * Drop ServerSideEncryptingAmazonS3MultipartUploadTest cherry-pick This upstream test (from apache#20331) targets the SDK v2-style ServerSideEncryptingAmazonS3/S3Utils API that isn't present on this 34.0.0-confluent fork yet (still on AWS SDK v1), so it doesn't compile. Keeping just the MinIO image fix for MinIOStorageResource.java and S3StorageConnectorTest.java, which is what's needed to unblock PR #495. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Gian Merlino <gianmerlino@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Frank Chen <frank.chen021@outlook.com>
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently the most natural endpoint to use for service health (e.g. if adding Druid services to a load balancer) is
/status/health. However, this does not play nicely with graceful shutdown mechanisms.When
druid.server.http.unannouncePropagationDelayis used, there is a delay between unannounce and server shutdown, which allows Druid's internal service discovery to stop sending traffic to a service before it shuts down its server. However,/status/healthcontinues to return OK until the server is shut down, so external load balancers cannot take advantage of this.This patch adds
/status/ready, an endpoint that is tied to announcement. It allows external load balancers to take advantage of this graceful shutdown mechanism.