Repository navigation
OBSDATA-13995 Backport connect timeout (#450) and service discovery fix (apache #19139) to druid 30 - #469
Merged
Sadananda Aithal (saithal-confluent) merged 2 commits intoJul 8, 2026
Conversation
The Netty ClientBootstrap had no connectTimeoutMillis set, so TCP
SYN to an unreachable host would hang for the OS default (~2 min
on Linux). The existing druid.broker.http.clientConnectTimeout
config (default 500ms) was only applied to the Jetty client used
by the Router, not the Netty client used by the Broker.
Thread connectTimeout through HttpClientConfig into the Netty
bootstrap via bootstrap.setOption("connectTimeoutMillis", ...).
Queries to unreachable Historicals now fail in 500ms instead of
hanging for minutes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sadananda Aithal (saithal-confluent)
requested a review
from a team
as a code owner
July 2, 2026 09:58
ravisharm
approved these changes
Jul 8, 2026
Sadananda Aithal (saithal-confluent)
merged commit Jul 8, 2026
455f3e5
into
30.0.1-confluent
2 checks passed
This was referenced Jul 20, 2026
Sadananda Aithal (saithal-confluent)
added a commit
that referenced
this pull request
Jul 20, 2026
…oker connect timeout (#474) Scoped revert of the kubernetes-extensions / service-discovery half of 455f3e5 (OBSDATA-13995 Backport connect timeout (#450) and service discovery fix (apache apache#19139) to druid 30 (#469)). The broker TCP connect-timeout fix (HttpClientConfig.java, HttpClientInit.java, HttpClientModule.java, DruidHttpClientConfig.java) is untouched and remains in place - it is unrelated and unaffected by this incident. The isPodReady() filter added to DefaultK8sApiClient.listPods()/ watchPods() requires k8s to already report a pod's container ready before that pod can be treated as self-discovered. But a pod's readiness/startup probe (still pointed at /status/selfDiscovered on every environment using this image) only passes once self-discovery succeeds - a circular dependency. This crash-restart-looped every overlord pod on druid-devel-44409 (running druid-34, which carries the identical filter) after it picked up an image containing this filter; druid-30 is exposed to the same risk on its next fresh pod rollout. The real fix (a new /status/ready endpoint backed by a purely local ServiceAnnouncementState flag, breaking the circularity) is up as PR #472 on this branch's line, but the filter and the fix must ship in the same image tag - landing them in different tags is what caused the incident, and leaving the filter in now blocks other in-flight work on this branch. Reverting here to unblock; the filter will be re-added together with PR #472 once both are ready to ship as a single image. This reverts the isPodReady() filter and its tests to their pre-455f3e5715 state; docs/development/extensions-core/kubernetes.md and website/.spelling are reverted alongside since both are attributable solely to the service-discovery change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backports two changes to the Druid 30 (
30.0.1-confluent) branch, both already on the 34 branch:OBSDATA-13995 — Wire TCP connect timeout into Netty HTTP client (from [OBSDATA-13995] Backport service discovery fix (#19139) and add broker TCP connect timeout #450, commit
dd70e94d7d)ClientBootstraphad noconnectTimeoutMillis, so a TCP SYN to an unreachable host would hang for the OS default (~2 min on Linux). The existingdruid.broker.http.clientConnectTimeout(default 500ms) was only applied to the Jetty client (Router), not the Netty client (Broker).connectTimeoutthroughHttpClientConfiginto the Netty bootstrap. Queries to unreachable Historicals now fail in 500ms instead of hanging.DruidHttpClientConfigon 30 lacked theclientConnectTimeoutfield +getClientConnectTimeout()getter (it pre-existed on 34). Added here (default 500ms) so the wiring compiles.Fix service discovery bug in
kubernetes-extensions(Fix service discovery bug inkubernetes-extensionsapache/druid#19139, cherry-pick of6ccac59acd)DefaultK8sApiClientremaps not-ready pods to aNOT_READYwatch event;BaseNodeRoleWatcher.childRemovedgains askipIfUnknownflag so k8s discovery removals don't log spurious errors for pods never added.Conflict resolution notes
BaseNodeRoleWatcher,DefaultK8sApiClient,docs/.../kubernetes.md— resolved by taking the Fix service discovery bug inkubernetes-extensionsapache/druid#19139 side.terminatingStateCheckDurationlogic already present on 30 was preserved alongside the newisPodReady/NOT_READYpath (same layering used on the 34 backport).Testing
mvn test-compile -pl server,extensions-core/kubernetes-extensions→ BUILD SUCCESS (main + test sources).BaseNodeRoleWatcherTest,DefaultK8sApiClientTest,K8sDruidNodeDiscoveryProviderTest) already pass on the 34 branch with identical code. Local test execution here was blocked by unrelated stale Maven processes holding the build lock; CI will run the full suite.🤖 Generated with Claude Code