Skip to content

VirtualStorage implementation for MSQ in druid 37 causing threadpool leak causing native threads exhausted #19904

Description

@simha03

We have using druid 37 and we are facing issue with threadpool not closing for MSQ as its not releasing the threads on task exit. we are exhausting the native threads due to this threads leak. once the threads hit the ulimit value it doesnt allow any new threads creation until we restart the indexer service. The issue is more visible in indexer as MSQ runs on same jvm unlike middle manager where each MSQ will fork different JVM.

[root@xxxxx ~]# ls /proc/32589/task | wc -l
254,869
[root@xxxxx ~]#

254,869 ← the leaker, ~99% thread are VirtualStorageOnDemandLoadingThread

Looks the issue was identified and fixed : dedicated ephemeral storage loading pool for tasks- #19658
#19658
I could see the backporting is done only to druid 38. Can we backport to druid 37 as its is a major break on druid stability running MSQ running on indexers ?

Activity

  1. simha03 commented on Aug 8, 2026

    @simha03
    Author

    @gianm @FrankChen021
    #19658 you had made the changes. Can you please check if this can also be backported to druid 37 ?

  2. FrankChen021 commented on Aug 8, 2026

    @FrankChen021
    Member

    @simha03 Thanks for reporting. I think for v37, it's not easy to backport #19658 because that PR depends on other changes in v38.

    A quick way is to upgrading to v38

  3. simha03 commented on Aug 8, 2026

    @simha03
    Author

    @FrankChen021 thanks for response.
    When can we expect druid 38 release ?

  4. FrankChen021 commented on Aug 8, 2026

    @FrankChen021
    Member

    this work started last month, and 38 release branch has been created, many bugs have already been backported to that branch. It will be soon released I believe, but I don't have a concrete date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions