docs: 38.0.0 release notes by 317brian · Pull Request #19945 · apache/druid

317brian · 2026-08-09T16:02:18Z

Release notes for 38.0.0. This should be caught up to August 9th

Fixes #XXXX.

Description

Fixed the bug ...

Renamed the class ...

Added a forbidden-apis entry ...

Release note

Key changed/added classes in this PR

MyFoo
OurBar
TheirBaz

This PR has:

techdocsmith

Great work. Left some comments

techdocsmith · 2026-08-14T21:53:32Z

+Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records
+the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match. 


Suggested change

Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records

the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match.

Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values don't match.

odd line break. "values can't match" reads awkwardly

techdocsmith · 2026-08-14T22:00:02Z

+
+#### Storage metrics
+
+The `storage/load/bytes` and `storage/virtual/load/bytes` metrics now measure once the load is complete. Previously, they measured when the load starts.


"they measured when the load starts" mix of past/present reads awkwardly. Consider revising

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

kfaraz · 2026-09-13T04:17:21Z

+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading


Shouldn't this section have a description?

Suggested change

#### Partial segment loading

#### Partial segment loading (experimental)

Historical tiers now support loading partial data from a segment. The exact data to be downloaded is dictated by partial load rules which may list one or more cluster groups or projections as eligible for download.

partial load rules are actually separate from the linked PRs (there are a handful of those PRs too). 19535 is about allowing historicals running in virtual storage mode, with v10 segments which are stored in deep storage without zip, to load on demand at the column level at query time (so dictated by what the query needs to read rather than load rules), when using Dart or MSQ (legacy engine still requires full download up front in virtual mode). The storage accounting of the historical cache is done at the 'bundle' level (base table, projections, cluster groups), where the cache has an entry for each bundle and we just lazy load columns contained in that bundle into it as the query needs.

Partial load rules work with this partial load functionality by allowing virtual mode historicals to eagerly load parts of segments at the 'bundle' level (base table, projections, cluster groups) and make those bundles sticky so that they cannot be evicted/are always 'hot' even if the rest of the segment can be loaded and evicted on demand.

All of this stuff is quite experimental, and will probably be in a lot nicer state next release.

Hmm, in that case, I feel we should just remove them from the release notes here, as you had suggested earlier. Otherwise, this will just create confusion.

kfaraz · 2026-09-13T04:17:33Z

+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments


Description missing?

Suggested change

#### Clustered segments

#### Clustered segments (experimental)

Clustered segments contain one or more cluster groups that allow quick pruning of data while querying by scanning only the groups which match a given query.

its probably worth mentioning that it only supports v10 segments, which are not really documented either. I think the tricky part with announcing some of this stuff is that probably the only way to use them is to look at PRs or look at the code since none of it is documented yet.

So, do you advise just removing these items from the release notes for now?

FrankChen021 · 2026-09-19T11:55:55Z

as discussed in #20236 (comment)
we should call out the deprecation of old pre-standard v0.4-v0.7 zstd frames support on both batch and kafka ingestion from this release and the support of these frames will be removed in next release.

See also #20110

kfaraz · 2026-10-01T04:14:33Z

+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments


Suggested change

#### Clustered segments

#### Clustered segments (experimental)

Clustered segments contain one or more cluster groups that allow quick pruning of data while querying by scanning only the groups which match a given query.

kfaraz · 2026-10-01T04:14:58Z

+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading


Suggested change

#### Partial segment loading

#### Partial segment loading (experimental)

Historical tiers now support loading partial data from a segment. The exact data to be downloaded is dictated by partial load rules which may list one or more cluster groups or projections as eligible for download.

…se notes

clintropolis · 2026-10-01T06:37:07Z

+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.


this is only really applicable to virtual storage mode historicals i think, though i suppose the docs for it don't mention that either

Thanks for calling it out, updating it.

clintropolis · 2026-10-01T06:54:21Z

+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals. 


partial load rules allow virtual storage mode historicals (also not documented i think, druid.segmentCache.virtualStorage and druid.segmentCache.virtualStoragePartialDownloadsEnabled both true) to eagerly load "parts" of v10 segments that are stored in deep storage unzipped (druid.storage.zip=false but only s3 supports that). There are matchers that must be set on the partial rule that can select the parts of the v10 segments (v10 segments are internally organized into 'bundles' which are sort of logical containers that hold all of the columns of a projection, where projection could be the base table (or individual cluster groups for clustered segments which when concatenated together form the base table projection), or individual aggregate projections).

I'm hesitant to call this stuff out since like the other partial load stuff it its going to be pretty tough to use without docs.

I wonder if we should consolidate all of this stuff in an 'experimental virtual storage mode improvements' section (partial load on demand and the partial load rules built on top of it) or just remove it as suggested in other thread

I think removing it for now would make more sense.

…for auto-scaler"

docs: 38.0.0 release notes

0b09914

github-actions Bot added the Area - Documentation label Aug 9, 2026

317brian added 6 commits August 9, 2026 09:09

upgrade notes

bf8b865

cleanup

33ebe91

spelling

cd03ab7

fix typo

e90250a

verify PRs that have 39 milestone

11acfca

codeticks for java_home

3a9b0f6

techdocsmith requested changes Aug 14, 2026

View reviewed changes

Apply suggestions from code review

df920fa

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

kfaraz reviewed Sep 13, 2026

View reviewed changes

Comment thread docs/release-info/release-notes.md Outdated

kfaraz added this to the 38.0.0 milestone Sep 22, 2026

FrankChen021 mentioned this pull request Sep 24, 2026

[DRAFT] 38.0.0 Release Notes #20416

Closed

kfaraz reviewed Oct 1, 2026

View reviewed changes

kfaraz added 2 commits October 1, 2026 13:34

Mark legacy zstd frames as deprecated

71671e8

Fix wording for supervisor restart item

994e993

kfaraz marked this pull request as ready for review October 1, 2026 08:11

kfaraz closed this Oct 1, 2026

kfaraz reopened this Oct 1, 2026

kfaraz reviewed Oct 1, 2026

View reviewed changes

Comment thread docs/release-info/release-notes.md Outdated

Remove sections for partial loading and clustered segments from relea…

517b6ae

…se notes

kfaraz reviewed Oct 1, 2026

View reviewed changes

Comment thread docs/release-info/release-notes.md Outdated

Fix spellings in zstd deprecation text

f4efd4a

kfaraz reviewed Oct 1, 2026

View reviewed changes

Comment thread docs/release-info/release-notes.md Outdated

Fix spelling

92c82ad

kfaraz reviewed Oct 1, 2026

View reviewed changes

Comment thread docs/release-info/release-notes.md Outdated

Add line after section, fix spelling

c0a9dd9

clintropolis reviewed Oct 1, 2026

View reviewed changes

kfaraz reviewed Oct 1, 2026

View reviewed changes

Comment thread docs/release-info/release-notes.md Outdated

Apply suggestion from @kfaraz

3271603

kfaraz reviewed Oct 1, 2026

View reviewed changes

Comment thread docs/release-info/release-notes.md Outdated

kfaraz reviewed Oct 1, 2026

View reviewed changes

Comment thread docs/release-info/release-notes.md Outdated

kfaraz added 2 commits October 1, 2026 17:27

Remove section for "new load rule types"

3cacda7

Update title for section "Kafka idle signal" -> "use processing rate …

7c21370

…for auto-scaler"

kfaraz approved these changes Oct 1, 2026

View reviewed changes

kfaraz merged commit 6723c21 into apache:38.0.0 Oct 1, 2026
8 checks passed

		Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records
		the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match.


		#### Storage metrics

		The `storage/load/bytes` and `storage/virtual/load/bytes` metrics now measure once the load is complete. Previously, they measured when the load starts.


		[#19559](https://github.com/apache/druid/pull/19559)

		#### Partial segment loading

-#### Partial segment loading
+#### Partial segment loading (experimental)
+Historical tiers now support loading partial data from a segment. The exact data to be downloaded is dictated by partial load rules which may list one or more cluster groups or projections as eligible for download.

-#### Clustered segments
+#### Clustered segments (experimental)
+Clustered segments contain one or more cluster groups that allow quick pruning of data while querying by scanning only the groups which match a given query.


		#### Segment prefetching for Dart

		Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.


		#### New load rule types

		Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.

Conversation

317brian commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fixed the bug ...

Renamed the class ...

Added a forbidden-apis entry ...

Release note

Key changed/added classes in this PR

Uh oh!

techdocsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FrankChen021 commented Sep 19, 2026

Uh oh!

kfaraz Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

317brian commented Aug 9, 2026 •

edited

Loading

kfaraz Oct 1, 2026 •

edited

Loading

kfaraz Oct 1, 2026 •

edited

Loading

kfaraz Oct 1, 2026 •

edited

Loading

kfaraz Oct 1, 2026 •

edited

Loading