Skip to content

ci: Upgrade zstd to match dependency of parquet-hadoop 1.18.1 - #20296

Closed
kfaraz wants to merge 2 commits into
apache:masterfrom
kfaraz:upgrade_zstd
Closed

kfaraz wants to merge 2 commits into
apache:masterfrom
kfaraz:upgrade_zstd

Conversation

@kfaraz

@kfaraz kfaraz commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

org.apache.parquet:parquet-hadoop:1.18.1 depends on com.github.luben:zstd-jni:1.5.7-15


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@FrankChen021

Copy link
Copy Markdown
Member

upgrading the zstd is deliberately closed in #20110 as it might cause incompatibility.

does parquet-hadoop:1.18.1 depenonds on any new API in com.github.luben:zstd-jni:1.5.7-15 ?

@FrankChen021

Copy link
Copy Markdown
Member

#20258
also deliberated excluded the zstd from the parquet-hadoop.

@FrankChen021 FrankChen021 left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 0
P2 1
P3 0
Total 1

Reviewed 2 of 2 changed files.


This is an automated review by Codex GPT-5.6-Luna(max)

Comment thread pom.xml
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<version>1.5.7-11</version>
<version>1.5.7-15</version>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Upgrading past -12 drops legacy Zstandard decoding

zstd-jni documents 1.5.7-12 as the last release that supports the legacy 0.4–0.7 formats; 1.5.7-15 removes them (see https://github.com/luben/zstd-jni#status-and-availability). Druid exposes .zst as a supported batch-ingestion format, and processing/src/main/java/org/apache/druid/utils/CompressionUtils.java:808-809 constructs ZstdCompressorInputStream from this same dependency, so existing legacy .zst inputs that worked with 1.5.7-11 will now fail during ingestion. Please retain a legacy-capable version (if Parquet works with it) or explicitly handle, document, and test this compatibility break before merging.

@kfaraz

kfaraz commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks a lot for calling this out, @FrankChen021 and for providing the context!

@kfaraz

kfaraz commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

does parquet-hadoop:1.18.1 depenonds on any new API in com.github.luben:zstd-jni:1.5.7-15 ?

Not that I am aware of. Closing this off for now unless someone in the community reports any implementation gap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants