Repository navigation
Conversation
|
upgrading the zstd is deliberately closed in #20110 as it might cause incompatibility. does parquet-hadoop:1.18.1 depenonds on any new API in com.github.luben:zstd-jni:1.5.7-15 ? |
|
#20258 |
| <groupId>com.github.luben</groupId> | ||
| <artifactId>zstd-jni</artifactId> | ||
| <version>1.5.7-11</version> | ||
| <version>1.5.7-15</version> |
There was a problem hiding this comment.
[P2] Upgrading past -12 drops legacy Zstandard decoding
zstd-jni documents 1.5.7-12 as the last release that supports the legacy 0.4–0.7 formats; 1.5.7-15 removes them (see https://github.com/luben/zstd-jni#status-and-availability). Druid exposes .zst as a supported batch-ingestion format, and processing/src/main/java/org/apache/druid/utils/CompressionUtils.java:808-809 constructs ZstdCompressorInputStream from this same dependency, so existing legacy .zst inputs that worked with 1.5.7-11 will now fail during ingestion. Please retain a legacy-capable version (if Parquet works with it) or explicitly handle, document, and test this compatibility break before merging.
|
Thanks a lot for calling this out, @FrankChen021 and for providing the context! |
Not that I am aware of. Closing this off for now unless someone in the community reports any implementation gap. |
org.apache.parquet:parquet-hadoop:1.18.1depends oncom.github.luben:zstd-jni:1.5.7-15This PR has: