build(zstd): replace aircompressor with zstd FFM bindings#165
Merged
Conversation
Swap the `vortex.zstd` encoding off `io.airlift:aircompressor-v3` onto `io.github.dfa1.zstd:zstd` 0.1 — FFM bindings to the native libzstd. The encoder and decoder now run fully zero-copy on arena segments via `ZstdCompressCtx.compress(arena, src)` / `ZstdDecompressCtx.decompress(dst, src)`, with no heap `byte[]` bounce on either side; the decoder writes each frame straight into its slice of the scan-arena output. I8/U8 encode input is now arena-allocated rather than heap-wrapped so it can feed the native call, and `buildLengthPrefixed` returns an arena segment. The 0.1 release ships no bundled native; libzstd is loaded from a `/native/<platform>/` classpath resource carried by a separate `zstd-native-<platform>` artifact. Two os-activated profiles pull in the matching native (osx-aarch64, linux-x86_64); add more platforms as needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Swaps the
vortex.zstdencoding offio.airlift:aircompressor-v3ontoio.github.dfa1.zstd:zstd0.1 — FFM bindings to the nativelibzstd. Drops everyaircompressor/airliftmention from the tree.What changed
dependencyManagement, the BOM, and 9 module poms (reader, writer, calcite, integration, cli, jdbc, csv, parquet, performance). Newzstd.version=0.1property.libzstdloads from a/native/<platform>/classpath resource shipped by a separatezstd-native-<platform>artifact. Two os-activated profiles pull in the matching native (osx-aarch64,linux-x86_64) at runtime scope. Other platforms need a profile added.ZstdCompressCtx.compress(arena, src)/ZstdDecompressCtx.decompress(dst, src), no heapbyte[]bounce. The decoder writes each frame straight into its slice of the scan-arena output; I8/U8 encode input is now arena-allocated;buildLengthPrefixedreturns an arena segment.Changedentry (consumers ofvortex.zstdmust declarezstd+ the platform native artifact).Testing
🤖 Generated with Claude Code