Skip to content

build(zstd): replace aircompressor with zstd FFM bindings#165

Merged
dfa1 merged 2 commits into
mainfrom
deps/zstd-ffm
Jun 26, 2026
Merged

build(zstd): replace aircompressor with zstd FFM bindings#165
dfa1 merged 2 commits into
mainfrom
deps/zstd-ffm

Conversation

@dfa1

@dfa1 dfa1 commented Jun 26, 2026

Copy link
Copy Markdown
Owner

Summary

Swaps the vortex.zstd encoding off io.airlift:aircompressor-v3 onto io.github.dfa1.zstd:zstd 0.1 — FFM bindings to the native libzstd. Drops every aircompressor/airlift mention from the tree.

What changed

  • Deps: replaced in parent dependencyManagement, the BOM, and 9 module poms (reader, writer, calcite, integration, cli, jdbc, csv, parquet, performance). New zstd.version=0.1 property.
  • Native: 0.1 bundles no native; libzstd loads from a /native/<platform>/ classpath resource shipped by a separate zstd-native-<platform> artifact. Two os-activated profiles pull in the matching native (osx-aarch64, linux-x86_64) at runtime scope. Other platforms need a profile added.
  • Encoder/decoder — fully zero-copy: compress/decompress run on arena segments via ZstdCompressCtx.compress(arena, src) / ZstdDecompressCtx.decompress(dst, src), no heap byte[] bounce. The decoder writes each frame straight into its slice of the scan-arena output; I8/U8 encode input is now arena-allocated; buildLengthPrefixed returns an arena segment.
  • CHANGELOG: user-facing Changed entry (consumers of vortex.zstd must declare zstd + the platform native artifact).

Testing

  • 11 zstd encoder/decoder unit tests pass.
  • 2 zstd interop ITs pass (Java writes zstd → Rust reads: i64 + utf8).
  • Full reactor compiles; broad unit suite green.

🤖 Generated with Claude Code

dfa1 and others added 2 commits June 26, 2026 15:21
Swap the `vortex.zstd` encoding off `io.airlift:aircompressor-v3` onto
`io.github.dfa1.zstd:zstd` 0.1 — FFM bindings to the native libzstd.

The encoder and decoder now run fully zero-copy on arena segments via
`ZstdCompressCtx.compress(arena, src)` / `ZstdDecompressCtx.decompress(dst,
src)`, with no heap `byte[]` bounce on either side; the decoder writes each
frame straight into its slice of the scan-arena output. I8/U8 encode input is
now arena-allocated rather than heap-wrapped so it can feed the native call,
and `buildLengthPrefixed` returns an arena segment.

The 0.1 release ships no bundled native; libzstd is loaded from a
`/native/<platform>/` classpath resource carried by a separate
`zstd-native-<platform>` artifact. Two os-activated profiles pull in the
matching native (osx-aarch64, linux-x86_64); add more platforms as needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dfa1 dfa1 merged commit 31c8e73 into main Jun 26, 2026
6 checks passed
@dfa1 dfa1 deleted the deps/zstd-ffm branch June 26, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant