Skip to content

[Iceberg] Commit taxi iceberg dataset - #594

Open
gabotechs wants to merge 1 commit into
gabrielmusat/upgrade-rust-toolchainfrom
gabrielmusat/add-taxi-iceberg-dataset
Open

[Iceberg] Commit taxi iceberg dataset#594
gabotechs wants to merge 1 commit into
gabrielmusat/upgrade-rust-toolchainfrom
gabrielmusat/add-taxi-iceberg-dataset

Conversation

@gabotechs

@gabotechs gabotechs commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Commits an Iceberg dataset to LFS for future testing. This is the same approach we use today for parquet files, but for Iceberg in this case, the upper PR in this stack will use it for integration testing.


Stack created with GitHub Stacks CLIGive Feedback 💬

@shinzoxD shinzoxD left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One factual fixture/documentation mismatch needs correction: I inspected all 91 Parquet column chunks (7 files × 13 columns) with PyArrow 25; each reports UNCOMPRESSED and parquet-rs 58.3.0, while the README calls the files Zstandard-compressed and shows COMPRESSION zstd. Please either regenerate the committed files with ZSTD or update the README/extraction description to match the actual fixture. The other integrity checks passed: 175,000 rows total, 25,000 per date partition, positive-value filters, field IDs 1–13, and Iceberg v2 manifest paths/counts/sizes all agree.

@gabotechs gabotechs changed the title Commit taxi iceberg dataset [Iceberg] Commit taxi iceberg dataset Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants