Skip to content

Support row group skipping for shredded variant columns #15510

Open

Description

@qlong

Feature Request / Improvement

Iceberg supports writing shredded variant columns to Parquet (PR #14297).

For a query like WHERE variant_get(v, '$.price', 'int') > 100, Iceberg can already skip Parquet files using manifest-level bounds, where the lower/upper bounds for $.price are stored as a Variant-encoded object on the v column's manifest entry (PR #15385).

For better performance, Iceberg should also skip row groups within a file: when $.price is shredded, the physical Parquet column v.typed_value.price.typed_value has standard Parquet min/max statistics. Iceberg should use those stats to skip row groups where max < 100, the same way ParquetMetricsRowGroupFilter handles regular column predicates.

The change is in parquet module, and should work with all engines.

Scope

  • Add BoundExtract handling to ParquetMetricsRowGroupFilter
  • Map variant path → physical shredded Parquet column
  • Unshredded variant columns are not in scope (binary stats are not useful for path-based filtering)
  • Test with Spark

Query engine

None

Willingness to contribute

  • I can contribute this improvement/feature independently
  • I would be willing to contribute this improvement/feature with guidance from the Iceberg community
  • I cannot contribute this improvement/feature at this time

Activity

  1. added
    improvementPR that improves existing functionality
    on Mar 4, 2026
  2. steveloughran commented on Apr 23, 2026

    @steveloughran
    Contributor

    i will have a go at this

  3. added 3 commits that reference this issue on Apr 27, 2026
  4. added 3 commits that reference this issue on May 11, 2026
  5. soumilshah199500 commented on May 19, 2026

    @soumilshah199500

    Hi there, just wanted to inquire when we plan to merge this. I wanted to rerun the tests again. as filters on shredding variant table are very slow

  6. added a commit that references this issue on May 21, 2026
  7. steveloughran commented on May 26, 2026

    @steveloughran
    Contributor

    @soumilshah199500 yes they are. As for schedule, lots to work on. See : https://steveloughran.github.io/benchmarking-variants/

    Some things we would love to have here are

    1. real world datasets (sanitized, obviously)
    2. example queries you want to run over them
  8. added a commit that references this issue on May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvementPR that improves existing functionality

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions