Skip to content

Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning #5553

Description

@peterxcli

Context

#5177 makes the native Parquet reader's TIMESTAMP_MILLIS -> microseconds conversion checked for top-level columns, matching Spark's millisToMicros (Math.multiplyExact). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark:

  1. Nested fields (struct children, list/map elements): the conversion inside parquet_convert_struct_to_struct / list / map recursion is unchecked.
  2. Scans whose data filters reference nested fields (GetStructField): the whole scan falls back to the safe cast (SparkParquetOptions.checked_timestamp_overflow = false, set in init_datasource_exec).

Why

Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (ParquetFilters supports nested column predicates). DataFusion currently can neither:

  • prune nested-field predicates — datafusion-pruning says "PruningPredicate does not support pruning on nested fields yet" (pruning_predicate.rs), nor
  • evaluate them as Parquet row filters — can_expr_be_pushed_down_with_schemas in datafusion-datasource-parquet/src/row_filter.rs classifies struct columns as non-pushable.

So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177: ts IN (...), ts <=> ..., and s.ts < ... reproducers).

Remaining divergence vs Spark

  • A nested-predicate scan that Spark fails to prune (mixed row-group statistics) throws ArithmeticException in Spark but returns NULL for overflowing top-level values in Comet.
  • A direct read of an overflowing nested TIMESTAMP_MILLIS field throws in Spark but returns NULL in Comet (pre-existing behavior on main).

Proposed work

Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the checked_timestamp_overflow fallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite in SparkPhysicalExprAdapter to GetStructField-wrapped conversions.

Regression coverage lives in ParquetReadSuite:

  • "TIMESTAMP_MILLIS overflow fails in native scan"
  • "TIMESTAMP_MILLIS overflow rows skipped by filter pruning do not fail"

Related: #5517 (error fidelity for the overflow exception).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingbugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions