Skip to content

native_datafusion: no error thrown for schema mismatch when reading Parquet with incompatible types #3720

Closed
#4229
Closed
native_datafusion: no error thrown for schema mismatch when reading Parquet with incompatible types#3720
#4229
Assignees
andygrove
Labels
native_datafusionSpecific to native_datafusion scan type
Milestone
0.17.0

Description

@andygrove
andygrove
opened on Mar 17, 2026

Description

When running Spark SQL tests with spark.comet.scan.impl=native_datafusion, several tests that expect errors on schema mismatch pass silently — DataFusion reads the data without throwing an exception, while Spark would throw a SparkException.

This affects cases where the read schema is incompatible with the physical Parquet schema:

  • Reading binary data as timestamp
  • Reading int as long
  • Reading TimestampLTZ as TimestampNTZ
  • Reading decimals with incompatible precision/scale
  • Reading int as bigint (row group skipping overflow)
  • Schema mismatch on vectorized reader (e.g., string column read as int)
  • Reading timestamp_ntz as array<timestamp_ntz>

DataFusion's Parquet reader is more permissive than Spark's and silently coerces or reads mismatched types instead of erroring.

Affected Tests

From ParquetIOSuite:

  • "SPARK-35640: read binary as timestamp should throw schema incompatible error"
  • "SPARK-35640: int as long should throw schema incompatible error"

From ParquetQuerySuite (via ParquetV1QuerySuite/ParquetV2QuerySuite):

  • "SPARK-36182: can't read TimestampLTZ as TimestampNTZ"
  • "SPARK-34212 Parquet should read decimals correctly"
  • "row group skipping doesn't overflow when reading into larger type"

From ParquetSchemaSuite:

  • "schema mismatch failure error message for parquet vectorized reader"
  • "SPARK-45604: schema mismatch failure error on timestamp_ntz to array<timestamp_ntz>"

Expected Behavior

DataFusion should detect schema incompatibilities and throw appropriate errors, matching Spark's behavior of rejecting incompatible type reads.

Parent Issue

Split from #3311.

Activity

  1. andygrove
    added 2 commits that reference this issue on Mar 17, 2026

    fix: enable native_datafusion Spark SQL tests for apache#3320, apache…

    82acd9c

    fix: enable native_datafusion Spark SQL tests for apache#3320, apache…

    5ee8e78

  2. andygrove commented on Mar 18, 2026

    @andygrove
    andygrove
    on Mar 18, 2026
    MemberAuthor

    Note that Spark 4 actually allows type promotion, so is closer to native_datafusion behavior.

    apache/spark#44368

  3. andygrove
    assigned
    vaibhawvipul
    on Mar 22, 2026
  4. andygrove
    added 3 commits that reference this issue on Mar 22, 2026

    fix: retag schema mismatch tests from apache#3311 to apache#3720

    5ccfc3b

    fix: retag schema mismatch tests from apache#3311 to apache#3720 in 3…

    e9502bd

    fix: replace all apache#3311 references with specific issues in 3.5.8…

    6efaa55
  5. mbutrovich
    added a commit that references this issue on Mar 23, 2026

    fix: replace all #3311 references with specific issues in 3.5.8 diff (#…

    cfa6e69
  6. vaibhawvipul
    added a commit that references this issue on Mar 24, 2026

    Version-aware schema validation and enable all apache#3720 tests in S…

    780e091

  7. parthchandra commented on Mar 24, 2026

    @parthchandra
    parthchandra
    on Mar 24, 2026
    Contributor

    See TypeUtil.checkParquetType for how primitive type checking is handled in native_iceberg_compat. There are additional checks for complex types in NativeBatchReader.checkParquetType

  8. vaibhawvipul
    added a commit that references this issue on Apr 4, 2026

    fix: detect schema incompatibilities in native_datafusion Parquet scan (

    30e946a

  9. vaibhawvipul commented on Apr 4, 2026

    @vaibhawvipul
    vaibhawvipul
    on Apr 4, 2026
    Contributor

    raised Fresh PR - #3902

    I closed the previous one as it got too messy to resolve merge conflicts and also I found a better approach.

  10. vaibhawvipul
    added a commit that references this issue on Apr 9, 2026

    fix: detect Parquet schema incompatibilities in native_datafusion sca…

    9bdc45f
  11. andygrove
    added a commit that references this issue on Apr 25, 2026

    docs: clarify silent Parquet schema coercion in native_datafusion

    4bcfc2b

  12. andygrove commented on Apr 25, 2026

    @andygrove
    andygrove
    on Apr 25, 2026
    MemberAuthor

    Added a permanent suite documenting the per-case, per-scan-impl, per-Spark-version behavior in #4087 (ParquetSchemaMismatchSuite). The suite encodes current behavior rather than fixing it, so the issue stays open. Highlights from running the matrix locally on Spark 3.4 / 3.5 / 4.0:

    Case                                   Spark 3.4  3.5    4.0    Comet native_datafusion              Comet native_iceberg_compat
1. BINARY -> TIMESTAMP                 throw      throw  throw  throw                                throw
2. INT32 -> INT64                      throw      throw  OK     OK (widened values)                  throw on 3.x / OK on 4.0
3. INT96 LTZ -> TIMESTAMP_NTZ          throw      throw  throw  OK (silent, possible wall-clock diff) throw on 3.x / OK on 4.0
4. Decimal(10,2) -> Decimal(5,0)       throw      throw  throw  OK (reads, values unverified)        throw
5. INT32 -> INT64 w/ rowgroup filter   throw      throw  OK     OK (1 row, no overflow)              throw on 3.x / OK on 4.0
6. STRING -> INT                       throw      throw  throw  OK (garbage values)                  throw
7. TIMESTAMP_NTZ -> ARRAY<...>         throw      throw  throw  throw                                throw
C1. INT8 -> INT32 (control)            OK         OK     OK     OK (widened values)                  OK (widened values)
C2. FLOAT -> DOUBLE (control)          OK         OK     OK     OK (widened values)                  throw on 3.x / OK on 4.0

    Observations relevant to the original issue list:

    • Binary read as timestamp: both scan impls now throw SparkException (case 1). native_datafusion raises CometNativeException ("column types must match schema types"), native_iceberg_compat raises SchemaColumnConvertNotSupportedException. Behavior aligns with Spark on all versions.
    • INT32 read as INT64: native_datafusion widens silently on all Spark versions (matches Spark 4.0, diverges from Spark 3.x). native_iceberg_compat throws on 3.x and accepts on 4.0 because COMET_SCHEMA_EVOLUTION_ENABLED defaults to true on 4.0.
    • TimestampLTZ read as TimestampNTZ: native_datafusion cannot detect the mismatch (INT96 carries no timezone metadata) and silently reads, possibly with a wrong wall-clock value. native_iceberg_compat throws on 3.x, but on 4.0 the isSpark40Plus guard in TypeUtil.checkParquetType bypasses the INT96 check and the read silently succeeds.
    • Decimal precision/scale (10,2 -> 5,0): native_datafusion succeeds without value validation (returned values were not verified by the test); native_iceberg_compat throws on all versions.
    • String read as INT: native_datafusion silently reinterprets the BINARY bytes of each string as INT32 garbage values without throwing on any Spark version. This is a real correctness gap (returns wrong answers without an error). native_iceberg_compat throws.
    • timestamp_ntz read as array<timestamp_ntz>: both scan impls throw on all versions (matches Spark).
    • Row group skipping regression guard (case 5): native_datafusion returns the correct single row when reading INT32 as INT64 with a filter constant beyond INT32 max, so no overflow occurs.

    Two cases worth follow-up issues if not already tracked:

    1. native_datafusion reading STRING as INT silently returns garbage. The vectorized native reader does not validate that the requested type is byte-compatible with the physical column.
    2. native_datafusion reading higher-precision decimal as lower-precision decimal succeeds without value validation; whether the values are correct under truncation/rounding rules has not been verified.

    The suite will surface either of these via test failure if the underlying behavior changes (intentional fix or accidental regression). When a fix lands, update the affected test(s) and the matrix in the same PR.

  13. 2 remaining items

  14. andygrove
    added this to the 0.16.0 milestone on Apr 27, 2026
  15. andygrove
    added 7 commits that reference this issue on Apr 29, 2026

    fix: fall back to Spark for TimestampType/TimestampNTZType schema mis…

    751af1e

    fix: add timestampNTZSafetyCheck to fall back to Spark for TimestampN…

    2c6bd00

    fix: fall back to Spark for Parquet scans containing TimestampNTZ

    9302868

    add suite to CI, update spark diff issue links for INT96→NTZ

    4f8a2b1

    docs: add type widening limitations for Spark 3.4/3.5

    adaeedd

    fix: format type-promotion error message to match Spark

    715b64b

    fix: unignore passing 3720 tests across all spark version diffs

    ce4adc4
  16. andygrove
    modified the milestones: 0.16.0 (May 2026), 0.17.0 (June 2026) on May 13, 2026
  17. andygrove
    added 2 commits that reference this issue on May 15, 2026

    docs: split umbrella apache#3720 into specific issues for type-promot…

    e7269c9

    fix: reject Parquet conversions Spark rejects (apache#4297, apache#4343…

    80836b1
  18. andygrove
    added
    native_datafusionSpecific to native_datafusion scan type
    on May 16, 2026
  19. andygrove
    added a commit that references this issue on May 16, 2026

    test: drop apache#3720 references from spark diffs in favor of specif…

    fa7a9e2
  20. andygrove
    closed this as completedin #4229on May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

native_datafusionSpecific to native_datafusion scan type

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions