Skip to content

[native_iceberg_compat] VariantShreddingSuite test failures with Spark 4.0.0 #2209

Description

@andygrove

Describe the bug

When switching from native_comet to native_iceberg_compat in Spark 4.0.0, I see some tests fail in VariantShreddingSuite. It appears to be an issue with selecting fields by index from within structs.

[info]   == Optimized Logical Plan ==
[info]   Project [v#216.0 AS variant_get(v, $.a)#211, v#216.1 AS variant_get(v, $.b)#212L, v#216.2 AS variant_get(v, $.c)#213, v#216.3 AS variant_get(v, $.d)#214]
[info]   +- Relation [v#216] parquet
[info]   
[info]   == Physical Plan ==
[info]   *(1) CometColumnarToRow
[info]   +- CometProject [variant_get(v, $.a)#211, variant_get(v, $.b)#212L, variant_get(v, $.c)#213, variant_get(v, $.d)#214], [v#216.0 AS variant_get(v, $.a)#211, v#216.1 AS variant_get(v, $.b)#212L, v#216.2 AS variant_get(v, $.c)#213, v#216.3 AS variant_get(v, $.d)#214]
[info]      +- CometScan [native_iceberg_compat] parquet [v#216] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/apache-spark-4.0.0/target/tmp/spark-5897c3a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<v:struct<0:int,1:bigint,2:double,3:decimal(9,4)>>
[info]   
[info]   == Results ==
[info]   
[info]   == Results ==
[info]   !== Correct Answer - 3 ==   == Spark Answer - 3 ==
[info]   !struct<>                   struct<variant_get(v, $.a):int,variant_get(v, $.b):bigint,variant_get(v, $.c):double,variant_get(v, $.d):decimal(9,4)>
[info]   ![1,2,3.3,4.4]              [null,null,null,null]
[info]   ![5,-6,null,null]           [null,null,null,null]
[info]    [null,null,null,null]      [null,null,null,null]

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingnative_iceberg_compatSpecific to native_iceberg_compat scan typepriority:criticalData corruption, silent wrong results, security issuesspark 4.0

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions