Repository navigation
[native_datafusion] [Spark SQL Tests] Missing files error handling differs from Spark #3314
Description
Summary
2 Spark SQL tests fail because native_datafusion produces different error behavior when files are missing or mutated.
Failing Tests
FileBasedDataSourceSuite: "Enabling/disabling ignoreMissingFiles using parquet" — native_datafusion throws "Object at location ... not found" instead of being silently ignoredSimpleSQLViewSuite: "alter temporary view should follow current storeAnalyzedPlanForView config" — file deleted after plan creation causes native error
Root Cause
nativeDataFusionScan already has a fallback check for ignoreMissingFiles, but the test scenario may not be triggering it (e.g., the config may be set after plan creation, or the fallback condition isn't matching). Native DataFusion's error message format also differs from Spark's expected error handling.
Related
Discovered in CI for #3307 (enable native_datafusion in auto scan mode).
Activity
skip additional native_datafusion test failures in Spark SQL tests
fix: route file-not-found errors through SparkError JSON path
chore: Enable spark SQL tests for issues now resolved in native_dataf…
chore: re-enable Spark SQL tests for fixed issues in 3.5.8 diff
andygrove commentedon Apr 23, 2026
Reopening. After rerunning under Spark 4.0.1 with
native_datafusionin auto scan mode, the following tests still fail:FileBasedDataSourceSuite: "Enabling/disabling ignoreMissingFiles using parquet" —null did not equal "FAILED_READ_FILE.FILE_NOT_EXIST"; native path does not surface the expected Spark error class.SimpleSQLViewSuite: "alter temporary view should follow current storeAnalyzedPlanForView config" —CometQueryExecutionException: FileNotFoundwhen the underlying file is deleted between planning and execution.ParquetV1QuerySuite: "Enabling/disabling ignoreCorruptFiles" —[INTERNAL_ERROR] Found unused message parameters of the error class '_LEGACY_ERROR_TEMP_2254'. (V2 variant passes.)fix: enable Spark 4 SQL tests previously ignored for issues #3313 and #…
test: unignore Spark 4.1.1 tests gated on closed issues apache#4045, a…