[maharanay22]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Resolves #4776

On Spark 4, BETWEEN and NULLIF are RuntimeReplaceable expressions whose replacement is a With expression that keeps the shared operand in a CommonExpressionDef. When replaceWhere targets partition columns, WriteIntoDelta deletes the matching files through DeleteCommand.performDelete, and that path calls DeltaLogUtils.rewritePartitionFilters. The rewrite turns every partition Attribute into Cast(UnresolvedAttribute(partitionValues.<col>)), including the one inside the CommonExpressionDef. Rebuilding the With node then calls nullable on the unresolved attribute, so the write fails with [INTERNAL_ERROR] Invalid call to nullable on unresolved object

This PR inlines With expressions in the partition filters before the attribute rewrite by replacing each CommonExpressionRef with the child of its definition. Partition filters are deterministic, so evaluating the operand more than once is safe. Spark's RewriteWithExpression works on logical plans and can't be applied to a standalone filter expression, so the inlining is done directly

How was this patch tested?

Added a test in DeltaSuite that runs replaceWhere on a partitioned table with part BETWEEN 2 AND 4, part NOT BETWEEN 2 AND 7 and NULLIF(part, 3) IS NULL and checks that only the matching partitions get replaced. It also checks that writing a row outside the BETWEEN range still fails with DELTA_REPLACE_WHERE_MISMATCH. Without the fix the test fails with the internal error from the issue, and with the fix it passes. The change compiles against Spark 4.0, 4.1 and 4.2

Does this PR introduce any user-facing changes?

Yes. replaceWhere with BETWEEN, NOT BETWEEN or NULLIF on partition columns used to fail with [INTERNAL_ERROR] Invalid call to nullable on unresolved object on Delta 4.x with Spark 4. Now it replaces the matching partitions as expected

[maharanay22]

@murali-db @tdas could you approve the CI run and take a look when you have time? Same goes for #7840, #7841 and #7843, thanks