Track evaluation masks for data-dependent errors beyond unbase64 operator fallback

[sunchao]

Problem and scope

Track Spark-compatible evaluation of expressions that can raise a data-dependent error when an enclosing operator or expression skips some input rows. The expression can return the correct result or error when called in isolation and still change a query's outcome if a batch evaluator calls it on rows Spark never evaluates.

#5532 demonstrates this for unbase64 under limits and semi/anti joins. #5533 adds an explicit serde policy, RequiresSparkEvaluationMask, for selected operator-level protections. Only UnBase64 opts in. The policy is deliberately not a complete classification of throwing expressions, and its presence does not prove every enclosing evaluation context is protected.

The review discussion identified the wider family: ANSI arithmetic overflow, invalid ANSI casts, decimal divide-by-zero, and element_at, plus per-row AND and CASE evaluation. This issue tracks validating and resolving that family; it does not claim that every expression/context combination below is currently broken. In particular, existing native conditional evaluation and targeted serde guards may already preserve a given mask.

Work to track

• Establish which expression instances can raise, accounting for eval mode, input types, nullable children, and Spark version. Start with ANSI arithmetic, ANSI casts, decimal division, array element_at, and unbase64. Determinism alone does not establish safety. Do not classify all casts or all element_at forms identically.
• Audit operator-level early termination: limits, already-ordered top-K/window limits, semi/anti join residuals, and empty-input join behavior. Check the actual consumption of each physical strategy. An intervening blocking sort, exchange, or aggregate can change which rows Spark evaluates.
• Audit per-row masks independently of operator masks. Representative shapes include WHERE k = 1 AND unbase64(bad) = X'616263' and CASE WHEN k = 1 THEN unbase64(bad) END, with malformed input only in excluded rows. Verify native execution, predicate placement/reordering, conditional branch selection, nullable operands, and whole-tree JVM dispatch before classifying a case as a mismatch.
• Audit aggregate FILTER separately. For example, determine whether max(unbase64(bad)) FILTER (WHERE k = 1) evaluates the argument only for qualifying rows on each execution path. Aggregate input evaluation is distinct from final-result evaluation beneath an outer limit; fix: preserve skipped unbase64 rows #5533 does not fix the aggregate FILTER case.
• Choose the compatibility/performance approach deliberately: conditional evaluation in native kernels, scoped fallback, or an explicitly documented divergence where a fix is deferred. Expanding a fallback policy to common ANSI arithmetic/casts can affect many valid-data queries, so measure the fallback footprint and end-to-end cost before broader enrollment.

Validation and completion criteria

• For each confirmed mismatch, record Spark/Comet versions and commits, configuration, executed physical plans, and expected versus observed rows/errors. Native execution and JVM dispatch should be tested separately; switching to batch JVM dispatch is not by itself proof of Spark's row-by-row consumption.
• Pair a skipped-invalid-input case with valid-input native controls and a case that deliberately consumes the invalid value and must raise. Keep any required physical input order explicit; SQL without ORDER BY does not guarantee a row order.
• Cover ANSI/legacy/try distinctions where relevant, AQE and reused stages, and supported Spark versions. Include already-correct conditional and element_at guards as regression controls.
• Add focused regressions for confirmed gaps, document remaining limitations in the compatibility guide, and keep RequiresSparkEvaluationMask enrollment and its scope explicit. Do not expand the policy simply because an expression is potentially throwing.

Related work

• #5532 / #5533: reproduced unbase64 operator-consumption mismatch and its scoped fix.
• #5078: per-expression ANSI behavior/error-fidelity audit. This tracking issue adds enclosing evaluation contexts; it does not replace the individual error-class issues.
• #5766: preserves element_at NULL short-circuiting for deterministic operands and declines nullable nondeterministic operands under ANSI. #5781 separately tracks repeated evaluation of stateful operands in other NULL guards.
• #5218 and #5608: fixed dispatcher NULL short-circuit cases that suppressed errors Spark would raise, the opposite outcome direction from eager evaluation of skipped rows.
• #5328: fallback-invariance testing provides a useful additional oracle, with execution-path assertions to avoid vacuous comparisons.
• #5962: moving reusable expressions before broadcast materialization also requires proving that error evaluation is preserved.