IN evaluates later candidates over the whole batch, so an ANSI error can fire for a row that already matched #6823

Describe the bug

Spark's In evaluates the candidates of each row in order and stops at the first one that matches. Comet's IN does not: DataFusion's InListExpr::evaluate, for a list it does not build as a static filter, and Comet's NestedPredicate evaluate each candidate over the whole batch, and stop only once every row of the batch has matched. A later candidate therefore runs for rows that an earlier candidate already matched. When that candidate can fail, as many expressions can with ANSI mode on, Comet raises an error for a row that Spark never evaluates it for.

Steps to reproduce

With spark.sql.ansi.enabled=true, and both rows in one file so that they reach the same batch:

CREATE TABLE t (id BIGINT, s STRING) USING parquet;
INSERT INTO t SELECT /*+ REPARTITION(1) */ * FROM VALUES (0L, 'bad'), (1L, '1') AS v(id, s);
SELECT id, id IN (0L, CAST(s AS BIGINT)) FROM t;

Spark returns [0, true], [1, true]. Comet fails with CAST_INVALID_INPUT for 'bad', which is only in the row where id = 0, a row that 0L already matched.

Since #6763, a CASE candidate that reads a column takes the same path, so SELECT id, id IN (0L, CASE WHEN id = 0 THEN CAST(s AS BIGINT) WHEN id = 2 THEN id END) FROM t fails the same way.

Expected behavior

The same results as Spark: a candidate is evaluated only for the rows that no earlier candidate matched, or a query whose candidates can fail falls back to Spark.

Additional context

Spark 4 runs with ANSI mode on by default, so this affects default configurations. This is one instance of the data-dependent errors tracked in #6006. Found while reviewing #6763.

Willingness to contribute

I can contribute a fix for this bug independently