unbase64 can fail on rows skipped by LIMIT and semi/anti joins

[sunchao]

Describe the bug

Comet can fail while decoding malformed Base64 on rows that Spark never evaluates. A limit may stop after the first row, or a semi/anti join may stop checking candidates after finding a match. Both native unbase64 and the JVM codegen dispatcher can evaluate a whole batch before that early termination takes effect.

This means enabling Comet can turn a successful Spark query into Last unit does not have enough valid bits, even when the malformed value does not contribute to the result.

Steps to reproduce

Start Spark with the Comet extension, native execution, and the Comet shuffle manager configured. Create the input with Comet disabled, then compare the same query with Comet disabled and enabled:

SET spark.comet.enabled = false;
SET spark.sql.shuffle.partitions = 1;

CREATE TABLE unbase64_limit_input USING parquet AS
SELECT /*+ COALESCE(1) */ bad
FROM VALUES ('YWJj'), ('A') AS v(bad);

SELECT hex(unbase64(bad)) FROM unbase64_limit_input LIMIT 1;

SET spark.comet.enabled = true;

SELECT hex(unbase64(bad)) FROM unbase64_limit_input LIMIT 1;

In the reproduced single-file scan order, YWJj is read before A. Spark returns 616263 (the bytes for abc). Comet instead throws on the second value. This example relies on that physical input order; it does not assume that SQL without ORDER BY guarantees ordering.

The failing Comet plan is:

CometCollectLimit 1
+- CometProject [hex(unbase64(bad))]
   +- CometNativeScan parquet

The same problem occurs when the decoder is in a filter below LIMIT, in semi/anti join residual conditions, or below an already ordered WindowGroupLimit. Compound inputs and to_binary(bad, 'base64') can reach the JVM dispatcher and still fail for the same reason.

Expected behavior

Match Spark's evaluation behavior: preserve successful queries when Spark skips the malformed value, while still raising an error when Spark actually decodes it. Ordinary native decoding should remain available when early termination cannot skip the value.

Additional context

Reproduced on Apache Comet 78defce020ac3c4d2b32ff7f172f3018b7587a48 with Spark 4.1.3, a fresh native/JVM build, and verified Comet physical operators. The relevant implementation is unchanged at a223ba14e9f1445b5d315962ae9005f7ec45b187.

Seven query shapes produced 28 Spark-versus-Comet mismatches across all four combinations of ANSI mode and AQE. Sixteen comparisons using valid inputs or deliberately consumed malformed values matched Spark. This is not specific to ANSI mode, and switching to JVM codegen dispatch alone does not preserve row-by-row evaluation.