Skip to content

Native string to float/double and decimal casts accept strings Spark rejects #6839

Closed

Description

@viirya

Describe the bug

The native string-to-floating-point and string-to-decimal casts accept some strings that Spark rejects. Spark returns NULL for them (or raises CAST_INVALID_INPUT under ANSI), while Comet returns a value. Found while reviewing #6826.

String to FLOAT/DOUBLE. Spark parses with Double.parseDouble / Float.parseFloat and falls back to Cast.processFloatingPointSpecialLiterals. Java only allows a d/D/f/F suffix after a decimal number, and accepts a signed NaN only in the exact case +NaN / -NaN. Comet's parse_string_to_float strips the suffix first and hands the rest to Rust's parser, which accepts nan, inf and infinity in any case, so 'NaNF', 'NaND', 'InfinityD', 'infF', '-nan' and '+nan' all cast to NaN or infinity.

String to DECIMAL. Spark parses with new java.math.BigDecimal(str), which rejects a scale (fractional digits minus exponent) outside the int range. Comet's parse_decimal_str / parse_string_to_decimal compute the scale in i32, which wraps in release builds (overflow checks are off), so '1e-2147483648', '1.0e-2147483647' and '0e-2147483648' cast to 0.00. Debug builds panic with attempt to subtract with overflow instead.

Steps to reproduce

CREATE TABLE t(s string) USING parquet;
INSERT INTO t VALUES ('NaNF'), ('InfinityD'), ('-nan'), ('1e-2147483648');
SELECT s, try_cast(s AS DOUBLE), try_cast(s AS DECIMAL(10,2)) FROM t;

Expected behavior

Spark returns NULL for every cast above. Under ANSI, CAST(s AS DOUBLE) and CAST(s AS DECIMAL(10,2)) raise CAST_INVALID_INPUT.

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions