Skip to content

DuckDB: a union branch selecting one column twice panics in prune or refuses #237

Closed

Description

@alxmrs

😈🧪 Adversarial tester

Since #229, on DuckDB, a union branch that selects the same column twice either panics or refuses, depending on where the branch sits. DataFusion gives correct gradients for all three queries below.

Repro (main at a223f33, duckdb 1.5.6):

import duckdb, ddxdb.duckdb as dd
con = duckdb.connect()
con.execute("CREATE TABLE a AS SELECT * FROM (VALUES (0, 1, 0.5::DOUBLE), (1, 2, -1.0::DOUBLE)) t(r, i, x)")
con.execute("CREATE TABLE b AS SELECT * FROM (VALUES (0, 1, 2.0::DOUBLE), (1, 3, 1.5::DOUBLE)) t(r, i, x)")
con.execute("CREATE TABLE w AS SELECT * FROM (VALUES (1, 1.5::DOUBLE), (2, 0.5::DOUBLE)) t(i, val)")
g = " SELECT r, grad(loss, x) AS g FROM a"
# 1. panics: index out of bounds: the len is 2 but the index is 2 (crates/ddx-ad/src/opt/prune.rs:322)
dd.sql(con, """WITH u AS (SELECT i AS k, x AS v FROM a
                UNION ALL SELECT k, v FROM (SELECT i AS k, i AS v FROM b UNION ALL SELECT i AS k, i AS v FROM b) n),
     loss AS (SELECT SUM(u.v * w.val * u.v) AS l FROM u JOIN w ON u.k = w.i)""" + g)
# 2. invalid Substrait plan: field 1 is referenced, but the input has only 1 columns
dd.sql(con, """WITH u AS (SELECT k, v FROM (SELECT i AS k, i AS v FROM b UNION ALL SELECT i AS k, i AS v FROM a) n
                UNION ALL SELECT i AS k, x AS v FROM a),
     loss AS (SELECT SUM(t * t) AS l FROM (SELECT k, SUM(v) AS t FROM u GROUP BY k) g)""" + g)
# 3. invalid Substrait plan: a UNION ALL whose inputs have different numbers of columns
dd.sql(con, """WITH u AS (SELECT i AS k, x AS v, x AS v2 FROM a UNION ALL SELECT i AS k, x AS v, 1.0 AS v2 FROM b),
     loss AS (SELECT SUM(v * v2) AS l FROM u)""" + g)

A flat union with an i AS k, i AS v branch works, and so does a repeated column with no union. My guess: DuckDB's producer writes a repeated column once and repeats it in the projection's emit. The union rewrite, or pruning after it, then counts the input's columns before the emit rather than after. Case 1 is a panic: in Python it raises PanicException, which derives from BaseException, so except Exception doesn't catch it.

Found by fuzzing UNION ALL on both engines. In 900 random cases this failed 9 times, 2 of them as panics.

🤖 Generated with Claude Code

https://claude.ai/code/session_012NE18ox5ivwTc7ZSbHZUGu

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions