You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
DuckDB: a union branch selecting one column twice panics in prune or refuses #237
Since #229, on DuckDB, a union branch that selects the same column twice either panics or refuses, depending on where the branch sits. DataFusion gives correct gradients for all three queries below.
importduckdb, ddxdb.duckdbasddcon=duckdb.connect()
con.execute("CREATE TABLE a AS SELECT * FROM (VALUES (0, 1, 0.5::DOUBLE), (1, 2, -1.0::DOUBLE)) t(r, i, x)")
con.execute("CREATE TABLE b AS SELECT * FROM (VALUES (0, 1, 2.0::DOUBLE), (1, 3, 1.5::DOUBLE)) t(r, i, x)")
con.execute("CREATE TABLE w AS SELECT * FROM (VALUES (1, 1.5::DOUBLE), (2, 0.5::DOUBLE)) t(i, val)")
g=" SELECT r, grad(loss, x) AS g FROM a"# 1. panics: index out of bounds: the len is 2 but the index is 2 (crates/ddx-ad/src/opt/prune.rs:322)dd.sql(con, """WITH u AS (SELECT i AS k, x AS v FROM a UNION ALL SELECT k, v FROM (SELECT i AS k, i AS v FROM b UNION ALL SELECT i AS k, i AS v FROM b) n), loss AS (SELECT SUM(u.v * w.val * u.v) AS l FROM u JOIN w ON u.k = w.i)"""+g)
# 2. invalid Substrait plan: field 1 is referenced, but the input has only 1 columnsdd.sql(con, """WITH u AS (SELECT k, v FROM (SELECT i AS k, i AS v FROM b UNION ALL SELECT i AS k, i AS v FROM a) n UNION ALL SELECT i AS k, x AS v FROM a), loss AS (SELECT SUM(t * t) AS l FROM (SELECT k, SUM(v) AS t FROM u GROUP BY k) g)"""+g)
# 3. invalid Substrait plan: a UNION ALL whose inputs have different numbers of columnsdd.sql(con, """WITH u AS (SELECT i AS k, x AS v, x AS v2 FROM a UNION ALL SELECT i AS k, x AS v, 1.0 AS v2 FROM b), loss AS (SELECT SUM(v * v2) AS l FROM u)"""+g)
A flat union with an i AS k, i AS v branch works, and so does a repeated column with no union. My guess: DuckDB's producer writes a repeated column once and repeats it in the projection's emit. The union rewrite, or pruning after it, then counts the input's columns before the emit rather than after. Case 1 is a panic: in Python it raises PanicException, which derives from BaseException, so except Exception doesn't catch it.
Found by fuzzing UNION ALL on both engines. In 900 random cases this failed 9 times, 2 of them as panics.
Activity
fix(ddx-ad): refuse a plan whose producer dropped a column beneath it…
fix(ddx-ad): refuse a plan whose producer dropped a column beneath it…