Repository navigation
DataFusion 55: plans that repartition can hang under a small memory pool — draft for upstream #71
Description
🪤 Claude (fuzz adversary)
DataFusion 55.2: a physical plan, reset with reset_plan_states and run again under a small memory pool, can hang. Plain DataFusion, no einfold. Found by #65's soak watchdog ([stall]): a case ran past 60 s, and a stage trace put it in the host's [rerun]. The first run succeeds; the rerun never finishes. Per AGENTS.md it's filed here first, for a maintainer to report upstream. I found no upstream issue about a reset plan hanging. The nearest are about state that reset_state fails to clear (apache/datafusion#19905, #26054, #26229).
Reproduce
From #65's branch:
EINFOLD_SEED=70019748 cargo test -p einfold-datafusion --test simulation --release \
-- --ignored --nocapture host_rerun_hanghost_rerun_hang builds that seed's tables and query, then runs plain DataFusion with dynamic filter pushdown off (reset_plan_states documents that it doesn't support plans with dynamic filters). For each setting it plans, collects, resets with reset_plan_states, collects again, and reports a rerun still running after 3 s. With 2 partitions, batch size 1 and an 8 KB FairSpillPool, 3 reruns in 8 hang. The query is a left join from d onto a fold of l and r, grouped again, over DOUBLE keys with NaNs and both zeros (seed 70019748's SQL and tables print with EINFOLD_SEED=70019748 cargo test … -- --nocapture simulation). The same tables as one-partition VALUES didn't hang, so the inputs' partitioning and small batches matter.
Why (unknown)
I haven't found the cause. A rerun that hangs only under memory pressure points at state that survives the reset and is waited on: a spill file, a memory reservation, or a channel in a RepartitionExec, as in the spill-pool deadlock fixed for 55.2 (apache/datafusion#24883). reset_plan_states is what RecursiveQueryExec calls between iterations, so a recursive query under a memory limit is the likely way to meet this outside a test. I didn't confirm that path.
For einfold
Nothing in einfold causes it. #65 steps around it with Known::ResetRerunUnderPool (no [rerun] under a memory pool), and keeps host_rerun_hang as the reproduction.
Activity
Rerun plans without dynamic filters, trace stages, and step around th…
[-]DataFusion 55: a plan reset with reset_plan_states can hang when run again under a small memory pool — draft for upstream[/-][+]DataFusion 55: plans that repartition can hang under a small memory pool — draft for upstream[/+]alxmrs commentedon Oct 28, 2026
🪤 Claude (fuzz adversary)
Correction: the hang isn't specific to
reset_plan_states. A second seed hangs on a query's first run. That makes the rerun just one more chance to hit the same hang. I've retitled the issue.Seed 95007199's query is a fold of a fold: two nested
GROUP BYs over joins,SELECT f.g0, d.a, SUM(f.v * d.v) FROM (SELECT l.a AS g0, r.b AS g1, SUM(l.v * r.v) AS v FROM l JOIN (SELECT * FROM r WHERE r.a IS NOT NULL) r ON (l.b IS NOT DISTINCT FROM r.a) GROUP BY r.b, l.a) f JOIN d ON (f.g1 IS NOT DISTINCT FROM d.b) GROUP BY f.g0, d.a. Plain DataFusion, run once per session, 20 sessions per setting, 5 s timeout:FairSpillPoolSo it takes a memory pool and repartitioning, at 2 partitions with small batches. Under gdb, every thread is idle when it hangs: both Tokio workers parked, the caller in
block_on. That's a future pending with no one left to wake it, a lost wakeup rather than a lock. With repartitioning required, my best guess is stillRepartitionExecspilling under memory pressure, a variant of apache/datafusion#24883 ("RepartitionExec deadlocks under a memory limit when its spill pool holds two open files"), whose fix is in 55.2. I haven't confirmed it.Reproduce from #65's branch (
host_hangprints the table above):EINFOLD_SEED=95007199 cargo test -p einfold-datafusion --test simulation --release -- --ignored --nocapture host_hang#65 now steps around it more widely (
Known::ResetRerunUnderPool): cases with a pool run without repartitioning, and skip[rerun].Step around #71 wherever a pool meets repartitioning, and reproduce i…
Rerun plans without dynamic filters, trace stages, and step around th…
Step around #71 wherever a pool meets repartitioning, and reproduce i…
Rerun plans without dynamic filters, trace stages, and step around th…
Step around #71 wherever a pool meets repartitioning, and reproduce i…
Rerun plans without dynamic filters, trace stages, and step around th…
Step around #71 wherever a pool meets repartitioning, and reproduce i…
Add a simulation soak of einfold in DataFusion, run nightly (#65)