Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24780 +/- ##
==========================================
+ Coverage 81.53% 81.80% +0.27%
==========================================
Files 1123 1130 +7
Lines 406102 417754 +11652
Branches 406102 417754 +11652
==========================================
+ Hits 331111 341748 +10637
- Misses 55628 55879 +251
- Partials 19363 20127 +764 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
nathanb9
marked this pull request as ready for review
August 30, 2026 19:37
Contributor
|
Thanks @nathanb9 -- I will try and give this a look |
|
Marked as draft after 14 days without activity: merge conflicts. Please address these items and mark Ready for review when ready. |
alxmrs
added a commit
to xqlsystems/xarray-sql
that referenced
this pull request
Oct 28, 2026
DataFusion plans a CTE again at every place that reads it, so a model whose CTEs read each other more than once has a plan that grows exponentially with its depth (#267). The 1-layer loss took 2.7 s at the demo's size and 108 s at nanoGPT's CPU size, and the 2-layer loss ran out of memory while planning. - The GPT's SQL builders return their CTEs as (name, sql) pairs, and every statement starts with the model's and the loss's CTEs, through _hosts.with_query. - Hosts gain query_with(ctes, select) and operators_with. DataFusion's runs each CTE on its own into an in-memory table that later CTEs read, under TODO(apache/datafusion#26237). It uses collect_partitioned and register_record_batches rather than DataFrame.cache(), which einfold can't yet plan a join of (xqlsystems/einfold#93). - The model is no longer the view logits. Creating a DataFusion view writes its CTEs out, so the 3-layer view didn't fit in 8 GB; ddx, given the CTEs inline, computes each shared one once. apache/datafusion#24780's streaming fan-out example doesn't fit these queries: by its own notes it can deadlock when both sides of a hash join read the shared plan, and doesn't nest. DataFusion 55 parses `AS MATERIALIZED` only in its PostgreSQL dialect, and ignores it. With ddxdb 0.4.4, at batch 4, on an Intel i7-8700: the 2-layer loss takes 0.07 s on DataFusion, and a 6-layer SGD step 2.4 s, against 0.30 s and 5.5 s on DuckDB; every step matches JAX. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
motivation
how the example works
StreamingFanoutExecexecutes the input once and serves consumer 0.StreamingFanoutReaderExecnodes serve additional consumers through bounded channels.extension points used
UserDefinedLogicalNodeCoreQueryPlannerandExtensionPlannerPhysicalOptimizerRuleExecutionPlannodes