What it buys: a recipe over files plans without a blocked planner
thread. The file reader is the one place in the crates that waits on a
future from inside a synchronous planner callback; with it gone,
block_on greps to zero in the tree and a file recipe plans the way
every other statement does.
What stands: the file reader is a table function, and
TableFunctionImpl::call_with_args is synchronous (datafusion-session
table.rs). Schema inference is not: ListingTableConfig::infer_schema
lists the URL and reads file headers from the object store. The reader
bridges the two with block_in_place around Handle::block_on
(crates/import/src/lib.rs, ReadFiles::call_with_args, the
infer_schema call). block_in_place requires the multi-thread
runtime and takes a worker out of the pool for the length of the
listing and the header reads.
The engine's own shape for this is to resolve what is asynchronous
before the synchronous planner runs: statement_to_plan walks the
statement for table references, awaits each, and only then plans
(datafusion session_state.rs). The session already has a pre-pass of
that shape for the shipped reads (crates/session/src/prepass.rs). A
recipe's SQL names its files as literal arguments of read_csv,
read_parquet and read_json, so the same walk finds them.
The config must keep inferring its own schema rather than being handed
one: a schema that is specified makes ListingTable decline the
session's statistics cache (datafusion-catalog-listing table.rs,
SchemaSource).
Done when: a walk over the recipe's statement resolves each file
call's ListingTable before planning, the table function answers from
what the walk resolved, block_on and block_in_place appear nowhere
in crates/import/src/lib.rs outside the blocking-driver calls, and
the import suite passes unchanged.
What it buys: a recipe over files plans without a blocked planner
thread. The file reader is the one place in the crates that waits on a
future from inside a synchronous planner callback; with it gone,
block_ongreps to zero in the tree and a file recipe plans the wayevery other statement does.
What stands: the file reader is a table function, and
TableFunctionImpl::call_with_argsis synchronous (datafusion-sessiontable.rs). Schema inference is not:ListingTableConfig::infer_schemalists the URL and reads file headers from the object store. The reader
bridges the two with
block_in_placearoundHandle::block_on(
crates/import/src/lib.rs,ReadFiles::call_with_args, theinfer_schemacall).block_in_placerequires the multi-threadruntime and takes a worker out of the pool for the length of the
listing and the header reads.
The engine's own shape for this is to resolve what is asynchronous
before the synchronous planner runs:
statement_to_planwalks thestatement for table references, awaits each, and only then plans
(datafusion
session_state.rs). The session already has a pre-pass ofthat shape for the shipped reads (
crates/session/src/prepass.rs). Arecipe's SQL names its files as literal arguments of
read_csv,read_parquetandread_json, so the same walk finds them.The config must keep inferring its own schema rather than being handed
one: a schema that is specified makes
ListingTabledecline thesession's statistics cache (datafusion-catalog-listing
table.rs,SchemaSource).Done when: a walk over the recipe's statement resolves each file
call's
ListingTablebefore planning, the table function answers fromwhat the walk resolved,
block_onandblock_in_placeappear nowherein
crates/import/src/lib.rsoutside the blocking-driver calls, andthe import suite passes unchanged.