Poor reported performance of DataFusion against DuckDB and Hyper

[alamb]

Describe the bug

There is a blog post that reports relatively poor performance of DataFusion compared to DuckDB and Hyper:

https://www.architecture-performance.fr/ap_blog/tpc-h-benchmark-of-hyper-duckdb-and-datafusion-on-parquet-files/

To Reproduce

I would like someone to try and reproduce the DataFusion performance reported in the blog and propose ways to improve the performance of DataFusion (perhaps by enabling some of the options that are off by default)

Expected behavior

No response

Additional context

@Dandandan suggests on slack that with parallelized scan of a parquet file this benchmark may go faster

[andygrove]

There was a question on slack asking if the Python bindings enable parallel scans.

The Python SessionContext constructor is here. It uses SessionConfig::default if no config is provided, so yes, parallel scans should be enabled if a recent version is being used.

https://github.com/apache/arrow-datafusion-python/blob/main/src/context.rs#L216-L233

[Dandandan]

Looking at https://github.com/apache/arrow-datafusion/blob/main/dev/changelog/19.0.0.md , it should be enabled since 19.0.0, while the article mentions it's using 20.0.0

[Dandandan]

FYI @djfrancesco we are looking into your article.
Is there any code you could share to replicate the results?

[djfrancesco]

@Dandandan thanks for your interest. I just created a public repo with the code used for the post : https://github.com/aetperf/sql_olap_bench
I am sorry because the code is a little messy... I might have use DataFusion in a wrong way. I tried several config options but without success. I write these blog posts mostly as notes to myself. I would be glad to update the post with better timings. The main command is the following:

python tpch_bench.py -d path_to_the_folder_tpch_100

It uses a function called run_queries_datafusion_on_parquet from the bench_tools module.

[alamb]

I am sorry because the code is a little messy... I might have use DataFusion in a wrong way. I tried several config options but without success. I write these blog posts mostly as notes to myself. I would be glad to update the post with better timings. The main command is the following:

No worries -- clearly if you can't get good performance out of DataFusion we have some things to improve (even if it is only UX).

Thank you for the article

[ozankabak]

Anybody has a preliminary idea why the figures look this bad? Mostly config related, algorithmic issues or code structure/performance related?

[alippai]

Related older post (in a sense that trying Datafusion with defaults yields questionable results): #5141

[mingmwang]

Looks like the blog is broken, I can not open it now.

[andygrove]

I took a quick look at the code, and have two observations:

1. The configs being created are not actually being used anywhere i.e. they are not being passed into the context
2. There is a comment that DataFusion does not support multiple Parquet files for a table, which is not true (unless this isn't available via the Python bindings?), but maybe this means the DataFusion tests are not comparable to the other tests if they are being run against one parquet file per table?

[andygrove]

Here is an example of configuring a context in Python:

    runtime = (
        RuntimeConfig().with_disk_manager_os().with_fair_spill_pool(10000000)
    )
    config = (
        SessionConfig()
        .with_create_default_catalog_and_schema(True)
        .with_default_catalog_and_schema("foo", "bar")
        .with_target_partitions(1)
        .with_information_schema(True)
        .with_repartition_joins(False)
        .with_repartition_aggregations(False)
        .with_repartition_windows(False)
        .with_parquet_pruning(False)
        .set("datafusion.execution.parquet.pushdown_filters", "true")
    )

    ctx = SessionContext(config, runtime)

[andygrove]

I created a quick PR to improve the docs around creating a context and will spend more time on this over the next few days. The UX could be improved.

apache/datafusion-python#321

[djfrancesco]

Thanks for your help @andygrove! I tried versions 20 and 21 and got this Error when using a SessionConfig instance:

      config.set("datafusion.execution.parquet.pushdown_filters", "true")
      ^^^^^^^^^^
  AttributeError: 'datafusion.SessionConfig' object has no attribute 'set'

Maybe it is a new feature ?