Add dfbench statistics command (#23975)

## Which issue does this PR close?

- Part of #8227

## Rationale for this change

Provide a repeatable way to measure how closely planning cardinality
statistics match runtime output for benchmark query suites.

The goal is not to have an automated benchmark suite, and not something
to enforce in the CI. Instead, this is meant to be a development tool
for developers to locally iterate over stats estimation improvements.

## What changes are included in this PR?

`dfbench statistics` accepts any Parquet data directory and SQL file or
directory, so it can be used with TPC-DS, TPC-H, or another compatible
query suite. It:

- prints an indented, per-operator estimate-versus-runtime report as
each query completes;
- reports q-error per operator and finite q-error p50/p75/p95/p99 across
the run;
- stores results by branch, comparing with the previous run by default
or a named branch using `--compare`.

> About q-error calculation: https://vldb.org/pvldb/vol9/p204-leis.pdf

### Example: verify a TPC-DS Q21 improvement

1. Run TPC-DS Q21 for storing the baseline

```shell
cargo run --bin dfbench statistics \
  --path benchmarks/data/tpcds_sf1 \
  --query_path datafusion/core/tests/tpc-ds \
  --query 21
```

2. Perform this small change in
https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/sorts/sort.rs#L1427-L1427:

```rust
-        Ok(Arc::new(stats.with_fetch(self.fetch, 0, 1)?))
+        Ok(Arc::new(stats.with_fetch(
+            self.fetch,
+            0,
+            self.topk_emitter_count(),
+        )?))
```

3. Run again the command from step 1.

```text
=== 21 ===
SortPreservingMergeExec: rows=Inexact(100) vs 100, q-error: previous=1.00x, current=1.00x, change=0.0%
  SortExec(TopK): rows=Inexact(1600) vs 1417, q-error: previous=14.40x, current=1.13x, change=✅ 92.2%
    ....
```

### Compare against a baseline from `main`

```shell
# Run on main first; the result is stored under main
cargo run --bin dfbench statistics \
  --path benchmarks/data/tpcds_sf1 \
  --query_path datafusion/core/tests/tpc-ds \
  --query 21

# Then, on a feature branch
cargo run --bin dfbench statistics \
  --path benchmarks/data/tpcds_sf1 \
  --query_path datafusion/core/tests/tpc-ds \
  --query 21 \
  --compare main
```

TPC-H works with its partitioned data layout as well:

```shell
cargo run --bin dfbench statistics \
  --path benchmarks/data/tpch_sf1 \
  --query_path benchmarks/queries \
  --query q1
```

## Are these changes tested?

No unit tests were added: this is a reporting CLI over planner
statistics and runtime metrics, exercised by the benchmark-suite smoke
runs above.

If reviewers think this type of code needs to be covered by unit tests
let me know.

## Are there any user-facing changes?

No, this is just for extending the `dfbench` command with a new `dfbench
statistics` subcommand

Author: gabotechs

benchmarks/src/bin/dfbench.rs

@@ -32,7 +32,8 @@ static ALLOC: snmalloc_rs::SnMalloc = snmalloc_rs::SnMalloc;
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
 use datafusion_benchmarks::{
-    cancellation, clickbench, dict, h2o, hj, imdb, nlj, smj, sort_tpch, tpcds, tpch,
+    cancellation, clickbench, dict, h2o, hj, imdb, nlj, smj, sort_tpch, statistics,
+    tpcds, tpch,
 };
 
 #[derive(Debug, Parser)]
@@ -52,6 +53,7 @@ enum Options {
     Imdb(imdb::RunOpt),
     Nlj(nlj::RunOpt),
     Smj(smj::RunOpt),
+    Statistics(statistics::RunOpt),
     SortPushdown(sort_pushdown::RunOpt),
     SortTpch(sort_tpch::RunOpt),
     Tpch(tpch::RunOpt),
@@ -73,6 +75,7 @@ pub async fn main() -> Result<()> {
         Options::Imdb(opt) => Box::pin(opt.run()).await,
         Options::Nlj(opt) => opt.run().await,
         Options::Smj(opt) => opt.run().await,
+        Options::Statistics(opt) => opt.run().await,
         Options::SortPushdown(opt) => opt.run().await,
         Options::SortTpch(opt) => opt.run().await,
         Options::Tpch(opt) => Box::pin(opt.run()).await,

benchmarks/src/lib.rs

@@ -29,6 +29,7 @@ pub mod sort_tpch;
 pub mod sql_benchmark;
 pub mod sql_benchmark_runner;
 pub mod sql_benchmark_suite;
+pub mod statistics;
 pub mod tpcds;
 pub mod tpch;
 pub mod util;