Project Hive Partition Columns With ProjectionExec #2291

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Part of #2079

Currently the values of Hive partitions are projected within each of the various file format specific physical operators. As described in #2079 this has a number of drawbacks

Describe the solution you'd like

Rather than handling partition projection within the file scan operator, I would like to propose modifying ListingTable to add a ProjectionExec within TableProvider::scan instead of relying on the FileFormat implementations to do this. This ProjectionExec would be created with a set of literal expressions corresponding to the partition values

Therefore instead of TableProvider::scan generating something like

AvroExec:

It would generate

UnionExec
  ProjectionExec: ... // Partition 1
    AvroExec: ...
  ProjectionExec: ... // Partition 2
    AvroExec: ...

Note: this will depend on #2289

Describe alternatives you've considered

The logic could instead be moved to FileFormat::create_physical_plan implementations, but I think it would be better to keep what is a catalog detail close to the catalog implementation.

FYI @matthewmturner @yjshen @rdettai