[SPARK-17636][SQL][WIP] Parquet predicate pushdown in nested fields

[dbtsai]

What changes were proposed in this pull request?

Support Parquet predicate pushdown in nested fields

How was this patch tested?

Existing tests and new tests are added.

[SparkQA]

Test build #96506 has started for PR 22535 at commit c95706f.

[dongjoon-hyun]

This PR will be a good performance improvement for Spark 2.5.0.

[viirya]

retest this please.

[SparkQA]

Test build #96528 has finished for PR 22535 at commit c95706f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @dbtsai . Could you fix the UT failure?

[info] ParquetFilterSuite:
...
[info] - SPARK-12218 Converting conjunctions into Parquet filter predicates *** FAILED *** (19 milliseconds)
[info]   Expected None, but got Some(lt(a, 10)) (ParquetFilterSuite.scala:802)

[dbtsai]

I'm breaking this PRs into three smaller PR. I'll fix the tests in those smaller PRs. Thanks.

[SparkQA]

Test build #99673 has finished for PR 22535 at commit c95706f.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #101203 has finished for PR 22535 at commit 49f19a9.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #101204 has finished for PR 22535 at commit eb400ed.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[aniket486]

@dbtsai are you still working on this?

[SparkQA]

Test build #115067 has finished for PR 22535 at commit eb400ed.

• This patch fails build dependency tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[dbtsai]

Going to close this in favor of new PR, #27728