Iceberg integration broken

[andygrove]

Describe the bug

I compiled the latest Iceberg from source and changed the Comet dependency version from 0.5.0 to 0.7.0.

When testing using the instructions in #1683, I get this runtime error:

Caused by: java.lang.NoSuchMethodError: 'org.apache.comet.parquet.ColumnReader org.apache.comet.parquet.Utils.getColumnReader(org.apache.spark.sql.types.DataType, org.apache.iceberg.shaded.org.apache.parquet.column.ColumnDescriptor, org.apache.comet.shaded.arrow.c.CometSchemaImporter, int, boolean, boolean)'

The issue seems to be that Iceberg is passing a shaded class org.apache.comet.shaded.arrow.c.CometSchemaImporter but Comet expects org.apache.arrow.c.CometSchemaImporter.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[andygrove]

@parthchandra @huaxingao Could I get some help with this next week?

[andygrove]

Oh, we seem to be accidentally shading this class in the Comet jar.

$ jar tvf comet-spark-spark3.4_2.12-0.7.0.jar | grep CometSchemaImporter
  2788 Wed Jan 22 15:10:16 MST 2020 org/apache/comet/shaded/arrow/c/CometSchemaImporter.class