Support for new logical type representation

[asfimport]

Latest parquet-format introduced a new representation for logical types. As of now this is not yet supported in parquet-mr, thus there's no way to use parametrized UTC normalized timestamp data types. When reading and writing Parquet files, besides 'converted_type' parquet-mr should use the new 'logicalType' field in SchemaElement to tell the current logical type annotation. To maintain backward compatibility, the semantic of converted_type shouldn't change.

Reporter: Nándor Kollár / @nandorKollar
Assignee: Nándor Kollár / @nandorKollar

Related issues:

• Add "Floating Timestamp" logical type (is duplicated by)
• Incorporate new logical type parameters into schema language (is duplicated by)
• Build failure when using Java 8 lambda expressions (depends upon)
• Incorporate new logical type parameters into schema language (is depended upon by)

PRs and other links:

• GitHub Pull Request #463

_{Note: This issue was originally created as PARQUET-1253. Please see the migration documentation for further details.}

[asfimport]

Nándor Kollár / @nandorKollar:
While working on the new logical type representation three questions came to mind:

• Despite there is a Thrift struct for UUID logical type in parquet-format, it is not included into the LogicalType union. Is this on purpose, or was omitted accidentally? How should parquet-mr handle those schemas, where UUID annotation is used, but there's no corresponding LogicalType mapping?

• Similar question with MAP_KEY_VALUE, but it is not implemented at all in the new representation. What should parquet-mr do with those schemas, which use it in the old representation?

• In parquet-format the comment for optional LogicalType logicalType says "The logical type of this SchemaElement; only valid for primitives." but I'm confused, because there's a Map and a List logical type, which - as far as I know - makes sense only on groups. What was the intention of this comment? Do I miss anything?

  @rdblue I can see that you worked on the new logical type representation, could you please help me to clarify these questions?

[asfimport]

Ryan Blue / @rdblue:
Not including the UUID logical type in that union is probably an accident.

MAP_KEY_VALUE is no longer used. It is noted in backward compatibility rules, but is not required for any types.

The comment "only valid for primitives" is incorrect. I think we can remove it. I'm not sure why the comment was there.

[asfimport]

Nándor Kollár / @nandorKollar:
Thanks Ryan for clarifying my questions!