[SPARK-57975][SQL] Add an opt-in lossless Arrow struct representation for nanosecond timestamps

[viirya]

What changes were proposed in this pull request?

This adds an opt-in Arrow mapping for the nanosecond timestamp types (TimestampNTZNanosType / TimestampLTZNanosType), selected by a new losslessTimestampNanos parameter on ArrowUtils.toArrowSchema / toArrowField (default false). When enabled, a nanosecond timestamp column maps to an Arrow struct of (epochMicros: int64, nanosWithinMicro: int16) -- TimestampNanosVal's own layout -- instead of the default single int64 of epoch-nanoseconds:

• Schema (ArrowUtils): the struct's epochMicros child is tagged through field metadata with the NTZ/LTZ kind and the column precision (following the geometry/variant struct tag pattern), so fromArrowField recovers the exact Spark type on read with no out-of-band information. Nested occurrences (array/struct/map/UDT sqlType) are covered by threading the flag through the recursive schema construction.
• Write (ArrowWriter): new TimestampNTZNanosStructWriter / TimestampLTZNanosStructWriter store the two components as-is -- no unit conversion, hence no overflow. TimestampNanosTypeOps.createArrowFieldWriter now dispatches on the vector shape instead of unconditionally casting to the native nanos vectors.
• Read (ArrowColumnVector): a dedicated TimestampNanosStructAccessor recognizes the tagged struct and serves getTimestampNTZNanos / getTimestampLTZNanos from the child vectors, including nested inside arrays, structs, and maps.

The default Timestamp(NANOSECOND) mapping and every existing caller are unchanged.

Why are the changes needed?

Spark defines the nanosecond timestamp types over years 0001-9999, and stores values losslessly as (epochMicros, nanosWithinMicro). The standard Arrow mapping packs the value into a single int64 of epoch-nanoseconds, which only covers roughly years 1677-2262: a common sentinel value like 9999-12-31 23:59:59.999999999 fails with DATETIME_OVERFLOW. Internal Arrow-based storage -- specifically the Arrow-backed Dataset cache proposed in #56334, where the default in-memory cache handles the full domain (SPARK-57735) -- needs a representation that covers the full domain of the types. This was raised in #56334 (comment).

Why an opt-in parameter instead of changing the default mapping? The mismatch is structural, so the two representations serve two permanently distinct needs:

• Interchange paths must keep the standard int64 encoding. toPandas(), Arrow UDFs, and Connect result sets hand the produced bytes directly to external consumers (pandas, PyArrow, arrow-rs clients) that only understand the standard Timestamp(NANOSECOND) encoding -- SPARK-57159 added that mapping precisely so pandas receives real timestamps. Moreover, those consumers' own timestamp domains are equally int64-bound (pandas datetime64[ns] is itself int64 epoch-nanos), so the reduced domain on interchange paths is inherent to the destination: even a struct encoding could not deliver year 9999 into datetime64[ns]. Failing loudly at write with DATETIME_OVERFLOW is the correct behavior there, not a limitation to be fixed.
• Internal storage is a closed write-then-read-back loop with no external consumer, where the only requirement is fidelity to Spark semantics -- hence the lossless struct.

Since Arrow's timestamp physical type is fixed at int64 by the Arrow format spec and Spark's type domain will not shrink, this is not a transitional state to be unified later. The per-call-site boolean follows the existing largeVarTypes pattern (one Spark type, two Arrow encodings, chosen by the consumer's needs), and only schema construction needs the flag: the struct is self-describing through its child-field metadata tag, so fromArrowField, ArrowWriter, and ArrowColumnVector recognize both shapes unconditionally and no mode mismatch is possible. Placing the encoding in the shared machinery (rather than a cache-private fork of schema/writer/reader) keeps it next to its sibling encodings, covered by the shared test suites, and reusable by any future internal Arrow storage.

Does this PR introduce any user-facing change?

No. The new mapping is opt-in via an internal API parameter that defaults to off; no existing behavior changes.

How was this patch tested?

New tests:

• ArrowUtilsSuite "timestamp nanos lossless struct": schema shape (struct of int64 + int16, non-null children), type/precision round-trip for NTZ/LTZ at p=7/8/9, LTZ requiring no time zone, nested array/struct/map coverage, user-metadata preservation, precision fallback for a missing/invalid tag, no misfire on an untagged struct with the same child names, and the default mapping staying unchanged.
• ArrowWriterSuite "timestamp nanos lossless struct round-trip covers the full value domain": write-and-read-back through ArrowWriter + ArrowColumnVector for values including 9999-12-31T23:59:59.999999999 and 0001-01-01T00:00:00.000000001 (both far outside the int64 epoch-nanos range) plus nulls, for NTZ/LTZ at p=9 and p=7.
• ArrowWriterSuite "timestamp nanos lossless struct round-trip inside nested types": the same extreme values inside array<...>, struct<...>, and map<int, ...>.

Existing regression suites pass: ArrowUtilsSuite, ArrowWriterSuite, ArrowConvertersSuite, ColumnVectorSuite, ColumnarBatchSuite.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

This pull request and its description were written by Claude Code.

[dongjoon-hyun]

+1, LGTM.

[viirya]

Merge Summary:

• merged into master 7dc70c1
• merged into branch-4.x 5220ac3

Posted by merge_spark_pr.py

[viirya]

Thanks @dongjoon-hyun