Presumably when this returns false for any reason, we fallback to the default cache driver, that should be made clear in the docs if it isn't already

e.g. the doc says supports all Spark SQL data types but this implementation would seem to falsify that claim ;)

Well, I guess the claim may be true, and the fallthrough at the end might be defensive... In which case maybe we'd want to log a surprisingly unsupported type :)

Good catch -- "supports all Spark SQL data types" did overstate it, and the type list was also incomplete (it omitted Time, intervals, Geometry/Geography, Variant, Null, and UDTs, which are all supported). Fixed the doc to list the actually-supported set and to describe the unsupported-type behavior.

One clarification on the mechanism, re: your fallback question: isSupportedByArrow here only gates supportsColumnarInput, which the cache framework uses to choose the columnar-vs-row input path into this same serializer -- it isn't a fallback to the default cache driver (there's no such per-type fallback). A truly unsupported type isn't silently dropped either: toArrowSchema throws UNSUPPORTED_DATATYPE when the cache is materialized. The docs now state this explicitly.

cc @MaxGekk to take a look ^^

[P2] Please reconcile this capability whitelist with current master before merging. master now maps and reads/writes TimestampNTZNanosType and TimestampLTZNanosType through Arrow, but this predicate still rejects them. The serializer then takes the row path, falls through to ObjectColumnStats (whose ColumnType has no nanos-timestamp case), and fails materialization with UNSUPPORTED_DATATYPE; row output also lacks a typed reader or fallback. The synthetic merge is conflict-free but still has this behavioral incompatibility.

Good catch that master now maps these through Arrow. I looked at the cache paths, though, and the physical value for TimestampNTZNanosType/TimestampLTZNanosType is a TimestampNanosVal, not a plain Long, so the cache's stats collector and fast columnar reader can't treat them as long-backed without a dedicated, precision-aware path -- that's net-new support rather than a predicate tweak. Importantly there's no parity regression: the default cache serializer doesn't support these types either. I verified on current master that df.cache() of a TIMESTAMP_NTZ(9) column throws not support type: TimestampNTZNanosType(9) with the default serializer (its ColumnBuilder has no case for them). The Arrow serializer now rejects them with a clear checkSupportedSchema error at materialization. I'd rather add real support as a focused follow-up than land a half-correct fast path here -- let me know if you'd prefer it gated differently in the meantime.

Follow-up filed: SPARK-57735 / #56842 adds nanosecond-timestamp support to the default in-memory cache (DefaultCachedBatchSerializer), which is the prerequisite -- once that lands, the Arrow cache can route these types through the same statistics machinery rather than rejecting them here.

[P2] The prerequisite cited here has now landed in this exact tree: DefaultCachedBatchSerializer has nanosecond timestamp builders in ColumnBuilder.scala, matching ColumnType cases and TimestampNanosColumnStats. This predicate still rejects both TimestampNTZNanosType and TimestampLTZNanosType, so the Arrow cache is now less capable than the default cache even though the guide says it covers every default-supported type. Since #56842 was the stated gate, please reconcile this before merging by completing Arrow-cache support or narrowing the compatibility claim.

[ 🤖 posted by Codex on behalf of sunchao using the code-review-for-me skill 🤖 ]

Done -- with the prerequisite merged (and now in this tree after rebase), this PR adds full Arrow-cache support for TimestampNTZNanosType/TimestampLTZNanosType: the capability predicate accepts them, statistics route to the same TimestampNanosColumnStats the default cache uses (min/max bounds, so cached nanos timestamps prune), the columnar-input stats path collects nanos bounds from the Arrow vectors, and reads work through both the vectorized path (ArrowColumnVector handles TimeStampNanoVector/TimeStampNanoTZVector natively) and the row path (via the columnar-to-row fallback). The write path was already in place via ArrowWriter's nanos writers, including their structured epoch-nanos overflow error. The type-coverage machinery now includes both types, so they are exercised by the cache-and-read-back tests under both reader modes.

[P2] The follow-up support is still not parity with the default cache for valid nanos-timestamp values outside Arrow's signed-INT64 epoch-nanos window. Spark defines both TimestampNTZNanosType and TimestampLTZNanosType over years 0001-9999, and the default cache stores the (epochMicros, nanosWithinMicro) pair losslessly. This serializer instead routes non-Arrow input through ArrowWriter, which packs the value into one Long. I reproduced this with LocalDateTime.of(9999, 12, 31, 23, 59, 59, 999999999): the default serializer materializes the row, while the Arrow serializer fails with DATETIME_OVERFLOW because only roughly 1677-2262 is representable. There is no fallback after selecting this serializer, and the guide omits this restriction while claiming coverage of every type supported by the default cache. Please either use a lossless cache representation or document the reduced domain beside CalendarIntervalType and narrow the parity claim.

The prerequisite is now built: SPARK-57975 / #57053 (approved, pending CI) adds an opt-in lossless Arrow representation for the nanosecond timestamp types -- a struct of (epochMicros: int64, nanosWithinMicro: int16), i.e. TimestampNanosVal's own layout with no unit conversion -- covering the full 0001-9999 domain, including the 9999-12-31T23:59:59.999999999 value from your repro. The interchange mapping stays untouched: its consumers (pandas datetime64[ns] etc.) are themselves int64-bound, so the reduced domain there is inherent to the destination; the struct is for internal storage only.

Once that lands, this PR will opt in via losslessTimestampNanos = true at the cache's schema-construction sites. The struct is self-describing (tagged child metadata), so both read paths already work through ArrowColumnVector with no format flag; the stats collector will read the struct children directly (which also removes the epoch-nanos conversion from the stats path); and the zero-copy eligibility check will route native int64-nanos Arrow input through the row-conversion path, like the existing large-var-types handling. That restores full value-domain parity with the default cache for these types.

Done. The cache now stores nanosecond timestamps in SPARK-57975's lossless struct encoding, so the full 0001-9999 domain round-trips -- including the 9999-12-31T23:59:59.999999999 value from your repro -- restoring value-domain parity with the default serializer. Wiring details: the stats collector reads the struct components directly and compares with TimestampNanosVal ordering (which also removes the epoch-nanos conversion, and with it the precision-truncation asymmetry MaxGekk flagged); the zero-copy input eligibility check now routes interchange-shaped int64-nanos vectors (e.g. a Python UDF output feeding a cache) through the row-conversion path, like the existing large-var-types handling; both reader modes are covered by the existing type round-trip machinery. The docs' value-range note is updated accordingly.

[P2] Please do not advertise CalendarIntervalType as fully supported unless the cache representation is lossless over Spark's full value domain. Arrow's IntervalMonthDayNanoWriter multiplies CalendarInterval.microseconds by 1000 with Math.multiplyExact, but Spark permits the full Long range. I reproduced this with new CalendarInterval(0, 0, Long.MaxValue / 1000L + 1L): the default cache returns COUNT=1, while the Arrow cache aborts materialization with ArithmeticException: long overflow. Please use a lossless representation or reject and document this type rather than claiming parity with the default serializer.

Fixed with a clear diagnostic. Caching a CalendarInterval whose microseconds exceed +/-(Long.MaxValue / 1000) now throws an explanatory error (naming the type and the nanosecond-conversion limit) instead of an opaque ArithmeticException: long overflow. The check is installed only when the schema actually contains a CalendarInterval column, so there is no per-row cost for other schemas. Arrow's IntervalMonthDayNano is nanosecond-based and cannot losslessly hold the full Long microsecond domain, so I documented the value-range limit rather than changing the shared Arrow writer; the default serializer's lack of this restriction is noted in the docs.

[P2] The diagnostic fix still covers only top-level intervals. hasCalendarInterval checks attr.dataType == CalendarIntervalType, while isSupportedByArrow accepts CalendarInterval recursively inside arrays, structs, maps, and UDTs, and their recursive Arrow writers still reach Math.multiplyExact(microseconds, 1000L). For example, an array<interval> containing Long.MaxValue / 1000 + 1 still escapes withIntervalOverflowTranslation and fails with raw ArithmeticException: long overflow, contrary to the guide and this reply. Please recurse through complex types and UserDefinedType.sqlType, with nested overflow coverage.

[ 🤖 posted by Codex on behalf of sunchao using the code-review-for-me skill 🤖 ]

Fixed. hasCalendarInterval now recurses through arrays, structs, maps, and UserDefinedType.sqlType (a dedicated traversal mirroring isSupportedByArrow, since DataType.existsRecursively does not descend into UDT sql types), so nested intervals get the same clear diagnostic. Added an array<interval> overflow case to the test.

[P2] Please narrow the CalendarInterval overflow translation to the actual interval conversion. The new schema-wide catch wraps the entire batch loop whenever any interval field exists, and SparkArithmeticException extends ArithmeticException. I reproduced this with one partition containing two rows under ANSI mode: range(0, 2, 1, 1).selectExpr("1 / (1 - id) AS bad", "make_interval() AS i").cache().count(). Without the harmless interval column, the second row fails as DIVIDE_BY_ZERO; with it, the same failure is caught inside the batch loop and rethrown as DATETIME_OVERFLOW claiming a CalendarInterval overflow. Adding an unrelated interval column must not change the error condition or SQLSTATE. Please catch the raw Math.multiplyExact failure at IntervalMonthDayNanoWriter, or preserve structured/non-interval arithmetic exceptions unchanged.

Confirmed -- SparkArithmeticException extends ArithmeticException, and the batch-loop try covers rowIter.next() where upstream lazy evaluation runs, so an unrelated ANSI error can indeed be re-labeled. Adding a harmless interval column must not change the error condition.

Rather than narrowing the catch, we plan to make the translation unnecessary and delete it outright: following the same approach as SPARK-57975 / #57053 (opt-in lossless struct representation for nanosecond timestamps), a follow-up will store CalendarInterval losslessly as a struct of (months: int32, days: int32, microseconds: int64) -- the type's own field layout, mirroring the default cache's CALENDAR_INTERVAL ColumnType. With no * 1000 conversion on the cache path, the overflow (and withIntervalOverflowTranslation along with this mis-attribution bug) ceases to exist, and the cache regains the default serializer's full value domain for this type as well. The shared IntervalMonthDayNanoWriter will also get a structured DATETIME_OVERFLOW at the Math.multiplyExact site (the TimestampNTZNanosWriter pattern) for the interchange paths that keep the standard MonthDayNano encoding.

Follow-up filed: SPARK-58005 / #57088 implements the plan above. It extends SPARK-57975's opt-in lossless encoding to CalendarInterval -- a struct of (months, days, microseconds), the type's own layout, so the * 1000 conversion and its overflow do not exist on that path -- and, independently, moves the overflow translation into IntervalMonthDayNanoWriter itself: the structured DATETIME_OVERFLOW is now raised by a catch scoped to the single Math.multiplyExact expression (the TimestampNTZNanosWriter pattern), so it structurally cannot re-label an unrelated ArithmeticException from upstream evaluation. Once both land, this PR will opt the cache into the lossless encoding and delete withIntervalOverflowTranslation entirely, which removes the mis-attribution you found along with the wrapper; your ANSI DIVIDE_BY_ZERO repro then surfaces DIVIDE_BY_ZERO regardless of interval columns in the schema.

Done. With SPARK-57975 and SPARK-58005 merged, this PR now opts the cache into the lossless encodings (losslessInternalTypes = true at the schema-construction sites) and deletes withIntervalOverflowTranslation entirely -- on the cache path there is no longer a nanosecond conversion that can overflow, so the mis-attribution you found is gone along with the wrapper: your ANSI DIVIDE_BY_ZERO repro surfaces DIVIDE_BY_ZERO regardless of interval columns. The overflow diagnostic test is replaced by a full-domain round-trip test (microseconds = Long.MaxValue etc., top-level and nested), and the docs now state the cache matches the default serializer's full value domain for this type.

[P2] This capability check accepts any UDT whose sqlType is supported, but the cache statistics path does not unwrap UDTs. A valid UDT backed by VariantType passes here, then createColumnStats constructs ObjectColumnStats; ColumnType(udt.sqlType) has no VariantType case and throws UNSUPPORTED_DATATYPE during materialization. The default cache serializer does unwrap the same UDT and supports it, so this also contradicts the documented claim that Arrow covers every default-supported type. Please unwrap UDTs when selecting statistics collectors, or narrow this predicate and the documentation.

Fixed. createColumnStats now unwraps UDTs (case udt: UserDefinedType[_] => createColumnStats(udt.sqlType)), so a Variant- or Geometry-backed UDT gets the right collector instead of falling through to ObjectColumnStats and throwing UNSUPPORTED_DATATYPE. This keeps the capability check and the statistics path in agreement.