GH-48961: [Docs][Python] Doctest fails on pandas 3.0

[tadeja]

Rationale for this change

See issue #48961
Pandas 3.0.0 string storage type changes https://github.com/pandas-dev/pandas/pull/62118/changes
and https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#dedicated-string-data-type-by-default

What changes are included in this PR?

Updating several doctest examples from string to large_string.

Are these changes tested?

Yes, locally.

Are there any user-facing changes?

No.

Closes #48961

• GitHub Issue: [Docs][Python] Doctest fails on pandas 3.0 #48961

[AlenkaF]

Thank you @tadeja for looking into this!

One question regarding the bump of the Python version in Sphinx&Numpydoc job. I think it would be good if the examples worked for users with new or old pandas version. What if we use ... (ELLIPSIS) instead of changing the string type? Or even better, we could not use pandas where possible and instead create a pyarrow table directly, like so:

arrow/python/pyarrow/table.pxi

Lines 1812 to 1814 in 95a3ed4

	>>> table = pa.Table.from_arrays([[2, 4, 5, 100],
	... ["Flamingo", "Horse", "Brittle stars", "Centipede"]],
	... names=['n_legs', 'animals'])

[rok]

Agreed that it doesn't make sense for us to "test Pandas logic" especially in our docs. Agreed with @AlenkaF to instantiate the table in pyarrow. Using ellipsis in this case would hide the type and potentially increase user confusion :).

[AlenkaF]

Note that some examples are demonstrating conversion from pandas to pyarrow so in that case we might remove the string column and only keep the integer ones?

[rok]

This looks good to me now. I think (hope) removing pandas from examples that don't require streamlines things for readers.

[rok]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 186c0a9

Submitted crossbow builds: ursacomputing/crossbow @ actions-ca47b1b8be

Task	Status
preview-docs

[tadeja]

@AlenkaF this is ready for final review.

• Generated doc pages: pyarrow.Table page and pyarrow.RecordBatch

• Both Sphinx jobs ran and completed doctests with success;
  AMD64 Conda Python 3.12 Sphinx Documentation
  pandas 3.0.0 pypi_0 pypi
  ================== 385 passed, 2 skipped, 1 warning in 6.24s ===================
  and
  AMD64 Conda Python 3.10 Sphinx & Numpydoc
  pandas 2.3.3 pypi_0 pypi
  ======================== 385 passed, 2 skipped in 5.63s ========================

• The two trivial cases where pandas 2.3.3 output expects None but pandas 3.0.0 expects NaN
1 4 None 2022.0
1 4 NaN 2022.0
  get best resolved by populating pa.array with a string instead: first case and second case.

• Note that I additionally removed pandas and replaced with pyarrow table for these three examples: def itercolumns, def remove_column and def join (although these are currently not causing failures as there isn't string vs. large_string in their output).
  But there are more unnecessary pandas examples remaining that could be simplified in the future (num_columns, num_rows etc).

[AlenkaF]

Thank you for the changes, looks great!
I only added one minor suggestion. Then I am happy to merge!

[tadeja]

@AlenkaF this is ready now!
(the unrelated change in .rst allowed getting CI test with pandas 3.0 to run. Only when there are changes in docs/** then this optional CI job with pandas 3.0 would be running Docs / AMD64 Conda Python 3.12 Sphinx Documentation).

[AlenkaF]

Thanks! I have opened an issue for the unrelated CI failure: #49044

[pitrou]

So are we ok that Pandas strings convert to Arrow large strings? It's a bit less memory-efficient, WDYT @jorisvandenbossche ?

[jorisvandenbossche]

It is essentially pandas that decides this, although in theory the pyarrow from_pandas() functionality could override that (the new string dtype in pandas is actually using pyarrow large_string under the hood, so the conversion to pyarrow with large_string is zero copy while converting to string would involve some conversion).

The reason we went for large_string instead of string in pandas is because with string we would easily run into the size limits of it, without having more advanced automatic chunking logic in pandas or pyarrow (for example, at the time there were still issues with take() potentially failing with string ("offset overflow while concatenating arrays"), I haven't fully followed to know if all those issues have been resolved now)

See pandas-dev/pandas#56259 for context on the pandas side

[jorisvandenbossche]

Not too important, but for future changes: if we want to keep the docstring examples as simple as possible, I would say that keeping the dict-type creation (as it is with pd.DataFrame) is easier than the arrays. So this could also be something like:

        >>> table = pa.table({'year': [None, 2022, 2019, 2021],
        ...                   'n_legs': [2, 4, 5, 100],
        ...                   'animals': ["Flamingo", "Horse", None, "Centipede"]})

i.e. essentially just swapping pd.DataFrame(..) with pa.table(..)

(also, the df creation can now be removed from the example, because it is no longer used)