Match pandas ordering, dtypes, and validation in unstack/_pivot by galipremsagar · Pull Request #23368 · NVIDIA/cudf

galipremsagar · 2026-07-21T14:10:09Z

Description

Split out of #23255 (4/6). Depends on #23365 (fidelity), the stack PR (#23370), and #23366 (groupby agg) — 8 of the 29 un-xfailed pandas-tests need those fixes too, so this PR's pandas-tests job goes green once they merge.

Order result rows/columns by the removed level's codes (level order preserved, missing keys first) instead of sorted values with nulls last, by encoding the integer code columns instead of the level values.
Propagate the source frame's column-axis level names into the result instead of hardcoding None; also fixes the ValueError: Length of names must match number of levels crash when unstacking MultiIndex-column frames.
Promote integer source columns to float64 when the reshape introduces missing cells (pandas' block semantics), gated on mode.pandas_compatible; pivot_table/crosstab opt out via a module-private _unstack parameter when fill_value fills the cells afterwards.
Preserve unused categories of the removed level in the result's column levels (pandas GH 17845); also fixes a libcudf Column sizes don't match crash for indexes with unused categorical categories.
Validate the level on flat-index frames (KeyError) and duplicated index names (ValueError) like pandas; pivot with values= drops the original columns-axis names.

Fixes 29 pandas-tests; their xfail entries are removed, three remaining test_stack_unstack.py entries get real failure reasons, and two classic categorical unstack params are un-xfailed. Attribution verified per node id against isolated builds: 21 pass with only this change, 4 need the stack PR, 2 need stack+fidelity, 2 need the groupby-agg PR.

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

- Order result rows/columns by the removed level's codes (level order preserved, missing keys first) instead of sorted values with nulls last, by encoding the integer code columns instead of level values. - Propagate the source frame's column-axis level names into the result instead of hardcoding None; also fixes the 'Length of names must match number of levels' crash for MultiIndex-column frames. - Promote integer source columns to float64 when the reshape introduces missing cells (pandas' block semantics), gated on mode.pandas_compatible; pivot_table/crosstab opt out via a module-private _unstack parameter when fill_value fills the cells. - Preserve unused categories of the removed level (pandas GH 17845); also fixes a libcudf 'Column sizes don't match' crash for unused categorical categories. - Validate the level on flat-index frames (KeyError) and duplicated index names (ValueError) like pandas; pivot with values= drops the original columns-axis names. Fixes 29 pandas-tests; their xfail entries are removed, and three remaining test_stack_unstack entries get real failure reasons. Two classic categorical unstack params are un-xfailed. Depends on the MultiIndex-column-fidelity, stack, and groupby-agg split PRs of NVIDIA#23255 for entangled tests.

copy-pr-bot · 2026-07-21T14:10:13Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

galipremsagar · 2026-07-21T14:14:05Z

/okay to test 2f3a872

coderabbitai · 2026-07-21T14:20:49Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e5eed024-1918-4e6b-85b2-7498a0b356ad

📥 Commits

Reviewing files that changed from the base of the PR and between 2f3a872 and a73c75a.

📒 Files selected for processing (4)

python/cudf/cudf/core/multiindex.py
python/cudf/cudf/core/reshape.py
python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
python/cudf/cudf/tests/dataframe/methods/test_swaplevel.py

💤 Files with no reviewable changes (1)

python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py

🚧 Files skipped from review as they are similar to previous changes (1)

python/cudf/cudf/core/reshape.py

📝 Walkthrough

Summary by CodeRabbit

Bug Fixes
- Improved pivot, pivot_table, and unstack compatibility with pandas.
- Preserved expected ordering for MultiIndex levels, including unused categories and missing keys.
- Correctly promotes integer results when reshaping introduces missing values, while respecting fill_value.
- Fixed MultiIndex.swaplevel to preserve level names consistently.
Tests
- Added coverage for sparse integer pivot tables, categorical indexes, and stacked MultiIndex workflows.

Walkthrough

Pivot and unstack now use encoded axes for pandas-compatible ordering, preserve MultiIndex metadata, and conditionally promote integer results when missing cells occur. Pivot-table routing, reshape tests, and expected-failure mappings were updated accordingly.

Changes

Reshape compatibility

Layer / File(s)	Summary
Pivot encoding and missing-cell promotion `python/cudf/cudf/core/reshape.py`	`pivot` and `_pivot` now exchange encoded axes, preserve column level metadata, and promote integer inputs when pandas-compatible reshaping introduces missing cells.
Unstack and pivot_table integration `python/cudf/cudf/core/reshape.py`	`unstack` re-encodes MultiIndex levels for ordering and category preservation, and `pivot_table` routes through `_unstack` with fill-aware promotion.
MultiIndex metadata and validation `python/cudf/cudf/core/multiindex.py`, `python/cudf/cudf/tests/dataframe/methods/test_swaplevel.py`, `python/cudf/cudf/tests/reshape/*`	`swaplevel` preserves reordered level names; reshape tests cover sparse integer pivot tables, categorical unstack cases, and stacked-name round trips.
Pandas testing expectations `python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py`	Expected-failure reasons and entries for reshape, crosstab, sort, and pivot-table cases were updated.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

rapidsai/cudf#23255: Directly overlaps with pivot/unstack encoding and missing-cell dtype handling.

Suggested labels: improvement

Suggested reviewers: rjzamora, mroeschke, brandon-b-miller, vyasr

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main reshape-related changes to unstack and _pivot behavior.
Description check	✅ Passed	The description directly matches the implemented unstack/_pivot fidelity updates and test changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/cudf/cudf/core/reshape.py`:
- Around line 1374-1381: Update the flat-index branch around
_validate_index_level to validate every level value, including integers, by
removing the is_integer(level) guard. Preserve the existing level normalization
and let _validate_index_level handle positional integer bounds and name-based
validation before df.T.stack executes.
- Around line 1219-1228: Update the _pivot call in pivot() to pass
promote_ints_on_missing=True, ensuring integer columns introduced with missing
values are promoted to float64 under pandas-compatible behavior. Preserve the
existing selection, encoding, and reshape arguments.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 30df6cae-74e0-42c2-9fb4-52ec15d610db

📥 Commits

Reviewing files that changed from the base of the PR and between 738fb74 and 2f3a872.

📒 Files selected for processing (4)

python/cudf/cudf/core/reshape.py
python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
python/cudf/cudf/tests/reshape/test_pivot_table.py
python/cudf/cudf/tests/reshape/test_unstack.py

…ndas-ordering # Conflicts: # python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py

- Pass promote_ints_on_missing=True from pivot(): pandas' pivot also promotes integer values to float64 when the reshape introduces missing cells (it reshapes via unstack internally). - Gate the unused-categories column rebuild on the cheap result._data.multiindex flag and only materialize the removed level / pandas column index inside that block.

swaplevel rebuilt the result from the level columns' accessor keys and only restored names when every name was None. An index whose levels are keyed positionally but carry real names - e.g. the index produced by an unstack/stack round trip - came back with the positional keys as its names, and name-based level lookups silently swapped nothing. Resolve the levels with _level_index_from_level (name first, then positional with bounds checking, like pandas) and always carry the swapped names over from self.names. Fixes the re-enabled pandas test tests/frame/test_stack_unstack.py::TestStackUnstackMultiLevel::test_stack[True] against the merged stack changes and makes invalid levels raise KeyError/IndexError like pandas.

galipremsagar · 2026-07-23T17:09:39Z

/okay to test a73c75a

galipremsagar · 2026-07-23T17:13:33Z

/merge

galipremsagar requested a review from a team as a code owner July 21, 2026 14:10

galipremsagar requested review from rjzamora and vyasr July 21, 2026 14:10

github-actions Bot assigned galipremsagar Jul 21, 2026

github-actions Bot added Python Affects Python cuDF API. cudf.pandas Issues specific to cudf.pandas labels Jul 21, 2026

github-project-automation Bot added this to cuDF Python Jul 21, 2026

GPUtester moved this to In Progress in cuDF Python Jul 21, 2026

galipremsagar added bug Something isn't working breaking Breaking change labels Jul 21, 2026

coderabbitai Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread python/cudf/cudf/core/reshape.py

Comment thread python/cudf/cudf/core/reshape.py

galipremsagar mentioned this pull request Jul 21, 2026

Fix DataFrame.stack/unstack pandas incompatibilities #23255

Closed

3 tasks

galipremsagar added the 3 - Ready for Review Ready for review by team label Jul 21, 2026

mroeschke approved these changes Jul 23, 2026

View reviewed changes

Comment thread python/cudf/cudf/core/reshape.py

Comment thread python/cudf/cudf/core/reshape.py

coderabbitai Bot mentioned this pull request Jul 23, 2026

Fix DataFrame.stack level resolution, ordering, and dtype preservation #23370

Merged

3 tasks

galipremsagar added 4 commits July 23, 2026 16:38

Merge remote-tracking branch 'upstream/release/26.08' into unstack-pa…

f182e7f

…ndas-ordering # Conflicts: # python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py

Merge branch 'release/26.08' into unstack-pandas-ordering

a73c75a

galipremsagar added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team labels Jul 23, 2026

NVIDIA deleted a comment from copy-pr-bot Bot Jul 23, 2026

rapids-bot Bot merged commit b1fbdda into NVIDIA:release/26.08 Jul 23, 2026
128 checks passed

github-project-automation Bot moved this from In Progress to Done in cuDF Python Jul 23, 2026

josephine-wolf-oberholtzer added this to cuDF Python Aug 12, 2026

josephine-wolf-oberholtzer moved this to Done in cuDF Python Aug 12, 2026

Navigation Menu

Match pandas ordering, dtypes, and validation in unstack/_pivot - #23368

Match pandas ordering, dtypes, and validation in unstack/_pivot#23368
rapids-bot[bot] merged 5 commits into
NVIDIA:release/26.08from
galipremsagar:unstack-pandas-ordering

galipremsagar commented Jul 21, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 21, 2026

Uh oh!

galipremsagar commented Jul 21, 2026

Uh oh!

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading

Summary by CodeRabbit

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

galipremsagar commented Jul 23, 2026

Uh oh!

galipremsagar commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

galipremsagar commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 21, 2026

Uh oh!

galipremsagar commented Jul 21, 2026

Uh oh!

coderabbitai Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

galipremsagar commented Jul 23, 2026

Uh oh!

galipremsagar commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

galipremsagar commented Jul 21, 2026 •

edited

Loading

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading