docs: add a user-facing Celeborn integration guide

[andygrove]

What is the problem the feature request solves?

Comet's Celeborn integration landed across #5501, #5513, #5531, #5537, #5604, #5627, #5665 and #5668. By surface area it is a first-class integration comparable to Iceberg: its own shuffle manager, its own reader and writer, its own config namespace, and its own CI suites. Its user-facing documentation is a subsection of the tuning guide.

What exists today:

• docs/source/user-guide/latest/tuning.md, "Remote Shuffle with Celeborn" (lines 393-467). The substance is largely there, but it is written as a specification of behavior rather than as a guide for someone setting it up.
• docs/source/user-guide/latest/configs.md, Shuffle section: prose on the two spark.comet.shuffle.rss.* settings plus their auto-generated table rows.

The gaps:

1. Not discoverable. The Integrations toctree in docs/source/user-guide/latest/index.rst lists Iceberg, Iceberg Writes, S3 Credential Providers and Kubernetes. Celeborn is absent, so someone evaluating whether Comet works with their remote shuffle service has to full-text search or happen to scroll the tuning guide.

2. No setup instructions. The tuning guide says to "provide a compatible Celeborn Spark client matching the application's Spark and Scala versions on both the driver and executors", but never names the artifact (org.apache.celeborn:celeborn-client-spark-3-shaded_<scala-binary-version>), how to supply it, or which Celeborn versions Comet is tested against. pom.xml pins celeborn.version at 0.7.0 solely for the reflection-compatibility test, and nothing user-facing states that.

3. The headline compatibility fact is buried. Released 0.6.0 and 0.7.0 clients do not provide the push-completion guarantee native shuffle requires, so on every released client today an application retains the delegated Spark/Celeborn path and the spark.comet.shuffle.mode=native recipe is not reachable. That is the first thing a prospective user needs to know, and it currently appears in the second paragraph of a tuning subsection, after the recipe.

4. No way to verify what happened. Nothing tells a user how to distinguish an exchange that went native-remote from one delegated to ordinary Spark/Celeborn shuffle, or one locally materialized after an oversized row (fix: recover native Celeborn shuffle from oversized rows #5668). metrics.md has no Celeborn or RSS content.

5. No troubleshooting. The conditions that silently retain ordinary shuffle are real and numerous: spark.io.encryption.enabled=true, an effective fallback policy of ALWAYS or a reached AUTO threshold, dynamic allocation with both spark.shuffle.service.enabled and spark.dynamicAllocation.shuffleTracking.enabled off, auto or jvm shuffle mode, unsupported child operators, data types or partitioning. They are scattered through the prose as constraints rather than presented as symptoms with causes.

Describe the potential solution

Add docs/source/user-guide/latest/celeborn.md and list it in the Integrations toctree, modeled on the Iceberg guide. A suggested shape:

• Support status up front: which Celeborn versions work with which Comet shuffle mode, and what the delegated path does and does not give you on the released clients.
• Setup: the client artifact and coordinates, driver and executor placement, the shuffle manager setting, the full config recipe, and the fact that Celeborn is an optional application dependency Comet does not bundle.
• What is accelerated: AQE reducer coalescing and mapper-range reads are supported; Celeborn physical-skew chunk reads are not.
• Verifying: how to confirm from the plan and from metrics which path an exchange took.
• Troubleshooting: the retention and fallback conditions as a symptom-to-cause table.
• Tuning: spark.comet.shuffle.rss.maxFrameBytes and spark.comet.shuffle.rss.maxInFlightBytes, referring to the tuning guide rather than duplicating it.

Then reduce the tuning-guide section to a short pointer, keeping the genuinely tuning-specific material there so the two pages do not drift.

Additional context

Adding Celeborn-specific rows to metrics.md may be worth splitting into a separate issue if the metric names need work first.

[andygrove]

@sunchao @pingzh would you be able to contribute some docs for the Celeborn integration?

[pingzh]

take

[pingzh]

@andygrove i can work on it