fix: explain ObjectHashAggregate fallback when Comet shuffle is disabled

[0lai0]

Which issue does this PR close?

Closes #5500.

Rationale for this change

CometObjectHashAggregateExec deliberately declines when Comet shuffle is disabled, because converting would split the aggregate across a Comet partial and a Spark final. That decline was a bare return None inside convert, with no withFallbackReason call, so the reason never reached the user.

Under spark.comet.explain.fallback.strict.enabled=true, planning throws:

java.lang.IllegalStateException: Comet did not convert ObjectHashAggregate but recorded no fallback reason on the operator or any of its expressions.

Under the production default the generic catch-all stands in for the real cause, which is worse because it is wrong. Comet supports this operator fine, the user just turned shuffle off:

FALLBACK_REASONS = Set(ObjectHashAggregate is not supported)

The check simply lived in the wrong place. CometExecRule.isOperatorEnabled turns an Unsupported(Some(reason)) from getSupportLevel into a withFallbackReason call centrally, so a check placed there is explained for free. CometCollectLimitExec and CometTakeOrderedAndProjectExec gate on the same predicate and already do it that way.

What changes are included in this PR?

• Move the shuffle guard from convert into getSupportLevel, returning Unsupported(Some(...)). convert becomes the same doConvert call CometHashAggregateExec.convert already is.
• Remove the duplicate copy of the predicate from CometExecRule.canAggregateBeConverted. The line above it already calls isOperatorEnabled, so the block was dead.
• Document the prerequisite on the ObjectHashAggregateExec row of operators.md.
• Add three regression tests.

The guard sits after the two test-knob checks so a test that disables partial or final aggregates still sees its own reason. The message names no config key, because isCometShuffleEnabled is a conjunction of spark.comet.shuffle.enabled, the shuffle manager, and the Celeborn check, and naming one would misdirect when another is the cause. That is also why this PR does not delegate to CometShuffleExchangeExec.isCometShuffleEnabledReason, which computes its Celeborn branch from op.outputPartitioning.numPartitions rather than the hardcoded 1 that isCometShuffleEnabled uses, and so could return no reason while the predicate is still false.

Not visible in the diff

The Final aggregate now carries the reason too.
getSupportLevel runs for every stage, so the Final node gets it where previously it recorded nothing at all. I confirmed this with a probe rather than by reasoning about it. The message is worded stage-neutrally for that reason. Eligibility is unchanged: hasFallbackReason is consulted only for shuffle exchanges and CometNativeScan, never for aggregates.

operators.md rows 74 and 75 are still blank.
CollectLimitExec and TakeOrderedAndProjectExec gate on the same predicate and do not mention it, so the three rows now disagree. Changing user-facing text for two already-merged operators is out of scope here, but it is worth picking up separately.

How are these changes tested?

Three new tests in CometExecRuleSuite, all of which fail on unfixed code:

1. The partial ObjectHashAggregateExec records the shuffle reason, not the generic message, and the reason survives into ExtendedExplainInfo. Run with strict mode both enabled and disabled as the issue asks, with a fresh plan per iteration because fallback reasons accumulate on tags.
2. getSupportLevel returns Unsupported with the reason when shuffle is off and Compatible when it is on. Before the fix it returned Compatible(None,None).
3. With shuffle enabled the query still converts, pinning the issue's "without changing aggregate execution eligibility" requirement.

[sunchao]

Correctness

Problem, approach, and compatibility

Reviewed the four-file authored change at 717d4239 against authoritative base bc74cc79, with merge-base 7e198439. The changed files and relevant caller files are unchanged by the four commits between merge-base and base.

The existing shuffle-disabled guard returned None without recording why ObjectHashAggregateExec stayed in Spark. Moving it into getSupportLevel lets isOperatorEnabled attach the specific reason before conversion declines. That addresses both the strict-mode planning exception and the generic message shown in normal mode. The partial/final test-config checks retain precedence, and the new reason is worded for either stage.

I found no change to execution eligibility. Both production conversion and the unsafe-partial predictor call isOperatorEnabled, so the moved guard still protects both paths. Removing the predictor's duplicate guard preserves its Boolean result. The common aggregate conversion logic and the checks for incompatible partial/final buffers are unchanged. Newly recorded aggregate diagnostics do not become an additional conversion veto.

The maintained Spark 3.5/4.0 sources confirm why the boundary matters: typed imperative aggregates expose binary intermediate buffers and deserialize them when merging. collect_list uses that contract. Keeping the existing refusal prevents the same Spark/Comet buffer mismatch as before. The change does not alter aggregate value handling, including NULLs, empty input, output types, ANSI behavior, or overflow. Maintained Spark 3.4/4.1 sources were unavailable, so those versions are not independently qualified here.

Validation

The new tests cover strict and lenient diagnostics, the aggregate-specific reason in extended explain, direct support classification, and conversion with shuffle enabled. Fresh plans prevent previously attached tags from satisfying a later case. These are appropriate planner tests for this change.

CI has not executed. At the supplied 2026-09-08 02:07:07 UTC snapshot, all four workflows were action_required, with no head or merge checks. The CI run has zero jobs. I did not run local builds or tests. The cached GitHub merge also uses an older base (7f1e0018), so it is not evidence for the assigned pair. This approval is based on source review and the test design, with CI still outstanding. No actionable correctness P1/P2 found.

Performance

This moves a planning-time eligibility check and adds diagnostic tagging on the fallback path. It introduces no row-processing work, native allocation, additional shuffle, or new aggregate implementation. The predictor no longer maintains a separate copy of the predicate. There is no performance claim to validate, and a new-expression microbenchmark is not applicable to this diagnostic fix. No actionable performance finding.

Design

Putting the refusal in getSupportLevel follows the existing CollectLimit and TakeOrderedAndProject pattern and keeps reason recording centralized. Reusing the exact composite shuffle predicate preserves its manager and Celeborn conditions. A mode-neutral message avoids directing users to one configuration key when several prerequisites can cause the refusal. The operator documentation now states the prerequisite and links to the tuning guidance. No additional design change is required.

Abstraction & complexity

The change removes duplicated eligibility logic and uses the existing support-level and explain mechanisms. The small test helpers build fresh plans and select the partial stage without adding a production abstraction. The remaining conversion method delegates directly to the shared aggregate implementation. No actionable complexity or abstraction finding.

[comphead]

can we concise the comments?