Repository navigation
docs: name the AQE query stages on the operator support page - #6660
Merged
andygrove merged 1 commit intoOct 5, 2026
Merged
Conversation
The Wrapper nodes section counted every AQE query stage as a wrapper but named only ResultQueryStage, so a search for TableCacheQueryStage found nothing and the stage looked like an operator Comet does not support. Name all four stages, matching the contributor guide's list of nodes Comet does not convert.
sunchao
approved these changes
Oct 5, 2026
sunchao
left a comment
Member
There was a problem hiding this comment.
Summary
- Prior state and problem: The operator support page referred to AQE query stages without naming
TableCacheQueryStage, making its handling unclear. - Design approach: Explicitly name all four stages in the existing wrapper paragraph.
- Correctness / compatibility analysis: Spark sources across supported 3.4–4.2 lines confirm the stage names and version boundaries.
TableCacheQueryStageExecappears in 3.5 andResultQueryStageExecin 4.0. Comet’s existing stage handling and coverage counting support the description. - Key design decisions: Keep scheduling wrappers separate from processing operators. The clarification adds no runtime overhead or implementation complexity.
- Implementation sketch: One paragraph changes in
docs/source/user-guide/latest/operators.md. - Behavioral changes worth calling out: Documentation becomes more searchable. Comparison with
branch-1.1found no runtime behavior change attributable to this PR. - Suggested improvements: None. No introduced P1/P2 issues found within this review.
Reviewed the entire diff from 0dec85749827fc63b30ad66a1bd074c2c313bdb2 to 3235ec97866607caecfdf2b8fc12b17dca92e747. Confirmed the PR is not a draft. Snapshot and live discussion checks contained no reviews or comments, and the snapshot contained no review threads.
Routed skills: review-comet-pr; no sibling skill applies to this documentation-only change.
Exact-head CI: check-pr-title and label passed. Preflight and Analyze Actions remain in progress. No completed failures were reported.
Validation: git diff --check passed. No builds, runtime tests, documentation rendering, or Prettier checks were run locally. CI is not yet final.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue. This is a small docs fix.
Rationale for this change
TableCacheQueryStageExecisn't on the Spark Operator Support page, so it looks like an operator Comet doesn't support yet. It's an AQE query stage, which does no data processing, and Comet already handles it: #5733 lets AQE wrap Comet's cache scan in the stage, and #6208 keeps the operators above it native after AQE re-plans them. The page's Wrapper nodes section already counts it as one of "the AQE query stages", but it names onlyResultQueryStage, so searching the page forTableCacheQueryStagefinds nothing.What changes are included in this PR?
The Wrapper nodes paragraph in
operators.mdnow names all four AQE query stages:ShuffleQueryStage,BroadcastQueryStage,TableCacheQueryStage(Spark 3.5 and later), andResultQueryStage(Spark 4.0 and later). These are the stages that the contributor guide's "Operators That Should Not Be Converted" section lists.How are these changes tested?
Docs only. Prettier passes on the changed file.