Repository navigation
Docs: Document adaptive split sizing configurations - #16557
Conversation
|
Local Rendering: |
|
@RussellSpitzer @karuppayya Could you PTAL? Thanks! |
|
@kevinjqliu @huaxingao Could you take a look. Thanks! |
| | spark.sql.iceberg.merge-schema | false | Enables modifying the table schema to match the write schema. Only adds columns missing columns | | ||
| | spark.sql.iceberg.report-column-stats | true | Report Puffin Table Statistics if available to Spark's Cost Based Optimizer. CBO must be enabled for this to be effective | | ||
| | spark.sql.iceberg.read.adaptive-split-size.enabled | Table default | Enables adaptive split sizing for read operations. When enabled, split size is automatically adjusted based on scan size and parallelism | | ||
| | spark.sql.iceberg.read.adaptive-split-size.parallelism | max(spark.default.parallelism, spark.sql.shuffle.partitions) | Overrides the parallelism used for adaptive split sizing. Must be greater than 0 | |
There was a problem hiding this comment.
Rather than documenting spark config, I would just say "Spark's default parallelism"
There was a problem hiding this comment.
Thanks @RussellSpitzer , have modified.
Updated Local rendering.
There was a problem hiding this comment.
On another thought, the default value does not exactly correspond to spark's default parallelism in this case, as it is maximum of both values spark.default.parallelism and spark.sql.shuffle.partitions, thought of documenting it explicitly. Please do provide your thoughts on this. Thanks!
There was a problem hiding this comment.
I just wanted to avoid any description of non-iceberg properties and keep the default to as few characters as possible. If you want to switch it back it's probably fine.
There was a problem hiding this comment.
Thank you! Have changed it back to actual configs for now as it seemed appropriate, please do let know if this is ok to check in for now? CI has passed.
61b4540 to
a16d6cd
Compare
|
Merged, Thank you @pratham76 for the PR and @karuppayya for the Review |
3 participants
Documenting the adaptive split sizing configurations that were added in #16088 but were missed to be documented.
Closes #16556