Repository navigation
Conversation
Replay materializes each event log line as a String with no length bound, so a corrupt or unexpectedly large log can exhaust the memory of the process replaying it. Cap lines at spark.history.fs.eventLog.maxLineLength (default 512m, <= 0 disables): longer lines are drained and skipped with a warning while the remaining events still replay. Co-authored-by: Cursor <cursoragent@cursor.com>
A single version does not tell an operator which maintenance releases carry a backported config, so name the set. Declare the version this branch first ships in, and add the row to the monitoring table -- until now the note only reached the config doc, where nobody reading the docs sees it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com> Co-Authored-By: Holden Karau <holden@pigscanfly.ca>
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Sep 25, 2026
…ory server ### What changes were proposed in this pull request? Backport of SPARK-59407 to branch-3.5, based on master commit 38258cb (#58700). Skip excessively long event log lines during History Server replay: `ReplayListenerBus` reads lines with a per-line byte cap (`spark.history.fs.eventLog.maxLineLength`, default 512m) instead of materializing unbounded lines; over-long lines are drained and skipped with a warning. Branch adaptations vs master: the config declares `.version("3.5.10")` and drops `.withBindingPolicy(...)` (ConfigBindingPolicy exists only on 4.2+); the new warning uses plain string interpolation since branch-3.5 predates the structured logging framework. ### Why are the changes needed? Individual corrupt or unexpectedly large event log lines can make loading a previous Spark job in the History Server slow or exhaust memory; bounding the per-line length keeps replay memory bounded. ### Does this PR introduce _any_ user-facing change? Yes: new configuration `spark.history.fs.eventLog.maxLineLength` (default 512m). Event log lines longer than the cap are skipped with a warning instead of being read into memory during replay; setting it to 0 or a negative value disables the limit. ### How was this patch tested? New test in `ReplayListenerSuite` covering that an over-long line is skipped while neighboring events still replay. Full core module test suite run locally on this branch, green. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude and Cursor Made with [Cursor](https://cursor.com) Closes #59058 from holdenk/f040-compressed-file-handling-branch-3.5. Lead-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
Contributor
Author
|
Merged to 3.5 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Backport of SPARK-59407 to branch-3.5, based on master commit 38258cb (#58700).
Skip excessively long event log lines during History Server replay:
ReplayListenerBusreads lines with a per-line byte cap (spark.history.fs.eventLog.maxLineLength, default 512m) instead of materializing unbounded lines; over-long lines are drained and skipped with a warning.Branch adaptations vs master: the config declares
.version("3.5.10")and drops.withBindingPolicy(...)(ConfigBindingPolicy exists only on 4.2+); the new warning uses plain string interpolation since branch-3.5 predates the structured logging framework.Why are the changes needed?
Individual corrupt or unexpectedly large event log lines can make loading a previous Spark job in the History Server slow or exhaust memory; bounding the per-line length keeps replay memory bounded.
Does this PR introduce any user-facing change?
Yes: new configuration
spark.history.fs.eventLog.maxLineLength(default 512m). Event log lines longer than the cap are skipped with a warning instead of being read into memory during replay; setting it to 0 or a negative value disables the limit.How was this patch tested?
New test in
ReplayListenerSuitecovering that an over-long line is skipped while neighboring events still replay. Full core module test suite run locally on this branch, green.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude and Cursor
Made with Cursor