SQLAppStatusListenerMemoryLeakSuite "no memory leak" is flaky in the Spark SQL tests #6890

Describe the bug

org.apache.spark.sql.execution.ui.SQLAppStatusListenerMemoryLeakSuite ("no memory leak"), which runs in the sql_core-1 shard of the Spark SQL tests, fails intermittently:

[info] - no memory leak *** FAILED *** (1 second, 338 milliseconds)
[info]   statusStore.listener.get.noLiveData() was false (SQLAppStatusListenerSuite.scala:1123)

It is the only failing test in the shard when it happens (nightly 2026-10-10, Spark 4.0: Tests: succeeded 7847, failed 1).

The cause is a race in the Spark test itself. Upstream Spark fixed it on master, but none of the Spark versions Comet's tests run against has the fix.

The race. In DAGScheduler.failJobAndIndependentStages, the job waiter is notified before SparkListenerJobEnd is posted:

cleanupStateForJobAndIndependentStages(job)
job.listener.jobFailed(error)   // unblocks the failing action on the test thread
listenerBus.post(SparkListenerJobEnd(job.jobId, clock.getTimeMillis(), JobFailed(error)))

The test runs 100 failing jobs (df.foreach(_ => throw new RuntimeException("Oops"))), then calls sc.listenerBus.waitUntilEmpty() and asserts noLiveData(). If the last JobEnd is posted after the bus has been drained, the last failed execution never reaches its end-event count and stays in liveExecutions, so the assertion sees live data. In the 2026-10-10 log the assertion fails 22 ms after the last Oops task failure, and the shard log has no dropped-event messages.

Steps to reproduce

Not deterministic. It occurred in 2 of 252 CI executions of the shard (see below). Upstream reports that it reproduces under SBT when the test is run repeatedly.

Expected behavior

The suite passes consistently.

Additional context

Sightings. Every nightly, merge-queue and label run from 2026-10-02 to 2026-10-12 was checked. The sql_core-1 job ran 252 times and the test failed in 2 of them, about 0.8%. Commit runs of PRs that carry a run-* label also execute the shards and were not counted.

By run type: 1 of 32 nightly executions, 1 of 39 label-run executions and 0 of 181 merge-queue executions. The other failed sql_core-1 jobs in the window failed other tests on PR branches, or had no failed step and no log (a lost runner).

Impact. CI only. It turns the nightly or a label run red. In the merge queue it would evict the PR, which has not happened yet.

Every Spark version Comet tests has the race. The assertion is unguarded and the DAGScheduler order above is the same in 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0, and in the 4.0.5-rc2, 4.1.4-rc2 and 4.2.1-rc1 release candidates. A patch-version bump will not fix it.

Upstream fix. apache/spark#56790 (SPARK-57711) wraps the final assertion in eventually(timeout(5.seconds), interval(10.milliseconds)). It was merged to master and branch-4.x on 2026-06-27 and is in 4.3.0-rc1. It is not on branch-4.2, branch-4.1, branch-4.0 or branch-3.5. Upstream fixed the test, and the scheduler order is unchanged in 4.3.0-rc1.

Is Comet involved? The suite builds its own SparkContext, and #6415 (2026-10-02) made suites like it run with Comet. The race is in Spark's scheduler and listener bus, and upstream hit the same failure on plain Spark SBT builds (example run). Whether running with Comet makes the window wider has not been measured.

Options.

1. Carry the upstream test change (one import and the eventually wrapper) in dev/diffs/{3.4.3,3.5.9,4.0.4,4.1.3,4.2.0}.diff. The diffs have to be regenerated as described in the Spark SQL tests guide, not edited by hand. This is the smallest change and matches upstream.
2. Do nothing and re-run the failed job when it happens.
3. Mark the test IgnoreComet. This also drops the check that a Comet plan leaves no live SQL status data behind, so it is the least attractive.