Skip to content

fix: run the Spoon pipeline on a thread with a configurable stack size - #6882

Open
monperrus wants to merge 2 commits into
INRIA:masterfrom
monperrus:fix/6804-deep-expression-stack-size
Open

fix: run the Spoon pipeline on a thread with a configurable stack size#6882
monperrus wants to merge 2 commits into
INRIA:masterfrom
monperrus:fix/6804-deep-expression-stack-size

Conversation

@monperrus

Copy link
Copy Markdown
Collaborator

Summary

Alternative to #6805 for #6804: give the Spoon pipeline a stack proportional to the problem, instead of rewriting individual visitors iteratively.

Problem

Spoon's visitors are recursive — one JVM frame per AST node on the path from the root to the node being visited. OpenJDK's valid DeepStringConcat.java has ~32,000 binary operators in one expression, which exhausts the default thread stack of a few hundred kilobytes.

AstParentConsistencyChecker and ProcessingVisitor are the two visitors named in the issue, but they are only the first two to be reached. On current master, with the 32k-operator file:

operation default stack 16 MB thread
buildModel() — parent check SOE OK
process() — processors SOE OK
prettyprint() SOE OK
getElements(filter) SOE OK
root.toString() SOE OK
root.clone() SOE OK
equals SOE OK

The recursion lives in CtScanner, EarlyTerminatingScanner, LexicalScopeScanner, CloneVisitor, EqualsVisitor, DefaultJavaPrettyPrinter and every user-written CtScanner subclass. Binary operators are also only the easiest deep chain — nested if/else and long .a().b().c() chains recurse the same way. Making visitors iterative one at a time does not converge.

Change

Launcher runs buildModel(), process(), prettyprint() and run() on a thread whose stack size is Environment#getStackSize().

  • Default 64 MB, 4x the measured requirement. The stack is reserved address space, not committed memory: pages are only committed as the stack actually grows, so it costs nothing on ordinary models.
  • setStackSize(0) runs the pipeline on the calling thread, restoring today's behaviour for embedders that manage their own threads.
  • Nested steps reuse the thread the outermost step created, so run() spawns one thread, not four.
  • Exceptions propagate to the caller unchanged: a RuntimeException or Error is rethrown as-is, anything else is wrapped in SpoonException.
  • The two new Environment methods are default, so third-party Environment implementations keep compiling.

+140 lines of main code, against 373 in #6805, and it covers seven operations rather than two.

Known limitation

A user calling element.toString() or getElements() on their own thread, outside the launcher, is not covered — they can size their own thread, and Environment#getStackSize()'s Javadoc says so. Nothing short of rewriting every visitor covers that case.

Validation

  • DeepExpressionTest#testDeepExpressionPipeline builds, processes and pretty-prints the 32k-operator expression, then asserts that the processor saw all 32,000 operators and that the printed file contains all 32,000 +. It runs in ~6 s.
  • Verified non-vacuous: with setStackSize(0) the same test fails with StackOverflowError.
  • Three further tests cover setStackSize(0), single-thread reuse across nested steps, and rejection of a negative stack size.
  • Full suite and checkstyle left to CI.

Spoon's visitors are recursive: they use one JVM frame per AST node on the
path from the root to the node being visited. Legal but deeply nested
expressions, such as the ~32,000 binary operators of OpenJDK's
test/langtools/tools/javac/DeepStringConcat.java, therefore exhaust the
default thread stack of a few hundred kilobytes.

The recursion is not confined to one visitor: AstParentConsistencyChecker,
ProcessingVisitor, EarlyTerminatingScanner (model queries), CloneVisitor,
EqualsVisitor and DefaultJavaPrettyPrinter all overflow on the same input.
Rather than rewriting each of them iteratively, Launcher now runs
buildModel(), process(), prettyprint() and run() on a thread whose stack
size is Environment#getStackSize(), which defaults to 64 MB. The stack is
reserved address space, not committed memory, so this costs nothing on
ordinary models. Setting it to 0 restores the previous behaviour for
embedders that manage their own threads.

Measured on the reproducer: 16 MB is enough for every operation; the whole
pipeline, including pretty-printing, runs in about 5 s.

Fixes INRIA#6804

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@SirYwell

SirYwell commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

This has the potential of a significant slowdown when invoking the relevant methods for small elements, right?

On Java 21, we could use virtual threads which would be cheaper in that aspect. Other than that, are there more sophisticated patterns to avoid stack overflows for deep visitor-based interactions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants