Skip to content

Fix v2 scanner compatibility with Python 3.15 - #552

Open
manav4u wants to merge 5 commits into
lepture:v2from
manav4u:fix-v2-python315-scanner
Open

manav4u wants to merge 5 commits into
lepture:v2from
manav4u:fix-v2-python315-scanner

Conversation

@manav4u

@manav4u manav4u commented Nov 10, 2026

Copy link
Copy Markdown

Fixes #551. Target branch: v2.

Python 3.15, 3.14.1 and 3.13.10 reject capturing groups in re.Scanner. This prevents even mistune.html("X") from running on the v2 branch.

Keep the existing scanner on runtimes that accept it. When initialization raises the specific capture-rejection error, compile rules separately and choose the earliest match, using rule order to break ties. This preserves local captures and backreferences without rebuilding regex bytecode. Keep the older scanner's default character-class behavior, handle overlapping candidates, and separate shortcut-reference handling from the two-group reference-link rule.

Add eight regression cases for captures, backreferences, priority, overlapping candidates, empty matches, character classes, unrelated initialization errors and the inherited scan contract. Add Python 3.15 to the existing v2 CI matrix.

Validation on Linux:

  • Unmodified v2: 464 failed / 98 passed on Python 3.15.1, 3.14.1 and 3.13.10, with capture-rejection errors.
  • Candidate: 570 passed on Python 3.15.1, 3.14.1, 3.14.9, 3.13.10 and 3.13.17.
  • The full suite also passes when forcing the fallback on Python 3.14.9.
  • 74,648 bounded scanner comparisons preserved token identity and spans against the older scanner.

The fallback is not claimed to be uniformly faster. A small formatting-heavy sample took about 1.43x the older scanner's runtime; a plain-text sample and the CommonMark fixture corpus were faster in the same local check. The older-runtime path remains unchanged.

Python 2.7, PyPy and the full supported-version CI matrix were not run locally. Arbitrary third-party scanner subclasses are not covered by these tests.

@dairiki

dairiki commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Note that #553 presents an alternative PR to this one.

By way of comparison, the changes made by #553 are (IMHO) more minimal than the changes made here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants