Take the allocation-free bounds-only regexp path, rebuild captures lazily - #2
Merged
Conversation
…zily go-ruby-regexp gained a bounds-only match API (MatchBoundsAt / MatchBounds returning (begin, end, ok) with no MatchData) whose lazy-NFA subset runs on pooled, per-call-allocation-free state. It exists precisely for a StringScanner-style cursor: scan / skip / match? / scan_until re-match an advancing cursor over capture-free class/quantifier patterns and almost never read a capture group. The scanner previously built a full capture-bearing MatchData on every recording op (MatchAt / Match), paying a fresh capture array per token in the tokenizing hot loop even though a lexer never inspects the groups. Switch every regexp-driven op (Scan/ScanUntil/Skip/SkipUntil/Match/Check/CheckUntil) to the bounds-only primitive, which yields the whole-match [beg,end) span with no allocation, and defer building the MatchData until a group is actually read (Group / GroupName, i.e. Ruby's StringScanner#[]). The lazy rebuild reproduces the identical match the bounds-only path found: anchored ops via MatchAt(str, matchBeg) — byte-for-byte the call the old eager path made — and forward ops via Match(rest) over the same rest slice. So captures stay byte-identical to MRI after every op, including scan_until / skip / match? / check_until (MRI's #[] works after all of them), while a scan/skip loop that never touches a group now allocates nothing per token. Bump the go-ruby-regexp pin to the match-optimized pseudo-version. Add a differential-oracle scenario reading capture groups after scan_until / skip / match? / check_until so the lazy forward-rebuild path is covered and its MRI parity is asserted. Coverage stays at 100%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
go-ruby-regexpgained a bounds-only match API (MatchBoundsAt/MatchBoundsreturning(begin, end, ok)with noMatchData) whose lazy-NFA subset runs on pooled, per-call-allocation-free state — built precisely for a StringScanner-style cursor. This wiresstrscanonto it.Scan/ScanUntil/Skip/SkipUntil/Match/Check/CheckUntil) now takes the bounds-only primitive: the whole-match[beg,end)span with noMatchDataallocation on the tokenizing hot path.MatchDatais rebuilt lazily only when a group is actually read (Group/GroupName, i.e. Ruby'sStringScanner#[]), reproducing the identical match: anchored ops viaMatchAt(str, matchBeg)(byte-for-byte the old eager call), forward ops viaMatch(rest)over the samerestslice.#[]still works afterscan/scan_until/skip/match?/check_until(MRI exposes captures after all of them) — a new differential-oracle scenario asserts it against real MRI and covers the lazy forward-rebuild path.go-ruby-regexppin to the match-optimized pseudo-version.Impact (library-level benchmark, Apple M4 Max, same-session before→after)
go-vs-YJIT on the regexp-driven ops:
scan-tokenizescan_untilmatch?skipTests
go build/go vet/go test -racegreen; differential oracle vs real MRI 4.0.5 passes.🤖 Generated with Claude Code