Parser round 3: 9 grammar features toward 100% parse-conformance (Rails 94.31% → 96.82%) - #5
Merged
Merged
Conversation
A in value position (or command-argument position) followed by a single character or backslash escape is a one-character String literal (MRI: ), distinct from the ternary that follows a finished value. Escapes mirror double-quote semantics; a multi-byte UTF-8 rune is taken whole. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MRI accepts any non-alphanumeric, non-whitespace character as a percent-literal delimiter for every percent kind: %r"...", %Q'...', %q[...], %w'a b', %i<x y>, %(...), %!...!, %|...|, %@...@, etc. isPercentDelim now admits the full set (excluding alphanumerics, whitespace, multibyte leads, and '=' so a %= compound assignment is unaffected); %w/%i delimiter detection shares it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adjacent string literals concatenate at parse time across every form, including
interpolated pieces: "a" "b" folds to one StringLit, "a" "b#{x}" folds to
one StrInterp, and a backslash-continued line joins too. A string literal that
hugs a name (no intervening space) is a paren-less command argument — foo"bar",
step"x" do…end, obj.m"x", Foo"bar" — read as a call even when the name is a
local (MRI: x"y" is x("y")).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A bare ':' is unambiguously the ternary separator at the lexer level (a symbol ':x' and a label 'x:' are distinct tokens), so a line ending in it is incomplete and joins the next line, completing the multi-line ternary 'cond ?'<nl>'a :'<nl>'b'. The '?' branch was already a continuation operator; this adds the ':' branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MRI permits an assignment to sit inside a larger expression even though '=' binds looser than the binary operators: 'if a && b = c', 'while x && line = gets', 'r = a && @b = 1'. After parsing a binary operator's right operand, an '=' that follows an assignable target (local/ivar/cvar/gvar/const, or an attribute/index receiver call) is consumed as an assignment. '=='/'=>'/'=~' are distinct tokens, so only a real '=' is taken. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MRI allows a bare jump as the right operand of && / ||: 'x || return', 'expr &&
next', 'cond || break', 'begin; x || retry; …'. Only the argument-less form is
valid in this position ('x || return 5' is a syntax error), so the binary
right-operand path consumes return/break/next/retry only when value-less.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A bare return/break/next/retry now ends at ')' or ']', so a control-flow jump can sit inside a parenthesised group: '( x or next )[1]', '(set_param(p) and return) unless cur'. Previously the jump greedily consumed the closing delimiter as its value. The '(expr if cond)' modifier form already worked via the group's statement-sequence parse. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
retry is not accepted by MRI as a binary operand or an or/and operand, so valuelessJumpOperand now covers only return/break/next; corrects the round-3 operand tests accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The |params| list of a brace/do block may begin on the line(s) after the opener:
'foo {'<nl>'|x| … }', 'do'<nl>'|a, b| … end'. parseBlockRest now skips the
intervening newlines when the first significant token is a '|' (a body statement
can never start with a bare '|', so this is unambiguous), leaving param-less
blocks unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add parity/edge tests covering every char-literal escape, the percent-delimiter accept/reject branches (alphanumeric, whitespace, '=', multibyte), inline assignment to an existing local and the non-assignable fall-through, and the continued-block-params lookahead; simplify firstSignificantIs to drop a dead branch (the stream always ends in a non-NEWLINE EOF). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Round 3 — toward 100% parse-conformance
Nine MRI-verified grammar features, each landed with parity tests and 100% coverage maintained. Verified against MRI 4.0.x (
ruby -c/ruby -e) and exercised end-to-end throughrbgo(no compiler/lowering changes were needed — every feature reuses existing AST nodes).Features (with MRI-verified repros)
?x—?a,?|,?\n,?\s,?é, value- and command-arg position; stays ternary after a finished value (1 ?2 :3).%r"...",%Q'...',%q[...],%w'a b',%i<x y>,%(...),%!...!,%@...@;%=modulo/op-assign untouched.\-continued lines:"a" "b"→"ab","a" "b#{x}"→ oneStrInterp.step"Ensure ..." do…end,foo"bar",obj.m"x",Foo"bar"(a call even when the name is a local:x"y"==x("y")).:now continues:cond ?a :b.if a && b = c,while x && line = gets,r = a && @b = 1(only a real=, not==/=>/=~).x || return,expr && next,cond || break(MRI rejects a value, e.g.x || return 5, andretry).)/]:( x or next )[1],(set_param(p) and return) unless cur.|params|on a continued line —foo {|x| … },do|a, b| … end.Rails parse-conformance
Measured over 3460 Ruby files in
rails/rails:+87 files (+2.51 points).
Quality
go test ./...green,-raceclean.gofmt+go vetclean, CGO=0.rbgo / consumer follow-up
All nine repros run correctly through
rbgoend-to-end with no compiler changes. One unrelated, pre-existing consumer test (go-embedded-rubyTestStringInterpolationMalformed, which asserts the literal word "malformed" for"#{1 2}") already fails againstorigin/maintoday — the error text changed in Round 2, not here — and is left as a separate follow-up.🤖 Generated with Claude Code