Round 2: Rails/Puppet parse-conformance 83%→94% - #4
Merged
Conversation
A paren-less command call on a receiver (`obj.foo bar do … end`) greedily consumes its space-separated arguments and returned before it could pick up a trailing `do…end` block, so MRI-valid code such as `Model.set_callback :work, prepend: true do |_, inner| … end` failed to parse. Attach the block to the command call (and continue the postfix chain so the block-bearing call can itself be chained, `obj.foo bar do end.baz`), matching MRI. This is the single largest Rails/Puppet parse-conformance gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A single assignment target followed by multiple comma-separated values is an implicit array assignment in MRI (`args = "-t", "--server", master` ≡ `args = ["-t", "--server", master]`); the right-hand side may also carry a `*splat` (`a = *list, y`). Route the local/constant/ivar/cvar/gvar single assignment RHS through a new parseAssignRhs that gathers such a trailing list into an ArrayLit, leaving ordinary chained (`a = b = 1`) and single-value assignments unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MRI joins a line that ends in the low-precedence keyword operators `and`/`or` or in a trailing modifier keyword (`if`/`unless`/`while`/`until`) when its operand or condition sits on the next line (`x = 1 or\n fail`, `do_it unless\n cond`). Add these keyword token types to the lexer's trailing-operator continuation set alongside the infix operators it already handles; a keyword that is not the last token on its line (the ordinary `if cond` form) keeps its statement newline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add two related lexer gaps surfaced by the Rails/Puppet sweep: - `=begin` … `=end` block comments (each marker at column 0) are now skipped wholesale in skipSpaceAndComments, so documentation blocks no longer abort the parse. A `=begin` that is not at the start of a line stays an ordinary begin-expression (`x = begin … end`), and an identifier merely starting with "begin" is unaffected. - The bitwise op-assignments `|=`, `&=`, `^=`, and `>>=` were never lexed (only `<<=` was), so common code such as `mode |= flag` failed. Emit them as OPASSIGN like the other compound assignments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two related front-end gaps from the Rails/Puppet sweep:
- A bare `cmd` backtick command literal now lexes to the same XSTRING token
the `%x{…}` form produces (with \` and \\ resolved), so it parses to an XStr
node in value position.
- `def` now accepts the complete operator-method name set, including `===`,
`=~`, `&`, `|`, `^`, `>>`, `**`, the unary forms `+@`/`-@`/`~@`, and
`~`/`!`. `def /` (and `def %`) are special-cased in the lexer so the
operator name is not mis-lexed as a regexp / percent-literal opener; ordinary
regexp and division lexing elsewhere is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A quoted string key immediately followed by `:` is a symbol key — the quoted
analogue of a `name:` label — in both hash literals (`{'desc': 'x'}`,
`{"d-a-s-h": 2}`) and method-call keyword arguments (`tag(:div, "@click":
"f")`). Recognise it after parsing the key/argument expression: a plain string
becomes a SymbolLit key, an interpolated string a dynamic `"…".to_sym` key.
Ternaries over string operands are unaffected (the `:` is consumed by the
ternary before this check).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A `*splat` of an array is valid in a rescue class list (`rescue *EXCEPTIONS => e`), a when candidate list (`when *LIST`), and a multiple-assignment RHS (`path, headers = *args`, `a, b = 1, *rest`); each previously raised a parse error on the `*`. Route those positions through helpers that accept an optional leading splat, emitting a SplatArg node. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…equence parens
Three front-end edges from the Rails/Puppet sweep:
- Stabby lambdas accept unparenthesized parameters (`->x { }`, `-> ctx { }`,
`->a, b { }`, `-> message do … end`, `->x, y=1 { }`), parsed with the block
parameter grammar up to the block opener.
- Block and lambda parameter lists accept keyword parameters (`|a, b:|`,
`|channel, count: nil, timeout: 5|`), recorded with a trailing-colon sentinel
name and an optional default, mirroring the existing **rest handling.
- A parenthesised group is now a full statement sequence, so it accepts trailing
modifiers (`(expr if cond)`, `(x if y) || z`) and multiple
semicolon-separated statements (`(a; b)`), evaluating to its last expression
(a multi-statement group becomes a Begin).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A parenthesised `def` parameter list may span several lines (newlines after the open paren, around the commas, and before the close paren), as MRI allows: `def f(\n a,\n b = Encoding::UTF_8\n)`. Skip newlines at the parameter-list boundaries only when the list is parenthesised; a paren-less list still terminates at its newline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- `alias NewName OldName` and `undef name[, name…]` are now keyword statements (`alias :== :eql?`, `undef foo, bar`); each name item may be a symbol, an operator symbol, a bare method name, or — for alias — a global variable. They parse to new lightweight Alias / Undef AST nodes. The plain-identifier `alias_method` is unaffected. - A multiple-assignment right-hand value may itself be a (chained) assignment whose result is destructured (`a, b = c = [1, 2]`, `_, h, _ = resp = call(x)`); parse each masgn value with parseExprOrAssign. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…polation bodies
Three further Rails/Puppet gaps:
- `def` accepts an explicit receiver that is an instance/class/global variable
(`def @controller.foo`, `def @@reg.bar`, `def $g.baz`), not just
self/local/constant.
- `def` is a value-producing expression, so it is accepted as a paren-less
command argument (`module_function def server; end`, `private def bar; end`).
- A `#{…}` string-interpolation body is parsed as a full statement sequence, so
it admits trailing modifiers (`"#{'s' if n > 1}"`) and several
semicolon-separated statements; an empty `#{}` yields nil.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- `A::B = v` and `::Top::X = v` assign to a scope-resolved constant; they parse to a new ScopedConstAssign node (previously the `=` after a ScopedConst was unexpected). - A class superclass is now any expression, so `class A < self`, `class A < Struct.new(:x, :y)`, and `class A < ns::Base` parse. A bare-constant superclass still uses the plain Super name; anything else goes into SuperExpr. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tests covering the remaining new branches to keep the package at 100% statement coverage: block-comment marker boundaries (near-miss, EOF, glued letters, not-at-column-0), backtick escape passthrough and unterminated forms, non-string hash key before a bare colon, alias/undef bad-item errors, and an interpolated quoted symbol key in a call. Also fix lexBacktick so a backslash at end of input yields ILLEGAL (an unterminated literal) rather than a truncated XSTRING. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Round-2 front-end work driving go-ruby-parser toward 100% parse-conformance,
measured against the Rails and Puppet corpora with MRI 4.0.x (
ruby -c) as theoracle. Every feature has MRI-verified repros and MRI-parity table tests; the
package stays at 100% statement coverage (CI gate) and
go test -race,gofmt, andgo vetare all clean.Parse-conformance delta (files that parse cleanly)
Features landed (each MRI-verified)
do…endblock on a receiver command call —obj.foo bar do … endnowattaches the block to the command call (and continues the postfix chain so a
block-bearing command can be chained). This was the single largest gap.
args = "-t", "--server", x≡args = [...], including a*splat(a = *list, y).and/or/if/unless/while/untiljoins the next line.=begin…=endblock comments, plus the previously-missing bitwiseop-assignments
|=,&=,^=,>>=.`cmd`→ the%x{}XStr node) and thefull operator-method
defset (===,=~,&,|,^,>>,**,the unary
+@/-@/~@,~,!, anddef /without regexp confusion).*splatin rescue/when lists and masgn RHS —rescue *EX => e,when *LIST,path, headers = *args.{'a': 1},{"d-a-s-h": 2},tag(:div, "@click": "f"), interpolated"x#{y}":→ a dynamic.to_symkey.->x { },-> ctx { },->a, b { },-> message do … end.if/unlessand statement sequences inside parens —(expr if cond),(x if y) || z,(a; b).|a, b:|,|c, count: nil, timeout: 5|.Additional gaps closed by the corpus sweep
alias NewName OldName/undef name[, …]keyword statements (newAlias/Undefnodes).a, b = c = expr,_, h, _ = resp = call(x).defwith an instance/class/global-variable receiver (def @ctrl.foo);defas a paren-less command argument (
module_function def foo; end).#{…}interpolation bodies parsed as full statements (trailing modifiers,multiple statements).
A::B = v(newScopedConstAssignnode);expression superclasses
class A < Struct.new(:x),class A < self.rbgo (go-embedded-ruby) compiler follow-ups
Verified end-to-end against rbgo via a throwaway
replace(NOT committed).Almost everything lowers with the existing compiler — the do-block, RHS array
(non-splat), backtick, keyword continuation, operator def, block kwargs, lambda
args, and statement interpolation all run. The new AST shapes the compiler does
not yet handle (separate go-embedded-ruby work):
*ast.Alias,*ast.Undef,*ast.ScopedConstAssign— new node kinds.*ast.SplatArgin masgn / RHS-array value position (a, b = *x,a = *x, y)— the compiler currently handles SplatArg only inside call arguments.
Testing
go test -race ./...green.-coverpkggate.🤖 Generated with Claude Code