Skip to content

Round 2: Rails/Puppet parse-conformance 83%→94% - #4

Merged
tannevaled merged 13 commits into
mainfrom
parser-100pct-round2
Jun 26, 2026
Merged

Round 2: Rails/Puppet parse-conformance 83%→94%#4
tannevaled merged 13 commits into
mainfrom
parser-100pct-round2

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Summary

Round-2 front-end work driving go-ruby-parser toward 100% parse-conformance,
measured against the Rails and Puppet corpora with MRI 4.0.x (ruby -c) as the
oracle. Every feature has MRI-verified repros and MRI-parity table tests; the
package stays at 100% statement coverage (CI gate) and go test -race,
gofmt, and go vet are all clean.

Parse-conformance delta (files that parse cleanly)

Corpus Before After Gain
Rails 83.1% (2844/3423) 94.3% (3228/3423) +384 files
Puppet 82.5% (1778/2156) 93.6% (2017/2156) +239 files

Features landed (each MRI-verified)

  1. do…end block on a receiver command callobj.foo bar do … end now
    attaches the block to the command call (and continues the postfix chain so a
    block-bearing command can be chained). This was the single largest gap.
  2. Single-target implicit array assignmentargs = "-t", "--server", x
    args = [...], including a *splat (a = *list, y).
  3. Trailing keyword-operator / modifier continuation — a line ending in
    and/or/if/unless/while/until joins the next line.
  4. =begin=end block comments, plus the previously-missing bitwise
    op-assignments |=, &=, ^=, >>=.
  5. Backtick command literals (`cmd` → the %x{} XStr node) and the
    full operator-method def set (===, =~, &, |, ^, >>, **,
    the unary +@/-@/~@, ~, !, and def / without regexp confusion).
  6. *splat in rescue/when lists and masgn RHSrescue *EX => e,
    when *LIST, path, headers = *args.
  7. Quoted string-key symbol shorthand{'a': 1}, {"d-a-s-h": 2},
    tag(:div, "@click": "f"), interpolated "x#{y}": → a dynamic .to_sym key.
  8. Stabby lambda with unparenthesized params->x { }, -> ctx { },
    ->a, b { }, -> message do … end.
  9. Modifier if/unless and statement sequences inside parens
    (expr if cond), (x if y) || z, (a; b).
  10. Keyword args in block params|a, b:|, |c, count: nil, timeout: 5|.
  11. Multi-line / scoped-default parenthesised parameter lists.

Additional gaps closed by the corpus sweep

  • alias NewName OldName / undef name[, …] keyword statements (new Alias /
    Undef nodes).
  • Chained masgn RHS — a, b = c = expr, _, h, _ = resp = call(x).
  • def with an instance/class/global-variable receiver (def @ctrl.foo); def
    as a paren-less command argument (module_function def foo; end).
  • #{…} interpolation bodies parsed as full statements (trailing modifiers,
    multiple statements).
  • Scope-resolved constant assignment A::B = v (new ScopedConstAssign node);
    expression superclasses class A < Struct.new(:x), class A < self.

rbgo (go-embedded-ruby) compiler follow-ups

Verified end-to-end against rbgo via a throwaway replace (NOT committed).
Almost everything lowers with the existing compiler — the do-block, RHS array
(non-splat), backtick, keyword continuation, operator def, block kwargs, lambda
args, and statement interpolation all run. The new AST shapes the compiler does
not yet handle (separate go-embedded-ruby work):

  • *ast.Alias, *ast.Undef, *ast.ScopedConstAssign — new node kinds.
  • *ast.SplatArg in masgn / RHS-array value position (a, b = *x, a = *x, y)
    — the compiler currently handles SplatArg only inside call arguments.

Testing

  • New MRI-parity table tests per feature; go test -race ./... green.
  • 100% statement coverage with the CI's -coverpkg gate.
  • Validated on the Rails and Puppet corpora (numbers above).

🤖 Generated with Claude Code

tannevaled and others added 13 commits June 26, 2026 14:20
A paren-less command call on a receiver (`obj.foo bar do … end`) greedily
consumes its space-separated arguments and returned before it could pick up a
trailing `do…end` block, so MRI-valid code such as
`Model.set_callback :work, prepend: true do |_, inner| … end` failed to parse.

Attach the block to the command call (and continue the postfix chain so the
block-bearing call can itself be chained, `obj.foo bar do end.baz`), matching
MRI. This is the single largest Rails/Puppet parse-conformance gap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A single assignment target followed by multiple comma-separated values is an
implicit array assignment in MRI (`args = "-t", "--server", master` ≡
`args = ["-t", "--server", master]`); the right-hand side may also carry a
`*splat` (`a = *list, y`). Route the local/constant/ivar/cvar/gvar single
assignment RHS through a new parseAssignRhs that gathers such a trailing list
into an ArrayLit, leaving ordinary chained (`a = b = 1`) and single-value
assignments unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MRI joins a line that ends in the low-precedence keyword operators `and`/`or`
or in a trailing modifier keyword (`if`/`unless`/`while`/`until`) when its
operand or condition sits on the next line (`x = 1 or\n  fail`, `do_it
unless\n  cond`). Add these keyword token types to the lexer's
trailing-operator continuation set alongside the infix operators it already
handles; a keyword that is not the last token on its line (the ordinary
`if cond` form) keeps its statement newline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add two related lexer gaps surfaced by the Rails/Puppet sweep:

- `=begin` … `=end` block comments (each marker at column 0) are now skipped
  wholesale in skipSpaceAndComments, so documentation blocks no longer abort the
  parse. A `=begin` that is not at the start of a line stays an ordinary
  begin-expression (`x = begin … end`), and an identifier merely starting with
  "begin" is unaffected.
- The bitwise op-assignments `|=`, `&=`, `^=`, and `>>=` were never lexed
  (only `<<=` was), so common code such as `mode |= flag` failed. Emit them as
  OPASSIGN like the other compound assignments.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two related front-end gaps from the Rails/Puppet sweep:

- A bare `cmd` backtick command literal now lexes to the same XSTRING token
  the `%x{…}` form produces (with \` and \\ resolved), so it parses to an XStr
  node in value position.
- `def` now accepts the complete operator-method name set, including `===`,
  `=~`, `&`, `|`, `^`, `>>`, `**`, the unary forms `+@`/`-@`/`~@`, and
  `~`/`!`. `def /` (and `def %`) are special-cased in the lexer so the
  operator name is not mis-lexed as a regexp / percent-literal opener; ordinary
  regexp and division lexing elsewhere is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A quoted string key immediately followed by `:` is a symbol key — the quoted
analogue of a `name:` label — in both hash literals (`{'desc': 'x'}`,
`{"d-a-s-h": 2}`) and method-call keyword arguments (`tag(:div, "@click":
"f")`). Recognise it after parsing the key/argument expression: a plain string
becomes a SymbolLit key, an interpolated string a dynamic `"…".to_sym` key.
Ternaries over string operands are unaffected (the `:` is consumed by the
ternary before this check).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A `*splat` of an array is valid in a rescue class list (`rescue *EXCEPTIONS =>
e`), a when candidate list (`when *LIST`), and a multiple-assignment RHS
(`path, headers = *args`, `a, b = 1, *rest`); each previously raised a parse
error on the `*`. Route those positions through helpers that accept an optional
leading splat, emitting a SplatArg node.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…equence parens

Three front-end edges from the Rails/Puppet sweep:

- Stabby lambdas accept unparenthesized parameters (`->x { }`, `-> ctx { }`,
  `->a, b { }`, `-> message do … end`, `->x, y=1 { }`), parsed with the block
  parameter grammar up to the block opener.
- Block and lambda parameter lists accept keyword parameters (`|a, b:|`,
  `|channel, count: nil, timeout: 5|`), recorded with a trailing-colon sentinel
  name and an optional default, mirroring the existing **rest handling.
- A parenthesised group is now a full statement sequence, so it accepts trailing
  modifiers (`(expr if cond)`, `(x if y) || z`) and multiple
  semicolon-separated statements (`(a; b)`), evaluating to its last expression
  (a multi-statement group becomes a Begin).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A parenthesised `def` parameter list may span several lines (newlines after the
open paren, around the commas, and before the close paren), as MRI allows:
`def f(\n  a,\n  b = Encoding::UTF_8\n)`. Skip newlines at the parameter-list
boundaries only when the list is parenthesised; a paren-less list still
terminates at its newline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- `alias NewName OldName` and `undef name[, name…]` are now keyword statements
  (`alias :== :eql?`, `undef foo, bar`); each name item may be a symbol, an
  operator symbol, a bare method name, or — for alias — a global variable. They
  parse to new lightweight Alias / Undef AST nodes. The plain-identifier
  `alias_method` is unaffected.
- A multiple-assignment right-hand value may itself be a (chained) assignment
  whose result is destructured (`a, b = c = [1, 2]`, `_, h, _ = resp =
  call(x)`); parse each masgn value with parseExprOrAssign.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…polation bodies

Three further Rails/Puppet gaps:

- `def` accepts an explicit receiver that is an instance/class/global variable
  (`def @controller.foo`, `def @@reg.bar`, `def $g.baz`), not just
  self/local/constant.
- `def` is a value-producing expression, so it is accepted as a paren-less
  command argument (`module_function def server; end`, `private def bar; end`).
- A `#{…}` string-interpolation body is parsed as a full statement sequence, so
  it admits trailing modifiers (`"#{'s' if n > 1}"`) and several
  semicolon-separated statements; an empty `#{}` yields nil.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- `A::B = v` and `::Top::X = v` assign to a scope-resolved constant; they parse
  to a new ScopedConstAssign node (previously the `=` after a ScopedConst was
  unexpected).
- A class superclass is now any expression, so `class A < self`, `class A <
  Struct.new(:x, :y)`, and `class A < ns::Base` parse. A bare-constant
  superclass still uses the plain Super name; anything else goes into SuperExpr.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tests covering the remaining new branches to keep the package at 100%
statement coverage: block-comment marker boundaries (near-miss, EOF, glued
letters, not-at-column-0), backtick escape passthrough and unterminated forms,
non-string hash key before a bare colon, alias/undef bad-item errors, and an
interpolated quoted symbol key in a call. Also fix lexBacktick so a backslash at
end of input yields ILLEGAL (an unterminated literal) rather than a truncated
XSTRING.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 6b7342b into main Jun 26, 2026
9 checks passed
@tannevaled
tannevaled deleted the parser-100pct-round2 branch June 26, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant