Skip to content

fix(lexer): interpret \xHH hex and \NNN octal escapes in interpolating strings - #9

Merged
tannevaled merged 1 commit into
mainfrom
fix-hex-octal-escapes
Jun 30, 2026
Merged

fix(lexer): interpret \xHH hex and \NNN octal escapes in interpolating strings#9
tannevaled merged 1 commit into
mainfrom
fix-hex-octal-escapes

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Problem

Interpolating string literals ("...", %Q{...}/%{...}, interpolating heredocs) did not interpret \xHH (hex byte), \NNN (octal byte), or \uHHHH/\u{...} (Unicode) escapes — they dropped the backslash and leaked the literal characters. The basic escapes (\n, \t, \0, …) already worked, so the escape interpreter existed but was incomplete.

source before MRI 4.0.5 / after
"\x41" "x41" "A"
"\101" "101" "A" (octal)
"\u{41}" "u{41}" "A"
"\n", "\0" newline / NUL unchanged (already worked)

Fix

lexer.scanStringSegment is the escape interpreter shared by every interpolating string form (%Q/%{} and interpolating heredocs re-lex their body as a double-quoted string, so they funnel through it). The missing cases were added there, matching MRI 4.0.5 byte-for-byte:

  • \xHH — 1–2 hex digits, greedy (\x41A, \xff0xFF, \x40x04). A \x with no hex digit degrades to a literal x (MRI raises a SyntaxError; this entry point has no error channel).
  • \NNN octal — 1–3 octal digits, greedy, masked to one byte (\101A, \12\n, \3770xFF, \4000x00).
  • \uHHHH (exactly four hex digits) and \u{cp cp …} (whitespace-separated codepoints), each emitted as UTF-8; out-of-range / surrogate values fall back to U+FFFD.

Single-quoted strings, %q{}, and the regexp pass-through are deliberately untouched.

Tests / coverage

New lexer/string_escape_test.go covers hex/octal/unicode plus edge cases: max digits, single digit, greedy boundary, non-octal-digit boundary, no hex digit after \x/\u, octal overflow (\400), surrogate/out-of-range codepoints, empty/unterminated braces, and the %Q/%{}/interpolation-tail forms. Every case was verified against ruby -e on MRI 4.0.5.

  • Module coverage stays at 100% (CI -coverpkg gate).
  • gofmt / go vet clean, CGO_ENABLED=0 build green, -race tests pass.

Known divergence

\x / \u with no hex digit (and an out-of-range \u) are SyntaxErrors in MRI; since scanStringSegment has no error channel they degrade gracefully (literal fallback / U+FFFD) rather than failing the parse.

🤖 Generated with Claude Code

…g strings

Double-quoted and other interpolating string literals dropped the
backslash and leaked the literal characters for `\xHH` (hex byte) and
`\NNN` (octal byte) escapes: `"\x41"` lexed to `"x41"` and `"\101"` to
`"101"` instead of `"A"`. The `\uHHHH` / `\u{...}` Unicode escapes were
broken the same way (`"\u{41}"` leaked `"u{41}"`). Only the basic escapes
(`\n`, `\t`, `\0`, ...) were handled.

scanStringSegment is the escape interpreter shared by every interpolating
form — `"..."`, `%Q{...}`/`%{...}`, and interpolating heredocs all funnel
through it (the latter two re-lex their body as a double-quoted string).
This adds the missing cases there, matching MRI 4.0.5 byte-for-byte:

- `\xHH`: 1–2 hex digits, greedy (`\x41`->"A", `\xff`->0xFF, `\x4`->0x04);
  a `\x` with no hex digit degrades to a literal `x` (MRI raises a
  SyntaxError, for which this entry point has no channel).
- `\NNN` octal: 1–3 octal digits, greedy, masked to one byte
  (`\101`->"A", `\12`->"\n", `\377`->0xFF, `\400`->0x00).
- `\uHHHH` (exactly four hex digits) and `\u{cp cp ...}` (whitespace-
  separated codepoints), each emitted as UTF-8; out-of-range / surrogate
  values fall back to U+FFFD.

Single-quoted strings, `%q{}`, and the regexp pass-through are untouched.

Adds string_escape_test.go covering hex/octal/unicode plus edge cases
(max digits, single digit, greedy boundary, non-digit boundary, no hex
digit after `\x`/`\u`, octal overflow, surrogate/out-of-range codepoints,
empty/unterminated braces) and the `%Q`/`%{}`/interpolation-tail forms.
Module coverage stays at 100%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit a831e71 into main Jun 30, 2026
9 checks passed
@tannevaled
tannevaled deleted the fix-hex-octal-escapes branch June 30, 2026 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant