fix(lexer): interpret \xHH hex and \NNN octal escapes in interpolating strings - #9
Merged
Merged
Conversation
…g strings
Double-quoted and other interpolating string literals dropped the
backslash and leaked the literal characters for `\xHH` (hex byte) and
`\NNN` (octal byte) escapes: `"\x41"` lexed to `"x41"` and `"\101"` to
`"101"` instead of `"A"`. The `\uHHHH` / `\u{...}` Unicode escapes were
broken the same way (`"\u{41}"` leaked `"u{41}"`). Only the basic escapes
(`\n`, `\t`, `\0`, ...) were handled.
scanStringSegment is the escape interpreter shared by every interpolating
form — `"..."`, `%Q{...}`/`%{...}`, and interpolating heredocs all funnel
through it (the latter two re-lex their body as a double-quoted string).
This adds the missing cases there, matching MRI 4.0.5 byte-for-byte:
- `\xHH`: 1–2 hex digits, greedy (`\x41`->"A", `\xff`->0xFF, `\x4`->0x04);
a `\x` with no hex digit degrades to a literal `x` (MRI raises a
SyntaxError, for which this entry point has no channel).
- `\NNN` octal: 1–3 octal digits, greedy, masked to one byte
(`\101`->"A", `\12`->"\n", `\377`->0xFF, `\400`->0x00).
- `\uHHHH` (exactly four hex digits) and `\u{cp cp ...}` (whitespace-
separated codepoints), each emitted as UTF-8; out-of-range / surrogate
values fall back to U+FFFD.
Single-quoted strings, `%q{}`, and the regexp pass-through are untouched.
Adds string_escape_test.go covering hex/octal/unicode plus edge cases
(max digits, single digit, greedy boundary, non-digit boundary, no hex
digit after `\x`/`\u`, octal overflow, surrogate/out-of-range codepoints,
empty/unterminated braces) and the `%Q`/`%{}`/interpolation-tail forms.
Module coverage stays at 100%.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Interpolating string literals (
"...",%Q{...}/%{...}, interpolating heredocs) did not interpret\xHH(hex byte),\NNN(octal byte), or\uHHHH/\u{...}(Unicode) escapes — they dropped the backslash and leaked the literal characters. The basic escapes (\n,\t,\0, …) already worked, so the escape interpreter existed but was incomplete."\x41""x41""A""\101""101""A"(octal)"\u{41}""u{41}""A""\n","\0"Fix
lexer.scanStringSegmentis the escape interpreter shared by every interpolating string form (%Q/%{}and interpolating heredocs re-lex their body as a double-quoted string, so they funnel through it). The missing cases were added there, matching MRI 4.0.5 byte-for-byte:\xHH— 1–2 hex digits, greedy (\x41→A,\xff→0xFF,\x4→0x04). A\xwith no hex digit degrades to a literalx(MRI raises aSyntaxError; this entry point has no error channel).\NNNoctal — 1–3 octal digits, greedy, masked to one byte (\101→A,\12→\n,\377→0xFF,\400→0x00).\uHHHH(exactly four hex digits) and\u{cp cp …}(whitespace-separated codepoints), each emitted as UTF-8; out-of-range / surrogate values fall back to U+FFFD.Single-quoted strings,
%q{}, and the regexp pass-through are deliberately untouched.Tests / coverage
New
lexer/string_escape_test.gocovers hex/octal/unicode plus edge cases: max digits, single digit, greedy boundary, non-octal-digit boundary, no hex digit after\x/\u, octal overflow (\400), surrogate/out-of-range codepoints, empty/unterminated braces, and the%Q/%{}/interpolation-tail forms. Every case was verified againstruby -eon MRI 4.0.5.-coverpkggate).gofmt/go vetclean,CGO_ENABLED=0build green,-racetests pass.Known divergence
\x/\uwith no hex digit (and an out-of-range\u) areSyntaxErrors in MRI; sincescanStringSegmenthas no error channel they degrade gracefully (literal fallback / U+FFFD) rather than failing the parse.🤖 Generated with Claude Code