feat(vm): String encoding — scrub family, valid_encoding?, force_encoding, -@ - #345
Merged
Conversation
…ding Horodate: 2026-08-10 09:35 CEST Advance String multibyte encoding support (MRI 4.0.5 parity): - valid_encoding? is now per-encoding: structural UTF-8/16/32 validators, US-ASCII 7-bit rule, always-valid ASCII-8BIT, and x/text-backed legacy encodings validated by decode (US-ASCII/Big5 no longer report as valid). - force_encoding raises FrozenError on a frozen receiver and resolves the "internal" special name through Encoding.default_internal (BINARY when unset). - Encoding.default_internal is now stateful (setter remembers it, reader reflects it); rbgo strings still work in UTF-8. - scrub / scrub! rewritten to be encoding-aware: they walk the receiver in encoding-appropriate units (UTF-8, US-ASCII, UTF-16LE/BE, UTF-32LE/BE), preserve the receiver's encoding, no-op on ASCII-8BIT, honour explicit-String and block replacements, and raise TypeError / ArgumentError as MRI does. scrub! mutates in place and returns self. - Add String#-@ (frozen copy) / #+@ (mutable copy); wire the -"literal" operator (OpNeg) to String#-@. ruby/spec deltas (rbgo, local): string/scrub 14->24, string/valid_encoding 6->7, string/force_encoding 7->11, string/uminus ->6/6. No regressions. New code is covered 100% (incl. error branches). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Horodate: 2026-08-10 10:05 CEST
pack("C*") yields an ASCII-8BIT string, on which scrub is now correctly a
no-op (MRI-faithful). The test asserted the old encoding-blind behavior; tag
the input UTF-8 so it still exercises the U+FFFD / explicit-replacement paths.
Verified against ruby 4.0.5.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rite) Horodate: 2026-08-10 10:20 CEST The only remaining caller (decodeToUTF8, encode :invalid=>:replace) passes already-invalid input, so the utf8.ValidString early return was unreachable. Removing it keeps scrubUTF8 at 100% coverage; behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Horodate: 2026-08-10 10:45 CEST Measured 12408 passing after the String encoding work (scrub family, valid_encoding?, force_encoding, -@); lock in with the usual N-30 margin for known ratchet flakiness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # scripts/conformance/rubyspec/FLOOR
tannevaled
added a commit
that referenced
this pull request
Aug 10, 2026
* feat(vm): String#grapheme_clusters/#byteindex/#bytesplice Horodate: 2026-08-10 11:48 CEST Implement the String grapheme + byte-index residuals deferred by the encoding PR (#345): - #grapheme_clusters / #each_grapheme_cluster: UAX#29 extended grapheme cluster segmentation (GB1-GB13/GB999), covering combining marks, Hangul L/V/T conjoining, ZWJ emoji sequences and regional-indicator flags. Grapheme_Cluster_Break + Extended_Pictographic tables are generated from the running Ruby's Unicode database (scripts/gen_gcb.rb) so segmentation matches MRI exactly. Encoding is preserved on each piece; the no-block form returns an Enumerator. - #byteindex / #byterindex: byte-offset of a String or Regexp match, with negative/oversized byte offsets, character-boundary IndexError, and the $~ side effect for Regexp (byterindex finds the last, incl. overlapping, match at or before the offset). - #bytesplice: all MRI 4.0 arities (index/length[/str_index/str_length] and range[/str_range]) with in-place replacement, boundary checks and IndexError/RangeError/TypeError/ArgumentError branches. Every behavior and error message verified against ruby 4.0.5. New code at 100% coverage (whole-package gate shows only the pre-existing partials). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(conformance): bump ruby/spec FLOOR 12553 -> 12798 Horodate: 2026-08-10 11:50 CEST Measured 12828 passing examples with the grapheme + byte-index String methods in place (275 above the old floor); lock in at N-30 for infra flake margin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Horodate: 2026-08-10 10:47 CEST
Advances String multibyte encoding support toward MRI 4.0.5 parity, focused on the scrub family, per-encoding validity, and
force_encoding. Verified againstruby 4.0.5.Implemented
String#valid_encoding?is now per-encoding: structural UTF-8/UTF-16(LE/BE)/UTF-32(LE/BE) validators, the US-ASCII 7-bit rule, always-valid ASCII-8BIT, and x/text-backed legacy encodings validated by decode. (US-ASCII / Big5 tagged strings no longer wrongly report valid.)String#force_encodingraisesFrozenErroron a frozen receiver and resolves the"internal"special name throughEncoding.default_internal(BINARY when unset).Encoding.default_internalis now stateful (setter remembers it, reader reflects it); rbgo strings still operate in UTF-8.String#scrub/#scrub!rewritten to be encoding-aware: they walk the receiver in encoding-appropriate units (UTF-8, US-ASCII, UTF-16LE/BE, UTF-32LE/BE), preserve the receiver's encoding, no-op on ASCII-8BIT (MRI-faithful), honour explicit-String and block replacements, and raiseTypeError/ArgumentErroras MRI does.scrub!mutates in place and returns self.String#-@(frozen copy) /#+@(mutable copy); the-"literal"operator (OpNeg) now dispatchesString#-@.Measured deltas (rbgo, local ruby/spec)
string/scrub14→24,string/valid_encoding6→7,string/force_encoding7→11,string/uminus→6/6,encoding/default_internal4→10.Deferred (documented residual)
each_grapheme_cluster/grapheme_clusters(needs full UAX#29 segmentation + encoding round-trips).byteindex/byterindex/bytesplice(regexp/$~/\G/encoding-compat surface — kept out to protect this PR's scope and coverage gate).valid_encoding?"all encodings" spec example: encodings with no rbgo/x-text validator (CP949, Emacs-Mule, EUC-TW) fall back to valid.Encoding.default_external=dynamicfind('external'/'filesystem')(touches shared lookup; left to avoid regression risk).scrub!"preserves instance variables" example depends on a pre-existing String-ivar storage gap, unrelated to scrub.Verification
grep -vE '100.0%$'shows only the pre-approved pre-existing partials (cmpFloat, rangeSize, newFiber, fiberResume, ioGetsParagraph, registerIOClassMethods, registerNetHTTP, classOf, ivarTable, registerSleep). All new code is 100% incl. error branches.go test ./...exit 0 (fixedTestPBStringScrub, which had asserted the old encoding-blind behavior —pack("C*")is ASCII-8BIT, on which scrub is now correctly a no-op).gofmtclean;go vet ./internal/vm/clean. No prelude edits. No shadowed defs.🤖 Generated with Claude Code