When MRB_UTF8_STRING is undefined, String#scrub is a no-op that
returns the receiver as-is (the #else branch added in ccb62ceb57).
The unit tests assumed UTF-8 semantics unconditionally, so a build
of just mruby-string-ext without UTF-8 strings failed all five scrub
tests.
Guard the UTF-8-dependent assertions with `skip unless "あ".length
== 1`, and add a paired test that asserts the no-op behaviour on the
non-UTF-8 build (skipped on UTF-8 builds).
Verified against the build config from the report:
- UTF-8 (host-debug): 5 tests pass, 1 skip
- non-UTF-8 (noutf8): 5 skip, 1 test pass
Closes#6860.
Co-authored-by: Claude <noreply@anthropic.com>
Replaces each maximal run of invalid UTF-8 bytes with a replacement
string (U+FFFD by default), returning a valid UTF-8 copy. Mirrors
CRuby's String#scrub (Feature #6752) -- the recovery counterpart to
the existing String#valid_encoding? detection API.
Validation matches utf8code() in src/string.c after the RFC 3629 /
Unicode D93b conformance fixup (#2708): overlong encodings, UTF-16
surrogates, and codepoints above U+10FFFF are all treated as invalid.
This is stricter than the existing mrb_utf8len()-based check used by
valid_encoding?, so a string can report valid_encoding? = true and
still get scrubbed; aligning valid_encoding? is a follow-up.
The block form lives in mrblib on top of two C primitives -- __scrub
and __scrub_chunks -- to avoid VM re-entry from C per CLAUDE.md.
Non-String block return values are coerced via to_s (CRuby raises
TypeError instead; the choice is locked in by test).
Closes#6859.
Co-authored-by: Claude <noreply@anthropic.com>
Make `Integer#chr("UTF-8")` reject UTF-16 surrogate code points
(U+D800..U+DFFF), and make `String#ord` reject ill-formed UTF-8 byte
sequences (overlong encodings, surrogates encoded as UTF-8, and code
points above U+10FFFF), matching CRuby and RFC 3629.
The `utf8code()` helper now decodes the code point first and then
validates the range per byte length:
len=2: cp >= 0x80 (rejects overlong)
len=3: cp >= 0x800 and not D800..DFFF
len=4: 0x10000 <= cp <= 0x10FFFF
close#2708
Co-authored-by: Claude <noreply@anthropic.com>
String#prepend(s, s) read RSTRING_LEN(argv[i]) in the copy loop after
mrb_str_resize had already updated the receiver's length, causing the
memcpy to write past the allocated buffer.
Detect self-references with mrb_obj_eq() and read from the memmoved
original data at p + total_prepend_len using the captured self_len.
This also handles mixed cases like s.prepend("X", s) where earlier
writes would otherwise corrupt the source of later reads.
Co-authored-by: Claude <noreply@anthropic.com>
Currently, `Integral#chr` in mruby changes behavior by `MRB_UTF8_STRING`
setting.
before this patch:
$ bin/mruby -e 'p 171.chr' #=> "\xab" (`MRB_UTF8_STRING` is disabled)
$ bin/mruby -e 'p 171.chr' #=> "«" (`MRB_UTF8_STRING` is enabled)
This behavior is incompatible with Ruby, and a little inconvenient because
it can't be interpreted as ASCII-8BIT with `MRB_UTF8_STRING`, I think.
So add encoding argument according to Ruby.
after this patch:
$ bin/mruby -e 'p 171.chr' #=> "\xab"
$ bin/mruby -e 'p 171.chr("ASCII-8BIT")' #=> "\xab"
$ bin/mruby -e 'p 171.chr("UTF-8")' #=> "«"
Allow only `String` for encoding because mruby doesn't have `Encoding`
class, and `"ASCII-8BIT"` (`"BINARY"`) and `"UTF-8"` (only with
`MRB_UTF8_STRING`) are valid value (default is `"ASCII-8BIT"`).
Because they're defined in both `mruby-string-ext` and `mruby-numeric-ext`
(they seem more natural to define in N, but `mruby-string-ext` depends on
`Integral#chr`).
Example:
$ bin/mruby -e '
p "あa".byteslice(1)
p "bar".byteslice(3)
p "bar".byteslice(4..0)
'
Before this patch:
"a"
""
RangeError (4..0 out of range)
After this patch (same as Ruby):
"\x81"
nil
nil
We have added internal convenience method `__to_str` which
does string type check.
The issue #3854 was fixed but fundamental flaw of lack of stack
depth check along with fibers still remains. Use `MRB_GC_FIXED_ARENA`
for workaround.
The ISO standard does not include implicit type conversion using
`to_int`. This implicit conversion often causes vulnerability.
There will be no more attacks like #4120.
In addition, we have added internal convenience method `__to_int` which
does type check and conversion (from floats).
mruby restriction:
`String#squeeze` can take more than 1 pattern arguments in CRuby,
in that case, the intersection of patterns will be used to match.
But in mruby, it doesn't take multiple patterns.
This patch is based on `mruby/c` implementation by Hirohito Higashi.
We might need to add `#tr_s`, `#squeeze` and `#delete` as well.
Adding them should not be too hard using functions we implemented here.
- String#ljust and String#rjust are now C functions to improve performance
- infinite loop because of an empty padding argument is now prevented (ArgumentError is raised)
- extra tests for ljust/rjust added
Yukihiro "Matz" Matsumoto