86b1c8ff87
string.c (utf8_strlen): add assertion utf8_len <= byte_len
2024-05-03 07:50:40 +09:00
fa213a2d89
string.c (search_nonascii): prevent buffer overrun; ref #6255
2024-04-29 16:35:08 +09:00
714ef4c4fd
string.c (mrb_utf8_strlen): handle invalid UTF-8 sequence; fix #6255
...
Previous SWAR version assumes valid UTF-8 to count number of code points
in the string, but we need to handle invalid sequence as well. We now
use `search_nonascii` to skip counting single byte characters for
performance. The new version is even faster than SWAR version (probably
because `search_nonascii` uses SSE2 on Intel compatible CPU (which I use).
2024-04-29 15:37:22 +09:00
aa28f48c3b
string.c (search_nonascii): simplify fallback implementation by SWAR
2024-04-02 19:50:13 +09:00
cbb31e6a7c
string.c (mrb_utf8_strlen): use SWAR technique for performance
2024-03-30 22:50:12 +09:00
8130bbc479
string.c (mrb_str_len_to_dbl): should return zero for invalid string
...
We should not use uninitialized value; fix #6183
2024-02-24 23:03:15 +09:00
3eeeb8fabf
string.c (memsearch_swar): skip a local variable
2024-02-15 14:01:49 +09:00
9c840a68e2
string.c (memsearch_swar): boundary check by the latest position
2024-02-13 22:36:03 +09:00
abcad36bae
string.c (memsearch_swar): fixed a buffer overflow
2024-02-09 19:09:01 +09:00
868aeaf48c
string.c (memsearch_swar): fixed a buffer overflow
2024-02-09 18:52:44 +09:00
cac698fe88
string.c (memsearch_swar): changed the function name
...
For your information, SWAR stands for SIMD within a register.
2024-01-31 21:37:39 +09:00
f8527f0670
string.c (mrb_memsearch_ss): update a comment regarding alignment issue
2024-01-31 21:36:05 +09:00
d8144aef78
string.c: remove a comment regarding unsigned char*
2024-01-31 21:34:54 +09:00
363f795432
string.c: remove unnecessary type casts
...
We don't need `unsigned` for string pointers.
2024-01-31 15:06:04 +09:00
e61a2881a7
string.c (mrb_memsearch_ss): fix potential buffer overflow; fix #6158
...
- stop using `bigint*`
- stop integer pointer dereferences
- use `memcpy` to integer variables
- add reminder search for shorter patterns
2024-01-31 07:48:24 +09:00
219cfd63e1
string.c (mrb_memsearch_ss): update integer prefixes (LLU -> ULL)
2024-01-31 07:42:52 +09:00
6ba65ea0f0
string.c (mrb_memsearch_ss): update comment
2024-01-31 01:54:37 +09:00
44a5882bc2
string.c (mrb_memsearch_ss): remove useless alignment adjustment
...
Ref #6158
2024-01-31 01:27:10 +09:00
be5448fd16
string.c (mrb_memsearch): refactor m==1 (use memchr) case
2024-01-31 01:17:12 +09:00
d8e1aed646
string.c (search_nonascii): need not to check n>3 on 32bit platforms
2024-01-30 12:35:56 +09:00
e859b430cc
fixup! string.c (search_nonascii): add faster search using SSE2
2024-01-27 11:59:29 +09:00
dee1ded7b5
string.c (mrb_memsearch_ss): support bigendians
2024-01-27 06:23:16 +09:00
06d9a54760
string.c (mrb_memsearch): remove simple search and quick search
...
Since mrb_memsearch_ss() is fast enough for most of the cases, we try to
simplify the code.
2024-01-27 06:22:55 +09:00
2943ca2685
string.c (mrb_memsearch_ss): faster integer-wise substring search
...
The function is based on @WojciechMula's code from the repository
https://github.com/WojciechMula/sse4-strstr.git Since it's licensed
under 2 clause BSD, we updated LEGAL file too.
2024-01-27 06:16:33 +09:00
08cd281200
string.c (search_nonascii): skip alignment adjustment for some CPUs
...
Some CPUs (e.g. x86) allow unaligned access to the memory.
2024-01-26 08:22:33 +09:00
68ab95ea57
string.c (mrb_memsearch): take char* instead of void*
...
Avoid useless `void*`.
2024-01-25 22:32:31 +09:00
cf0cd6785c
string.c: rename MRB_QS_SHORT_STRING_LENGTH
...
The new name is MRB_SEARCH_SHORT_STRING_LENGTH since it's no longer used
in the mrb_memsearch_qs(). FYI, 'qs' stands for 'quick search'.
2024-01-23 23:28:45 +09:00
fb2ebcde7c
string.c (search_nonascii): add faster search using SSE2
2024-01-23 23:28:45 +09:00
4108b85c62
string.c (search_nonascii): faster search using integer match
...
If you define `SIMPLE_SEARCH_NONASCII`, you can use old, naive
implementation of search_nonascii(). You may want to use the old one for
code size constraint for example.
2024-01-23 23:28:45 +09:00
772f13520c
Fixed buffer overflow in mrb_memsearch()
2024-01-20 09:44:47 +09:00
f13101124c
string.c: add cast to remove warnings
2024-01-19 18:24:34 +09:00
69cf074778
string.c (mrb_memsearch): move simple search from mrb_memsearch_qs()
2024-01-19 15:21:43 +09:00
57fd0edaa7
mruby.h: rename ASCII flag to SINGLE_BYTE
...
This flag means all the characters in the string can be represented by a
single byte, i.e., the string does not contain any multi-byte character.
Those characters are likely ASCII characters, but may be a part of broken
UTF-8 sequence, so the term 'ASCII' is not sufficient.
2024-01-18 22:50:39 +09:00
fb8bc1954b
string.c: improve performance of chars2bytes/bytes2chars; ref #6143
2024-01-11 18:23:50 +09:00
f646228dbe
string.c (str_index_str_by_char): simplify using str_index_str()
...
Instead of its own version of quick search, now we use str_index_str()
and adjust character position. This change makes searching 4 times
faster in some cases; ref #6143
2024-01-11 12:23:06 +09:00
e42f3b36f9
string.c (chars2bytes): simplify the condition to detect break
2024-01-11 11:49:49 +09:00
787439455c
string.c: remove the macro BYTES_ALIGN_CHECK which is no longer used
2024-01-10 17:31:04 +09:00
c2f148e15d
string.c (mrb_memsearch_qs): reduce the scope of local variables
2024-01-10 16:21:39 +09:00
31f2d93657
string.c (bytes2chars): skip calling mrb_utf8len() if possible
...
If (ch < 0x80) the length of the character (in bytes) should be 1, so we
don't have to call mrb_utf8len().
2024-01-10 15:36:07 +09:00
4c859e754d
string.c (bytes2chars): skip scanning if the string is ASCII only
2024-01-10 14:40:39 +09:00
df0a5e838d
string.c (chars2bytes): use early return
2024-01-10 12:41:28 +09:00
e39e4ea26d
string.c (chars2bytes): small refactoring
2024-01-10 11:09:47 +09:00
1d243bc58b
string.c (bytes2chars): takes a string object instead of char*
2024-01-10 10:53:26 +09:00
59a1d74937
string.c: inline str_range_to_bytes
2024-01-09 18:22:38 +09:00
9135f0991b
string.c: avoid RSTRING_CHAR_LEN() if possible
...
Current code scan the string twice (once from RSTRING_CHAR_LEN, and once
from chars2bytes), but those scans are not necessary. Just point the end
of the string.
2024-01-09 11:44:12 +09:00
f3ebef1172
mruby-string (chars2bytes): reduce the scope of local variables
2024-01-09 11:00:44 +09:00
f1d01a6107
string.c (mrb_utf8len): should return 1 for invalid chars; ref #6143
2024-01-09 10:39:03 +09:00
64a4bf0e28
string.c: avoid infinite loop; fix #6143
...
`mrb_utf8len` returns 0 for invalid characters.
2024-01-09 10:37:52 +09:00
a5e83075d4
string.c: reduce the scope of a local variable
2024-01-09 08:22:26 +09:00
c080d16150
string.c: use presym to define String class
2023-11-01 22:23:35 +09:00
Yukihiro "Matz" Matsumoto