Pre-allocate visited array and thread lists at compile time and
reuse them across re_exec calls. A cache_in_use flag detects
re-entrancy and falls back to malloc when needed.
Eliminates 3 malloc + 3 free per NFA execution for the common
non-re-entrant case. Combined with the literal fast path, brings
literal match? from 4.7x to 2.5x vs CRuby.
Co-authored-by: Claude <noreply@anthropic.com>
Pure literal patterns (/hello/, /abc/) are detected at compile
time and matched using memchr+memcmp directly, completely
bypassing Pike VM setup (no malloc, no visited array, no thread
lists). Engine-only time drops from ~400ns to ~80ns.
Co-authored-by: Claude <noreply@anthropic.com>
Compute a 128-bit bitmap of bytes that could start a match.
For patterns like /cat|dog|fox/ the bitmap contains only {b,c,d,f},
skipping positions where no alternative can match. For /\d+/ only
{'0'-'9'} are set.
Used when no literal prefix is available (alternation, character
class patterns). Falls back gracefully when too many bytes match.
Key improvements vs CRuby ratio:
alternation miss: 9.5x -> 3.6x
\d+ medium string: 3.0x -> 1.4x
Co-authored-by: Claude <noreply@anthropic.com>
Move pool_copy inside each match condition so captures are only
copied for threads that advance to the next step, skipping the
copy for non-matching threads.
Co-authored-by: Claude <noreply@anthropic.com>
Extract the leading literal bytes from compiled bytecode and use
memchr + memcmp to skip positions where the prefix cannot match.
Both Pike VM and backtracking engine benefit.
For /needle/ in a 2006-char string: 29x faster (1.97s -> 0.07s),
now on par with CRuby/Oniguruma.
Co-authored-by: Claude <noreply@anthropic.com>
The flag is set not only for non-greedy quantifiers but also for
lookahead, lookbehind, and backreferences. The new name accurately
reflects its purpose: indicating that the backtracking engine is
required instead of the Pike VM.
Co-authored-by: Claude <noreply@anthropic.com>
Add (?<=...) positive and (?<!...) negative lookbehind support.
The sub-pattern must have a fixed byte length (no quantifiers or
alternation), computed at compile time and stored in the instruction.
At execution time, the engine backs up by that many bytes and runs
the sub-pattern forward. Maximum lookbehind length is 255 bytes.
Co-authored-by: Claude <noreply@anthropic.com>
When match? calls re_exec with captures=NULL, the Pike VM now
skips all capture pool operations: no pool_copy, no RE_SAVE
writes, no pool compaction. Only a single dummy pool slot is
allocated. This significantly reduces work for boolean matching.
Co-authored-by: Claude <noreply@anthropic.com>
Major changes to the NFA execution engine:
- Thread captures stored in a flat pool sized to actual ncap
(e.g. 4 ints for 1 capture group vs 64 fixed), dramatically
reducing per-thread copy cost
- Generation counter for visited[] eliminates per-step memset
of the entire bytecode-length array
- Pool compaction between steps reclaims dead thread slots
- Backtracking engine also uses dynamic ncap-sized captures
Co-authored-by: Claude <noreply@anthropic.com>
implement positive and negative lookahead in the backtracking engine:
- (?=pattern): succeeds if pattern matches at current position
without consuming characters
- (?!pattern): succeeds if pattern does NOT match at current position
new bytecodes RE_LOOKAHEAD and RE_NEG_LOOKAHEAD implemented in
the backtracking engine via nested bt_match calls.
Co-authored-by: Claude <noreply@anthropic.com>
non-greedy patterns now correctly match the shortest possible
string. patterns with non-greedy quantifiers are dispatched to
the backtracking engine which naturally handles non-greedy
semantics.
the Pike VM continues to be used for purely greedy patterns
(O(n*m) guarantee).
Co-authored-by: Claude <noreply@anthropic.com>
add a recursive backtracking engine that handles \1-\9
backreferences. the Pike VM (NFA) is used for patterns without
backreferences; patterns with backreferences automatically
fall back to the backtracking engine.
the backtracking engine has a step limit (MRB_REGEXP_STEP_LIMIT,
default 1M) to prevent ReDoS on pathological patterns.
also adds SAVE backtracking (save/restore capture positions on
failed branches) for correct submatch tracking.
Co-authored-by: Claude <noreply@anthropic.com>
add tests for: empty pattern, nested captures, word boundary \b,
non-capturing groups (?:), sub/gsub with block, scan with captures,
split with regexp, case/when with regexp, date reformatting.
known limitation: non-greedy quantifiers (*?, +?) currently behave
as greedy. needs match priority tracking (TODO for Phase 2).
Co-authored-by: Claude <noreply@anthropic.com>
Yukihiro "Matz" Matsumoto