Yukihiro "Matz" Matsumoto 73255d3b70 mruby-numeric-ext: avoid signed overflow on Integer#gcd/lcm with MRB_INT_MIN ...

Negating MRB_INT_MIN (-2^63) is signed overflow (UB) because
2^63 does not fit in mrb_int.  Both `mrb_int_gcd` and `int_lcm`
took the absolute value via `if (x < 0) x = -x`, which trips on
MRB_INT_MIN.

Reported by ClusterFuzz testcase
clusterfuzz-testcase-minimized-mruby_fuzzer-5137605569347584.

* mrb_int_gcd: cast each input to mrb_uint before negating; the
  Euclidean reduction runs in unsigned.  The cast back at the
  end yields MRB_INT_MIN only when the mathematical gcd is 2^63
  (i.e., gcd(MIN, 0) or gcd(MIN, MIN)).
* int_gcd: detect the negative return value from mrb_int_gcd
  and raise via mrb_int_overflow, since the true result does
  not fit.
* int_lcm: short-circuit raise when either operand is
  MRB_INT_MIN (after the existing zero check), since the abs
  would overflow and the lcm with any non-zero operand could
  not fit anyway.

Co-authored-by: Claude <noreply@anthropic.com>

2026-05-19 12:05:33 +09:00

..

numeric_ext.c

mruby-numeric-ext: avoid signed overflow on Integer#gcd/lcm with MRB_INT_MIN

2026-05-19 12:05:33 +09:00