admin/mruby-mruby
1
0
Fork 0
mirror of https://github.com/mruby/mruby synced 2026-06-08 16:11:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

mruby-numeric-ext: avoid signed overflow on Integer#gcd/lcm with MRB_INT_MIN

Browse Source 
Negating MRB_INT_MIN (-2^63) is signed overflow (UB) because
2^63 does not fit in mrb_int.  Both `mrb_int_gcd` and `int_lcm`
took the absolute value via `if (x < 0) x = -x`, which trips on
MRB_INT_MIN.

Reported by ClusterFuzz testcase
clusterfuzz-testcase-minimized-mruby_fuzzer-5137605569347584.

* mrb_int_gcd: cast each input to mrb_uint before negating; the
  Euclidean reduction runs in unsigned.  The cast back at the
  end yields MRB_INT_MIN only when the mathematical gcd is 2^63
  (i.e., gcd(MIN, 0) or gcd(MIN, MIN)).
* int_gcd: detect the negative return value from mrb_int_gcd
  and raise via mrb_int_overflow, since the true result does
  not fit.
* int_lcm: short-circuit raise when either operand is
  MRB_INT_MIN (after the existing zero check), since the abs
  would overflow and the lcm with any non-zero operand could
  not fit anyway.

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Yukihiro "Matz" Matsumoto
2026-05-19 12:05:33 +09:00
parent db2845aae0
commit 73255d3b70
1 changed files with 20 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mrbgems/mruby-numeric-ext/src/numeric_ext.c
+20 -8
View File
@@ -84,16 +84,20 @@ mrb_value mrb_int_pow(mrb_state *mrb, mrb_value x, mrb_value y);
static mrb_int
mrb_int_gcd(mrb_int x, mrb_int y)
{
if (x < 0) x = -x;
if (y < 0) y = -y;
/* Negate via unsigned so MRB_INT_MIN doesn't overflow.
The cast back at the end produces MRB_INT_MIN only when the
true result is 2^63 (gcd of MRB_INT_MIN with itself or 0);
callers detect that case from the negative return value. */
mrb_uint ux = (x < 0) ? -(mrb_uint)x : (mrb_uint)x;
mrb_uint uy = (y < 0) ? -(mrb_uint)y : (mrb_uint)y;
while (y != 0) {
mrb_int temp = y;
y = x % y;
x = temp;
while (uy != 0) {
mrb_uint temp = uy;
uy = ux % uy;
ux = temp;
}
return x;
return (mrb_int)ux;
}
/*
@@ -122,7 +126,11 @@ int_gcd(mrb_state *mrb, mrb_value x)
if (!mrb_integer_p(y)) {
mrb_raisef(mrb, E_TYPE_ERROR, "can't convert %Y into Integer", y);
}
return mrb_int_value(mrb, mrb_int_gcd(mrb_integer(x), mrb_integer(y)));
mrb_int g = mrb_int_gcd(mrb_integer(x), mrb_integer(y));
/* g < 0 only when the mathematical result is 2^63 (= |MRB_INT_MIN|),
which does not fit in mrb_int. */
if (g < 0) mrb_int_overflow(mrb, "gcd");
return mrb_int_value(mrb, g);
}
/*
@@ -158,6 +166,10 @@ int_lcm(mrb_state *mrb, mrb_value x)
if (a == 0 || b == 0) return mrb_int_value(mrb, 0);
/* Negation of MRB_INT_MIN is UB and the lcm with any non-zero
operand would not fit in mrb_int anyway. */
if (a == MRB_INT_MIN || b == MRB_INT_MIN) mrb_int_overflow(mrb, "lcm");
gcd_val = mrb_int_gcd(a, b);
if (a < 0) a = -a;
if (b < 0) b = -b;