mruby-bigint: fix missing trim in urshift/ulshift when n==0

When shift amount is 0, urshift() and ulshift() called mpz_set() which
copies data without trimming leading zero limbs. This caused bigint
values to have inflated sz fields, making ucmp() comparisons incorrect.

For example, a 256-bit remainder from division could have sz=18 instead
of sz=8 because the divisor had 18 limbs. This made it compare greater
than values with fewer limbs, even when numerically smaller.

The bug also caused memory leaks when the incorrect comparison led to
taking wrong code paths in division, triggering size overflow exceptions
after memory was allocated.

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Yukihiro "Matz" Matsumoto
2026-01-12 21:04:13 +09:00
parent 78fe8a0476
commit f2f385f572
+6 -2
View File
@@ -1714,8 +1714,10 @@ urshift(mpz_ctx_t *ctx, mpz_t *c1, mpz_t *a, size_t n)
{
mrb_assert(n < DIG_SIZE);
if (n == 0)
if (n == 0) {
mpz_set(ctx, c1, a);
trim(c1);
}
else if (uzero_p(a)) {
zero(c1);
}
@@ -1740,8 +1742,10 @@ static void
ulshift(mpz_ctx_t *ctx, mpz_t *c1, mpz_t *a, size_t n)
{
mrb_assert(n < DIG_SIZE);
if (n == 0)
if (n == 0) {
mpz_set(ctx, c1, a);
trim(c1);
}
else if (uzero_p(a)) {
zero(c1);
}