bigint.c: fix FPE caused by inconsistent zero sign state

Fix two functions that could create bigints with sn != 0 but value of 0:

- mpz_mod_limb: single-limb case set r->sn = x->sn even when result was 0
- mpz_mul_2exp: set z->sn = sn unconditionally after zero-producing ops

This inconsistent state caused GCD loop (!zero_p(&b)) to continue with
a zero divisor, eventually causing FPE in mpz_mod_limb with m = 0.

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Yukihiro "Matz" Matsumoto
2025-12-29 23:39:30 +09:00
parent 9f0950da13
commit 27c9037d48
+8 -2
View File
@@ -1650,7 +1650,10 @@ mpz_mod_limb(mpz_ctx_t *ctx, mpz_t *r, mpz_t *x, mp_limb m)
/* Single limb case - simple modulo */
mp_limb result = x->p[0] % m;
mpz_set_int(ctx, r, result);
r->sn = x->sn;
if (result == 0)
r->sn = 0;
else
r->sn = x->sn;
return;
}
@@ -2071,7 +2074,10 @@ mpz_mul_2exp(mpz_ctx_t *ctx, mpz_t *z, mpz_t *x, mrb_int e)
else {
mpz_move(ctx, z, &y);
}
z->sn = sn;
if (uzero_p(z))
z->sn = 0;
else
z->sn = sn;
}
}