mirror of
https://github.com/mruby/mruby
synced 2026-06-08 16:11:16 +00:00
mruby-bigint: maintain canonical sn=0 when trim reduces sz to 0
mpz_t treats sn (sign) as the canonical "is zero" flag (zero_p(x) := (x)->sn == 0). When an arithmetic operation produces a value whose limbs trim to zero size, sn must be reset to 0 to preserve the invariant. Several call sites already enforced this locally (e.g. mpz_sub line 616); make trim() responsible so every caller benefits. Without this, an inconsistent zero bignum (sn!=0, sz=0) can flow into mpz_sqr, miss the zero_p guard, and reach mpz_init_heap with hint=0 where mpn_zero(NULL, 0) invokes UB (memset() declares its first argument nonnull). The trip survives at runtime on glibc but is formally undefined behavior, flagged by UBSan via Integer#pow(b,e,m) with specific operands. close #6849 Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -373,6 +373,8 @@ trim(mpz_t *x)
|
||||
while (x->sz && x->p[x->sz-1] == 0) {
|
||||
x->sz--;
|
||||
}
|
||||
/* Maintain invariant: sz == 0 implies sn == 0 (zero is canonical). */
|
||||
if (x->sz == 0) x->sn = 0;
|
||||
}
|
||||
|
||||
/* z = x + y, without regard for sign */
|
||||
|
||||
Reference in New Issue
Block a user