mruby-bigint: gate mpz_mod's Barrett path on the algorithm's precondition

Barrett reduction requires x < 2^(2*bits(m)); the gate condition only
required x->sz >= y->sz + 2, which let x.sz reach 25 limbs against a
4-limb modulus. When the precondition is violated, mpz_barrett_reduce
silently truncates high limbs and returns garbage.

Integer#remainder, which routes through mpz_mod, was affected:
(3**500).remainder((2**100)+3) returned the wrong value. Integer#%
took the udiv path via mpz_mmod and was unaffected.

Add x->sz <= 2 * y->sz to the gate so out-of-range inputs fall
through to the general udiv path.

Co-authored-by: Claude <noreply@anthropic.com>

mrbgems/mruby-bigint/core/bigint.c

@@ -3149,8 +3149,11 @@ mpz_mod(mpz_ctx_t *ctx, mpz_t *r, mpz_t *x, mpz_t *y)
     return;
   }
 
-  /* Barrett reduction for moderate-sized moduli (>= 4 limbs where setup is worthwhile) */
-  if (y->sz >= 4 && y->sz <= 16 && x->sz >= y->sz + 2) {
+  /* Barrett reduction for moderate-sized moduli (>= 4 limbs where setup is worthwhile).
+   * Barrett's precondition is x < 2^(2*bits(m)); inputs beyond ~2*m.sz limbs
+   * violate it and the algorithm silently truncates high limbs. Fall through
+   * to general division for those. */
+  if (y->sz >= 4 && y->sz <= 16 && x->sz >= y->sz + 2 && x->sz <= 2 * y->sz) {
     mpz_t mu;
     mpz_init_temp(ctx, &mu, y->sz + 1);
     mpz_barrett_mu(ctx, &mu, y);

mrbgems/mruby-bigint/test/bigint.rb

@@ -164,6 +164,18 @@ assert 'Bigint Integer#pow(e, m) - Montgomery path' do
   assert_equal ((5**300) ** 7) % m2, (5**300).pow(7, m2)
 end
 
+assert 'Bigint Integer#remainder large operand' do
+  # Regression: mpz_mod's Barrett path didn't enforce its precondition
+  # x < 2^(2*bits(m)), so it silently truncated high limbs when x was
+  # much larger than m^2, producing the wrong remainder. Integer#%
+  # took the udiv path and worked, but Integer#remainder went through
+  # mpz_mod and was broken.
+  m = (2**100) + 3
+  assert_equal (3**500) % m, (3**500).remainder(m)
+  assert_equal (5**500) % ((2**150) + 1), (5**500).remainder((2**150) + 1)
+  assert_equal (2**400) % ((2**130) + 1), (2**400).remainder((2**130) + 1)
+end
+
 assert 'Bigint abs' do
   n = 1<<65
   assert_equal 36893488147419103232, n.abs