The Difference
Since Ruby1.9, the keyword arguments were emulated by Ruby using the hash
object at the bottom of the arguments. But we have gradually moved toward
keyword arguments separated from normal (positinal) arguments.
At the same time, we value compatibility, so that Ruby3.0 keyword
arguments are somewhat compromise. Basically, keyword arguments are
separated from positional arguments, except when the method does not
take any formal keyword arguments, given keyword arguments (packed
in the hash object) are considered as the last argument.
And we also allow non symbol keys in the keyword arguments. In that
case, those keys are just passed in the `**` hash (or raise
`ArgumentError` for unknown keys).
The Instruction Changes
We have changed `OP_SEND` instruction. `OP_SEND` instruction used to
take 3 operands, the register, the symbol, the number of (positional)
arguments. The meaning of the third operand has been changed. It is now
considered as `n|(nk<<4)`, where `n` is the number of positional
arguments, and `nk` is the number of keyword arguments, both occupies
4 bits in the operand.
The number `15` in both `n` and `nk` means variable sized arguments are
packed in the object. Positional arguments will be packed in the array,
and keyword arguments will be packed in the hash object. That means
arguments more than 14 values are always packed in the object.
Arguments information for other instructions (`OP_SENDB` and `OP_SUPER`)
are also changed. It works as the third operand of `OP_SEND`. the
difference between `OP_SEND` and `OP_SENDB` is just trivial. It assigns
`nil` to the block hidden arguments (right after arguments).
The instruction `OP_SENDV` and `OP_SENDVB` are removed. Those
instructions are replaced by `OP_SEND` and `OP_SENDB` respectively with
the `15` (variable sized) argument information.
Calling Convention
When calling a method, the stack elements shall be in the order of the
receiver of the method, positional arguments, keyword arguments and the
block argument. If the number of positional or keyword arugument (`n` or
`nk`) is zero, corresponding arguments will be empty. So when `n=0` and
`nk=0` the stack layout (from bottom to top) will be:
+-----------------------+
| recv | block (or nil) |
+-----------------------+
The last elements `block` should be explicitly filled before `OP_SEND`
or assigned to `nil` by `OP_SENDB` internally. In other words, the
following have exactly same behavior:
OP_SENDB clears `block` implicitly:
```
OP_SENDB reg sym 0
```
OP_SEND clears `block` implicitly:
```
OP_LOADNIL R2
OP_SEND R2 sym 0
```
When calling a method with only positional arguments (n=0..14) without
keyword arguments, the stack layout will be like following:
+--------------------------------------------+
| recv | arg1 | ... | arg_n | block (or nil) |
+--------------------------------------------+
When calling a method with arguments packed in the array (n=15) which
means argument splat (*) is used in the actual arguments, or more than
14 arguments are passed the stack layout will be like following:
+-------------------------------+
| recv | array | block (or nil) |
+-------------------------------+
The number of the actual arguments is determined by the length of the
argument array.
When keyword arguments are given (nk>0), keyword arguments are passed
between positional arguments and the block argument. For example, when
we pass one positional argument `1` and one keyword argument `a: 2`,
the stack layout will be like:
+------------------------------------+
| recv | 1 | :a | 2 | block (or nil) |
+------------------------------------+
Note that keyword arguments consume `2*nk` elements in the stack when
`nk=0..14` (unpacked).
When calling a method with keyword arguments packed in the hash object
(nk=15) which means keyword argument splat (**) is used or more than
14 keyword arguments in the actual arguments, the stack layout will
be like:
+------------------------------+
| recv | hash | block (or nil) |
+------------------------------+
Note for mruby/c
When mruby/c authors try to support new keyword arguments, they need
to handle the new meaning of the argument information operand. If they
choose not to support keyword arguments in mruby/c, it just raise
error when `nk` (taken by `(c>>4)&0xf`) is not zero. And combine
`OP_SENDV` behavior with `OP_SEND` when `n` is `15`.
If they want to support keyword arguments seriously, contact me at
<matz@ruby.or.jp> or `@yukihiro_matz`. I can help you.
Otherwise `target_class` can be lost when it differs from `proc`'s
`target_class`, e.g. when called from `instance_eval`.
Also we should not pass `target_class` to `MRB_OBJ_ALLOC` since it
checks instance type from the class, and `target_class` may not have
proper information. ref #5272
Previously the following code did not produce the expected results:
```ruby
bx = binding
block = bx.eval("a = 1; proc { a }")
bx.eval("a = 2")
p block.call # Expect 2 but return 1 due to a bug
```
The previous implementation of `Binding#eval` evaluated the code and then merged the top layer variables.
This patch will parse and expand the variable space before making a call to `eval`.
This means that the call to `Binding#eval` will do the parsing twice.
In addition, the following changes will be made:
- Make `mrb_parser_foreach_top_variable()`, `mrb_binding_extract_proc()` and `mrb_binding_extract_env()` functions private global functions.
- Remove the `posthook` argument from `mrb_exec_irep()`.
The `posthook` argument was introduced to implement the `binding` method.
This patch is unnecessary because it uses a different implementation method.
ref #5362fixed#5491
The following methods will be made static.
- `Class#new`
- `Proc#call`
- `Kernel#catch`
Previously, static const RProc could not be registered as a method, but this has been changed to allow it.
- Added to `mruby-binding-core`
- `Binding#local_variable_defined?`
- `Binding#local_variable_get`
- `Binding#local_variable_set`
- `Binding#local_variables`
- `Binding#receiver`
- `Binding#source_location`
- `Binding#inspect`
- Added to `mruby-proc-binding`
- `Proc#binding`
The reason for separating `Proc#binding` is that core-mrbgems has a method that returns a closure object to minimize possible problems with being able to manipulate internal variables.
By separating it as different mrbgem, each user can judge this problem and incorporate it arbitrarily.
Instead of including `mruby/presym.h` everywhere, we provided the
fallback `mruby/presym.inc` under `include/mruby` directory, and specify
`-I<build-dir>/include` before `-I<top-dir>/include` in `presym.rake`.
So even when someone drops `-I<build-dir>/include` in compiler options,
it just compiles without failure.
Previously I used the `RData` object to avoid a memory leak in `mrb_irep` if `src/load.c` failed.
ref: https://github.com/mruby/mruby/pull/4250
commit: f1523d2404
Considering that the `RProc` object will be created in the subsequent process, it is preferable to create the `RProc` object from the beginning.
Along with this, the inside of `read_irep()` is replaced with the processing centered on the `RProc` object.
The global function that returns the `mrb_irep` pointer is still provided for compatibility.
This enhances self-containment.
Previously `mrb_context::stack` had the current call level stack, but now it owns it.
The `mrb_context::stack` field, which is no longer needed, will be removed.
To be also able to build mruby without presym in the future. However,
`MRB_QSYM` has been removed and changed as follows:
### Example
| Type | Symbol | Previous Style | New Style |
|---------------------------|--------|------------------|----------------|
| Operator | & | MRB_QSYM(and) | MRB_OPSYM(and) |
| Class Variable | @@foo | MRB_QSYM(00_foo) | MRB_CVSYM(foo) |
| Instance Variable | @foo | MRB_QSYM(0_foo) | MRB_IVSYM(foo) |
| Method with Bang | foo! | MRB_QSYM(foo_b) | MRB_SYM_B(foo) |
| Method with Question mark | foo? | MRB_QSYM(foo_p) | MRB_SYM_Q(foo) |
| Mmethod with Equal | foo= | MRB_QSYM(foo_e) | MRB_SYM_E(foo) |
This change makes it possible to define, for example, `MRB_IVSYM(foo)` as
`mrb_intern_lit(mrb, "@" "foo")`, which is useful if we support building
without presym in the future.
`mrb_get_arg1()` raises `ArgumentError` if the method does not receive one
argument.
And replaces all `mrb_get_args(mrb, "o", &arg)` by the new function.
commit 2d7d545c4c4bfce7fdcbcbe9baaeb437915742f0
Merge: 625a1249 b178914b
Author: Yukihiro "Matz" Matsumoto <matz@ruby.or.jp>
Date: Fri Jun 5 14:35:13 2020 +0900
Merge branch 'fix-mrb_open-with-nomem' of https://github.com/dearblue/mruby into dearblue-fix-mrb_open-with-nomem
commit b178914b11
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sat Jan 19 22:22:44 2019 +0900
Fix invalid pointer free inside other heap's block
1. `e = mrb_obj_alloc(...)`
2. `e->stack = mrb->c->stack` (`mrb->c->stack` is anywhere in the range `stbase...stend`)
3. And raised exception by `mrb_malloc()`!
4. `mrb_free(e->stack)` by GC part (wrong free)
commit 52e3d5d858
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sat Jan 19 21:55:36 2019 +0900
Fix memory leak for temporary symbols when out of memory
commit 4c5499b88e
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 11:42:07 2019 +0900
Fix uninitialized pointer dereference for debug section
commit 8e993167de
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 11:41:09 2019 +0900
Fix memory leak for temporary filenames when out of memory
commit 8b422577e6
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 10:57:51 2019 +0900
Fix memory leak for irep when out of memory
commit 6b35ebf49a
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 10:55:50 2019 +0900
Fix uninitialized pointer dereference when do not finished initializing irep
commit 2531f2631e
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 10:48:15 2019 +0900
Fix NULL pointer dereference when do not finished initializing irep
commit e2d6896eba
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sat Jan 19 12:54:19 2019 +0900
Fix memory leak for irep when out of memory by `mrb_proc_new()`
commit b6214ff8a0
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sat Jan 19 12:53:07 2019 +0900
Fix memory leak for `khash_t` in `kh_init_size()` when out of memory by `kh_alloc()`
commit 19162dd6c1
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 02:15:07 2019 +0900
Fix memory leak for symbol string when out of memory in `kh_put()`
commit 15e67297ff
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 02:12:24 2019 +0900
Fix keep wrong symbol index when out of memory
commit 3f8e2b3752
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 02:08:13 2019 +0900
Fix keep wrong symbol capacity when out of memory
commit a3cfe755ab
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sat Jan 19 10:11:37 2019 +0900
Fix NULL pointer dereference `mrb->c` by `mark_context()`
commit d9c7b6be6e
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 15:25:09 2019 +0900
Fix protect exception for print error message
commit 100642750e
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 11:59:02 2019 +0900
Protect exception for mruby core initialization
commit 7a0418304e
Author: dearblue <dearblue@users.noreply.github.com>
Date: Fri Jan 18 20:38:27 2019 +0900
Fix memory leak for string object when out of memory
The `mrb_str_pool()` function has a path to call `malloc()` twice.
If occurs `NoMemoryError` exception in second `malloc()`,
first `malloc()` pointer is not freed.
commit fef1c152ce
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sat Jan 19 13:05:09 2019 +0900
Fix stack overflow when out of memory
As a result of this change, no backtrace information is set
for NoMemoryError (`mrb->nomem_err`).
Detailes:
When generating a backtrace, called `mrb_intern_lit()`,
`mrb_str_new_cstr()` and `mrb_obj_iv_set()` function with
`exc_debug_info()` function in `src/error.c`.
If a `NoMemoryError` exception occurs at this time,
the `exc_debug_info()` function will be called again,
and in the same way `NoMemoryError` exception raised will result
in an infinite loop to occurs stack overflow (and SIGSEGV).
commit da7d7f881b
Author: dearblue <dearblue@users.noreply.github.com>
Date: Sun Jan 20 12:00:38 2019 +0900
Fix NULL pointer dereference `mrb->nomem_err` when not initialized
Add internal functions (not `static`):
* `mrb_raise_nomemory()`
* `mrb_core_init_abort()`
We needed to preserve the original method name somewhere. We kept it in
the `env` structure pointed from aliased methods. #1457 and #1531 tried
to address this issue. But this patch is more memory efficient.
Limitation: this fix does not support `super` from methods defined by
`define_method`. This limitation may be addressed in the future, but
it's low priority.
The binary sizes (gems are only `mruby-bin-mruby`) are reduced slightly in
my environment than before the introduction of new specifiers/modifiers
(5116789a) with this change.
------------+-------------------+-------------------+--------
BINARY | BEFORE (5116789a) | AFTER (This PR) | RATIO
------------+-------------------+-------------------+--------
mruby | 593416 bytes | 593208 bytes | -0.04%
libmruby.a | 769048 bytes | 767264 bytes | -0.23%
------------+-------------------+-------------------+--------
BTW, I accidentally changed `tasks/toolchains/visualcpp.rake` at #4613,
so I put it back.
Now the method tables (in classes/modules and caches) keeps C function
pointers without wrapping in `struct RProc` objects. For the sake of
portability, `mrb_method_t` is represented by the struct and union, but
if the most significant bit of the pointer is not used by the platform,
`mrb_method_t` should be packed in `uintptr_t` to reduce memory usage.
`MRB_METHOD_TABLE_INLINE` is turned on by default for linux.
Instead of `irep` links, we added a `upper` link to `struct RProc`.
To make a space for the `upper` link, we moved `target_class` reference.
If a `Proc` does not have `env`, `target_class` is saved in an `union`
shared with `env` (if a `Proc` has env, you can tell it by `MRB_PROC_ENV_P()).
Otherwise `target_class` is referenced from `env->c`. We removed links
in `env` as well.
This change removes 2 members from `mrb_irep` struct, thus saving 2
words per method/proc/block. This also fixes potential memory leaks
due to the circular references caused by a link from `mrb_irep`.
Yukihiro "Matz" Matsumoto