Commit Graph

1036 Commits

Author SHA1 Message Date
KOBAYASHI Shuji 89f591485b Change name and usage of presym macros
To be also able to build mruby without presym in the future. However,
`MRB_QSYM` has been removed and changed as follows:

### Example

|       Type                | Symbol |  Previous Style  |   New Style    |
|---------------------------|--------|------------------|----------------|
| Operator                  | &      | MRB_QSYM(and)    | MRB_OPSYM(and) |
| Class Variable            | @@foo  | MRB_QSYM(00_foo) | MRB_CVSYM(foo) |
| Instance Variable         | @foo   | MRB_QSYM(0_foo)  | MRB_IVSYM(foo) |
| Method with Bang          | foo!   | MRB_QSYM(foo_b)  | MRB_SYM_B(foo) |
| Method with Question mark | foo?   | MRB_QSYM(foo_p)  | MRB_SYM_Q(foo) |
| Mmethod with Equal        | foo=   | MRB_QSYM(foo_e)  | MRB_SYM_E(foo) |

This change makes it possible to define, for example, `MRB_IVSYM(foo)` as
`mrb_intern_lit(mrb, "@" "foo")`, which is useful if we support building
without presym in the future.
2020-11-13 13:41:20 +09:00
KOBAYASHI Shuji ff2ef44c0e Remove unused argument of mrb_obj_iv_tbl_memsize 2020-11-12 20:19:41 +09:00
Yukihiro "Matz" Matsumoto 6a5e97b448 Merge pull request #5121 from shuujii/reduce-memory-usage-of-Hash-object
Reduce memory usage of Hash object
2020-11-10 21:56:42 +09:00
Sutou Kouhei a00eabbf5d Fix typo in backward compatible mrb_exc_new_str_lit() macro 2020-11-10 16:03:05 +09:00
KOBAYASHI Shuji f2d8db39be Reduce memory usage of Hash object
## Implementation Summary

* Change entry list from segmented list to flat array.
* Change value of hash bucket from pointer to entry to index of entry list,
  and represent it by variable length bits according to capacity of hash
  buckets.
* Store management information about entry list and hash table to `struct
  RHash` as much as possible.

## Benchmark Summary

Only the results of typical situations on 64-bit Word-boxing are present
here. For more detailed information, including consideration, see below
(although most of the body is written in Japanese).

* https://shuujii.github.io/mruby-hash-benchmark

### Memory Usage

Lower value is better.

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |          344B |          256B |   0.74419x |
|        40 |        1,464B |          840B |   0.57377x |
|       200 |        8,056B |        3,784B |   0.46971x |
|       500 |       17,169B |        9,944B |   0.57949x |

### Performance

Higher value is better.

#### `mrb_hash_set`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  1.41847M i/s |  1.36004M i/s |   0.95881x |
|        40 |  0.39224M i/s |  0.31888M i/s |   0.81296x |
|       200 |  0.03780M i/s |  0.04290M i/s |   1.13494x |
|       500 |  0.01225M i/s |  0.01314M i/s |   1.07275x |

#### `mrb_hash_get`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 26.05920M i/s | 30.19543M i/s |   1.15872x |
|        40 | 44.26420M i/s | 32.75781M i/s |   0.74005x |
|       200 | 44.55171M i/s | 31.56926M i/s |   0.70860x |
|       500 | 39.19250M i/s | 29.73806M i/s |   0.75877x |

#### `mrb_hash_each`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 25.11964M i/s | 30.34167M i/s |   1.20789x |
|        40 | 11.74253M i/s | 13.25539M i/s |   1.12884x |
|       200 |  2.01133M i/s |  2.97214M i/s |   1.47770x |
|       500 |  0.87411M i/s |  1.21178M i/s |   1.38631x |

#### `Hash#[]=`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.50095M i/s |  0.56490M i/s |   1.12764x |
|        40 |  0.19132M i/s |  0.18392M i/s |   0.96129x |
|       200 |  0.03624M i/s |  0.03256M i/s |   0.89860x |
|       500 |  0.01527M i/s |  0.01236M i/s |   0.80935x |
#### `Hash#[]`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 11.53211M i/s | 12.78806M i/s |   1.10891x |
|        40 | 15.26920M i/s | 13.37529M i/s |   0.87596x |
|       200 | 15.28550M i/s | 13.36410M i/s |   0.87430x |
|       500 | 14.57695M i/s | 12.75388M i/s |   0.87494x |

#### `Hash#each`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.30462M i/s |  0.27080M i/s |   0.88898x |
|        40 |  0.12912M i/s |  0.11704M i/s |   0.90642x |
|       200 |  0.02638M i/s |  0.02402M i/s |   0.91071x |
|       500 |  0.01066M i/s |  0.00959M i/s |   0.89953x |

#### `Hash#delete`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  7.84167M i/s |  6.96419M i/s |   0.88810x |
|        40 |  6.91292M i/s |  7.41427M i/s |   1.07252x |
|       200 |  3.75952M i/s |  7.32080M i/s |   1.94727x |
|       500 |  2.10754M i/s |  7.05963M i/s |   3.34970x |

#### `Hash#shift`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 14.66444M i/s | 13.18876M i/s |   0.89937x |
|        40 | 11.95124M i/s | 11.10420M i/s |   0.92913x |
|       200 |  5.53681M i/s |  7.88155M i/s |   1.42348x |
|       500 |  2.96728M i/s |  5.40405M i/s |   1.82121x |

#### `Hash#dup`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.15063M i/s |  5.37889M i/s |  35.71024x |
|        40 |  0.06515M i/s |  3.38196M i/s |  51.91279x |
|       200 |  0.01359M i/s |  1.46538M i/s | 107.84056x |
|       500 |  0.00559M i/s |  0.75411M i/s | 134.88057x |

### Binary Size

Lower value is better.

|    File    |   Baseline    |      New      |  Factor   |
|:-----------|--------------:|--------------:|----------:|
| mruby      |      730,408B |      734,176B |  1.00519x |
| libmruby.a |    1,068,134B |    1,072,846B |  1.00441x |

## Other Fixes

The following issues have also been fixed in the parts where there was some
change this time.

* [Heap use-after-free in `Hash#value?`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-hash-value-md)
* [Heap use-after-free in `ht_hash_equal`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_equal-md)
* [Heap use-after-free in `ht_hash_func`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_func-md)
* [Heap use-after-free in `mrb_hash_merge`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-mrb_hash_merge-md)
* [Self-replacement does not work for `Hash#replace`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-self-replacement-does-not-work-for-hash-replace-md)
* [Repeated deletes and inserts increase memory usage of `Hash`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-repeated-deletes-and-inserts-increase-memory-usage-of-hash-md)
* [`Hash#rehash` does not reindex completely](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-hash-rehash-does-not-reindex-completely-md)
* `mrb_hash_delete_key` does not cause an error for frozen object
* `mrb_hash_new_capa` does not allocate required space first
* [`mrb_os_memsize_of_hash_table` result is incorrect](https://github.com/mruby/mruby/pull/5032#discussion_r457994075)
2020-11-10 15:21:49 +09:00
Yukihiro "Matz" Matsumoto 352c4755b2 Merge pull request #5118 from dearblue/mrb_value
Define `mrb_value` in struct for Word/NaN boxing
2020-11-10 11:50:59 +09:00
dearblue e993b83c50 Fixed build NaN boxing with 32-bit CPU mode
`SET_CPTR_VALUE()` requires the `p` field on 32-bit CPU mode.
2020-11-08 14:15:19 +09:00
dearblue 926ec1dc7e Define mrb_value in struct for Word/NaN boxing
This is to make it possible to distinguish between `mrb_value` and `mrb_sym` in C++ "Function overloading" and C11 "Generic selection".
2020-11-08 14:14:00 +09:00
Yukihiro "Matz" Matsumoto d13df1536d Add a new instruction OP_LOADI32.
That loads 32 bit integer bypassing pool access.
2020-11-04 14:01:04 +09:00
Yukihiro "Matz" Matsumoto b7e8406f6c Add new instructions to handle symbols/literals >255; fix #5109
New instructions:
  * OP_LOADL16
  * OP_LOADSYM16
  * OP_STRING16

Size of pools, symbols are `int16_t` but offset representation in the
bytecode was 8 bits. Size of child `irep` array is `int16_t`, too but
this change does not address it.
2020-11-03 14:58:44 +09:00
mimaki bc9566eb64 Add preview to version. 2020-11-01 13:42:35 +09:00
Hiroshi Mimaki 0589749952 mruby 3.0.0-preview. 2020-10-16 10:43:03 +09:00
Yukihiro "Matz" Matsumoto 5069fb15e4 Cause error explicitly from MRB_TRY() with cxx_exception; ref #5088
`MRB_TRY()` does not work when compiled by C compiler with `cxx_exception`
configuration. We should explicitly warn.
2020-10-12 18:20:29 +09:00
Yukihiro "Matz" Matsumoto 896338e219 Unify mrb_str_to_str to mrb_obj_as_string.
Redirect `mrb_str_to_str` to `mrb_obj_as_string` via C macro.
Inspired by #5082
2020-10-12 18:20:26 +09:00
Yukihiro "Matz" Matsumoto f1d02bfc15 Clarify the meaning of MRB_IREP_STATIC; ref #5084 2020-10-12 18:20:26 +09:00
Yukihiro "Matz" Matsumoto c1f5fdafb6 Remove duplicated constant IREP_TT_SFLAG; #5084 2020-10-12 18:20:25 +09:00
Yukihiro "Matz" Matsumoto 3ff2757bf7 Fix mrb_int_mul_overflow() to check either operand being zero. 2020-10-12 18:20:22 +09:00
Yukihiro "Matz" Matsumoto 9b8e5658ce Fix integer casting on 64 bit platforms.
On platforms where `sizeof(long)` is 4, casting `(long)` can lose data
or sign information.
2020-10-12 18:20:18 +09:00
Yukihiro "Matz" Matsumoto 471479e723 Change float representation in mrb binary files.
From human readable (ASCII) string representation to binary dump of
IEEE754 in little endian.
2020-10-12 18:20:06 +09:00
Yukihiro "Matz" Matsumoto 397b005715 Replace the implementation of method tables in classes/modules.
They are basically the copy of instance variable tables. On my Linux
box, memory consumption of `mrbtest` measured by `valgrind` is:

- old: 17,683,830 bytes
- new: 14,283,749 bytes
2020-10-12 18:20:06 +09:00
Yukihiro "Matz" Matsumoto 7e3d22f095 Raname mrb_exc_new_str_lit() to mrb_exc_new_lit().
It uses `mrb_str_new_lit()` internally, but it doesn't need to express
it in the name of the function (macro).
2020-10-12 18:20:06 +09:00
Yukihiro "Matz" Matsumoto 9d613159b5 Add mrb_integer() definition for MRB_NAN_BOXING. 2020-10-12 18:20:04 +09:00
Yukihiro "Matz" Matsumoto 2b188ed8a1 Reorganize Integer system.
- Integrate `Fixnum` and `Integer`
- Remove `Integral`
- `int / int -> int`
- Replace `mrb_fixnum()` to `mrb_int()`
- Replace `mrb_fixnum_value()` to `mrb_int_value()`.
- Use `mrb_integer_p()` instead of `mrb_fixnum_p()`
2020-10-12 18:19:54 +09:00
Yukihiro "Matz" Matsumoto 5a7cbebc8a Remove MRB_NO_FLOAT_INLINE and MRB_WBOX_FLOAT_INLINE configuration.
They are not used from the beginning.
2020-10-12 16:21:49 +09:00
Yukihiro "Matz" Matsumoto 55163a8a0a Rename MRB_TT_FIXNUM to MRB_TT_INTEGER.
We still have `#define MRB_TT_FIXNUM MRB_TT_INTEGER` for compatibility.
2020-10-12 16:21:47 +09:00
dearblue 80fe9838d2 Integrate Fixnum class into Integer class
* The `Fixnum` constant is now an alias for the `Integer` class.
* Remove `struct mrb_state::fixnum_class` member.
  If necessary, use `struct mrb_state::integer_class` instead.
2020-10-12 16:21:44 +09:00
Yukihiro "Matz" Matsumoto 424afa4446 Make Proc#parameters to support keyword arguments; fix #5066
TODO: Unlike CRuby, mruby's `Proc#parameters` does not distinguish
required keyword arguments and optional keyword arguments currently.
2020-10-12 16:21:42 +09:00
Yukihiro "Matz" Matsumoto 8a87549315 Rename float configuration option names.
- `MRB_WITHOUT_FLOAT` => `MRB_NO_FLOAT`
- `MRB_USE_FLOAT` => `MRB_USE_FLOAT32`

The former is to use `USE_XXX` naming convention. The latter is to make
sure `float` is 32bit float and not floating point number in general.
2020-10-12 16:21:40 +09:00
Yukihiro "Matz" Matsumoto 72d071540c Rename MRB_METHOD_T_STRUCT to MRB_USE_METHOD_T_STRUCT.
It's the first change of renaming configuration options to `MRB_XXX` to
`MRB_USE_XXX` or `MRB_NO_XXX`.
2020-10-12 16:21:38 +09:00
Yukihiro "Matz" Matsumoto 8813ebec07 Skip array embedding if MRB_NO_BOXING and MRB_32BIT; fix #4382
On some platforms, `sizeof(mrb_value) > sizeof(void*)*3`, which makes
`MRB_ARY_EMBED_LEN_MAX` zero. And zero sized array cause compile errors.
2020-10-12 16:21:36 +09:00
Yukihiro "Matz" Matsumoto 763b39d00b Simplify NaN boxing definitions.
Remove `#ifdef` from `union mrb_value_`.
2020-10-12 16:21:35 +09:00
Yukihiro "Matz" Matsumoto 107c777341 Rename OP_JUW instruction to OP_JMPUW. 2020-10-12 16:21:35 +09:00
Yukihiro "Matz" Matsumoto 500f721f70 Fix typo _hander -> _handler. 2020-10-12 16:21:34 +09:00
dearblue c1f112c49a Replace global jump with catch handler implementation
When a global jump occurs, look at the catch handler table to determine where to jump.
In that case, `pc` already shows the following instruction, but since the table shows `begin_offset ... end_offset`, the comparison is done with `begin_offset < pc && pc <= end_offset`.
If there is a corresponding handler, move `pc` to `handler.target_offset` and continue running the VM.

When a global jump across `ensure` is made by `return`, `break`, `next`, `redo` and `retry`, the extended `RBreak` object saves and restores the C-level execution position.
This extended `RBreak` can have tag information, which makes it a pseudo coroutine (the "tag" mimics CRuby).

The implementation of pseudo coroutines by `RBreak` is summarized by `CHECKPOINT_RESTORE ... CHECKPOINT_MAIN ... CHECKPOINT_END` and `throw_tagged_break` / `unwind_ensure` macros.
The restart of processing is branched by `RBREAK_TAG_FOREACH(DISPATCH_CHECKPOINTS)`.

- Not only `rescue` blocks but also `ensure` blocks are now sandwiched between `OP_EXCEPT` and `OP_RAISEIF`.

- Remove the function `ecall()`.
  It is no longer necessary to re-enter the VM to perform an "ensure block".

  This will resolves #1888.

- Added instruction `OP_JUW` (Jump while UnWind).

  It jumps unconditionally like `OP_JMP`, but searches the catch handler table and executes the ensure block.
  Since it searches the catch handler table, it is much heavier than `OP_JMP`.
2020-10-12 16:21:33 +09:00
dearblue 0d7b4deccf Removed push/pop instructions for rescue/ensure
`OP_PUSHERR`, `OP_POPERR`, `OP_EPUSH` and `OP_EPOP` are removed.
2020-10-12 16:21:32 +09:00
dearblue f467b02d45 Extended OP_EXCEPT and OP_RAISE (OP_RAISEIF) instructions
- `OP_EXCEPT` checks if `mrb->exc` is `NULL`, `MRB_TT_EXCEPTION` or
  `MRB_TT_BREAK`.
  If `mrb->exc` is `NULL`, it will be replaced with `nil`.

- If `OP_RAISE` is `nil`, it does nothing and the immediately
  following instruction is executed (like `OP_NOP`).
  Also, in case of `RBreak` object, it moves to the processing for
  `break`.
  With this change, the instruction name is changed from
  `OP_RAISE` to `OP_RAISEIF`.
2020-10-12 16:21:32 +09:00
dearblue a54a3df32c Extended mruby binary format
The catch handler table is combined with iseq block.
This is to prevent the structure from growing by adding a field for the
catch handler table to the `mrb_irep` structure.

"iseq block" and "catch handler table":
  [number of catch handler table (2 bytes)]
  [number of byte code (4 bytes)]
  [iseq (any bytes)]
  [catch handlers (multiple of 7 bytes)]

catch handler:
  [catch type (1 byte)]
  [begin offset (2 bytes)]
  [end offset (2 bytes)]
  [target offset (2 bytes)]

catch type: enum mrb_catch_type (0 = rescue, 1 = ensure)
begin offset: Includes the specified instruction address
end offset: Does not include the specified instruction address
target offset: replaces pc with the specified instruction address

This table is not expanded by `read_irep_record_1()`.
The necessary elements are expanded one by one when used.
2020-10-12 16:21:32 +09:00
Yukihiro "Matz" Matsumoto 6fbc03b3a8 Revert 4c001673b
Probably I misunderstand strict aliasing rule of C++. The fix in
4c001673b was other way around.
2020-10-12 16:21:30 +09:00
Yukihiro "Matz" Matsumoto 4e56e0defc Rename union mrb_value to union mrb_value_.
Since some compiler complains when we `typedef` `mrb_value`.
2020-10-12 16:21:30 +09:00
Yukihiro "Matz" Matsumoto 53e46be7e2 Simplify mrb_value structure for MRB_WORD_BOXING. 2020-10-12 16:21:29 +09:00
Yukihiro "Matz" Matsumoto 73e289c54e Simplify mrb_value structure for MRB_NAN_BOXING. 2020-10-12 16:21:29 +09:00
Yukihiro "Matz" Matsumoto 425b142168 Refine MRB_NAN_BOXING on 32bit architecture.
You don't need pointer tweaking on 32bit architecture, where pointers
fit in 32bit (lower half of mrb_value).
2020-10-12 16:21:27 +09:00
Yukihiro "Matz" Matsumoto 5e9dc72bc5 Pack mrb_value into uint64_t when MRB_NAN_BOXING. 2020-10-12 16:21:25 +09:00
Yukihiro "Matz" Matsumoto 3d8a38bea4 You don't need to keep index in local variables info in irep. 2020-10-12 16:21:22 +09:00
Yukihiro "Matz" Matsumoto 639946a006 Enable method cache by default.
Introduced `MRB_NO_METHOD_CACHE` which is inverse of `MRB_METHOD_CACHE`
that should be enabled intestinally. In addition, the default cache is
made bigger (128 -> 256).
2020-10-12 16:21:22 +09:00
Yukihiro "Matz" Matsumoto 6db0162ec7 Upgrade RITE_VM_VERSION to 0300 (means mruby 3.0). 2020-10-12 16:21:20 +09:00
Yukihiro "Matz" Matsumoto dec34d6c7b Split MRB_BINARY_FORMAT to major and minor.
The minor versions should be upper compatible. So mere opcode, section
addition can be done without breaking compiled binary.
2020-10-12 16:21:20 +09:00
Yukihiro "Matz" Matsumoto 8f0ac27196 Update opcode reference and comment.
- no OP_EXT_ anymore
- OP_LOADI16 in right position
2020-10-12 16:21:19 +09:00
Yukihiro "Matz" Matsumoto fd10c72319 Remove OP_EXT[123] from operands. 2020-10-12 16:21:19 +09:00
Yukihiro "Matz" Matsumoto ce30a4e04b Add const to irep structure to place data on ROM. 2020-10-12 16:21:16 +09:00