Commit Graph

937 Commits

Author SHA1 Message Date
Yukihiro "Matz" Matsumoto af13cfb7d5 Merge pull request #5122 from shuujii/change-name-and-usage-of-presym-macros
Change name and usage of presym macros
2020-11-13 14:05:26 +09:00
KOBAYASHI Shuji 89f591485b Change name and usage of presym macros
To be also able to build mruby without presym in the future. However,
`MRB_QSYM` has been removed and changed as follows:

### Example

|       Type                | Symbol |  Previous Style  |   New Style    |
|---------------------------|--------|------------------|----------------|
| Operator                  | &      | MRB_QSYM(and)    | MRB_OPSYM(and) |
| Class Variable            | @@foo  | MRB_QSYM(00_foo) | MRB_CVSYM(foo) |
| Instance Variable         | @foo   | MRB_QSYM(0_foo)  | MRB_IVSYM(foo) |
| Method with Bang          | foo!   | MRB_QSYM(foo_b)  | MRB_SYM_B(foo) |
| Method with Question mark | foo?   | MRB_QSYM(foo_p)  | MRB_SYM_Q(foo) |
| Mmethod with Equal        | foo=   | MRB_QSYM(foo_e)  | MRB_SYM_E(foo) |

This change makes it possible to define, for example, `MRB_IVSYM(foo)` as
`mrb_intern_lit(mrb, "@" "foo")`, which is useful if we support building
without presym in the future.
2020-11-13 13:41:20 +09:00
KOBAYASHI Shuji ff2ef44c0e Remove unused argument of mrb_obj_iv_tbl_memsize 2020-11-12 20:19:41 +09:00
Yukihiro "Matz" Matsumoto 6a5e97b448 Merge pull request #5121 from shuujii/reduce-memory-usage-of-Hash-object
Reduce memory usage of Hash object
2020-11-10 21:56:42 +09:00
Sutou Kouhei a00eabbf5d Fix typo in backward compatible mrb_exc_new_str_lit() macro 2020-11-10 16:03:05 +09:00
KOBAYASHI Shuji f2d8db39be Reduce memory usage of Hash object
## Implementation Summary

* Change entry list from segmented list to flat array.
* Change value of hash bucket from pointer to entry to index of entry list,
  and represent it by variable length bits according to capacity of hash
  buckets.
* Store management information about entry list and hash table to `struct
  RHash` as much as possible.

## Benchmark Summary

Only the results of typical situations on 64-bit Word-boxing are present
here. For more detailed information, including consideration, see below
(although most of the body is written in Japanese).

* https://shuujii.github.io/mruby-hash-benchmark

### Memory Usage

Lower value is better.

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |          344B |          256B |   0.74419x |
|        40 |        1,464B |          840B |   0.57377x |
|       200 |        8,056B |        3,784B |   0.46971x |
|       500 |       17,169B |        9,944B |   0.57949x |

### Performance

Higher value is better.

#### `mrb_hash_set`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  1.41847M i/s |  1.36004M i/s |   0.95881x |
|        40 |  0.39224M i/s |  0.31888M i/s |   0.81296x |
|       200 |  0.03780M i/s |  0.04290M i/s |   1.13494x |
|       500 |  0.01225M i/s |  0.01314M i/s |   1.07275x |

#### `mrb_hash_get`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 26.05920M i/s | 30.19543M i/s |   1.15872x |
|        40 | 44.26420M i/s | 32.75781M i/s |   0.74005x |
|       200 | 44.55171M i/s | 31.56926M i/s |   0.70860x |
|       500 | 39.19250M i/s | 29.73806M i/s |   0.75877x |

#### `mrb_hash_each`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 25.11964M i/s | 30.34167M i/s |   1.20789x |
|        40 | 11.74253M i/s | 13.25539M i/s |   1.12884x |
|       200 |  2.01133M i/s |  2.97214M i/s |   1.47770x |
|       500 |  0.87411M i/s |  1.21178M i/s |   1.38631x |

#### `Hash#[]=`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.50095M i/s |  0.56490M i/s |   1.12764x |
|        40 |  0.19132M i/s |  0.18392M i/s |   0.96129x |
|       200 |  0.03624M i/s |  0.03256M i/s |   0.89860x |
|       500 |  0.01527M i/s |  0.01236M i/s |   0.80935x |
#### `Hash#[]`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 11.53211M i/s | 12.78806M i/s |   1.10891x |
|        40 | 15.26920M i/s | 13.37529M i/s |   0.87596x |
|       200 | 15.28550M i/s | 13.36410M i/s |   0.87430x |
|       500 | 14.57695M i/s | 12.75388M i/s |   0.87494x |

#### `Hash#each`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.30462M i/s |  0.27080M i/s |   0.88898x |
|        40 |  0.12912M i/s |  0.11704M i/s |   0.90642x |
|       200 |  0.02638M i/s |  0.02402M i/s |   0.91071x |
|       500 |  0.01066M i/s |  0.00959M i/s |   0.89953x |

#### `Hash#delete`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  7.84167M i/s |  6.96419M i/s |   0.88810x |
|        40 |  6.91292M i/s |  7.41427M i/s |   1.07252x |
|       200 |  3.75952M i/s |  7.32080M i/s |   1.94727x |
|       500 |  2.10754M i/s |  7.05963M i/s |   3.34970x |

#### `Hash#shift`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 14.66444M i/s | 13.18876M i/s |   0.89937x |
|        40 | 11.95124M i/s | 11.10420M i/s |   0.92913x |
|       200 |  5.53681M i/s |  7.88155M i/s |   1.42348x |
|       500 |  2.96728M i/s |  5.40405M i/s |   1.82121x |

#### `Hash#dup`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.15063M i/s |  5.37889M i/s |  35.71024x |
|        40 |  0.06515M i/s |  3.38196M i/s |  51.91279x |
|       200 |  0.01359M i/s |  1.46538M i/s | 107.84056x |
|       500 |  0.00559M i/s |  0.75411M i/s | 134.88057x |

### Binary Size

Lower value is better.

|    File    |   Baseline    |      New      |  Factor   |
|:-----------|--------------:|--------------:|----------:|
| mruby      |      730,408B |      734,176B |  1.00519x |
| libmruby.a |    1,068,134B |    1,072,846B |  1.00441x |

## Other Fixes

The following issues have also been fixed in the parts where there was some
change this time.

* [Heap use-after-free in `Hash#value?`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-hash-value-md)
* [Heap use-after-free in `ht_hash_equal`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_equal-md)
* [Heap use-after-free in `ht_hash_func`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_func-md)
* [Heap use-after-free in `mrb_hash_merge`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-mrb_hash_merge-md)
* [Self-replacement does not work for `Hash#replace`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-self-replacement-does-not-work-for-hash-replace-md)
* [Repeated deletes and inserts increase memory usage of `Hash`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-repeated-deletes-and-inserts-increase-memory-usage-of-hash-md)
* [`Hash#rehash` does not reindex completely](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-hash-rehash-does-not-reindex-completely-md)
* `mrb_hash_delete_key` does not cause an error for frozen object
* `mrb_hash_new_capa` does not allocate required space first
* [`mrb_os_memsize_of_hash_table` result is incorrect](https://github.com/mruby/mruby/pull/5032#discussion_r457994075)
2020-11-10 15:21:49 +09:00
Yukihiro "Matz" Matsumoto 352c4755b2 Merge pull request #5118 from dearblue/mrb_value
Define `mrb_value` in struct for Word/NaN boxing
2020-11-10 11:50:59 +09:00
dearblue e993b83c50 Fixed build NaN boxing with 32-bit CPU mode
`SET_CPTR_VALUE()` requires the `p` field on 32-bit CPU mode.
2020-11-08 14:15:19 +09:00
dearblue 926ec1dc7e Define mrb_value in struct for Word/NaN boxing
This is to make it possible to distinguish between `mrb_value` and `mrb_sym` in C++ "Function overloading" and C11 "Generic selection".
2020-11-08 14:14:00 +09:00
Yukihiro "Matz" Matsumoto d13df1536d Add a new instruction OP_LOADI32.
That loads 32 bit integer bypassing pool access.
2020-11-04 14:01:04 +09:00
Yukihiro "Matz" Matsumoto b7e8406f6c Add new instructions to handle symbols/literals >255; fix #5109
New instructions:
  * OP_LOADL16
  * OP_LOADSYM16
  * OP_STRING16

Size of pools, symbols are `int16_t` but offset representation in the
bytecode was 8 bits. Size of child `irep` array is `int16_t`, too but
this change does not address it.
2020-11-03 14:58:44 +09:00
mimaki bc9566eb64 Add preview to version. 2020-11-01 13:42:35 +09:00
Hiroshi Mimaki 0589749952 mruby 3.0.0-preview. 2020-10-16 10:43:03 +09:00
Yukihiro "Matz" Matsumoto 5069fb15e4 Cause error explicitly from MRB_TRY() with cxx_exception; ref #5088
`MRB_TRY()` does not work when compiled by C compiler with `cxx_exception`
configuration. We should explicitly warn.
2020-10-12 18:20:29 +09:00
Yukihiro "Matz" Matsumoto 896338e219 Unify mrb_str_to_str to mrb_obj_as_string.
Redirect `mrb_str_to_str` to `mrb_obj_as_string` via C macro.
Inspired by #5082
2020-10-12 18:20:26 +09:00
Yukihiro "Matz" Matsumoto f1d02bfc15 Clarify the meaning of MRB_IREP_STATIC; ref #5084 2020-10-12 18:20:26 +09:00
Yukihiro "Matz" Matsumoto c1f5fdafb6 Remove duplicated constant IREP_TT_SFLAG; #5084 2020-10-12 18:20:25 +09:00
Yukihiro "Matz" Matsumoto 3ff2757bf7 Fix mrb_int_mul_overflow() to check either operand being zero. 2020-10-12 18:20:22 +09:00
Yukihiro "Matz" Matsumoto 9b8e5658ce Fix integer casting on 64 bit platforms.
On platforms where `sizeof(long)` is 4, casting `(long)` can lose data
or sign information.
2020-10-12 18:20:18 +09:00
Yukihiro "Matz" Matsumoto 471479e723 Change float representation in mrb binary files.
From human readable (ASCII) string representation to binary dump of
IEEE754 in little endian.
2020-10-12 18:20:06 +09:00
Yukihiro "Matz" Matsumoto 397b005715 Replace the implementation of method tables in classes/modules.
They are basically the copy of instance variable tables. On my Linux
box, memory consumption of `mrbtest` measured by `valgrind` is:

- old: 17,683,830 bytes
- new: 14,283,749 bytes
2020-10-12 18:20:06 +09:00
Yukihiro "Matz" Matsumoto 7e3d22f095 Raname mrb_exc_new_str_lit() to mrb_exc_new_lit().
It uses `mrb_str_new_lit()` internally, but it doesn't need to express
it in the name of the function (macro).
2020-10-12 18:20:06 +09:00
Yukihiro "Matz" Matsumoto 9d613159b5 Add mrb_integer() definition for MRB_NAN_BOXING. 2020-10-12 18:20:04 +09:00
Yukihiro "Matz" Matsumoto 2b188ed8a1 Reorganize Integer system.
- Integrate `Fixnum` and `Integer`
- Remove `Integral`
- `int / int -> int`
- Replace `mrb_fixnum()` to `mrb_int()`
- Replace `mrb_fixnum_value()` to `mrb_int_value()`.
- Use `mrb_integer_p()` instead of `mrb_fixnum_p()`
2020-10-12 18:19:54 +09:00
Yukihiro "Matz" Matsumoto 5a7cbebc8a Remove MRB_NO_FLOAT_INLINE and MRB_WBOX_FLOAT_INLINE configuration.
They are not used from the beginning.
2020-10-12 16:21:49 +09:00
Yukihiro "Matz" Matsumoto 55163a8a0a Rename MRB_TT_FIXNUM to MRB_TT_INTEGER.
We still have `#define MRB_TT_FIXNUM MRB_TT_INTEGER` for compatibility.
2020-10-12 16:21:47 +09:00
dearblue 80fe9838d2 Integrate Fixnum class into Integer class
* The `Fixnum` constant is now an alias for the `Integer` class.
* Remove `struct mrb_state::fixnum_class` member.
  If necessary, use `struct mrb_state::integer_class` instead.
2020-10-12 16:21:44 +09:00
Yukihiro "Matz" Matsumoto 424afa4446 Make Proc#parameters to support keyword arguments; fix #5066
TODO: Unlike CRuby, mruby's `Proc#parameters` does not distinguish
required keyword arguments and optional keyword arguments currently.
2020-10-12 16:21:42 +09:00
Yukihiro "Matz" Matsumoto 8a87549315 Rename float configuration option names.
- `MRB_WITHOUT_FLOAT` => `MRB_NO_FLOAT`
- `MRB_USE_FLOAT` => `MRB_USE_FLOAT32`

The former is to use `USE_XXX` naming convention. The latter is to make
sure `float` is 32bit float and not floating point number in general.
2020-10-12 16:21:40 +09:00
Yukihiro "Matz" Matsumoto 72d071540c Rename MRB_METHOD_T_STRUCT to MRB_USE_METHOD_T_STRUCT.
It's the first change of renaming configuration options to `MRB_XXX` to
`MRB_USE_XXX` or `MRB_NO_XXX`.
2020-10-12 16:21:38 +09:00
Yukihiro "Matz" Matsumoto 8813ebec07 Skip array embedding if MRB_NO_BOXING and MRB_32BIT; fix #4382
On some platforms, `sizeof(mrb_value) > sizeof(void*)*3`, which makes
`MRB_ARY_EMBED_LEN_MAX` zero. And zero sized array cause compile errors.
2020-10-12 16:21:36 +09:00
Yukihiro "Matz" Matsumoto 763b39d00b Simplify NaN boxing definitions.
Remove `#ifdef` from `union mrb_value_`.
2020-10-12 16:21:35 +09:00
Yukihiro "Matz" Matsumoto 107c777341 Rename OP_JUW instruction to OP_JMPUW. 2020-10-12 16:21:35 +09:00
Yukihiro "Matz" Matsumoto 500f721f70 Fix typo _hander -> _handler. 2020-10-12 16:21:34 +09:00
dearblue c1f112c49a Replace global jump with catch handler implementation
When a global jump occurs, look at the catch handler table to determine where to jump.
In that case, `pc` already shows the following instruction, but since the table shows `begin_offset ... end_offset`, the comparison is done with `begin_offset < pc && pc <= end_offset`.
If there is a corresponding handler, move `pc` to `handler.target_offset` and continue running the VM.

When a global jump across `ensure` is made by `return`, `break`, `next`, `redo` and `retry`, the extended `RBreak` object saves and restores the C-level execution position.
This extended `RBreak` can have tag information, which makes it a pseudo coroutine (the "tag" mimics CRuby).

The implementation of pseudo coroutines by `RBreak` is summarized by `CHECKPOINT_RESTORE ... CHECKPOINT_MAIN ... CHECKPOINT_END` and `throw_tagged_break` / `unwind_ensure` macros.
The restart of processing is branched by `RBREAK_TAG_FOREACH(DISPATCH_CHECKPOINTS)`.

- Not only `rescue` blocks but also `ensure` blocks are now sandwiched between `OP_EXCEPT` and `OP_RAISEIF`.

- Remove the function `ecall()`.
  It is no longer necessary to re-enter the VM to perform an "ensure block".

  This will resolves #1888.

- Added instruction `OP_JUW` (Jump while UnWind).

  It jumps unconditionally like `OP_JMP`, but searches the catch handler table and executes the ensure block.
  Since it searches the catch handler table, it is much heavier than `OP_JMP`.
2020-10-12 16:21:33 +09:00
dearblue 0d7b4deccf Removed push/pop instructions for rescue/ensure
`OP_PUSHERR`, `OP_POPERR`, `OP_EPUSH` and `OP_EPOP` are removed.
2020-10-12 16:21:32 +09:00
dearblue f467b02d45 Extended OP_EXCEPT and OP_RAISE (OP_RAISEIF) instructions
- `OP_EXCEPT` checks if `mrb->exc` is `NULL`, `MRB_TT_EXCEPTION` or
  `MRB_TT_BREAK`.
  If `mrb->exc` is `NULL`, it will be replaced with `nil`.

- If `OP_RAISE` is `nil`, it does nothing and the immediately
  following instruction is executed (like `OP_NOP`).
  Also, in case of `RBreak` object, it moves to the processing for
  `break`.
  With this change, the instruction name is changed from
  `OP_RAISE` to `OP_RAISEIF`.
2020-10-12 16:21:32 +09:00
dearblue a54a3df32c Extended mruby binary format
The catch handler table is combined with iseq block.
This is to prevent the structure from growing by adding a field for the
catch handler table to the `mrb_irep` structure.

"iseq block" and "catch handler table":
  [number of catch handler table (2 bytes)]
  [number of byte code (4 bytes)]
  [iseq (any bytes)]
  [catch handlers (multiple of 7 bytes)]

catch handler:
  [catch type (1 byte)]
  [begin offset (2 bytes)]
  [end offset (2 bytes)]
  [target offset (2 bytes)]

catch type: enum mrb_catch_type (0 = rescue, 1 = ensure)
begin offset: Includes the specified instruction address
end offset: Does not include the specified instruction address
target offset: replaces pc with the specified instruction address

This table is not expanded by `read_irep_record_1()`.
The necessary elements are expanded one by one when used.
2020-10-12 16:21:32 +09:00
Yukihiro "Matz" Matsumoto 6fbc03b3a8 Revert 4c001673b
Probably I misunderstand strict aliasing rule of C++. The fix in
4c001673b was other way around.
2020-10-12 16:21:30 +09:00
Yukihiro "Matz" Matsumoto 4e56e0defc Rename union mrb_value to union mrb_value_.
Since some compiler complains when we `typedef` `mrb_value`.
2020-10-12 16:21:30 +09:00
Yukihiro "Matz" Matsumoto 53e46be7e2 Simplify mrb_value structure for MRB_WORD_BOXING. 2020-10-12 16:21:29 +09:00
Yukihiro "Matz" Matsumoto 73e289c54e Simplify mrb_value structure for MRB_NAN_BOXING. 2020-10-12 16:21:29 +09:00
Yukihiro "Matz" Matsumoto 425b142168 Refine MRB_NAN_BOXING on 32bit architecture.
You don't need pointer tweaking on 32bit architecture, where pointers
fit in 32bit (lower half of mrb_value).
2020-10-12 16:21:27 +09:00
Yukihiro "Matz" Matsumoto 5e9dc72bc5 Pack mrb_value into uint64_t when MRB_NAN_BOXING. 2020-10-12 16:21:25 +09:00
Yukihiro "Matz" Matsumoto 3d8a38bea4 You don't need to keep index in local variables info in irep. 2020-10-12 16:21:22 +09:00
Yukihiro "Matz" Matsumoto 639946a006 Enable method cache by default.
Introduced `MRB_NO_METHOD_CACHE` which is inverse of `MRB_METHOD_CACHE`
that should be enabled intestinally. In addition, the default cache is
made bigger (128 -> 256).
2020-10-12 16:21:22 +09:00
Yukihiro "Matz" Matsumoto 6db0162ec7 Upgrade RITE_VM_VERSION to 0300 (means mruby 3.0). 2020-10-12 16:21:20 +09:00
Yukihiro "Matz" Matsumoto dec34d6c7b Split MRB_BINARY_FORMAT to major and minor.
The minor versions should be upper compatible. So mere opcode, section
addition can be done without breaking compiled binary.
2020-10-12 16:21:20 +09:00
Yukihiro "Matz" Matsumoto 8f0ac27196 Update opcode reference and comment.
- no OP_EXT_ anymore
- OP_LOADI16 in right position
2020-10-12 16:21:19 +09:00
Yukihiro "Matz" Matsumoto fd10c72319 Remove OP_EXT[123] from operands. 2020-10-12 16:21:19 +09:00