f2d8db39be
Reduce memory usage of Hash object
...
## Implementation Summary
* Change entry list from segmented list to flat array.
* Change value of hash bucket from pointer to entry to index of entry list,
and represent it by variable length bits according to capacity of hash
buckets.
* Store management information about entry list and hash table to `struct
RHash` as much as possible.
## Benchmark Summary
Only the results of typical situations on 64-bit Word-boxing are present
here. For more detailed information, including consideration, see below
(although most of the body is written in Japanese).
* https://shuujii.github.io/mruby-hash-benchmark
### Memory Usage
Lower value is better.
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 344B | 256B | 0.74419x |
| 40 | 1,464B | 840B | 0.57377x |
| 200 | 8,056B | 3,784B | 0.46971x |
| 500 | 17,169B | 9,944B | 0.57949x |
### Performance
Higher value is better.
#### `mrb_hash_set`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 1.41847M i/s | 1.36004M i/s | 0.95881x |
| 40 | 0.39224M i/s | 0.31888M i/s | 0.81296x |
| 200 | 0.03780M i/s | 0.04290M i/s | 1.13494x |
| 500 | 0.01225M i/s | 0.01314M i/s | 1.07275x |
#### `mrb_hash_get`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 26.05920M i/s | 30.19543M i/s | 1.15872x |
| 40 | 44.26420M i/s | 32.75781M i/s | 0.74005x |
| 200 | 44.55171M i/s | 31.56926M i/s | 0.70860x |
| 500 | 39.19250M i/s | 29.73806M i/s | 0.75877x |
#### `mrb_hash_each`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 25.11964M i/s | 30.34167M i/s | 1.20789x |
| 40 | 11.74253M i/s | 13.25539M i/s | 1.12884x |
| 200 | 2.01133M i/s | 2.97214M i/s | 1.47770x |
| 500 | 0.87411M i/s | 1.21178M i/s | 1.38631x |
#### `Hash#[]=`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 0.50095M i/s | 0.56490M i/s | 1.12764x |
| 40 | 0.19132M i/s | 0.18392M i/s | 0.96129x |
| 200 | 0.03624M i/s | 0.03256M i/s | 0.89860x |
| 500 | 0.01527M i/s | 0.01236M i/s | 0.80935x |
#### `Hash#[]`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 11.53211M i/s | 12.78806M i/s | 1.10891x |
| 40 | 15.26920M i/s | 13.37529M i/s | 0.87596x |
| 200 | 15.28550M i/s | 13.36410M i/s | 0.87430x |
| 500 | 14.57695M i/s | 12.75388M i/s | 0.87494x |
#### `Hash#each`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 0.30462M i/s | 0.27080M i/s | 0.88898x |
| 40 | 0.12912M i/s | 0.11704M i/s | 0.90642x |
| 200 | 0.02638M i/s | 0.02402M i/s | 0.91071x |
| 500 | 0.01066M i/s | 0.00959M i/s | 0.89953x |
#### `Hash#delete`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 7.84167M i/s | 6.96419M i/s | 0.88810x |
| 40 | 6.91292M i/s | 7.41427M i/s | 1.07252x |
| 200 | 3.75952M i/s | 7.32080M i/s | 1.94727x |
| 500 | 2.10754M i/s | 7.05963M i/s | 3.34970x |
#### `Hash#shift`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 14.66444M i/s | 13.18876M i/s | 0.89937x |
| 40 | 11.95124M i/s | 11.10420M i/s | 0.92913x |
| 200 | 5.53681M i/s | 7.88155M i/s | 1.42348x |
| 500 | 2.96728M i/s | 5.40405M i/s | 1.82121x |
#### `Hash#dup`
| Hash Size | Baseline | New | Factor |
|----------:|--------------:|--------------:|-----------:|
| 16 | 0.15063M i/s | 5.37889M i/s | 35.71024x |
| 40 | 0.06515M i/s | 3.38196M i/s | 51.91279x |
| 200 | 0.01359M i/s | 1.46538M i/s | 107.84056x |
| 500 | 0.00559M i/s | 0.75411M i/s | 134.88057x |
### Binary Size
Lower value is better.
| File | Baseline | New | Factor |
|:-----------|--------------:|--------------:|----------:|
| mruby | 730,408B | 734,176B | 1.00519x |
| libmruby.a | 1,068,134B | 1,072,846B | 1.00441x |
## Other Fixes
The following issues have also been fixed in the parts where there was some
change this time.
* [Heap use-after-free in `Hash#value?`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-hash-value-md )
* [Heap use-after-free in `ht_hash_equal`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_equal-md )
* [Heap use-after-free in `ht_hash_func`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_func-md )
* [Heap use-after-free in `mrb_hash_merge`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-mrb_hash_merge-md )
* [Self-replacement does not work for `Hash#replace`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-self-replacement-does-not-work-for-hash-replace-md )
* [Repeated deletes and inserts increase memory usage of `Hash`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-repeated-deletes-and-inserts-increase-memory-usage-of-hash-md )
* [`Hash#rehash` does not reindex completely](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-hash-rehash-does-not-reindex-completely-md )
* `mrb_hash_delete_key` does not cause an error for frozen object
* `mrb_hash_new_capa` does not allocate required space first
* [`mrb_os_memsize_of_hash_table` result is incorrect](https://github.com/mruby/mruby/pull/5032#discussion_r457994075 )
2020-11-10 15:21:49 +09:00
8455344bf3
Skip too big left shift in flo_shift().
2020-11-06 12:52:18 +09:00
2a52ebcc56
Avoid negating MRB_INT_MIN which is impossible.
2020-11-06 12:51:44 +09:00
d93fc6593f
Fix wrong integer casting.
2020-11-06 12:42:43 +09:00
f77113e841
Fix a bug with printing (null) local variable name for a register.
2020-11-05 16:55:23 +09:00
d9bd77b180
Fix A.new([[1,2],3]).flatten to return Array.
2020-11-05 14:29:26 +09:00
4e535080dc
Use INT32_{MAX,MIN} to specify integer ranges.
2020-11-05 14:29:26 +09:00
afba74ff50
Add Array.new([1,2,3]) initialization.
2020-11-05 14:29:25 +09:00
aa4d8e9a2b
Avoid integer overflow in comparison.
2020-11-04 14:27:34 +09:00
0113db6716
Add cast to silence int conversion warnings.
2020-11-04 14:18:47 +09:00
b77a0f69a3
Fix a bug with big negative integers.
2020-11-04 14:15:51 +09:00
5d85045fcb
Update mruby3.md about new instructions.
2020-11-04 14:01:56 +09:00
d13df1536d
Add a new instruction OP_LOADI32.
...
That loads 32 bit integer bypassing pool access.
2020-11-04 14:01:04 +09:00
f8156ae175
Merge pull request #5113 from shuujii/reduce-recompilation-when-updating-presym-target-files
...
Reduce recompilation when updating presym target files
2020-11-03 18:13:56 +09:00
b08da9c30b
Merge pull request #5112 from shuujii/fix-dependencies-of-y.tab.o
...
Fix dependencies of `y.tab.o`
2020-11-03 18:12:50 +09:00
963b44286a
Use C99 __func__ instead of __FUNCTION__; #5107
2020-11-03 18:03:23 +09:00
82a2a5657c
Reduce recompilation when updating presym target files
...
Fix that updating presym target files would recompile almost all files even
if symbols did not change.
2020-11-03 16:03:46 +09:00
ea5d0a593c
Fix dependencies of y.tab.o
...
In the case of `y.tab.o`, `y.tab.d` was not read, so `y.tab.c` was not
recompiled by updating header files.
It seemed that the cause was overwriting the rake rule, so I deleted
unnecessary settings.
2020-11-03 15:56:08 +09:00
b7e8406f6c
Add new instructions to handle symbols/literals >255; fix #5109
...
New instructions:
* OP_LOADL16
* OP_LOADSYM16
* OP_STRING16
Size of pools, symbols are `int16_t` but offset representation in the
bytecode was 8 bits. Size of child `irep` array is `int16_t`, too but
this change does not address it.
2020-11-03 14:58:44 +09:00
0be5b266b6
format '%p' expects argument of type 'void *'; #5107
2020-11-02 23:04:12 +09:00
ac51a58683
ISO C does not allow extra ';' outside of a function; #5107
2020-11-02 23:04:12 +09:00
436765eb94
Revert "Remove long-time deprecated minirake."
...
`minirake` was still used in many occasions.
This reverts commit 3847c7eb64
2020-11-02 15:14:36 +09:00
ddfac9aba0
Search gem from MRUBY_ROOT if build config is in build_config.
2020-11-02 15:14:36 +09:00
9233d28ac7
Make Ranges frozen as Ruby3.0.
2020-11-02 15:14:36 +09:00
dc886873bc
Replace R-assignment by single-line pattern matching.
...
Since `R-assignment` in CRuby is abandoned. Single-line pattern matching
in `mruby` only matches single local variable at the moment. Currently
it works as a right assignment to a local variable. It will be enhanced
in the future.
2020-11-02 15:14:36 +09:00
70ee936367
Update the build description in mruby3.md.
2020-11-02 15:14:35 +09:00
c427776910
Merge pull request #5111 from shuujii/fix-parallel-build
...
Fix parallel build
2020-11-02 15:14:25 +09:00
9a5c78dc26
Fix parallel build
...
When `rake -m` and so on are used to build in parallel, building may be
started before presym files are generated. Then, for example, the following
error occurs and this issue is fixed.
```console
In file included from /Users/shuujii/mruby/mruby/include/mruby.h:92:
/mruby/mruby/include/mruby/presym.h:16:10: fatal error: '../build/presym.inc' file not found
#include <../build/presym.inc>
^~~~~~~~~~~~~~~~~~~~~
```
2020-11-02 14:13:33 +09:00
331b119313
Merge pull request #5110 from bshastry/fix-mruby-ossfuzz-build
...
ossfuzz: Fix mruby include paths
2020-11-01 22:19:39 +09:00
b5e4e283e8
ossfuzz: Fix mruby include paths
2020-11-01 13:24:03 +01:00
f000856d68
Merge pull request #5108 from mimaki/display_preview_version
...
Add `preview` to version.
2020-11-01 17:19:01 +09:00
943029954f
Fix mruby -v option test.
2020-11-01 14:19:19 +09:00
bc9566eb64
Add preview to version.
2020-11-01 13:42:35 +09:00
6cc29ed377
Fix the condition to detect core mrbgems.
2020-10-30 07:15:11 +09:00
e4d4192c3a
Exclude symbols from mruby-test gem only used for tests.
2020-10-29 23:16:54 +09:00
a5d05ea8b7
Use mrb dump format for non core mrbgems by default.
...
If you confirm the gem is cdump safe, add `spec.enable_cdump` in
`mrbgem.rake` file. Some external gems e.g. `mruby-mgem-template` do
not work with cdump.
2020-10-29 22:52:29 +09:00
86981dafb1
Use clear if instead of unless for condition.
2020-10-29 22:52:29 +09:00
5dff2194a2
Use instance variable @dir instead of access method dir.
2020-10-29 22:52:29 +09:00
78f7162508
Remove mrblib_dir and objs_dir configuration from gems.
...
Always use `mrblib` and 'src` for directory names.
2020-10-29 22:52:28 +09:00
ccca5e521c
Merge pull request #5105 from shuujii/fix-typo-in-Range-last
...
Fix typo in `Range#last`
2020-10-29 22:52:17 +09:00
212800a901
Merge pull request #5104 from shuujii/fix-typo-in-doc-guides-compile.md
...
Fix typo in `doc/guides/compile.md` [ci skip]
2020-10-29 22:51:43 +09:00
353ecd7119
Fix typo in Range#last
2020-10-29 18:40:42 +09:00
0736d7c99f
Fix typo in doc/guides/compile.md [ci skip]
2020-10-29 18:27:11 +09:00
385cebe0b9
Use `default' gembox by default.
...
I changed to use `full-core` gembox for debugging. It should have been
reverted before `mruby3` branch was merged.
2020-10-29 11:53:10 +09:00
3847c7eb64
Remove long-time deprecated minirake.
2020-10-29 11:49:05 +09:00
24ffa4386a
The path of the gem should be relative from MRUBY_ROOT.
...
It was relative from the directory of the build configuration file, but
the document says it's relative from `MRUBY_ROOT`. When the default
configuration file was `MRUBY_ROOT/build_config.rb`, it was OK for most
of the cases.
2020-10-29 11:49:05 +09:00
33742d941f
Move build_config.rb -> build_config/default.rb; ref #5098
...
In addition, update the documents referring `build_config.rb` which is
no longer used. The new `build_config.rb` describes the new configuration
structure in the comment.
2020-10-29 11:49:05 +09:00
c1ad4c4261
Restore original name for default build_config_name; ref #5098
2020-10-29 10:46:20 +09:00
41d5df03ed
Use Dir.glob instead of map.
2020-10-29 10:43:09 +09:00
32ffd81011
Add beginless/endless ranges tests.
2020-10-29 10:43:09 +09:00
KOBAYASHI Shuji