Files
mruby-mruby/include/mruby/hash.h
T
KOBAYASHI Shuji f2d8db39be Reduce memory usage of Hash object
## Implementation Summary

* Change entry list from segmented list to flat array.
* Change value of hash bucket from pointer to entry to index of entry list,
  and represent it by variable length bits according to capacity of hash
  buckets.
* Store management information about entry list and hash table to `struct
  RHash` as much as possible.

## Benchmark Summary

Only the results of typical situations on 64-bit Word-boxing are present
here. For more detailed information, including consideration, see below
(although most of the body is written in Japanese).

* https://shuujii.github.io/mruby-hash-benchmark

### Memory Usage

Lower value is better.

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |          344B |          256B |   0.74419x |
|        40 |        1,464B |          840B |   0.57377x |
|       200 |        8,056B |        3,784B |   0.46971x |
|       500 |       17,169B |        9,944B |   0.57949x |

### Performance

Higher value is better.

#### `mrb_hash_set`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  1.41847M i/s |  1.36004M i/s |   0.95881x |
|        40 |  0.39224M i/s |  0.31888M i/s |   0.81296x |
|       200 |  0.03780M i/s |  0.04290M i/s |   1.13494x |
|       500 |  0.01225M i/s |  0.01314M i/s |   1.07275x |

#### `mrb_hash_get`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 26.05920M i/s | 30.19543M i/s |   1.15872x |
|        40 | 44.26420M i/s | 32.75781M i/s |   0.74005x |
|       200 | 44.55171M i/s | 31.56926M i/s |   0.70860x |
|       500 | 39.19250M i/s | 29.73806M i/s |   0.75877x |

#### `mrb_hash_each`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 25.11964M i/s | 30.34167M i/s |   1.20789x |
|        40 | 11.74253M i/s | 13.25539M i/s |   1.12884x |
|       200 |  2.01133M i/s |  2.97214M i/s |   1.47770x |
|       500 |  0.87411M i/s |  1.21178M i/s |   1.38631x |

#### `Hash#[]=`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.50095M i/s |  0.56490M i/s |   1.12764x |
|        40 |  0.19132M i/s |  0.18392M i/s |   0.96129x |
|       200 |  0.03624M i/s |  0.03256M i/s |   0.89860x |
|       500 |  0.01527M i/s |  0.01236M i/s |   0.80935x |
#### `Hash#[]`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 11.53211M i/s | 12.78806M i/s |   1.10891x |
|        40 | 15.26920M i/s | 13.37529M i/s |   0.87596x |
|       200 | 15.28550M i/s | 13.36410M i/s |   0.87430x |
|       500 | 14.57695M i/s | 12.75388M i/s |   0.87494x |

#### `Hash#each`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.30462M i/s |  0.27080M i/s |   0.88898x |
|        40 |  0.12912M i/s |  0.11704M i/s |   0.90642x |
|       200 |  0.02638M i/s |  0.02402M i/s |   0.91071x |
|       500 |  0.01066M i/s |  0.00959M i/s |   0.89953x |

#### `Hash#delete`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  7.84167M i/s |  6.96419M i/s |   0.88810x |
|        40 |  6.91292M i/s |  7.41427M i/s |   1.07252x |
|       200 |  3.75952M i/s |  7.32080M i/s |   1.94727x |
|       500 |  2.10754M i/s |  7.05963M i/s |   3.34970x |

#### `Hash#shift`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 14.66444M i/s | 13.18876M i/s |   0.89937x |
|        40 | 11.95124M i/s | 11.10420M i/s |   0.92913x |
|       200 |  5.53681M i/s |  7.88155M i/s |   1.42348x |
|       500 |  2.96728M i/s |  5.40405M i/s |   1.82121x |

#### `Hash#dup`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.15063M i/s |  5.37889M i/s |  35.71024x |
|        40 |  0.06515M i/s |  3.38196M i/s |  51.91279x |
|       200 |  0.01359M i/s |  1.46538M i/s | 107.84056x |
|       500 |  0.00559M i/s |  0.75411M i/s | 134.88057x |

### Binary Size

Lower value is better.

|    File    |   Baseline    |      New      |  Factor   |
|:-----------|--------------:|--------------:|----------:|
| mruby      |      730,408B |      734,176B |  1.00519x |
| libmruby.a |    1,068,134B |    1,072,846B |  1.00441x |

## Other Fixes

The following issues have also been fixed in the parts where there was some
change this time.

* [Heap use-after-free in `Hash#value?`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-hash-value-md)
* [Heap use-after-free in `ht_hash_equal`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_equal-md)
* [Heap use-after-free in `ht_hash_func`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_func-md)
* [Heap use-after-free in `mrb_hash_merge`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-mrb_hash_merge-md)
* [Self-replacement does not work for `Hash#replace`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-self-replacement-does-not-work-for-hash-replace-md)
* [Repeated deletes and inserts increase memory usage of `Hash`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-repeated-deletes-and-inserts-increase-memory-usage-of-hash-md)
* [`Hash#rehash` does not reindex completely](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-hash-rehash-does-not-reindex-completely-md)
* `mrb_hash_delete_key` does not cause an error for frozen object
* `mrb_hash_new_capa` does not allocate required space first
* [`mrb_os_memsize_of_hash_table` result is incorrect](https://github.com/mruby/mruby/pull/5032#discussion_r457994075)
2020-11-10 15:21:49 +09:00

245 lines
6.2 KiB
C

/**
** @file mruby/hash.h - Hash class
**
** See Copyright Notice in mruby.h
*/
#ifndef MRUBY_HASH_H
#define MRUBY_HASH_H
#include "common.h"
/**
* Hash class
*/
MRB_BEGIN_DECL
/* offset of `iv` must be 3 words */
struct RHash {
MRB_OBJECT_HEADER;
#ifdef MRB_64BIT
uint32_t size;
struct iv_tbl *iv;
uint32_t ea_capa;
uint32_t ea_n_used;
#else
struct iv_tbl *iv;
uint32_t size;
#endif
union {
struct hash_entry *ea;
struct hash_table *ht;
};
};
#define mrb_hash_ptr(v) ((struct RHash*)(mrb_ptr(v)))
#define mrb_hash_value(p) mrb_obj_value((void*)(p))
size_t mrb_os_memsize_of_hash_table(mrb_value obj);
MRB_API mrb_value mrb_hash_new_capa(mrb_state *mrb, mrb_int capa);
MRB_API mrb_value mrb_ensure_hash_type(mrb_state *mrb, mrb_value hash);
MRB_API mrb_value mrb_check_hash_type(mrb_state *mrb, mrb_value hash);
/*
* Initializes a new hash.
*
* Equivalent to:
*
* Hash.new
*
* @param mrb The mruby state reference.
* @return The initialized hash.
*/
MRB_API mrb_value mrb_hash_new(mrb_state *mrb);
/*
* Sets a keys and values to hashes.
*
* Equivalent to:
*
* hash[key] = val
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @param key The key to set.
* @param val The value to set.
* @return The value.
*/
MRB_API void mrb_hash_set(mrb_state *mrb, mrb_value hash, mrb_value key, mrb_value val);
/*
* Gets a value from a key. If the key is not found, the default of the
* hash is used.
*
* Equivalent to:
*
* hash[key]
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @param key The key to get.
* @return The found value.
*/
MRB_API mrb_value mrb_hash_get(mrb_state *mrb, mrb_value hash, mrb_value key);
/*
* Gets a value from a key. If the key is not found, the default parameter is
* used.
*
* Equivalent to:
*
* hash.key?(key) ? hash[key] : def
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @param key The key to get.
* @param def The default value.
* @return The found value.
*/
MRB_API mrb_value mrb_hash_fetch(mrb_state *mrb, mrb_value hash, mrb_value key, mrb_value def);
/*
* Deletes hash key and value pair.
*
* Equivalent to:
*
* hash.delete(key)
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @param key The key to delete.
* @return The deleted value. This value is not protected from GC. Use `mrb_gc_protect()` if necessary.
*/
MRB_API mrb_value mrb_hash_delete_key(mrb_state *mrb, mrb_value hash, mrb_value key);
/*
* Gets an array of keys.
*
* Equivalent to:
*
* hash.keys
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @return An array with the keys of the hash.
*/
MRB_API mrb_value mrb_hash_keys(mrb_state *mrb, mrb_value hash);
/*
* Check if the hash has the key.
*
* Equivalent to:
*
* hash.key?(key)
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @param key The key to check existence.
* @return True if the hash has the key
*/
MRB_API mrb_bool mrb_hash_key_p(mrb_state *mrb, mrb_value hash, mrb_value key);
/*
* Check if the hash is empty
*
* Equivalent to:
*
* hash.empty?
*
* @param mrb The mruby state reference.
* @param self The target hash.
* @return True if the hash is empty, false otherwise.
*/
MRB_API mrb_bool mrb_hash_empty_p(mrb_state *mrb, mrb_value self);
/*
* Gets an array of values.
*
* Equivalent to:
*
* hash.values
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @return An array with the values of the hash.
*/
MRB_API mrb_value mrb_hash_values(mrb_state *mrb, mrb_value hash);
/*
* Clears the hash.
*
* Equivalent to:
*
* hash.clear
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @return The hash
*/
MRB_API mrb_value mrb_hash_clear(mrb_state *mrb, mrb_value hash);
/*
* Get hash size.
*
* Equivalent to:
*
* hash.size
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @return The hash size.
*/
MRB_API mrb_int mrb_hash_size(mrb_state *mrb, mrb_value hash);
/*
* Copies the hash. This function does NOT copy the instance variables
* (except for the default value). Use mrb_obj_dup() to copy the instance
* variables as well.
*
* @param mrb The mruby state reference.
* @param hash The target hash.
* @return The copy of the hash
*/
MRB_API mrb_value mrb_hash_dup(mrb_state *mrb, mrb_value hash);
/*
* Merges two hashes. The first hash will be modified by the
* second hash.
*
* @param mrb The mruby state reference.
* @param hash1 The target hash.
* @param hash2 Updating hash
*/
MRB_API void mrb_hash_merge(mrb_state *mrb, mrb_value hash1, mrb_value hash2);
#define RHASH(hash) ((struct RHash*)(mrb_ptr(hash)))
#define RHASH_IFNONE(hash) mrb_iv_get(mrb, (hash), MRB_SYM(ifnone))
#define RHASH_PROCDEFAULT(hash) RHASH_IFNONE(hash)
#define MRB_HASH_IB_BIT_BIT 5
#define MRB_HASH_AR_EA_CAPA_BIT 5
#define MRB_HASH_IB_BIT_SHIFT 0
#define MRB_HASH_AR_EA_CAPA_SHIFT 0
#define MRB_HASH_AR_EA_N_USED_SHIFT MRB_HASH_AR_EA_CAPA_BIT
#define MRB_HASH_SIZE_FLAGS_SHIFT (MRB_HASH_AR_EA_CAPA_BIT * 2)
#define MRB_HASH_IB_BIT_MASK ((1 << MRB_HASH_IB_BIT_BIT) - 1)
#define MRB_HASH_AR_EA_CAPA_MASK ((1 << MRB_HASH_AR_EA_CAPA_BIT) - 1)
#define MRB_HASH_AR_EA_N_USED_MASK (MRB_HASH_AR_EA_CAPA_MASK << MRB_HASH_AR_EA_N_USED_SHIFT)
#define MRB_HASH_DEFAULT (1 << (MRB_HASH_SIZE_FLAGS_SHIFT + 0))
#define MRB_HASH_PROC_DEFAULT (1 << (MRB_HASH_SIZE_FLAGS_SHIFT + 1))
#define MRB_HASH_HT (1 << (MRB_HASH_SIZE_FLAGS_SHIFT + 2))
#define MRB_RHASH_DEFAULT_P(hash) (RHASH(hash)->flags & MRB_HASH_DEFAULT)
#define MRB_RHASH_PROCDEFAULT_P(hash) (RHASH(hash)->flags & MRB_HASH_PROC_DEFAULT)
/* GC functions */
void mrb_gc_mark_hash(mrb_state*, struct RHash*);
size_t mrb_gc_mark_hash_size(mrb_state*, struct RHash*);
void mrb_gc_free_hash(mrb_state*, struct RHash*);
/* return non zero to break the loop */
typedef int (mrb_hash_foreach_func)(mrb_state *mrb, mrb_value key, mrb_value val, void *data);
MRB_API void mrb_hash_foreach(mrb_state *mrb, struct RHash *hash, mrb_hash_foreach_func *func, void *p);
MRB_END_DECL
#endif /* MRUBY_HASH_H */