Files
mruby-mruby/mrblib/hash.rb
T
KOBAYASHI Shuji f2d8db39be Reduce memory usage of Hash object
## Implementation Summary

* Change entry list from segmented list to flat array.
* Change value of hash bucket from pointer to entry to index of entry list,
  and represent it by variable length bits according to capacity of hash
  buckets.
* Store management information about entry list and hash table to `struct
  RHash` as much as possible.

## Benchmark Summary

Only the results of typical situations on 64-bit Word-boxing are present
here. For more detailed information, including consideration, see below
(although most of the body is written in Japanese).

* https://shuujii.github.io/mruby-hash-benchmark

### Memory Usage

Lower value is better.

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |          344B |          256B |   0.74419x |
|        40 |        1,464B |          840B |   0.57377x |
|       200 |        8,056B |        3,784B |   0.46971x |
|       500 |       17,169B |        9,944B |   0.57949x |

### Performance

Higher value is better.

#### `mrb_hash_set`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  1.41847M i/s |  1.36004M i/s |   0.95881x |
|        40 |  0.39224M i/s |  0.31888M i/s |   0.81296x |
|       200 |  0.03780M i/s |  0.04290M i/s |   1.13494x |
|       500 |  0.01225M i/s |  0.01314M i/s |   1.07275x |

#### `mrb_hash_get`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 26.05920M i/s | 30.19543M i/s |   1.15872x |
|        40 | 44.26420M i/s | 32.75781M i/s |   0.74005x |
|       200 | 44.55171M i/s | 31.56926M i/s |   0.70860x |
|       500 | 39.19250M i/s | 29.73806M i/s |   0.75877x |

#### `mrb_hash_each`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 25.11964M i/s | 30.34167M i/s |   1.20789x |
|        40 | 11.74253M i/s | 13.25539M i/s |   1.12884x |
|       200 |  2.01133M i/s |  2.97214M i/s |   1.47770x |
|       500 |  0.87411M i/s |  1.21178M i/s |   1.38631x |

#### `Hash#[]=`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.50095M i/s |  0.56490M i/s |   1.12764x |
|        40 |  0.19132M i/s |  0.18392M i/s |   0.96129x |
|       200 |  0.03624M i/s |  0.03256M i/s |   0.89860x |
|       500 |  0.01527M i/s |  0.01236M i/s |   0.80935x |
#### `Hash#[]`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 11.53211M i/s | 12.78806M i/s |   1.10891x |
|        40 | 15.26920M i/s | 13.37529M i/s |   0.87596x |
|       200 | 15.28550M i/s | 13.36410M i/s |   0.87430x |
|       500 | 14.57695M i/s | 12.75388M i/s |   0.87494x |

#### `Hash#each`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.30462M i/s |  0.27080M i/s |   0.88898x |
|        40 |  0.12912M i/s |  0.11704M i/s |   0.90642x |
|       200 |  0.02638M i/s |  0.02402M i/s |   0.91071x |
|       500 |  0.01066M i/s |  0.00959M i/s |   0.89953x |

#### `Hash#delete`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  7.84167M i/s |  6.96419M i/s |   0.88810x |
|        40 |  6.91292M i/s |  7.41427M i/s |   1.07252x |
|       200 |  3.75952M i/s |  7.32080M i/s |   1.94727x |
|       500 |  2.10754M i/s |  7.05963M i/s |   3.34970x |

#### `Hash#shift`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 | 14.66444M i/s | 13.18876M i/s |   0.89937x |
|        40 | 11.95124M i/s | 11.10420M i/s |   0.92913x |
|       200 |  5.53681M i/s |  7.88155M i/s |   1.42348x |
|       500 |  2.96728M i/s |  5.40405M i/s |   1.82121x |

#### `Hash#dup`

| Hash Size |   Baseline    |      New      |   Factor   |
|----------:|--------------:|--------------:|-----------:|
|        16 |  0.15063M i/s |  5.37889M i/s |  35.71024x |
|        40 |  0.06515M i/s |  3.38196M i/s |  51.91279x |
|       200 |  0.01359M i/s |  1.46538M i/s | 107.84056x |
|       500 |  0.00559M i/s |  0.75411M i/s | 134.88057x |

### Binary Size

Lower value is better.

|    File    |   Baseline    |      New      |  Factor   |
|:-----------|--------------:|--------------:|----------:|
| mruby      |      730,408B |      734,176B |  1.00519x |
| libmruby.a |    1,068,134B |    1,072,846B |  1.00441x |

## Other Fixes

The following issues have also been fixed in the parts where there was some
change this time.

* [Heap use-after-free in `Hash#value?`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-hash-value-md)
* [Heap use-after-free in `ht_hash_equal`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_equal-md)
* [Heap use-after-free in `ht_hash_func`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-ht_hash_func-md)
* [Heap use-after-free in `mrb_hash_merge`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-heap-use-after-free-in-mrb_hash_merge-md)
* [Self-replacement does not work for `Hash#replace`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-self-replacement-does-not-work-for-hash-replace-md)
* [Repeated deletes and inserts increase memory usage of `Hash`](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-repeated-deletes-and-inserts-increase-memory-usage-of-hash-md)
* [`Hash#rehash` does not reindex completely](https://gist.github.com/shuujii/30e4fcd5844a4112a0ecd4a5b3483101#file-hash-rehash-does-not-reindex-completely-md)
* `mrb_hash_delete_key` does not cause an error for frozen object
* `mrb_hash_new_capa` does not allocate required space first
* [`mrb_os_memsize_of_hash_table` result is incorrect](https://github.com/mruby/mruby/pull/5032#discussion_r457994075)
2020-11-10 15:21:49 +09:00

308 lines
6.9 KiB
Ruby

##
# Hash
#
# ISO 15.2.13
class Hash
##
# Equality---Two hashes are equal if they each contain the same number
# of keys and if each key-value pair is equal to (according to
# <code>Object#==</code>) the corresponding elements in the other
# hash.
#
# ISO 15.2.13.4.1
def ==(hash)
return true if self.equal?(hash)
unless Hash === hash
return false
end
return false if self.size != hash.size
self.each do |k,v|
return false unless hash.key?(k)
return false unless self[k] == hash[k]
end
return true
end
##
# Returns <code>true</code> if <i>hash</i> and <i>other</i> are
# both hashes with the same content compared by eql?.
#
# ISO 15.2.13.4.32 (x)
def eql?(hash)
return true if self.equal?(hash)
unless Hash === hash
return false
end
return false if self.size != hash.size
self.each do |k,v|
return false unless hash.key?(k)
return false unless self[k].eql?(hash[k])
end
return true
end
##
# Delete the element with the key +key+.
# Return the value of the element if +key+
# was found. Return nil if nothing was
# found. If a block is given, call the
# block with the value of the element.
#
# ISO 15.2.13.4.8
def delete(key, &block)
if block && !self.has_key?(key)
return block.call(key)
end
self.__delete(key)
end
##
# Calls the given block for each element of +self+
# and pass the key and value of each element.
#
# call-seq:
# hsh.each {| key, value | block } -> hsh
# hsh.each_pair {| key, value | block } -> hsh
# hsh.each -> an_enumerator
# hsh.each_pair -> an_enumerator
#
#
# If no block is given, an enumerator is returned instead.
#
# h = { "a" => 100, "b" => 200 }
# h.each {|key, value| puts "#{key} is #{value}" }
#
# <em>produces:</em>
#
# a is 100
# b is 200
#
# ISO 15.2.13.4.9
def each(&block)
return to_enum :each unless block
keys = self.keys
vals = self.values
len = self.size
i = 0
while i < len
block.call [keys[i], vals[i]]
i += 1
end
self
end
##
# Calls the given block for each element of +self+
# and pass the key of each element.
#
# call-seq:
# hsh.each_key {| key | block } -> hsh
# hsh.each_key -> an_enumerator
#
# If no block is given, an enumerator is returned instead.
#
# h = { "a" => 100, "b" => 200 }
# h.each_key {|key| puts key }
#
# <em>produces:</em>
#
# a
# b
#
# ISO 15.2.13.4.10
def each_key(&block)
return to_enum :each_key unless block
self.keys.each{|k| block.call(k)}
self
end
##
# Calls the given block for each element of +self+
# and pass the value of each element.
#
# call-seq:
# hsh.each_value {| value | block } -> hsh
# hsh.each_value -> an_enumerator
#
# If no block is given, an enumerator is returned instead.
#
# h = { "a" => 100, "b" => 200 }
# h.each_value {|value| puts value }
#
# <em>produces:</em>
#
# 100
# 200
#
# ISO 15.2.13.4.11
def each_value(&block)
return to_enum :each_value unless block
self.values.each{|v| block.call(v)}
self
end
##
# Return a hash which contains the content of
# +self+ and +other+. If a block is given
# it will be called for each element with
# a duplicate key. The value of the block
# will be the final value of this element.
#
# ISO 15.2.13.4.22
def merge(other, &block)
raise TypeError, "Hash required (#{other.class} given)" unless Hash === other
h = self.dup
if block
other.each_key{|k|
h[k] = (self.has_key?(k))? block.call(k, self[k], other[k]): other[k]
}
else
other.each_key{|k| h[k] = other[k]}
end
h
end
# internal method for Hash inspection
def _inspect(recur_list)
return "{}" if self.size == 0
return "{...}" if recur_list[self.object_id]
recur_list[self.object_id] = true
ary=[]
keys=self.keys
vals=self.values
size=keys.size
i=0
while i<size
ary<<(keys[i]._inspect(recur_list) + "=>" + vals[i]._inspect(recur_list))
i+=1
end
"{"+ary.join(", ")+"}"
end
##
# Return the contents of this hash as a string.
#
# ISO 15.2.13.4.30 (x)
def inspect
self._inspect({})
end
# ISO 15.2.13.4.31 (x)
alias to_s inspect
##
# call-seq:
# hsh.reject! {| key, value | block } -> hsh or nil
# hsh.reject! -> an_enumerator
#
# Equivalent to <code>Hash#delete_if</code>, but returns
# <code>nil</code> if no changes were made.
#
# 1.8/1.9 Hash#reject! returns Hash; ISO says nothing.
#
def reject!(&block)
return to_enum :reject! unless block
keys = []
self.each{|k,v|
if block.call([k, v])
keys.push(k)
end
}
return nil if keys.size == 0
keys.each{|k|
self.delete(k)
}
self
end
##
# call-seq:
# hsh.reject {|key, value| block} -> a_hash
# hsh.reject -> an_enumerator
#
# Returns a new hash consisting of entries for which the block returns false.
#
# If no block is given, an enumerator is returned instead.
#
# h = { "a" => 100, "b" => 200, "c" => 300 }
# h.reject {|k,v| k < "b"} #=> {"b" => 200, "c" => 300}
# h.reject {|k,v| v > 100} #=> {"a" => 100}
#
# 1.8/1.9 Hash#reject returns Hash; ISO says nothing.
#
def reject(&block)
return to_enum :reject unless block
h = {}
self.each{|k,v|
unless block.call([k, v])
h[k] = v
end
}
h
end
##
# call-seq:
# hsh.select! {| key, value | block } -> hsh or nil
# hsh.select! -> an_enumerator
#
# Equivalent to <code>Hash#keep_if</code>, but returns
# <code>nil</code> if no changes were made.
#
# 1.9 Hash#select! returns Hash; ISO says nothing.
#
def select!(&block)
return to_enum :select! unless block
keys = []
self.each{|k,v|
unless block.call([k, v])
keys.push(k)
end
}
return nil if keys.size == 0
keys.each{|k|
self.delete(k)
}
self
end
##
# call-seq:
# hsh.select {|key, value| block} -> a_hash
# hsh.select -> an_enumerator
#
# Returns a new hash consisting of entries for which the block returns true.
#
# If no block is given, an enumerator is returned instead.
#
# h = { "a" => 100, "b" => 200, "c" => 300 }
# h.select {|k,v| k > "a"} #=> {"b" => 200, "c" => 300}
# h.select {|k,v| v < 200} #=> {"a" => 100}
#
# 1.9 Hash#select returns Hash; ISO says nothing
#
def select(&block)
return to_enum :select unless block
h = {}
self.each{|k,v|
if block.call([k, v])
h[k] = v
end
}
h
end
end
##
# Hash is enumerable
#
# ISO 15.2.13.3
class Hash
include Enumerable
end