The split managed one ordering constraint: arena_choose() had to be
defined before arena_choose_maybe_huge() but after the tsd/tcache
inlines it depends on. After the malloc_dispatch refactor moved the
heaviest tcache-pulling inlines out of arena_inlines_b.h, the
remaining arena-side inlines all belong together. The merged
arena_inlines.h explicitly includes jemalloc_internal_inlines_a.h
and tcache.h (previously transitively pulled).
Fold historical *_types/_structs/_externs/_inlines splits where the
layering is no longer load-bearing.
- large_externs.h -> large.h: rename; it was a single-purpose
function-prototype file.
- background_thread_structs.h + background_thread_externs.h ->
background_thread.h: merge. background_thread_inlines.h stays
separate because it depends on arena_inlines_a.h.
- bin_types.h -> bin.h: BIN_SHARDS_MAX / N_BIN_SHARDS_DEFAULT join the
bin_t struct and the bin_dalloc_locked_info_t type. bin_inlines.h
stays separate so TUs that only need the bin_t type don't pull in
the tcache-flush inline closure. The two bin_stats_* helpers move
from bin.h into bin_inlines.h so all bin inlines live together.
- tsd_binshards.h (new): houses tsd_binshards_t and its zero
initializer. Lets tsd_internals.h pull it in for X-macro expansion
without dragging mutex.h -- mutex.h depends on TSD machinery, which
would form a cycle through bin.h.
jemalloc_internal_includes.h drops the references to the deleted/
merged headers.
During boot, some mutexes are not initialized yet, plus there's no point taking
many mutexes while everything is covered by the global init lock, so the locking
assumptions in some functions (e.g. background_thread_enabled_set()) can't be
enforced. Skip the lock owner check in this case.
Header files are now self-contained, which makes the relationships
between the files clearer, and crucially allows LSP tools like `clangd`
to function correctly in all of our header files. I have verified that
the headers are self-contained (aside from the various Windows shims) by
compiling them as if they were C files – in a follow-up commit I plan to
add this to CI to ensure we don't regress on this front.
Previously the calculation of sleep time between wakeups was implemented within
background_thread. This resulted in some parts of decay and hpa specific
logic mixing with background thread implementation. In this change, background
thread delegates this calculation to arena and it, in turn, delegates it to PAI.
The next step is to implement the actual calculation of time until deferred work
in HPA.
Passing is_background_thread down the decay path, so that background thread
itself won't attempt inactivity_check. This fixes an issue with background
thread doing trylock on a mutex it already owns.
To avoid background threads sleeping forever with idle arenas, we eagerly check
background threads' sleep time after extents_dalloc, and signal the thread if
necessary.
Added opt.background_thread to enable background threads, which handles purging
currently. When enabled, decay ticks will not trigger purging (which will be
left to the background threads). We limit the max number of threads to NCPUs.
When percpu arena is enabled, set CPU affinity for the background threads as
well.
The sleep interval of background threads is dynamic and determined by computing
number of pages to purge in the future (based on backlog).
Slobodan Predolac