* adding memory-file mapping to Windows.
* making memory-file mapping optional under windows, as it is fragile
* removing non-ascii
* saving.
* bumping up
* take 2
* work on the ondemand iterators
* guarding two SIMDJSON_ASSUME
* simplify following @jkeiser's comment
* adding safety rails to the iterators
* silencing a warning.
* updating the amalgamation files
* marking a few trivial functions as pure
* adding other marks
* additional marks
* vs will issue warnings, so don't use [[gnu::pure]] when __clang__ or __GNUC__ is not defined
Under compilers like MSVC the `#pragma message` about portability will be issued for each translation module; regardless of whether or not `SIMDJSON_PORTABILITY_H` has already been defined. As such a new define `SIMDJSON_NO_PORTABILITY_WARNING`, can be defined prior to the inclusion of `<simdjson.h>` to silence it.
* Initial PPC64 support
* Add travis CI
* Fix outdated cmake version for travis
* Fix indendtation
* Try another workaround for outdated cmake in travis
* Try beta cmake
* Add dash before beta
* Use builtin snaps
* Use cmake as rocksdb
* Test cmake on bionic
* Remove unnecessary things from travis
* Remove unnecessary things from travis
* Another try of compiler install
* Add all major compilers
* Add all major compilers
* Add all major compilers
* Tweak travis a bit
* Typo
* More robust travis
* Typos typos typos
* Add fewer compilers, add non specific build for clang and gcc, should be the final config
* CMAKE_FLAGS is in incorrect place
* Remove default implementation
* Limit build thread number
* Fall back prefix_xor to a usual implementation, no performance boost is noticed
* Test for power9 as it is the main architecture for OpenPOWER right now
* Add to documentation to build with power9 as the implementation is compatible but compiler optimizations is not
* Replace ARM with PPC in the comment
- Allow user to specify SIMDJSON_BUILTIN_IMPLEMENTATION
- Make cmake -DSIMDJSON_IMPLEMENTATION=haswell *only* specify haswell
- Move negative implementation selection to
-DSIMDJSON_EXCLUDE_IMPLEMENTATION
- Automatically select SIMDJSON_BUILTIN_IMPLEMENTATION if
SIMDJSON_IMPLEMENTATION is set
- Move implementation enablement mostly to implementation files
- Make implementation enablement and selection simpler and more robust
- Fix bug where programs linked against simdjson were not passed
SIMDJSON_XXX_IMPLEMENTATION or SIMDJSON_EXCEPTIONS
In the parse_many function, we have one thread doing the stage 1, while the main thread does stage 2. So if stage 1 and stage 2 take half the time, the parse_many could run at twice the speed. It is unlikely to do so. Still, we see benefits of about 40% due to threading.
To achieve this interleaving, we load the data in batches (blocks) of some size. In the current code (master), we create a new thread for each batch. Thread creation is expensive so our approach only works over sizeable batches. This PR improves things and makes parse_many faster when using small batches.
This fixes our parse_stream benchmark which is just busted.
This replaces the one-thread per batch routine by a worker object that reuses the same thread. In benchmarks, this allows us to get the same maximal speed, but with smaller processing blocks. It does not help much with larger blocks because the cost of the thread create gets amortized efficiently.
This PR makes parse_many beneficial over small datasets. It also makes us less dependent on the thread creation time.
Unfortunately, it is going to be difficult to say anything definitive in general. The cost of creating a thread varies widely depending on the OS. On some systems, it might be cheap, in others very expensive. It should be expected that the new code will depend less drastically on the performances of the underlying system, since we create juste one thread.
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Daniel Lemire <lemire@gmai.com>
Daniel Lemire