Under compilers like MSVC the `#pragma message` about portability will be issued for each translation module; regardless of whether or not `SIMDJSON_PORTABILITY_H` has already been defined. As such a new define `SIMDJSON_NO_PORTABILITY_WARNING`, can be defined prior to the inclusion of `<simdjson.h>` to silence it.
* Initial PPC64 support
* Add travis CI
* Fix outdated cmake version for travis
* Fix indendtation
* Try another workaround for outdated cmake in travis
* Try beta cmake
* Add dash before beta
* Use builtin snaps
* Use cmake as rocksdb
* Test cmake on bionic
* Remove unnecessary things from travis
* Remove unnecessary things from travis
* Another try of compiler install
* Add all major compilers
* Add all major compilers
* Add all major compilers
* Tweak travis a bit
* Typo
* More robust travis
* Typos typos typos
* Add fewer compilers, add non specific build for clang and gcc, should be the final config
* CMAKE_FLAGS is in incorrect place
* Remove default implementation
* Limit build thread number
* Fall back prefix_xor to a usual implementation, no performance boost is noticed
* Test for power9 as it is the main architecture for OpenPOWER right now
* Add to documentation to build with power9 as the implementation is compatible but compiler optimizations is not
* Replace ARM with PPC in the comment
- Allow user to specify SIMDJSON_BUILTIN_IMPLEMENTATION
- Make cmake -DSIMDJSON_IMPLEMENTATION=haswell *only* specify haswell
- Move negative implementation selection to
-DSIMDJSON_EXCLUDE_IMPLEMENTATION
- Automatically select SIMDJSON_BUILTIN_IMPLEMENTATION if
SIMDJSON_IMPLEMENTATION is set
- Move implementation enablement mostly to implementation files
- Make implementation enablement and selection simpler and more robust
- Fix bug where programs linked against simdjson were not passed
SIMDJSON_XXX_IMPLEMENTATION or SIMDJSON_EXCEPTIONS
In the parse_many function, we have one thread doing the stage 1, while the main thread does stage 2. So if stage 1 and stage 2 take half the time, the parse_many could run at twice the speed. It is unlikely to do so. Still, we see benefits of about 40% due to threading.
To achieve this interleaving, we load the data in batches (blocks) of some size. In the current code (master), we create a new thread for each batch. Thread creation is expensive so our approach only works over sizeable batches. This PR improves things and makes parse_many faster when using small batches.
This fixes our parse_stream benchmark which is just busted.
This replaces the one-thread per batch routine by a worker object that reuses the same thread. In benchmarks, this allows us to get the same maximal speed, but with smaller processing blocks. It does not help much with larger blocks because the cost of the thread create gets amortized efficiently.
This PR makes parse_many beneficial over small datasets. It also makes us less dependent on the thread creation time.
Unfortunately, it is going to be difficult to say anything definitive in general. The cost of creating a thread varies widely depending on the OS. On some systems, it might be cheap, in others very expensive. It should be expected that the new code will depend less drastically on the performances of the underlying system, since we create juste one thread.
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Daniel Lemire <lemire@gmai.com>
* JsonStream threaded prototype
* JsonStream Threaded version working. Still supporting non-threaded version.
* Fix where invalid files would enter infinite loop.
* SingleHeader update
* I will remove -pthread in cmake for now.
* Attempt at resolving the -pthread issue
* Attempt 1 - fn targeting
GCC won't work with templates with different targets, need to specialize all the way up the call stack.
* Compiles properly with cmake. Does not with the Makefile.
* Compilation works with Makefile
* instruction_set changes to architecture
* some aesthetic changes
* fix amalgation and tests + aesthetic changes
* This now compiles and passes tests under CLANG
* Minor correction.
* Trying to make it work on ARM
* Adding missing namespace
* Missing bracket
* Fixing minor compilation issues.
* Getting parse to use runtime dispatch
* Fixing amalgamation script.
* Making sure that NEON is supported.
* Fixing typo
* Merging https://github.com/lemire/simdjson/pull/229
* Manual merge of
https://github.com/lemire/simdjson/pull/229
by @jkeiser (second part)
* Trying another way.
* Removing the paral.
* Fixing the make file
* Let us make the practice run long enough.
* Resolved the awful slowness.
* Cleaning the README.md
* With runtime dispatching, we should not need flags anymore.
* Changing isa detection file's name + fixing typos.
* Improving portability.
* Revisiting faulty logic regarding same-page overruns.
* Disabling same-page overruns under VS.
* Clarifying the documentation
* Fix for issue 131 + being more explicit regarding memory realloc.
* Fix for issue 137.
* removing "using namespace std" throughout. Fix for 50
* Introducing typed malloc/free.
* Introducing a custom class (padded_string) that solves several minor usability issues.
* Updating amalgamation for testing.
N. Kolotov