* adding memory-file mapping to Windows.
* making memory-file mapping optional under windows, as it is fragile
* removing non-ascii
* saving.
* bumping up
* take 2
When parser.number_as_string(true) is set and a number exceeds uint64
range, write the raw digits to the string buffer and emit a BIGINT ('Z')
tape tag instead of returning BIGINT_ERROR. Default behavior unchanged.
Changes:
- tape_type.h: add BIGINT = 'Z'
- dom_parser_implementation.h: add _number_as_string flag
- parser.h: add number_as_string() getter/setter
- parser-inl.h: propagate flag before parse
- tape_builder.h: visit_number checks flag on BIGINT_ERROR
- tape_writer.h: add append_bigint helper
- serialization-inl.h: handle BIGINT in serialization (raw digits)
- tape.md: document big integer tape format
Builds on PR #2139 which added BIGINT_ERROR detection.
All 119 existing tests pass — default behavior unchanged.
Closes#167
Co-authored-by: harshagd <harshagd@amazon.com>
* marking a few trivial functions as pure
* adding other marks
* additional marks
* vs will issue warnings, so don't use [[gnu::pure]] when __clang__ or __GNUC__ is not defined
* Rename simdjson_really_inline -> simdjson_inline
I want to change the simdjson_really_inline macro to sometimes not force
inlining. After that upcoming change, the name simdjson_really_inline
will no longer makes sense.
Rename simdjson_really_inline to simdjson_inline. This patch should not
change semantics; simdjson_inline still forces inlining as before.
Some functions still need to be really inlined for ABI reasons.
(GCC's -Wpsabi complains otherwise.) Leave those functions marked as
simdjson_really_inline.
* Improve build times for debug builds
simdjson_inline is used for most simdjson functions. It forces inlining.
In unoptimized/debug builds, this can lead to a lot of machine code
being generated (especially with Address Sanitizer), causing slow
compilation.
Change simdjson_inline to force inlining only for optimized builds.
Sometimes, the programmer might want a slightly-optimized build and want
fast compilation (e.g. GCC's -Og mode). Allow simdjson users to define
the simdjson_inline macro themselves (e.g. on the command line:
-Dsimdjson_inline=inline) in cases where the default behavior is
undesired.
This patch reduced build times by over 75% for ondemand_object_tests.cpp
with GCC 9.4.0 and CMAKE_BUILD_TYPE=Debug on my AMD 5950X:
Before: 6.885 6.683 6.971 6.957 6.949 seconds (5 samples)
After: 1.492 1.551 1.494 1.490 1.531 seconds (5 samples)
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
* Add document_stream constructors and iterate_many
* Attempt to implement streaming.
* Kind of fixed next() for getting next document
* Temporary save.
* Putting in working order.
* Add working doc_index and add function next_document()
* Attempt to implement streaming.
* Re-anchoring json_iterator after a call to stage 1
* I am convinced it should be a 'while'.
* Add source() with test.
* Add truncated_bytes().
* Fix casting issues.
* Fix old style cast.
* Fix privacy issue.
* Fix privacy issues.
* Again
* .
* Add more tests. Add error() for iterator class.
* Fix source() to not included whitespaces between documents.
* Fixing CI.
* Fix source() for multiple batches. Add new tests.
* Fix batch_start when document has leading spaces. Add new tests for that.
* Add new tests.
* Temporary save.
* Working hacky multithread version.
* Small fix in header files.
* Correct version (not working).
* Adding a move assignment to ondemand::parser.
* Fix attempt by changing std::swap.
* Moving DEFAULT_BATCH_SIZE and MINIMAL_BATCH_SIZE.
* Update doc and readme tests.
* Update basics.md
* Update readme_examples tests.
* Fix exceptions in test.
* Partial setup for amazon_cellphones.
* Benchmark with vectors.
* Benchmark with maps
* With vectors again.
* Fix for weighted average.
* DOM benchmark.
* Fix typos. Add On Demand benchmark.
* Add large amazon_cellphones benchmark for DOM
* Add benchmark for On demand.
* Fix broken read_me test.
* Add parser.threaded to enable/disable thread usage.
Co-authored-by: Daniel Lemire <lemire@gmail.com>
Introduce cmake option SIMDJSON_DISABLE_DEPRECATED_API (default Off)
which turns off deprecated simdjson api functions by setting the macro
SIMDJSON_DISABLE_DEPRECATED_API.
For non-cmake users, users will have to set SIMDJSON_DISABLE_DEPRECATED_API
by some other means to disable the api.
Closes#1264
* Adding new files.
* Better.
* Fixing minifier and adding tests.
* Adding benchmarks.
* Including the array header.
* Replacing old stream-based code by the new code.
* Doubling up the itoa.
* Hidden away to_chars in internal namespace.
* Removing the repetitions.
* Documented the atoi functions.
* Tuning the escape sequences.
* Moving the operators off the main namespace.
* Added more tests.
* Tweaking the implementation so that it works with and without exp.
* The string_builder template and mini_formatter class
are not part of our public API and are subject to change
at any time!
* Adding a benchmark and some optimization.
* Cleaning.
* Strictly speaking, this header is needed.
* Adding test.
* Saving.
* With exceptions.
* Added extensive tests.
* Better documentation.
* Tweaking CI
* Cleaning.
* Do not assume make.
* Let us make the build verbose
* Reorg
* I do not understand how circle ci works.
* Breaking it up.
* Better syntax.
* This allows the users to disable threading.
* This would disable bash scripts under FreeBSD. (#1118)
* This would disable bash scripts under FreeBSD.
* Let us also disable GIT.
* Let us try to just disable GIT
* Nope. We must have both bash and git disabled.
* This allows the users to disable threading.
In the parse_many function, we have one thread doing the stage 1, while the main thread does stage 2. So if stage 1 and stage 2 take half the time, the parse_many could run at twice the speed. It is unlikely to do so. Still, we see benefits of about 40% due to threading.
To achieve this interleaving, we load the data in batches (blocks) of some size. In the current code (master), we create a new thread for each batch. Thread creation is expensive so our approach only works over sizeable batches. This PR improves things and makes parse_many faster when using small batches.
This fixes our parse_stream benchmark which is just busted.
This replaces the one-thread per batch routine by a worker object that reuses the same thread. In benchmarks, this allows us to get the same maximal speed, but with smaller processing blocks. It does not help much with larger blocks because the cost of the thread create gets amortized efficiently.
This PR makes parse_many beneficial over small datasets. It also makes us less dependent on the thread creation time.
Unfortunately, it is going to be difficult to say anything definitive in general. The cost of creating a thread varies widely depending on the OS. On some systems, it might be cheap, in others very expensive. It should be expected that the new code will depend less drastically on the performances of the underlying system, since we create juste one thread.
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Daniel Lemire <lemire@gmai.com>
Daniel Lemire