* work on the ondemand iterators
* guarding two SIMDJSON_ASSUME
* simplify following @jkeiser's comment
* adding safety rails to the iterators
* silencing a warning.
* updating the amalgamation files
* improving the documentation of raw json access
* minor fix
* documentation update
* guarding for exceptions
* fixing exception issue
* fix test
* update.
* saving comments
* more technical fixes
* correcting path in ci test
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Make null-like value test consistent with docs
According to the docs: `INCORRECT_TYPE If the JSON value begins with 'n'
and is not 'null'.`
* Match document::is_null behavior to documentation
* Test other token beginning with n with is_null
* This documents the big-int feature, and adds a few tests.
* moving check_if_integer
* trimming the example.
* More trimming.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Allowing users to write directly to std::optional<std::string>
* better fallback
* do not force the cast to std::string
* missing header
* removing abort
* Rename simdjson_really_inline -> simdjson_inline
I want to change the simdjson_really_inline macro to sometimes not force
inlining. After that upcoming change, the name simdjson_really_inline
will no longer makes sense.
Rename simdjson_really_inline to simdjson_inline. This patch should not
change semantics; simdjson_inline still forces inlining as before.
Some functions still need to be really inlined for ABI reasons.
(GCC's -Wpsabi complains otherwise.) Leave those functions marked as
simdjson_really_inline.
* Improve build times for debug builds
simdjson_inline is used for most simdjson functions. It forces inlining.
In unoptimized/debug builds, this can lead to a lot of machine code
being generated (especially with Address Sanitizer), causing slow
compilation.
Change simdjson_inline to force inlining only for optimized builds.
Sometimes, the programmer might want a slightly-optimized build and want
fast compilation (e.g. GCC's -Og mode). Allow simdjson users to define
the simdjson_inline macro themselves (e.g. on the command line:
-Dsimdjson_inline=inline) in cases where the default behavior is
undesired.
This patch reduced build times by over 75% for ondemand_object_tests.cpp
with GCC 9.4.0 and CMAKE_BUILD_TYPE=Debug on my AMD 5950X:
Before: 6.885 6.683 6.971 6.957 6.949 seconds (5 samples)
After: 1.492 1.551 1.494 1.490 1.531 seconds (5 samples)
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
* Naive implementation for doubles in string.
* Add double from string in atom doc.
* Simplification (removed all *_from_string())
* Add int and uint parsing in string.
* Make duplicates instead.
* Make tests exceptionless.
* Add missing declarations.
* Add more tests (errors, JSON pointer).
* Add crypto json tests.
* Update doc.
* Update doc after review.
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
* Add document_stream constructors and iterate_many
* Attempt to implement streaming.
* Kind of fixed next() for getting next document
* Temporary save.
* Putting in working order.
* Add working doc_index and add function next_document()
* Attempt to implement streaming.
* Re-anchoring json_iterator after a call to stage 1
* I am convinced it should be a 'while'.
* Add source() with test.
* Add truncated_bytes().
* Fix casting issues.
* Fix old style cast.
* Fix privacy issue.
* Fix privacy issues.
* Again
* .
* Add more tests. Add error() for iterator class.
* Fix source() to not included whitespaces between documents.
* Fixing CI.
* Fix source() for multiple batches. Add new tests.
* Fix batch_start when document has leading spaces. Add new tests for that.
* Add new tests.
* Temporary save.
* Working hacky multithread version.
* Small fix in header files.
* Correct version (not working).
* Adding a move assignment to ondemand::parser.
* Fix attempt by changing std::swap.
* Moving DEFAULT_BATCH_SIZE and MINIMAL_BATCH_SIZE.
* Update doc and readme tests.
* Update basics.md
* Update readme_examples tests.
* Fix exceptions in test.
* Partial setup for amazon_cellphones.
* Benchmark with vectors.
* Benchmark with maps
* With vectors again.
* Fix for weighted average.
* DOM benchmark.
* Fix typos. Add On Demand benchmark.
* Add large amazon_cellphones benchmark for DOM
* Add benchmark for On demand.
* Fix broken read_me test.
* Add parser.threaded to enable/disable thread usage.
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Changing the name of the function to 'to_json_string' from 'to_string' to avoid confusion.
* Moving to a fast string_view model
* Making it exception-safe.
* Tweaking.
* Workaround for exceptions.
* more robust to_json_string (#1651)
* WIP.
* Fuzzing timeout (bug fix) (#1650)
* prove pull request #1648 introduces an infinite loop
* Interesting bug!
* Tweak.
Co-authored-by: Paul Dreik <github@pauldreik.se>
* It should now work.
* Moving car examples to exception mode
* Simplifying somewhat.
* I forgot to abandon. Let us do that.
* Adding more tests.
* WIP.
* It should now work.
* Moving car examples to exception mode
* Simplifying somewhat.
* I forgot to abandon. Let us do that.
* Adding more tests.
Co-authored-by: Paul Dreik <github@pauldreik.se>
Co-authored-by: Paul Dreik <github@pauldreik.se>
* Adding 'count_elements' method.
* Actually reporting errors.
* removing white space.
* Removing white space again.
* Adding an extra example.
* Prettier.
* Making the functionality more error-proof.
* Avoiding exceptions.
* Various fixes including extending count_elements to value types.
* Various fixes.
* Minor fixes.
* Correcting comment.
* Trimming white spaces.
When find_field_unordered is used on an empty object, it calls
json_iterator::reenter_child. reenter_child asserts that it doesn't
rewind too far back by consulting parser->start_positions.
When the On Demand parser sees an empty object, it fails to update
parser->start_positions. This means that the assertion in
json_iterator::reenter_child reads stale data, or potentially
uninitialized memory. Reading uninitialized memory can cause spurious
assertion failures and Valgrind memcheck reports:
Running missing_keys_for_empty_top_level_object ...
==170679== Conditional jump or move depends on uninitialised value(s)
==170679== at 0x4943D7: reenter_child (json_iterator-inl.h:208)
==170679== by 0x4943D7: find_field_unordered_raw (value_iterator-inl.h:197)
==170679== by 0x4943D7: find_field_unordered (object-inl.h:13)
==170679== by 0x4943D7: find_field_unordered (object-inl.h:96)
==170679== by 0x4943D7: find_field_unordered (value-inl.h:110)
==170679== by 0x4943D7: find_field_unordered (document-inl.h:105)
==170679== by 0x4943D7: object_tests::missing_keys_for_empty_top_level_object() (ondemand_object_tests.cpp:117)
==170679== by 0x4CA761: object_tests::run() (ondemand_object_tests.cpp:1085)
==170679== by 0x8BA314: int test_main<bool ()>(int, char**, bool ( const&)()) (test_ondemand.h:81)
==170679== by 0x4CA9C8: main (ondemand_object_tests.cpp:1119)
==170679==
Fix the read of uninitialized or stale memory by updating
parser->start_positions regardless of whether we see an empty object or
an object with some keys.
This commit only affects builds where development checks
(SIMDJSON_DEVELOPMENT_CHECKS) are enabled. Builds where development
checks are disabled are unaffected by this bug.
Daniel Lemire