* Initial work on JSON builder
* moving the files back to ondemand for now.
* tweak
* more later
* update
* minor edits
* dropping vs arm (missing support)
* adding tests. we still specialized write_string_escaped
* tweaking
* fix typo
* tweaking the approach
* minor fix
* missing store
* another missing store
* Attempt at fixing failing serialization tests. (#2292)
* Fixing appeand_float typo (#2294)
* applying a couple of fixes
* updating single header
* fix for pre C++17 if constexpr
* Fixing unused argument problem and updating the singleheader file
* various pedantic fixes
* Sketch of builder
* reordering.
* simplify
* Adding draft of static reflection based deserialization
* Updating simdjson singleheader
* patching the automated deserialization.
* automated
* Adding support for smart pointers of user defined types.
* Adding specialization for smart pointers for basic types. I think it is highly likely that this can be done in a more generic way.
* Referncing a later version of rapidjson that fixed the issue related with assignment attempt of a const variable for GenericStringRef class.
* guarding the tests
* adding documentation for string_builder
* saving
* rename to 'append'
* saving
* non-functional benchmarks (#2342)
* non-functional benchmarks
* Fix typo
* various fixes
* tweaking
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
* tuning
* various minor fixes
* minor tweak
* minor simplification
* updating amal
* adding a cast
* update
* fancy casting
* removing dead code
* Pushing latest changes. CITM benchmark is still not working.
* Still not working, but now I am getting only 10 errors.
* add static reflection benchmark to 'large random' benchmark and allows (#2349)
deserialization (with static reflection) from objects and arrays.
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Removing std::map from CitmCatalog definition, since that is not currently supported.
* Added free to rust bench, segfault is still happening..
* The syntax changed: ^E became ^^E. (#2350)
* The syntax changed: ^E became ^^E.
* guarding
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Adding support for string_view_keyed_map types.
* Adding concepts as a conditional include.
* updating single-header
* Adding concepts to ondemand deps
* rust benchmark is finally working
* Fixing small typo in docs.
* adding docker config and instructions so that our users can test the static reflection (#2358)
* adding docker config and instructions so that our users can test the
static reflection
* completing the instructions
* pruning white spaces
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* minor optimizations on the JSON builder branch
* avoiding undef behaviour
* saving
* somewhat nicer builder
* make it possible to run just one benchmark
* adding linux perf
* fixing minor issue
* updating swar
* Adding real world compilation benchmark (#2379)
* Adding compilation benchmark for json parsing with and without reflection
* Moving it to the benchmark folder, also reducing a bit the number of iterations.
* Removing script from root folder.
* Reducing number of iterations
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Making the script more customizable and also test whether the compiler being used supports reflection before actually running the benchmark
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Using define_static_string from https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p3491r2.html (#2389)
* Applying changes needed after latest reflection paper updates.
* Working, but no template for yet.
* Updating single-header to incldue the use of define_static_string.
* copying over master
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
* tag_invoke based custom types (#2219)
* tag_invoke based custom types
Now you can use tag_invoke to add a custom type or a group of custom types.
* Fixing macro usage + Fixing noexcept
* Fixing the usage of #include
We don't need <concepts> at all seems like it
* Fixing tag_invoke impl for MSVC
* Making `tag_invoke` to support `ondemand::document` as well + docs (#2228)
* Making `tag_invoke` to support `ondemand::document` as well + docs
* Fix typos and doc update by @lemire
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Better docs by @lemire
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Preserving the old, disallowing in the new
I'm disabling `document::get() &&` if the user has provided a `tag_invoke`d version; otherwise, we retain the compatibility.
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* fix: correct small issues with deserialize (#2232)
* Extending the deserialization code with more defaults + docs (#2233)
* Make custom types easier with some predefined cases + docs
* missing include
* adding Ubuntu 24 CXX 20
* using concepts all the way
* minor tweak
* tiny tweak
* tweaks
* more tweaking
* saving
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Making `tag_invoke` a "put" as opposed to a "get" (#2256)
* fix: add tests related to issue 2227 (#2229)
* fix: add tests related to issue 2227
* avoiding name clash
* pedantic fix
* deprecate rvalue get on document
* selectively deprecating
* Fix ndjson spec link (#2234)
* fix ndjson spec link
The link in the readme of parse_many links to a casino spam site
* fix link
* [no-ci] Update README.md
* Make simdjson compile again
* Enable SIMDJSON_SINGLEHEADER=OFF in VS Code
With singleheader on, clangd can't find the right
include files.
* Add missing include directives to static build targets of simdjson. (#2240)
* adding a warning
* adding warning regarding SIMDJSON_BUILD_STATIC_LIB
* release candidate
* pedantic viable size
* Making tag_invoke a feeder instead of a producer
* adding missing undef silencer (#2253)
* Ignore pragma once when amalgamating source files (#2248)
With gcc it causes an error in `simdjson.cpp`:
```
simdjson.cpp:548:9: warning: #pragma once in main file
548 | #pragma once
| ^~~~
```
It had previously been commented out in:
https://github.com/simdjson/simdjson/commit/6ef555e6fb79363fae057a9a46b52cd208d9e305
However, this was lost in an upgrade:
https://github.com/simdjson/simdjson/commit/2a4ff7346813b120f2b5b40e95d69352b593cc9c
* Update CI (#2254)
* adding missing undef silencer
* Updating CI
* more fixes
* fix
* big endian fix
* Moving to the new tag_invoke signature
* Fix nlohmann ambiguity on C++23-enabled clang
* Revert "Merge branch 'master' of https://github.com/simdjson/simdjson into builder_development_branch_extra"
This reverts commit 3eeecbab34, reversing
changes made to 6858b208b4.
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>
* update CI on the builder_development_branch (no code change) (#2262)
* typo
* General madness simpler, no simpler!!! (#2267)
* Minimal tag_invokes for STL types
* simpler madness
* adding a comment
* missing file
* minor tweaks to style
* fixing incorrect max/min usage
* updating single
* simplify
* validating the idea
* putting back the concept
* moving the include
* guarding
* Cheap General Madness (#2268)
* Some General Concepts and their deserializations
* Resolving ambiguity
* Add missing #include
* C++20 custom deserializer: better documentation (#2269)
* mostly a documentation update.
* missing cpp
* [no-ci] fix comment
* various minor fixes
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
---------
Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: M. Bahoosh <moisrex@gmail.com>
* minor update
* More documentation regarding builder (#2270)
* minor update
* more improvment to our documentation (builder branch)
* putting back missing functions
* merge candidate
---------
Co-authored-by: M. Bahoosh <moisrex@gmail.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>
Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
* Documenting our support of JSON Path and JSON Pointer vs. Unicode characters
* using more standard terminology (nitpicking)
* specifying UTF-8 encoding
* allow char8_t when compiling as C++20
* casting
* minor doc corrections
Related to #1904
Users of the simdjson library will see json2msgpack as an available
example of how to recursively process json with the ondemand parser,
and checking for trailing tokens in a document is one part of json validation.
These checks shouldn't affect benchmark results performance.
The benchmark is run on the 631KB twitter.json file.
Load 2 bytes and compare the 2 bytes against `"\u"`
Compilers with optimizations turned on will turn this into a 16-bit load
then 16-bit compare on supported platforms
(with smaller compiled code size).
Make it obvious to the compiler that it's reading two
consecutive bytes of the same pointer
Add parse_surrogate_pairs to show the difference exists.
See discussion in #1896
* Rename simdjson_really_inline -> simdjson_inline
I want to change the simdjson_really_inline macro to sometimes not force
inlining. After that upcoming change, the name simdjson_really_inline
will no longer makes sense.
Rename simdjson_really_inline to simdjson_inline. This patch should not
change semantics; simdjson_inline still forces inlining as before.
Some functions still need to be really inlined for ABI reasons.
(GCC's -Wpsabi complains otherwise.) Leave those functions marked as
simdjson_really_inline.
* Improve build times for debug builds
simdjson_inline is used for most simdjson functions. It forces inlining.
In unoptimized/debug builds, this can lead to a lot of machine code
being generated (especially with Address Sanitizer), causing slow
compilation.
Change simdjson_inline to force inlining only for optimized builds.
Sometimes, the programmer might want a slightly-optimized build and want
fast compilation (e.g. GCC's -Og mode). Allow simdjson users to define
the simdjson_inline macro themselves (e.g. on the command line:
-Dsimdjson_inline=inline) in cases where the default behavior is
undesired.
This patch reduced build times by over 75% for ondemand_object_tests.cpp
with GCC 9.4.0 and CMAKE_BUILD_TYPE=Debug on my AMD 5950X:
Before: 6.885 6.683 6.971 6.957 6.949 seconds (5 samples)
After: 1.492 1.551 1.494 1.490 1.531 seconds (5 samples)
* Patch for possible AVX-512 overflow.
* Updating the test for new padding.
* Preparing new version.
* replace binary integer literals with hex literals for C++11 compatibility (#1855)
Binary integer literals are a C++14 feature, so those are not supported
in C++11 and should be replaced by hexadecimal literals instead.
Fixes#1854.
Co-authored-by: Dirk Stolle <striezel-dev@web.de>
* Let us time minify
* Making AVX-512 available by default.
* Silencing some maybe-uninitialized warning under GCC (warning appears in the standard library).
* Making the Python amalgamation script a bit more Windows friendly.
* We do not try to silence -Wmaybe-uninitialized under clang.
Including <iostream> has two problems:
* Compile times are worse because of over-inclusion
* Binary sizes are worse when statically linking libstdc++ because
iostreams cannot be dead-code-stripped
simdjson only needs std::ostream. Include the header declaring only what
we need (<ostream>), omitting stuff we don't need (std::cout and its
initialization, for example).
This commit should not change behavior, but it might break users who
assume that including <simdjson/simdjson.h> will make std::cout
available (such as many of simdjson's own files).
* Adding test.
* Verifies and fix issue 1668. This commit updates the previous behavior of the
On Demand stream support by return a value type (document_reference) instead
of a reference to a document. This allows us to bridge with the usually simdjson
error system, with its simdjson_result types.
* Minor reformat.
* Adds a test with initial tests passing.
* Adding an example.
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
* Add document_stream constructors and iterate_many
* Attempt to implement streaming.
* Kind of fixed next() for getting next document
* Temporary save.
* Putting in working order.
* Add working doc_index and add function next_document()
* Attempt to implement streaming.
* Re-anchoring json_iterator after a call to stage 1
* I am convinced it should be a 'while'.
* Add source() with test.
* Add truncated_bytes().
* Fix casting issues.
* Fix old style cast.
* Fix privacy issue.
* Fix privacy issues.
* Again
* .
* Add more tests. Add error() for iterator class.
* Fix source() to not included whitespaces between documents.
* Fixing CI.
* Fix source() for multiple batches. Add new tests.
* Fix batch_start when document has leading spaces. Add new tests for that.
* Add new tests.
* Temporary save.
* Working hacky multithread version.
* Small fix in header files.
* Correct version (not working).
* Adding a move assignment to ondemand::parser.
* Fix attempt by changing std::swap.
* Moving DEFAULT_BATCH_SIZE and MINIMAL_BATCH_SIZE.
* Update doc and readme tests.
* Update basics.md
* Update readme_examples tests.
* Fix exceptions in test.
* Partial setup for amazon_cellphones.
* Benchmark with vectors.
* Benchmark with maps
* With vectors again.
* Fix for weighted average.
* DOM benchmark.
* Fix typos. Add On Demand benchmark.
* Add large amazon_cellphones benchmark for DOM
* Add benchmark for On demand.
* Fix broken read_me test.
* Add parser.threaded to enable/disable thread usage.
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Add first working version of rapidjson_sax for partial tweets.
* Add cleaner and faster rapidjson_sax
* Add nlohmann_json_sax.
* Replace array of bool by bitsets.
* Replace strdup to copy string in rapidjson_sax.
* Change std::string_view assignment in rapidjson_sax.
* Add rapidjson_sax.h and fix typo in rapidjson.h
* Add nlohmann_json_sax.h and add user key check for screen_name in rapidjson_sax
* Change std::string_view assignement for text and screen_name.
* Add rapidjson_sax.h .
* Add nlohmann_json_sax.h . Fix typos distinct_user_id/nlohmann_json_sax.h, find_tweet/rapidjson.h and find_tweet/rapidjson_sax.h .
* Add extra check for id key when looking for find_id.
* This moves all DOM (benchmark + test) files to a subdir
* Missing file.
* CMake + DLL is not pretty.
* Capitalizing AND
* Fixing mismatch endif
* Flipping the order.
* onedemand => ondemand
* Remove CMP0025 policy
This policy is already set to NEW by the minimum required version.
* Use HOMEPAGE_URL in the project call
* Use VERSION in the project call
* Detect if this is the top project
* Port simdjson-user-cmakecache to a CMake script
* Create a developer mode
The SIMDJSON_DEVELOPER_MODE option set to ON will enable targets that
are only useful for developers of simdjson.
* Consolidate root CML commands into logical sections
* Warn about intended use of developer mode
* Prettify the just_ascii test
* Remove redundant CMake variables
* Inline CML contents from include and src
* Raise minimum CMake requirement to 3.14
* Define proper install rules
* Restore thread support variable
* Add BUILD_SHARED_LIBS as a top level only option
* Force developer mode to be on in CI
* Include flags earlier in developer mode
* Set CMAKE_BUILD_TYPE conditionally
CMAKE_BUILD_TYPE is used only by single configuration generators and is
otherwise completely ignored.
* Remove useless static/shared options
simdjson now uses the CMake builtin BUILD_SHARED_LIBS to switch the
built artifact's type.
* Remove unused CMAKE_MODULE_PATH variable
* Refactor implementation switching into a module
* Factor exception option out into a module
* Reformat simdjson-flags.cmake
* Rename simdjson-flags to developer-options
* Accumulate properties into an include module
This is done this way to avoid using utility targets that must be
exported and installed, which could potentially be misused by users of
the library.
* Port impl definitions to props
* Port exception options to props
* Lift normal options to the top
* Port developer options to props
* Remove simdjson-flags from benchmark
* Document the developer mode in HACKING
* Fix include path in installed config file
* Fix formatting of prop commands
* Fix tests that include .cpp files
* Change GCC AVX fixes back to compile options
* Deprecate SIMDJSON_BUILD_STATIC
* Always link fuzz targets to simdjson
* Install CMake from simdjson's debian repo
* Add gnupg for apt-key
* Make sure ASan link flags come first
* Pass CI env variable to cmake invocation
* Install package for apt-add-repository
* Remove return() from flush macro
* Use directory level commands instead of props
* Restore the github repository variable
* Set developer mode unconditionally for checkperf
The CI env variable is only set in the CI and this target is always run
in developer mode.
* Attempt to fix ODR violation in parsing checks
These tests were compiling the simdjson.cpp file again and linking to
the simdjson library target causes ODR violations.
Instead of linking to the target, just inherit its props.
* Move variables before the source dir
* Mark props to be flushed after adding more
* Use props for every command for the library
* Use keyword form for linking libs
* Handle deprecation of SIMDJSON_JUST_LIBRARY
* Handle deprecations in a separate module
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Truncate final unclosed string.
* Adding more precise remarks.
* Better documentation and more robust code.
* ARM + PPC corrections.
* Patching ARM implementation with new stage1_mode parameter.
* Fixed most problems.
* Correcting white spaces and adding a remark.
* This adds the truncated_bytes() method to the stream instances.
Daniel Lemire