If you provide a custom serializer for range types it is currently never used due to the requires clause for string_builder::append with ranges is overly broad
* Adding compile time json path
* using string_view
* Adding support for compile-time json pointer as well.
* Removing unnecessary comment
* Tests now working, still will re-review.
* Adding documentation on the compile-time json path/pointer parsing feature.
* Adding benchmark showing the significant performance advantage of using compiled paths whenever you have them a priori.
* going for JSONPath (correct wording).
* minor update (mostly doc)
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Resolves a build failure on the loong64 architecture caused by a narrowing conversion error.
The compiler, with the -Werror=narrowing flag, was flagging the implicit conversion from 'int' (the return
type of to_bitmask()) to 'uint64_t'.
This is fixed by adding an explicit static_cast to uint64_t in include/simdjson/lsx/stringparsing_defs.h.
Signed-off-by: Zhou Qiankang <wszqkzqk@qq.com>
* Using iterators instead of subscript operators and size. This helps us work with a broader range of containers.
* Adding list test
* Using std::ranges::input_range<T> as suggested by moisrex
* modifying simdjson::from to avoid exceptions when needed.
* moved the function
* moving the strings.
* more moving around
* updating cmake version in ci
* This is a small reorg of the new convert code so that we only expose 'simdjson::from'.
This can be changed in a future release, but we don't want our users to start depending
on code that we might need to change.
* marking simdjson::from as experimental
* updating tests to match recent changes
* improving the documentation of raw json access
* minor fix
* documentation update
* guarding for exceptions
* fixing exception issue
* fix test
* update.
* saving comments
* more technical fixes
* correcting path in ci test
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* introducing a thread-local parser
* adding functionality to release the memory
* some more documentation.
* fixing build
* adding benchmarks for 'from'
* generalizing the code somewhat.
* adding tests, fixing the benchmark (now with arrays and streams), and a
minor update to document_stream
* adding missing files (I forgot to check them).
* We cannot use [[nodiscard]] without guarding it, it is C++17
* fixing the cmake
* marking it as experimental
* removing ranges support (it is too experimental)
* putting back documentation.
* guarding SIMDJSON_CONSTEVAL more carefully.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* clang format
* added `chars()` method
* implemented vector with small buffer instead of `std::vector`
* added missing <utility> header
* minor fixes
---------
Co-authored-by: Pavel Novikov <dev-ape@yandex.ru>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
The test was failing in CI with g++-13 because it only handled the
exception case. However, the array() method is marked noexcept and
returns a simdjson_result that may contain an error code instead of
throwing an exception.
This fix checks for both cases:
1. If array_result.error() is not SUCCESS, verify it's INCORRECT_TYPE
2. If no error is returned initially, the exception may be thrown when
iterating over the result
This ensures the test passes regardless of whether the error is
reported via error code or exception.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added (void)position; to suppress unused parameter warning when compiling with SIMDJSON_CLANG_VISUAL_STUDIO defined, where the position parameter isn't used in the SIMDJSON_ASSUME statements.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Resolved conflicts by regenerating the amalgamated single-header
files (simdjson.h, simdjson.cpp, and singleheader.zip) using the
amalgamate.py script after merging latest changes from master.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit addresses multiple issues:
1. Fixed -Werror=effc++ warnings by using #pragma to disable the
warning for constructors that cannot initialize all members in
the member initialization list due to error handling requirements.
2. Added proper error tracking (m_error member) to handle cases where
document initialization fails, preventing segfaults when using
invalid documents.
3. Fixed lifetime issues in tests where temporary auto_parser objects
were being used, causing dangling references. Tests now properly
store the parser object before using it.
4. Simplified range adaptor tests that were expecting features not
yet implemented in simdjson's ondemand API.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The issue was that we were trying to initialize ondemand::document
directly from simdjson_result<ondemand::document> in the member
initializer list. This caused a segfault in C++20 builds.
The fix explicitly handles the simdjson_result in the constructor
body, checking for errors and using value_unsafe() to extract the
document. This avoids potential issues with implicit conversions
and ensures proper error handling.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The compiler was warning about member initialization order mismatch.
C++ initializes members in the order they are declared in the class,
not the order they appear in the initializer list.
Fixed by reordering member declarations to match the initialization
order needed: m_doc must be initialized before m_parser since we
need to call parser.iterate() before moving the parser.
This fixes the -Werror=reorder compilation error in CI.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The issue was that we were calling m_parser.iterate() after moving
the parser, which could leave it in an invalid state. In C++20,
this might behave differently than C++17.
Fixed by reordering the member initializer list to call
parser.iterate() BEFORE moving the parser into m_parser.
This ensures the document is created while the parser is still valid.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The issue might be related to how brace initialization vs parentheses
initialization handles implicit conversion from simdjson_result<document>
to document. This could be compiler-specific behavior.
Using parentheses initialization to ensure the conversion operator
is called properly.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The issue was that the auto_parser constructor was using implicit
conversion from simdjson_result<document> to document, which could
cause issues with certain implementations (particularly fallback).
Changed to use value_unsafe() to explicitly extract the document
after the parser is fully initialized. This ensures the document
is in a valid state for subsequent operations.
This fixes the ondemand_convert_tests failure in CI with clang++-16.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove constexpr from functions that call non-constexpr methods
- The no_errors and to<T> adaptors were marked constexpr but call
simdjson_result methods that are not constexpr in C++20
- This was causing compilation failures in CI for C++20 builds
- Tests now compile and pass with both C++17 and C++20
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Instead of disabling the feature, provide C++20-compatible implementation
of the pipe operators for ranges support. The range_adaptor_closure is
C++23-only, so we implement our own pipe operators for C++20.
This preserves the core functionality of the PR while ensuring
compatibility across different compiler versions.
The test_no_errors() and to_clean_array() tests depend on the C++23
ranges features that we disabled. This commit conditionally compiles
these tests out when ranges support is disabled.
The ranges features were causing compatibility issues across different
compilers and platforms. Disabling them for now until C++23 support
is more widespread.
This should fix the remaining Ubuntu and Windows CI failures.
Add defined() check before comparing the value to avoid preprocessor
errors in compilers where this macro doesn't exist (like g++-13
with certain configurations).
- Fix deprecated reflect_value warning by using reflect_constant
- Fix std::const_iterator C++23 requirement by using auto_iterator
- Fix C++23 std::ranges::range_adaptor_closure availability check
- Add convert.h to main simdjson.h includes
These changes ensure compatibility across different C++ standards
and compiler versions, fixing the Ubuntu CI failures.
Co-Authored-By: Claude <noreply@anthropic.com>
* Adding type validation, enhancing optional type support and adding test a few more tests.
* Adding support for string-based enum serlalization and deserialization.
* Removing unintentional endline.
* Removing trailing whitespace.
* Adding simpler api as suggested by moisrex.
* Removing explicit optiona<int> and optional<std::string> references and using concepts instead! Credit goes to Lemire for pointing this out and suggesting a concepts based approach here.
* Removing tests that are not relevant for this branch.
* Removing api related changes. That will be done by moisrex.
* Removing unnecessary new endlines.
* Removing tests related to api changes and cleaning-up irrelevant tests.
* removing broken reference
* Removing trailing whitespace
* Initial work on JSON builder
* moving the files back to ondemand for now.
* tweak
* more later
* update
* minor edits
* dropping vs arm (missing support)
* adding tests. we still specialized write_string_escaped
* tweaking
* fix typo
* tweaking the approach
* minor fix
* missing store
* another missing store
* Attempt at fixing failing serialization tests. (#2292)
* Fixing appeand_float typo (#2294)
* applying a couple of fixes
* updating single header
* fix for pre C++17 if constexpr
* Fixing unused argument problem and updating the singleheader file
* various pedantic fixes
* Sketch of builder
* reordering.
* simplify
* Adding draft of static reflection based deserialization
* Updating simdjson singleheader
* patching the automated deserialization.
* automated
* Adding support for smart pointers of user defined types.
* Adding specialization for smart pointers for basic types. I think it is highly likely that this can be done in a more generic way.
* Referncing a later version of rapidjson that fixed the issue related with assignment attempt of a const variable for GenericStringRef class.
* guarding the tests
* adding documentation for string_builder
* saving
* rename to 'append'
* saving
* non-functional benchmarks (#2342)
* non-functional benchmarks
* Fix typo
* various fixes
* tweaking
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
* tuning
* various minor fixes
* minor tweak
* minor simplification
* updating amal
* adding a cast
* update
* fancy casting
* removing dead code
* Pushing latest changes. CITM benchmark is still not working.
* Still not working, but now I am getting only 10 errors.
* add static reflection benchmark to 'large random' benchmark and allows (#2349)
deserialization (with static reflection) from objects and arrays.
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Removing std::map from CitmCatalog definition, since that is not currently supported.
* Added free to rust bench, segfault is still happening..
* The syntax changed: ^E became ^^E. (#2350)
* The syntax changed: ^E became ^^E.
* guarding
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Adding support for string_view_keyed_map types.
* Adding concepts as a conditional include.
* updating single-header
* Adding concepts to ondemand deps
* rust benchmark is finally working
* Fixing small typo in docs.
* adding docker config and instructions so that our users can test the static reflection (#2358)
* adding docker config and instructions so that our users can test the
static reflection
* completing the instructions
* pruning white spaces
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* minor optimizations on the JSON builder branch
* avoiding undef behaviour
* saving
* somewhat nicer builder
* make it possible to run just one benchmark
* adding linux perf
* fixing minor issue
* updating swar
* Adding real world compilation benchmark (#2379)
* Adding compilation benchmark for json parsing with and without reflection
* Moving it to the benchmark folder, also reducing a bit the number of iterations.
* Removing script from root folder.
* Reducing number of iterations
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Making the script more customizable and also test whether the compiler being used supports reflection before actually running the benchmark
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Using define_static_string from https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p3491r2.html (#2389)
* Applying changes needed after latest reflection paper updates.
* Working, but no template for yet.
* Updating single-header to incldue the use of define_static_string.
* copying over master
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
* We run on practically all systems.
* Update doc/basics.md
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
---------
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* tag_invoke based custom types (#2219)
* tag_invoke based custom types
Now you can use tag_invoke to add a custom type or a group of custom types.
* Fixing macro usage + Fixing noexcept
* Fixing the usage of #include
We don't need <concepts> at all seems like it
* Fixing tag_invoke impl for MSVC
* Making `tag_invoke` to support `ondemand::document` as well + docs (#2228)
* Making `tag_invoke` to support `ondemand::document` as well + docs
* Fix typos and doc update by @lemire
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Better docs by @lemire
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Preserving the old, disallowing in the new
I'm disabling `document::get() &&` if the user has provided a `tag_invoke`d version; otherwise, we retain the compatibility.
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* fix: correct small issues with deserialize (#2232)
* Extending the deserialization code with more defaults + docs (#2233)
* Make custom types easier with some predefined cases + docs
* missing include
* adding Ubuntu 24 CXX 20
* using concepts all the way
* minor tweak
* tiny tweak
* tweaks
* more tweaking
* saving
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Making `tag_invoke` a "put" as opposed to a "get" (#2256)
* fix: add tests related to issue 2227 (#2229)
* fix: add tests related to issue 2227
* avoiding name clash
* pedantic fix
* deprecate rvalue get on document
* selectively deprecating
* Fix ndjson spec link (#2234)
* fix ndjson spec link
The link in the readme of parse_many links to a casino spam site
* fix link
* [no-ci] Update README.md
* Make simdjson compile again
* Enable SIMDJSON_SINGLEHEADER=OFF in VS Code
With singleheader on, clangd can't find the right
include files.
* Add missing include directives to static build targets of simdjson. (#2240)
* adding a warning
* adding warning regarding SIMDJSON_BUILD_STATIC_LIB
* release candidate
* pedantic viable size
* Making tag_invoke a feeder instead of a producer
* adding missing undef silencer (#2253)
* Ignore pragma once when amalgamating source files (#2248)
With gcc it causes an error in `simdjson.cpp`:
```
simdjson.cpp:548:9: warning: #pragma once in main file
548 | #pragma once
| ^~~~
```
It had previously been commented out in:
https://github.com/simdjson/simdjson/commit/6ef555e6fb79363fae057a9a46b52cd208d9e305
However, this was lost in an upgrade:
https://github.com/simdjson/simdjson/commit/2a4ff7346813b120f2b5b40e95d69352b593cc9c
* Update CI (#2254)
* adding missing undef silencer
* Updating CI
* more fixes
* fix
* big endian fix
* Moving to the new tag_invoke signature
* Fix nlohmann ambiguity on C++23-enabled clang
* Revert "Merge branch 'master' of https://github.com/simdjson/simdjson into builder_development_branch_extra"
This reverts commit 3eeecbab34, reversing
changes made to 6858b208b4.
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>
* update CI on the builder_development_branch (no code change) (#2262)
* typo
* General madness simpler, no simpler!!! (#2267)
* Minimal tag_invokes for STL types
* simpler madness
* adding a comment
* missing file
* minor tweaks to style
* fixing incorrect max/min usage
* updating single
* simplify
* validating the idea
* putting back the concept
* moving the include
* guarding
* Cheap General Madness (#2268)
* Some General Concepts and their deserializations
* Resolving ambiguity
* Add missing #include
* C++20 custom deserializer: better documentation (#2269)
* mostly a documentation update.
* missing cpp
* [no-ci] fix comment
* various minor fixes
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
---------
Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: M. Bahoosh <moisrex@gmail.com>
* minor update
* More documentation regarding builder (#2270)
* minor update
* more improvment to our documentation (builder branch)
* putting back missing functions
* merge candidate
---------
Co-authored-by: M. Bahoosh <moisrex@gmail.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>
Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
* Add support for JSONPath with '$' prefix and integrate into dom::at_path (#2266)
- Updated `json_path_to_pointer_conversion` to support JSONPath starting with the '$' prefix, while maintaining compatibility with the existing implementation.
- Moved `json_path_to_pointer_conversion` to a separate header file for better modularity and to support JSONPath queries in `dom` mode.
- Implemented `at_path` functionality in `dom` mode to enable querying JSON using JSONPath.
- Added unit tests to validate the new JSONPath support in `dom` mode and ensure compatibility with both standard and existing JSONPath formats.
* two minor fixes
* more tests and documentation
* damn compiler warnings
* more documentation fixes
---------
Co-authored-by: Zhengguo Yang <yangzhgg@gmail.com>
* Make null-like value test consistent with docs
According to the docs: `INCORRECT_TYPE If the JSON value begins with 'n'
and is not 'null'.`
* Match document::is_null behavior to documentation
* Test other token beginning with n with is_null
* This PR does the following:
1. Upgrade cxxopts.
2. Allows field::unescape_key to take in a string parameter (syntaxic sugar).
3. Adds a dev. check to detect a string buffer overflow (indicating broken code). Note that this is unrecoverable and indicates bad code.
* tweak
* marking a few trivial functions as pure
* adding other marks
* additional marks
* vs will issue warnings, so don't use [[gnu::pure]] when __clang__ or __GNUC__ is not defined
1. Allows processing inclomplete, damaged, corrupted json to some extent.
2. Pariity with the Presto Java functionality.
3. Protected with SIMDJSON_EXPERIMENTAL_ALLOW_INCOMPLETE_JSON define.
4. Does not interfere with the normal path (can co-exist).
5. Tested in production forkflow.
```
src/implementation.cpp:193:20: error: no template named 'is_trivially_destructible' in namespace 'std'; did you mean 'is_trivially_move_constructible'?
static_assert(std::is_trivially_destructible<detect_best_supported_implementation_on_first_use>::value, "detect_best_supported_implementation_on_first_use should be trivially destructible");
~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
is_trivially_move_constructible
```
* In some cases, clang might try to cast an ondemand::document to an
ondemand::document, instead of calling the move constructor. So we
can disable the template cast.
* In some cases, clang might get confused when constructing an ondemand
document in a constructor: instead of calling the move constructor, it
somehow ends up trying to cast a document to a document. We can easily
disallow this behavior with std::enable_if.
* making compatible with C++11
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* This documents the big-int feature, and adds a few tests.
* moving check_if_integer
* trimming the example.
* More trimming.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Documenting our support of JSON Path and JSON Pointer vs. Unicode characters
* using more standard terminology (nitpicking)
* specifying UTF-8 encoding
* allow char8_t when compiling as C++20
* casting
* minor doc corrections
* First implementation of the conversion logic.
- Not yet tested
- Not yet compiled
- Not yet used in ondemand array/object/etc...
* Adding at_path to ondemand arrays
* Adding at_path to array, document and value.
* Adding at_path to ondemand object as well.
Pending:
- Building
- Adding tests for each of the on-demand classes usage of at_path
- Properly documenting the subset of json path that is currently supported.
* Adding a simple json_path test to on_demand
* Still trying to compile the ondemand_readme_examples test with the at_path() call
* Fixing linking issues with array::at_path
* Fixing issues with the path -> pointer conversion and removing comments
* Adding a clone of json_pointer to test json_path extensively
* Adding ondemand_json_path_tests (all tests passing)
* Adding documentation for at_path()
* Removing some newlines
* Removing simdjson_result from the return type of the string conversion function. Now json_path_to_pointer_conversion returns a std::string.
* Fixing typo
* Making string concatenation explicit to avoid ubuntu gcc12 issue with -O3 flag.
* Addressing latest reviews
* Addressing latest reviews
* documenting the support for custom types
* fixes
* getting around -Weffc++
* calling get_raw_json_string()
* documenting the support for custom types
* fixes
* getting around -Weffc++
* calling get_raw_json_string()
* Allowing users to write directly to std::optional<std::string>
* better fallback
* do not force the cast to std::string
* missing header
* removing abort
* Standard compatibility fixes
* missing commit
* Should work.
* Fix.
* Fix.
* Should work now.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Add comma separated value parsing
* Fix failing tests
* Make tests work for exceptions
* Fix test
* Fix try catch making test fail
---------
Co-authored-by: Yong Xiang Ng <yxng@drwholdings.com>
* Adding CXX 20 to CI
* side-stepping new CXX 20 guard.
* Going another way
* Saving.
* Explicit.
* Saving...
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Add info and error logging
* Add tests for error logging
* Add logging for missing field
* Update docs for logging usage
* Fix style and pass by ref for string format args
* Make log_level explicit and simplify get log level from env
* Make log level int32_t
* Format enum class
* Fix ci
* Move enum to header
* Fix compilation for noexception build in test case
* Disable warnings and putenv
---------
Co-authored-by: Yong Xiang Ng <yxng@drwholdings.com>
* Adding support for AVX-512 on macOS.
* Fix.
* Fix.
* Minor fix
* Setting the variable to zero.
* Fixing include
* Checking if the OS supports AVX-512
* Tweaking.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
This fixes a bug that caused simdjsonTargets.cmake not to be included in
CPack-generated packages, which—unlike `cmake --install`—does not
pick up this mislabeled install component.
I git-grepped through the code base, after this change all components
are either `simdjson_Development` or `simdjson_Runtime`.
* This adds SIMDJSON_DEVELOPMENT_CHECKS to the DOM API to help users
in the scenario of issue 1914.
* More documentation and warnings.
* Updating following comments by Tyson
Related to #1904
Users of the simdjson library will see json2msgpack as an available
example of how to recursively process json with the ondemand parser,
and checking for trailing tokens in a document is one part of json validation.
These checks shouldn't affect benchmark results performance.
The benchmark is run on the 631KB twitter.json file.
Make it less likely to accidentally introduce tabs, trailing whitespace,
carriage returns, non-utf8 in files, or files without trailing newlines.
https://editorconfig.org/ has plugins for various editors/IDEs and is
enabled by default in some IDEs.
* Fixing issue 1898 Preserve sign for number with underflowing exponent (#1900)
Before this commit, simdjson parsed "-1e-999" and "-0e-999" and "-1e-342"
as 0.0.
After this commit, those JSON strings get parsed as -0.0.
(https://en.wikipedia.org/wiki/Signed_zero)
The old behavior was inconsistent with the way simdjson parsed "-0.0" as -0.0.
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Co-authored-by: Tyson Andre <tysonandre775@hotmail.com>
Load 2 bytes and compare the 2 bytes against `"\u"`
Compilers with optimizations turned on will turn this into a 16-bit load
then 16-bit compare on supported platforms
(with smaller compiled code size).
Make it obvious to the compiler that it's reading two
consecutive bytes of the same pointer
Add parse_surrogate_pairs to show the difference exists.
See discussion in #1896
Closes#1894
Reject low surrogates outside of the range U+DC00—U+DFFF
Related to https://unicodebook.readthedocs.io/unicode_encodings.html#utf-16-surrogate-pairs
A surrogate pair should consist of a high surrogate and low surrogate.
They're used to represent 0x010000-0x10FFFF in the JSON spec because
the JavaScript specification originally only supported `\uXXXX`.
Previously, simdjson would accept some combinations of valid high
surrogates and invalid low surrogates due to a bug in the check.
(e.g. `\uD888\u1234` was accepted)
U+D800—U+DBFF (1,024 code points): high surrogates
U+DC00—U+DFFF (1,024 code points): low surrogates
* build: add pkg-config support
The CMake build script now generates a simple pkg-config files that can
be easily used by non-CMake users.
The file is generated from a template file that gets filled in at
configure time.
As CMake doesn't have anything similar to Meson's pkg-config generator
the file is quite static, i.e. new simdjson public defines/dependencies
won't be picked up automatically.
This approach also suffers from one minor issue, mentioned in
[jtojnar/cmake-snips][]; in short, it doesn't work well when users
specify CMAKE_INSTALL_INCLUDEDIR and similar as absolute paths. It's not
a big deal, and it will easily fixable once you'll require CMake >=3.20.
Fixes#1763
[jtojnar/cmake-snips]: https://github.com/jtojnar/cmake-snips#concatenating-paths-when-building-pkg-config-files
* build: handle absolute paths in .pc generation
As mentioned in the previous commit message, correct concatenation of
paths is only available in CMake >=3.20, so handling absolute paths in
pkg-config file generation requires using jtojnar's JoinPaths module.
* ci: add debian job
This new jobs compiles simdjson on Debian Testing, a semi-rolling
release, so that new compilers are always tested.
This job also tests the pkg-config file introduced in commit
1096c3b299
* Rename simdjson_really_inline -> simdjson_inline
I want to change the simdjson_really_inline macro to sometimes not force
inlining. After that upcoming change, the name simdjson_really_inline
will no longer makes sense.
Rename simdjson_really_inline to simdjson_inline. This patch should not
change semantics; simdjson_inline still forces inlining as before.
Some functions still need to be really inlined for ABI reasons.
(GCC's -Wpsabi complains otherwise.) Leave those functions marked as
simdjson_really_inline.
* Improve build times for debug builds
simdjson_inline is used for most simdjson functions. It forces inlining.
In unoptimized/debug builds, this can lead to a lot of machine code
being generated (especially with Address Sanitizer), causing slow
compilation.
Change simdjson_inline to force inlining only for optimized builds.
Sometimes, the programmer might want a slightly-optimized build and want
fast compilation (e.g. GCC's -Og mode). Allow simdjson users to define
the simdjson_inline macro themselves (e.g. on the command line:
-Dsimdjson_inline=inline) in cases where the default behavior is
undesired.
This patch reduced build times by over 75% for ondemand_object_tests.cpp
with GCC 9.4.0 and CMAKE_BUILD_TYPE=Debug on my AMD 5950X:
Before: 6.885 6.683 6.971 6.957 6.949 seconds (5 samples)
After: 1.492 1.551 1.494 1.490 1.531 seconds (5 samples)
* Patch for possible AVX-512 overflow.
* Updating the test for new padding.
* Preparing new version.
* replace binary integer literals with hex literals for C++11 compatibility (#1855)
Binary integer literals are a C++14 feature, so those are not supported
in C++11 and should be replaced by hexadecimal literals instead.
Fixes#1854.
Co-authored-by: Dirk Stolle <striezel-dev@web.de>
* Let us time minify
* Making AVX-512 available by default.
* Silencing some maybe-uninitialized warning under GCC (warning appears in the standard library).
* Making the Python amalgamation script a bit more Windows friendly.
* We do not try to silence -Wmaybe-uninitialized under clang.
* Optimized the arm64 implementation of simd8x64::compress
This is ~35% faster on the fast_minify benchmarks on Apple M1
* Return byte-count from simd8x64::compress
This avoids a redundant popcount on ARM, for ~3% faster minify
on Apple M1
* This exposes 'rewind' for object and array instances.
* Putting really_inline back to count_elements()
* Update array.h
* Adding empty array rewind.
* Adds "is_empty" method to arrays.
* More fragmentation.
* Tweaking implementation.
* Fixing issue with get_value() on document instances.
* Changing the name of the new rewind functions to reset.
Including <iostream> has two problems:
* Compile times are worse because of over-inclusion
* Binary sizes are worse when statically linking libstdc++ because
iostreams cannot be dead-code-stripped
simdjson only needs std::ostream. Include the header declaring only what
we need (<ostream>), omitting stuff we don't need (std::cout and its
initialization, for example).
This commit should not change behavior, but it might break users who
assume that including <simdjson/simdjson.h> will make std::cout
available (such as many of simdjson's own files).
* Adding test.
* Verifies and fix issue 1668. This commit updates the previous behavior of the
On Demand stream support by return a value type (document_reference) instead
of a reference to a document. This allows us to bridge with the usually simdjson
error system, with its simdjson_result types.
* Minor reformat.
* Adds a test with initial tests passing.
* Adding an example.
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
* Naive implementation for doubles in string.
* Add double from string in atom doc.
* Simplification (removed all *_from_string())
* Add int and uint parsing in string.
* Make duplicates instead.
* Make tests exceptionless.
* Add missing declarations.
* Add more tests (errors, JSON pointer).
* Add crypto json tests.
* Update doc.
* Update doc after review.
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
* Add document_stream constructors and iterate_many
* Attempt to implement streaming.
* Kind of fixed next() for getting next document
* Temporary save.
* Putting in working order.
* Add working doc_index and add function next_document()
* Attempt to implement streaming.
* Re-anchoring json_iterator after a call to stage 1
* I am convinced it should be a 'while'.
* Add source() with test.
* Add truncated_bytes().
* Fix casting issues.
* Fix old style cast.
* Fix privacy issue.
* Fix privacy issues.
* Again
* .
* Add more tests. Add error() for iterator class.
* Fix source() to not included whitespaces between documents.
* Fixing CI.
* Fix source() for multiple batches. Add new tests.
* Fix batch_start when document has leading spaces. Add new tests for that.
* Add new tests.
* Temporary save.
* Working hacky multithread version.
* Small fix in header files.
* Correct version (not working).
* Adding a move assignment to ondemand::parser.
* Fix attempt by changing std::swap.
* Moving DEFAULT_BATCH_SIZE and MINIMAL_BATCH_SIZE.
* Update doc and readme tests.
* Update basics.md
* Update readme_examples tests.
* Fix exceptions in test.
* Partial setup for amazon_cellphones.
* Benchmark with vectors.
* Benchmark with maps
* With vectors again.
* Fix for weighted average.
* DOM benchmark.
* Fix typos. Add On Demand benchmark.
* Add large amazon_cellphones benchmark for DOM
* Add benchmark for On demand.
* Fix broken read_me test.
* Add parser.threaded to enable/disable thread usage.
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Changing the name of the function to 'to_json_string' from 'to_string' to avoid confusion.
* Moving to a fast string_view model
* Making it exception-safe.
* Tweaking.
* Workaround for exceptions.
* more robust to_json_string (#1651)
* WIP.
* Fuzzing timeout (bug fix) (#1650)
* prove pull request #1648 introduces an infinite loop
* Interesting bug!
* Tweak.
Co-authored-by: Paul Dreik <github@pauldreik.se>
* It should now work.
* Moving car examples to exception mode
* Simplifying somewhat.
* I forgot to abandon. Let us do that.
* Adding more tests.
* WIP.
* It should now work.
* Moving car examples to exception mode
* Simplifying somewhat.
* I forgot to abandon. Let us do that.
* Adding more tests.
Co-authored-by: Paul Dreik <github@pauldreik.se>
Co-authored-by: Paul Dreik <github@pauldreik.se>
* upload corpus to https://www.pauldreik.se/ from the x64 github action job (keep the github action cache)
* drop the github action cache (which was not working anyway) for power fuzzer and download the fuzz corpus from https://www.pauldreik.se/ instead
* resurrect arm64 fuzzing on drone CI, downloading the fuzz corpus from https://www.pauldreik.se/
* update the fuzzing documentation
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Add working JSON pointer for array of atoms.
* Add working JSON pointer for object with key-atom pairs.
* Add first version of JSON pointer.
* Update tests (2 tests).
* Make tests exceptionless.
* Fix builing issues.
* Add more tests. Add json_pointer validation in array-inl.h and object-inl.h and empty json_pointer in document-inl.h.
* Fix errors in tests.
* Review.
* Add missing comment.
* Adds compile-test for Visual Studio + ARM and turn developer mode throughout CI.
* Correcting YAML error.
* Disabling google benchmarks under Windows ARM.
* Turning off exceptions under ARM.
* First try at implementing max_capacity for simdjson_ondemand.
* Add max_capacity check.
* Update doc.
* Add one more example in doc for fixed capacity.
* Make allocate() public.
* Remove whitespace
* Found culprit whitespace.
* Duplicating variable.
* Adding 'count_elements' method.
* Actually reporting errors.
* removing white space.
* Removing white space again.
* Adding an extra example.
* Prettier.
* Making the functionality more error-proof.
* Avoiding exceptions.
* Various fixes including extending count_elements to value types.
* Various fixes.
* Minor fixes.
* Correcting comment.
* Trimming white spaces.
* Add first working version of rapidjson_sax for partial tweets.
* Add cleaner and faster rapidjson_sax
* Add nlohmann_json_sax.
* Replace array of bool by bitsets.
* Replace strdup to copy string in rapidjson_sax.
* Change std::string_view assignment in rapidjson_sax.
* Add rapidjson_sax.h and fix typo in rapidjson.h
* Add nlohmann_json_sax.h and add user key check for screen_name in rapidjson_sax
* Change std::string_view assignement for text and screen_name.
* Add rapidjson_sax.h .
* Add nlohmann_json_sax.h . Fix typos distinct_user_id/nlohmann_json_sax.h, find_tweet/rapidjson.h and find_tweet/rapidjson_sax.h .
* Add extra check for id key when looking for find_id.
When find_field_unordered is used on an empty object, it calls
json_iterator::reenter_child. reenter_child asserts that it doesn't
rewind too far back by consulting parser->start_positions.
When the On Demand parser sees an empty object, it fails to update
parser->start_positions. This means that the assertion in
json_iterator::reenter_child reads stale data, or potentially
uninitialized memory. Reading uninitialized memory can cause spurious
assertion failures and Valgrind memcheck reports:
Running missing_keys_for_empty_top_level_object ...
==170679== Conditional jump or move depends on uninitialised value(s)
==170679== at 0x4943D7: reenter_child (json_iterator-inl.h:208)
==170679== by 0x4943D7: find_field_unordered_raw (value_iterator-inl.h:197)
==170679== by 0x4943D7: find_field_unordered (object-inl.h:13)
==170679== by 0x4943D7: find_field_unordered (object-inl.h:96)
==170679== by 0x4943D7: find_field_unordered (value-inl.h:110)
==170679== by 0x4943D7: find_field_unordered (document-inl.h:105)
==170679== by 0x4943D7: object_tests::missing_keys_for_empty_top_level_object() (ondemand_object_tests.cpp:117)
==170679== by 0x4CA761: object_tests::run() (ondemand_object_tests.cpp:1085)
==170679== by 0x8BA314: int test_main<bool ()>(int, char**, bool ( const&)()) (test_ondemand.h:81)
==170679== by 0x4CA9C8: main (ondemand_object_tests.cpp:1119)
==170679==
Fix the read of uninitialized or stale memory by updating
parser->start_positions regardless of whether we see an empty object or
an object with some keys.
This commit only affects builds where development checks
(SIMDJSON_DEVELOPMENT_CHECKS) are enabled. Builds where development
checks are disabled are unaffected by this bug.
* Verifies bug with missing keys.
* Allowing search from any key.
* Workaround for buggy msys
* Restricting how we can end key searches.
* Adding a few tests.
* Under ARM, it is slightly better to reverse the word once and then extract the bits.
* Guarding the zero_leading_bit call to avoid sanitizer warnings.
* This moves all DOM (benchmark + test) files to a subdir
* Missing file.
* CMake + DLL is not pretty.
* Capitalizing AND
* Fixing mismatch endif
* Flipping the order.
* onedemand => ondemand
* Remove CMP0025 policy
This policy is already set to NEW by the minimum required version.
* Use HOMEPAGE_URL in the project call
* Use VERSION in the project call
* Detect if this is the top project
* Port simdjson-user-cmakecache to a CMake script
* Create a developer mode
The SIMDJSON_DEVELOPER_MODE option set to ON will enable targets that
are only useful for developers of simdjson.
* Consolidate root CML commands into logical sections
* Warn about intended use of developer mode
* Prettify the just_ascii test
* Remove redundant CMake variables
* Inline CML contents from include and src
* Raise minimum CMake requirement to 3.14
* Define proper install rules
* Restore thread support variable
* Add BUILD_SHARED_LIBS as a top level only option
* Force developer mode to be on in CI
* Include flags earlier in developer mode
* Set CMAKE_BUILD_TYPE conditionally
CMAKE_BUILD_TYPE is used only by single configuration generators and is
otherwise completely ignored.
* Remove useless static/shared options
simdjson now uses the CMake builtin BUILD_SHARED_LIBS to switch the
built artifact's type.
* Remove unused CMAKE_MODULE_PATH variable
* Refactor implementation switching into a module
* Factor exception option out into a module
* Reformat simdjson-flags.cmake
* Rename simdjson-flags to developer-options
* Accumulate properties into an include module
This is done this way to avoid using utility targets that must be
exported and installed, which could potentially be misused by users of
the library.
* Port impl definitions to props
* Port exception options to props
* Lift normal options to the top
* Port developer options to props
* Remove simdjson-flags from benchmark
* Document the developer mode in HACKING
* Fix include path in installed config file
* Fix formatting of prop commands
* Fix tests that include .cpp files
* Change GCC AVX fixes back to compile options
* Deprecate SIMDJSON_BUILD_STATIC
* Always link fuzz targets to simdjson
* Install CMake from simdjson's debian repo
* Add gnupg for apt-key
* Make sure ASan link flags come first
* Pass CI env variable to cmake invocation
* Install package for apt-add-repository
* Remove return() from flush macro
* Use directory level commands instead of props
* Restore the github repository variable
* Set developer mode unconditionally for checkperf
The CI env variable is only set in the CI and this target is always run
in developer mode.
* Attempt to fix ODR violation in parsing checks
These tests were compiling the simdjson.cpp file again and linking to
the simdjson library target causes ODR violations.
Instead of linking to the target, just inherit its props.
* Move variables before the source dir
* Mark props to be flushed after adding more
* Use props for every command for the library
* Use keyword form for linking libs
* Handle deprecation of SIMDJSON_JUST_LIBRARY
* Handle deprecations in a separate module
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Truncate final unclosed string.
* Adding more precise remarks.
* Better documentation and more robust code.
* ARM + PPC corrections.
* Patching ARM implementation with new stage1_mode parameter.
* Fixed most problems.
* Correcting white spaces and adding a remark.
* This adds the truncated_bytes() method to the stream instances.
* This implementations string serialization for On Demand instances.
* Adding more documentation.
* Another remark.
* Marking the new functions as inline.
* casts apparently do not work.
* Upgrading the API.
* Making the code really free from exceptions.
* At another fix for exceptionless.
* Modify to_chars so that it does not pad integers with '.0'.
* Negative 0 cannot be expressed as an integer.
* Again, accomodating exceptionless usage.
* Using x <= -0 does not allow you to determine the sign since 0 <= -0. I am not sure where
this bug comes from.
This eliminates the possibility of inlining target failures for ondemand
Also makes it so we always compile common architectures needed by simdjson.cpp in simdjson.h, since amalgamation has no way to reason about whether to include / exclude it.
* This gives the CMake install the necessarily information (and flags) to know
whether we have a Windows DLL and in such cases how to handle the linkage.
This avoids a very unlikely buffer overrun that can occur in a particular kind of invalid JSON:
- the document is invalid with an unclosed top level array or object
- the last thing in the document is a number that ends at EOF
- the padding is filled entirely with numeric digits
* This adds a little test to see if we can compiler with very strict flags.
* Trimming a leftover old-style cast.
* More cleaning.
* A few more pedantic casts.
* Fixing issue 1243
* The tie must go.
* Having std::pair be a protected inheritance breaks on demand.
* Putting it back.
* You really want to use emplace.
* Fixing one botched test.
* Prettier test.
* Using safer code.
* Fixing unsafe code.
* Simplifying the fuzzer.
* Trying another way.
* Ok. It should work without exceptions.
* Removing trailing spaces.
* Minor edits regarding the On Demand documentation.
* Adding more instructions for CMake
* Tweaking.
* Adding changes requested by John.
* Bringing back detailed explanations of -march=native.
as described in the description for the allocate_padded_buffer function: // The caller is responsible to free the memory (e.g., delete [] (...)).
but your code used the function free.
I propose to fix this error.
* Entering a new UTF-8 test
* Maybe *I* had a bug in the tests.
* Replacing nulls with 1s.
* Let us try to be more verbose.
* Return 0.
* Fixing issue.
* Adding puzzler scenario.
* Fixing PPC64
Co-authored-by: Daniel Lemire <dlemire@rcs-power9-talos>
* Reenabling the optimized kernels (main branch).
* Defining SIMDJSON_CAN_ALWAYS_RUN_PPC64 and SIMDJSON_CAN_ALWAYS_RUN_ARM64
* Adding the bad UTF8 string from the fuzzer.
* Taking into account John's comments.
* Bumping the lib version.
* Update CMakeLists.txt
* first try
* use ubuntu 20.04, do the fuzzing
* new try at power fuzz
* hard code clang version
* setting env variables does not seem to work
* use fuzzer-no-link
* switch to Debian Buster for power fuzz
* use non-sanitizer build for power
* me not like yaml
* fix bad syntax
* add ndjson fuzzer
* reproduce #1310 in the newly added unit test
Had to replace the input, because:
1)
the fuzzer uses the first part of the input to determine
the batch_size to use, so that has to be cut off
2)
the master now protects against low values of batch_size
I also made the test not return early, so the error is triggered.
* bump boost.json and see if it works in simdjson CI
* enable boost json
* clean up
* add boost json to deps
* use boost if std::string_view is available
* add build with c++20
* use docker image which has the proper libc++ installed
Forks that would like to contribute via PRs from feature branches
needlessly run CI on those branches on top of the PRs.
This is a waste of resources.
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Fix Cirrus CI
CMake is not installed at this point yet.
* Add caching in Circle CI
* Add caching to the MinGW Github workflows
* Fix Circle CI config
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Bump minimum CMake version
* Remove unnecessary git checks
* Move benchmark options where they are used
* Declare helper functions for dependencies
The custom solution here is tailored for fast configure times, but only
works for dependencies on Github.
* Import dependencies using the declared commands
* Remove git submodules
* Call target_link_libraries properly
target_link_libraries must not be called without a requirement
specifier.
* Fix includes for competition
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Updating main branch for legacy libc++ support
* Adopting
* Removing unnecessary math header.
* Updating the single-header files so we can pass the new tests.
* Portable infinite-value detection is hard.
* Working toward disabling boost json selectively.
* Selectively disabling Boost JSON
* More work toward selectively disabling boost json.
Under compilers like MSVC the `#pragma message` about portability will be issued for each translation module; regardless of whether or not `SIMDJSON_PORTABILITY_H` has already been defined. As such a new define `SIMDJSON_NO_PORTABILITY_WARNING`, can be defined prior to the inclusion of `<simdjson.h>` to silence it.
Regardless of the compiler, Windows targets do not support -fPIC,
as position independent code is already implicitly enabled. Compiling
simdjson with Clang on Windows will error because -fPIC is an
unsupported option for target 'x86_64-pc-windows-msvc'.
fix uninteded early return in ondemand unit test loops
go through and fix warnings appearing in qtcreator,
qualify with std::, add const, abort on error
get rid of ulp_distance, not needed anymore when parsing is exact
Introduce cmake option SIMDJSON_DISABLE_DEPRECATED_API (default Off)
which turns off deprecated simdjson api functions by setting the macro
SIMDJSON_DISABLE_DEPRECATED_API.
For non-cmake users, users will have to set SIMDJSON_DISABLE_DEPRECATED_API
by some other means to disable the api.
Closes#1264
This builds the CI fuzzers with the intended clang version. It also allows users to set the clang version locally,
in case they need to.
It also switches the CI fuzzers to use an optimized sanitizer build, to do something oss-fuzz doesn't and get more done in the short time the CI fuzzer runs.
* Add script for CMake PPA
* Call the CMake PPA script in Drone CI
"apt-get update -qq" can be omitted, as that command is already called
by the script to pull in necessary packages for the CMake GPG keys.
* Remove sudo calls in the CMake PPA script
This script is intended to be run in Docker images, where the default
user is already root.
* Use echo instead of printf
* Use /etc/os-release instead of lsb_release
lsd_release could be installed, but os-release is just more convenient
to grab the version code from at this point.
* On Debian images grab CMake from buster-backports
It's not wise to mix Ubuntu PPAs with Debian and buster-backports has
CMake 3.16, which is recent enough for our purposes.
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Initial PPC64 support
* Add travis CI
* Fix outdated cmake version for travis
* Fix indendtation
* Try another workaround for outdated cmake in travis
* Try beta cmake
* Add dash before beta
* Use builtin snaps
* Use cmake as rocksdb
* Test cmake on bionic
* Remove unnecessary things from travis
* Remove unnecessary things from travis
* Another try of compiler install
* Add all major compilers
* Add all major compilers
* Add all major compilers
* Tweak travis a bit
* Typo
* More robust travis
* Typos typos typos
* Add fewer compilers, add non specific build for clang and gcc, should be the final config
* CMAKE_FLAGS is in incorrect place
* Remove default implementation
* Limit build thread number
* Fall back prefix_xor to a usual implementation, no performance boost is noticed
* Test for power9 as it is the main architecture for OpenPOWER right now
* Add to documentation to build with power9 as the implementation is compatible but compiler optimizations is not
* Replace ARM with PPC in the comment
Projects that link simdjson from MSVC with exceptions off will
include simdjson headers which transitively include STL headers.
The MSVC STL stipulates that _HAS_EXCEPTIONS=0 be defined or code
requiring exceptions will be enabled. This change adds a new job
to the appveyor build matrix to verify the build and tests with
exceptions disabled, and disables exceptions at the compiler level
when SIMDJSON_EXCEPTIONS is specified to OFF.
* Adding a distinct user id benchmark
* reenabling everything
* Removing an unnecessary "value()".
* Better tests of the examples and some fixes.
* Guarding exception code.
* Reenable the on-demand tests and allows us to convert a raw string into a C++ string.
* Fixing a 1-byte buffer overrun.
* More documentation.
* Adding more tests.
* Enabling the new tests
* Committing a nicer example.
* Not yet happy but this should fix our failures.
* Duh.
* Ok. Making it easier to get string_view instances from field instances.
* It is a struct.
* Trying to satisfy VS.
* Adopting John's name.
* add definitions for is_number and tie (by lemire)
* add fuzzer for element
* update fuzz documentation
* fix UB in creating an empty padded string
* don't bother null terminating padded_string, it is done by the std::memset already
* refactor fuzz data splitting into a separate class
* This would allow users to find out what builtin is.
* Trying another approach.
* Added instructions.
* Cleaning up the printout.
* Let us be less invasive.
* Adding a comment.
* This adds new tests regarding ordering.
* Updating the documentation with more examples.
* Adding compilation tests.
* Pruning code for exceptions.
* Guarding exceptionless.
* Remove our dependency on strtod_l by bundling our own slow path.
* Ok. Let us drop strtod entirely.
* Trimming down the powers to -342.
* Removing useless line.
* Many more comments.
* Adding some DLL exports.
* Let the gods help those who rely on windows+gcc.
* Marking the subnormals as unlikely. This is pretty much "performance neutral", but it might help just a bit with twitter.json.
* initial try at adding boost json to the benchmark
* clean up
* qualify memcpy etc. with std::
* clang format
* extra space
* update benchmark with help from Vinnie Falco from Boost.json
* add missing separators
This refactors the dynamic check of which implementations are supported at runtime.
It also reduces duplicated effort in the CI fuzzing job, the differential fuzzers don't need to run with different values of SIMDJSON_FORCE_IMPLEMENTATION.
There is also a convenience script to run the fuzzers locally, to quickly check that the fuzzers still build, run and no easy to find bugs are there. It should be handy not only when developing the fuzzers, but also when modifying simdjson.
- Allow user to specify SIMDJSON_BUILTIN_IMPLEMENTATION
- Make cmake -DSIMDJSON_IMPLEMENTATION=haswell *only* specify haswell
- Move negative implementation selection to
-DSIMDJSON_EXCLUDE_IMPLEMENTATION
- Automatically select SIMDJSON_BUILTIN_IMPLEMENTATION if
SIMDJSON_IMPLEMENTATION is set
- Move implementation enablement mostly to implementation files
- Make implementation enablement and selection simpler and more robust
- Fix bug where programs linked against simdjson were not passed
SIMDJSON_XXX_IMPLEMENTATION or SIMDJSON_EXCEPTIONS
* Make it possible to check that an implementation is supported at runtime.
* add CI fuzzing on arm 64 bit
This adds fuzzing on drone.io arm64
For some reason, leak detection had to be disabled. If it is enabled, the fuzzer falsely reports a crash at the end of fuzzing.
Closes: #1188
* Guarding the implementation accesses.
* Better doc.
* Updating cxxopts.
* Make it possible to check that an implementation is supported at runtime.
* Guarding the implementation accesses.
* Better doc.
* Updating cxxopts.
* We need to accomodate cxxopts
Co-authored-by: Paul Dreik <github@pauldreik.se>
This adds fuzzing on drone.io arm64
For some reason, leak detection had to be disabled. If it is enabled, the fuzzer falsely reports a crash at the end of fuzzing.
Closes: #1188
This adds a minifier fuzzer. There is also an utf-8 fuzzer, but it is disabled until #1187 is fixed.
Run all fuzzers bug the utf-8 one in the github CI fuzz.
* Adding new files.
* Better.
* Fixing minifier and adding tests.
* Adding benchmarks.
* Including the array header.
* Replacing old stream-based code by the new code.
* Doubling up the itoa.
* Hidden away to_chars in internal namespace.
* Removing the repetitions.
* Documented the atoi functions.
* Tuning the escape sequences.
* Moving the operators off the main namespace.
* Added more tests.
* Tweaking the implementation so that it works with and without exp.
* The string_builder template and mini_formatter class
are not part of our public API and are subject to change
at any time!
* Adding a benchmark and some optimization.
* Cleaning.
* Strictly speaking, this header is needed.
This adds a fuzzer for at_pointer() which recently had a bug.
The #1142 bug had been found with this fuzzer
Also, it polishes the github action job:
cross pollinate the fuzzer corpora (lets fuzzers reuse results from other fuzzers)
use github action syntax instead of bash checks
only run on push if on master
* This avoids locale-dependent number parsing at the standard library level.
* Adding missing cast.
* Inserting the missing "endif"
* Trial and error.
* Another attempt.
* Another tweak.
* Another fix.
* Restricting it even more.
* Tweaking our symbol checks.
* Somewhat smarter tests.
* Nice comments.
* Minor simplification.
* Adding cerr.
This adds a fuzzer which parses the same input using all the available implementations (haswell, westmere, fallback on x64).
This should get the otherwise uncovered sourcefiles (mostly fallback) to show up in the fuzz coverage.
For instance, the fallback directory has only one line covered.
As of the 20200909 report, 1866 lines are covered out of 4478.
Also, it will detect if the implementations behave differently:
by making sure they all succeed, or all error
turning the parsed data into text again, should produce equal results
While at it, I corrected some minor things:
clean up building too many variants, run with forced implementation (closes#815 )
always store crashes as artefacts, good in case the fuzzer finds something
return value of the fuzzer function should always be 0
reduce log spam
introduce max size for the seed corpus and the CI fuzzer
* Removes 5 KB of tables at the expense, and a load, at the expense
of a multiplication and a shift. I have not benchmarked this new
code, but my expectation is that it should be largely performance
neutral. The motivation is to reduce the size of the library slightly.
There is also a matter of elegance.
* Adding test.
* Saving.
* With exceptions.
* Added extensive tests.
* Better documentation.
* Tweaking CI
* Cleaning.
* Do not assume make.
* Let us make the build verbose
* Reorg
* I do not understand how circle ci works.
* Breaking it up.
* Better syntax.
* Specification is not followed.
* Fixes.
* Do not pass string_view by reference.
* Better documentation.
* The example is written for exceptions.
* Better documentation.
* Updating with deprecation.
* Updating example.
* Updating example.
* This allows the users to disable threading.
* This would disable bash scripts under FreeBSD. (#1118)
* This would disable bash scripts under FreeBSD.
* Let us also disable GIT.
* Let us try to just disable GIT
* Nope. We must have both bash and git disabled.
* This allows the users to disable threading.
* This would disable bash scripts under FreeBSD.
* Let us also disable GIT.
* Let us try to just disable GIT
* Nope. We must have both bash and git disabled.
The jobs were executed in powershell using the globally installed cmake.
This makes things actually run in a MSYS2 shell.
This also removes the msys/cygwin job because it doesn't build
(it complains about undeclared posix_memalign)
C++ 20 adds a new feature called "ranges", which provides components for dealing
with sequences of values: https://en.cppreference.com/w/cpp/ranges.
A range is like a normal object containing `begin` and `end`, except there are
also composable operations like maps, filters, joins, etc.
The iterator objects returned by a range's `begin` and `end` require a more
strict set of operations than is needed for a range-for loop.
This PR adds the extra operations needed to support turning `dom::array` and
`dom::object` into a range.
This PR does not depend on any C++ 20 behavior, the added operators are all
valid C++ 11, and are already part of the LegacyIterator concepts.
This PR adds extra code behind: `#if defined(__cpp_lib_ranges)` guards, which is
the new C++ 20 specified feature test macro for ranges support. When ranges
support is detected, extra compile time checks are added to ensure that
`dom::array` and `dom::object` satisfy the range concept. No runtime tests have
been added yet because these compile time checks should be sufficient.
If desired, the `static_assert` code could be moved out of the actual code
headers and put into a test file.
* lookup4
* Self-document lookup4 and clean up extra bits
* Maintenance, to match against upcoming PR.
Co-authored-by: Daniel Lemire <lemire@gmai.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
* The initial motivation behind basictests was for a quick set of sanity tests to check whether your code made sense. It
was not meant for thorough testing to find corner cases. However, over time, it grew to include such expensive tests.
This PR takes them out. It also allows us to bring back basictests to MinGW tests, since it is now cheap.
This is not an exercise in software engineering and making things prettier. This is a pragmatic change to improve our
test coverage and quality of life.
* Adds many more cheap tests.
Co-authored-by: Daniel Lemire <lemire@gmai.com>
According to https://gitlab.kitware.com/cmake/cmake/-/issues/17976,
CMAKE_VS_PLATFORM_TOOLSET is set only when using a Visual Studio generator.
When we use Ninja as the generator, CMAKE_VS_PLATFORM_TOOLSET will be empty.
As a result:
if(${CMAKE_VS_PLATFORM_TOOLSET} STREQUAL "v140")
will be treated as:
if( STREQUAL "v140")
We may also quote it like this:
if("${CMAKE_VS_PLATFORM_TOOLSET}" STREQUAL "v140")
but that won't make the warnings disappeared in VS2015.
Links to other files need to be either relative to themselves (doc/performance.md -> performance.md) or absolute (doc/performance.md -> /doc/performance.md). This change fixes the documentation when read on GitHub.
* Testing with GCC 10 and clang 10
* Fixing spurious space
* gcc10 does not need the cmake installation.
* We don't want to run the perf test on ARM. I ignore them systematically. ARM performance
should be assessed manually.
* Switching to GCC 10 and Clang 10
* Disabling some tests under sanitizers when they involve rapidjson or other parsers.
Co-authored-by: Daniel Lemire <lemire@gmai.com>
In the parse_many function, we have one thread doing the stage 1, while the main thread does stage 2. So if stage 1 and stage 2 take half the time, the parse_many could run at twice the speed. It is unlikely to do so. Still, we see benefits of about 40% due to threading.
To achieve this interleaving, we load the data in batches (blocks) of some size. In the current code (master), we create a new thread for each batch. Thread creation is expensive so our approach only works over sizeable batches. This PR improves things and makes parse_many faster when using small batches.
This fixes our parse_stream benchmark which is just busted.
This replaces the one-thread per batch routine by a worker object that reuses the same thread. In benchmarks, this allows us to get the same maximal speed, but with smaller processing blocks. It does not help much with larger blocks because the cost of the thread create gets amortized efficiently.
This PR makes parse_many beneficial over small datasets. It also makes us less dependent on the thread creation time.
Unfortunately, it is going to be difficult to say anything definitive in general. The cost of creating a thread varies widely depending on the OS. On some systems, it might be cheap, in others very expensive. It should be expected that the new code will depend less drastically on the performances of the underlying system, since we create juste one thread.
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Daniel Lemire <lemire@gmai.com>
This regresses performance and is ONLY here because the next
two commits are here; this lets us see the impact of removing
parser.error separately from the impact of the next commit.
When using Ninja as generator, it failed to build:
$ cmake .. -G Ninja
$ ninja
> ninja: error: '/jsonexamples/generated/miss-templates/*.txt', needed by 'jsonexamples/generated/utf-8.json', missing and no known rule to make it
Although it passes user-defined options, if the project is build in Debug mode or with Clang (since
CXX defaults to gcc on Linux) results can flactuate
If linked against Threads::Threads target while building static libraries, cmake cannot find the
threads library while trying to use the installed target afterwards
* This will change the default of the parse benchmark so that it work over hot buffers
by default, thus omitting memory allocation as part of the benchmark.
* Everyone should be using '-H' from now on.
amalgamation.sh shouldn't change contents of src/simdjson.cpp by forcing dmalloc.h that didn't exist in non-amalgamated version and shouldn't change order of includes by placing simdjson.h at the top
fixes#739
* Added bitexact implementations of _BitScanForward64 and _BitScanReverse64 for VS2019 32-bit builds
* Added bitexact implementations of _umul128 for VS2019 x86, arm, arm64 builds
* Implement mul_overflow for VS2019 arm64 builds
+ implement mul_overflow using __umulh (msvc/clang results: https://godbolt.org/z/smRwA7)
* Added Win32 for VS2019 to .appveyor.yml
* Update amalgamated headers (fix x86 builds with VS2019)
* This is an implementation of "size()" for arrays and objects.
* Adding benchmark
* Adding a size() remark in the documentation.
* Extending size() to result types.
To avoid using data belonging to a temporary, the parse functions are ref qualified to get a compile error if used on an rvalue. See https://github.com/simdjson/simdjson/issues/696
Compilation tests are also added, to make sure bad usage fails to compile.
Reviewed by jkeiser.
* Trying to correct the documentation so that it actually describes how the code behaves.
* tweaking the wording.
* Improving.
* Removing confusing sentence.
* Fixing formatting.
* Now with working example, tested.
* Added a smaller piece of code
* It is inconvenient to be unable to print a padded_string.
* Allows us to print the padded_string even when it is embedded in result object when exceptions are enabled.
* move from deprecated interface in fuzz dump raw tape
* update fuzz_dump to the non deprecated replacement
* replace use of deprecated api
* hopefully fix windows build
This enables the minify fuzzer, which has been disabled because it did not pass the oss-fuzz instrumentation test. Now it does, after changes in simdjson (https://github.com/lemire/simdjson/issues/186).
* get minify running (api change)
* disable benchmarks when compiling fuzzers
* catch exceptions from the minify fuzzer
* enable repeated corpus creation without recursive inclusion of zip
* remove leftover comment
* This is a first attempt at fixing issue 660. The hard part is not the fix by itself, the hard part is to make sure we never get caught with our pants down like that again.
I expect the CI tests will fail. Further commits will solve the issues.
* Setting the rpath properly.
* For the sanitize tests, we want verify the installation.
* Some users are interested, as a metric, in the number of documents parsed per second.
Obviously, this means reusing the same parser again and again.
* Adding a sentence
* This update the parsingcompetition benchmark so that it displays the number of documents parsed per second.
* Fallback should use our scalar code.
* parse should have a nicer error message.
* Making it so that "minify" can use different architectures.
* Let us change the minifier competition so that it tests all implementations.
* Documenting the untaken optimization opportunity.
Co-authored-by: John Keiser <john@johnkeiser.com>
* Currently, document::stream contains an attribute that is a reference:
```
document::parser &parser;
```
Yet we try to have it default on the move operator:
```
stream &operator=(document::stream &&other) = default;
stream &operator=(const document::stream &) = delete; // Disallow copying
```
```
stream(document::stream &&other) = default;
stream(const document::stream &) = delete; // Disallow copying
```
I am not sure what the move is supposed to do with the reference.
I cannot find where we test the copy constructor and assignment. This has been concerned that it is either dead code or buggy code.
* Remove non-working, unnecessary move constructors
* We still want to disallow copies.
Co-authored-by: John Keiser <john@johnkeiser.com>
* If we are going to have a google benchmark flag, we better make sure that we test it out minimal (it should build).
* Fix bench_dom_api
Co-authored-by: John Keiser <john@johnkeiser.com>
* Make architecture implementations virtual functions
- Easier to add new architectures (add implementation to implementation.cpp)
- Easier to add new algorithms / functions to architecture selection
(add to implementation.h, implement)
- Automatically select best implementation in static initialization
- Allow user to explicitly select implementation with a string (i.e.
parameter)
- Allow user to inspect current implementation name/description
- Allow user to list available implementations
- Eliminate architecture enum and architecture-based templating
- Add noexcept in non-inline functions
* Move implementation static methods to their own classes
* Detect best supported implementation on first use
* available_implementationsI() -> available_implementations
This creates a "document" class with only user-facing document state (no parser internals).
- document: user-facing document state
- document::iterator: iterator (equivalent of ParsedJsonIterator)
- document::parser: parser state plus a "docked" document we parse into (equivalent of ParsedJson)
Usage:
```c++
auto doc = simdjson::document::parse(buf, len); // less efficient but simplest
```
```c++
simdjson::document::parser parser; // reusable parser
parser.allocate_capacity(len);
simdjson::document* doc = parser.parse(buf, len); // pointer to doc inside parser
doc = parser.parse(buf2, len); // reuses all buffers and overwrites doc; more efficient
```
* Fix issue472: make JsonStream a template.
* Adding missing include.
* Tweaking headers and some minor formatting.
* Removing file from aggregation.
* Moving jsoncharutils
* Adding new header.
* Trying another header.
* Let us try to route around Visual Studio's nonesense.
* Fix for issue467
* Updating single-header
* Let us make it so that JsonStream is constructed from a padded_string which will avoid dangerous overruns.
* Fixing parse_stream
* Updating documentation.
* This revert the code back to how it was prior to the silly "run two stages" routine and instead
adds an option to benchmark the code over hot buffers. It turns out that it can be expensive,
when the files are large, to allocate the pages.
* Instead of emulating the whole parsing as stage 1 + stage 2, let us
benchmark the real thing.
* Adding explicit constructor.
* Adding warning to the benchmark user.
* Making re-running optional.
* dirent portable latest version
* improved
std::string argument passed by const reference
ctor added with std::string_view argument
`allocate_padded_buffer()` moved here with **optional** check on `length < 1`
* allocate_padded_buffer moved to padded_string.h
* string literal + integer means unintended and incorrect pointer arithmetic
fixes a clang warning. it could not be triggered, because it can only be
triggered if the string given to getopt is not covered among the
cases in the switch.
* handle review comment
* I think we can align the numbers better (so it is prettier).
* Remove space before %, align third line better
Co-authored-by: John Keiser <john@johnkeiser.com>
It's only a coincidence that it works in current uses: it doesn't do
what the name says. Particularly, if the high bit is 1 it will yield
0 even if the lower 4 bits would yield something else.
Only the simdjson library should optionally depend on threads,
the executables that link to simdjson will get the dependency
indirectly.
* add option for controlling threads (default is on)
* add CI testing with threading on/off for msvc, gcc and clang
* fix an unrelated copy paste comment error in the cirlce ci build conf
* JsonStream threaded prototype
* JsonStream Threaded version working. Still supporting non-threaded version.
* Fix where invalid files would enter infinite loop.
* SingleHeader update
* I will remove -pthread in cmake for now.
* Attempt at resolving the -pthread issue
* rough prototype working. Needs more test and fine tuning.
* prototype working on large files.
* prototype working on large files.
* Adding benchmarks
* jsonstream API adjustment
* type
* minor fixes and cleaning.
* minor fixes and cleaning.
* removing warnings
* removing some copies
* runtime dispatch error fix
* makefile linking src/jsonstream.cpp
* fixing arm stage 1 headers
* fixing stage 2 headers
* fixing stage 1 arm header
* making jsonstream portable
* cleaning imports
* including <algorithms> for windows compiler
* cleaning benchmark imports
* adding jsonstream to amalgamation
* merged main into branch
* bug fix where JsonStream would bug on rare cases.
* Addind a JsonStream Demo to Amalgamation
* Fix for https://github.com/lemire/simdjson/issues/345
* Follow up test and fix for https://github.com/lemire/simdjson/issues/345 (#347)
* Final (?) fix for https://github.com/lemire/simdjson/issues/345
* Verbose basictest
* Being more forgiving of powers of ten.
* Let us zero the tail end.
* add basic fuzzers (#348)
* add basic fuzzing using libFuzzer
* let cmake respect cflags, otherwise the fuzzer flags go unnoticed
also, integrates badly with oss-fuzz
* add new fuzzer for minification, simplify the old one
* add fuzzer for the dump example
* clang format
* adding Paul Dreik
* rough prototype working. Needs more test and fine tuning.
* prototype working on large files.
* prototype working on large files.
* Adding benchmarks
* jsonstream API adjustment
* type
* minor fixes and cleaning.
* Fixing issue 351 (#352)
* Fixing issues 351 and 353
* minor fixes and cleaning.
* removing warnings
* removing some copies
* Fix ARM compile errors on g++ 7.4 (#354)
* Fix ARM compilation errors
* Update singleheader
* runtime dispatch error fix
* makefile linking src/jsonstream.cpp
* fixing arm stage 1 headers
* fixing stage 2 headers
* fixing stage 1 arm header
* fix integer overflow in subnormal_power10 (#355)
detected by oss-fuzz
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18714
* Adding new test file, following https://github.com/lemire/simdjson/pull/355
* making jsonstream portable
* cleaning imports
* including <algorithms> for windows compiler
* cleaning benchmark imports
* adding jsonstream to amalgamation
* merged main into branch
* bug fix where JsonStream would bug on rare cases.
* Addind a JsonStream Demo to Amalgamation
* merging main
* rough prototype working. Needs more test and fine tuning.
* prototype working on large files.
* prototype working on large files.
* Adding benchmarks
* jsonstream API adjustment
* minor fixes and cleaning.
* minor fixes and cleaning.
* removing warnings
* removing some copies
* runtime dispatch error fix
* makefile linking src/jsonstream.cpp
* fixing arm stage 1 headers
* fixing stage 2 headers
* fixing stage 1 arm header
* making jsonstream portable
* cleaning imports
* including <algorithms> for windows compiler
* cleaning benchmark imports
* adding jsonstream to amalgamation
* bug fix where JsonStream would bug on rare cases.
* Addind a JsonStream Demo to Amalgamation
* rough prototype working. Needs more test and fine tuning.
* minor fixes and cleaning.
* adding jsonstream to amalgamation
* merged main into branch
* Addind a JsonStream Demo to Amalgamation
* merging main
* merging main
* make file fix
* initial oss-fuzz friendly build
parts taken from libfmt, which I wrote and have the copyright to
* fix build error
* add script for building a corpus zip
see https://google.github.io/oss-fuzz/getting-started/new-project-guide/#seed-corpus
* fix zip command
* drop setting the C++ standard
* disable the minify fuzzer, does not pass oss-fuzz check-build test
* fix integer overflow in subnormal_power10
detected by oss-fuzz
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18714
* invoke the build like oss fuzz does
* document what the scripts are for and how to use them
* add a page about fuzzing
* add basic fuzzing using libFuzzer
* let cmake respect cflags, otherwise the fuzzer flags go unnoticed
also, integrates badly with oss-fuzz
* add new fuzzer for minification, simplify the old one
* add fuzzer for the dump example
* clang format
- Removes templating from simd_input, utf8_checker, and parse_string
- Make drone gcc run a lot faster
- Make drone clang run a little faster (NOTE:
https://hub.docker.com/r/silkeh/clang helps even more, but I wasn't sure
whether we wanted to trust that)
- Make drone arm run in parallel to get results quicker
* Allow -f
* Support parse -s (force sse)
* Simplify flatten_bits
- Add directly to base instead of storing variable
- Don't modify base_ptr after beginning of function
- Eliminate base variable and increment base_ptr instead
* De-unroll the flatten_bits loops
* Decrease dependencies in stage 1
- Do all finalize_structurals work before computing the quote mask; mask
out the quote mask later
- Join find_whitespace_and_structurals and finalize_structurals into
single find_structurals call, to reduce variable leakage
- Rework pseudo_pred algorithm to refer to "primitive" for clarity and some
dependency reduction
- Rename quote_mask to in_string to describe what we're trying to
achieve ("mask" could mean many things)
- Break up find_quote_mask_and_bits into find_quote_mask and
invalid_string_bytes to reduce data leakage (i.e. don't expose quote bits
or odd_ends at all to find_structural_bits)
- Genericize overflow methods "follows" and "follows_odd_sequence" for
descriptiveness and possible lifting into a generic simd parsing library
* Mark branches as likely/unlikely
* Reorder and unroll+interleave stage 1 loop
* Nest the cnt > 16 branch inside cnt > 8
* Use generic each/reduce in simdutf8check
* Remove macros from generic simd_input uses
* Use array instead of members to store simd registers
* Default local checkperf to clone from .
* handle uint64 value in JSON
* Add integer_tests
* Add get_unsigned_integer() on ParsedJson::BasicIterator
* Write 'u' to tape when the value seems unsigned
* Add to handle 'u' element
* Brush up integer_tests.cpp
* Append tests/integer_tests in .gitignore
* Add comments to is_integer and is_unsigned_integer
* handle uint64 value in JSON
* Add integer_tests
* Add get_unsigned_integer() on ParsedJson::BasicIterator
* Write 'u' to tape when the value seems unsigned
* Add to handle 'u' element
* Brush up integer_tests.cpp
* Append tests/integer_tests in .gitignore
* Add comments to is_integer and is_unsigned_integer
* Get rid of dynamic allocation in ParsedJson::Iterator.
* Implement copy assignment operator for ParsedJson::Iterator.
* ParsedJson::Iterator is now a template class.
* Add -n and -w arguments
* Add Dockerfile that compares perf against master
* Add checkperf to .drone.yml
* Clone from github instead of .git since CI doesn't have .git
* stage1 compiles without macros
* cleaning
* amalgation is weird but works
* macros are removed from stringparsing
* amalgation fixed
* Huge macros are removed.
* clang-format
* Hiding the runtime dispatch pointer in a source file so it is not an exported symbol
* Disabling hard failure on style check.
* Fixes https://github.com/lemire/simdjson/issues/250
* Attempt 1 - fn targeting
GCC won't work with templates with different targets, need to specialize all the way up the call stack.
* Compiles properly with cmake. Does not with the Makefile.
* Compilation works with Makefile
* instruction_set changes to architecture
* some aesthetic changes
* fix amalgation and tests + aesthetic changes
* This now compiles and passes tests under CLANG
* Minor correction.
* Trying to make it work on ARM
* Adding missing namespace
* Missing bracket
* Fixing minor compilation issues.
* Getting parse to use runtime dispatch
* Fixing amalgamation script.
* Making sure that NEON is supported.
* Fixing typo
* Merging https://github.com/lemire/simdjson/pull/229
* Manual merge of
https://github.com/lemire/simdjson/pull/229
by @jkeiser (second part)
* Trying another way.
* Removing the paral.
* Fixing the make file
* Let us make the practice run long enough.
* Resolved the awful slowness.
* Cleaning the README.md
* With runtime dispatching, we should not need flags anymore.
* Changing isa detection file's name + fixing typos.
# Forward slash is used because this is used in CMake as is
simdjson_DEPENDENCY_CACHE_DIR:C:/dependencies
matrix:
- job_name:VS2019
CMAKE_ARGS:-A %Platform%
- job_name:VS2019ARM
CMAKE_ARGS:-A ARM64 -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_CROSSCOMPILING=1 -D SIMDJSON_GOOGLE_BENCHMARKS=OFF# Does Google Benchmark builds under VS ARM?
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017). If you wish to support features that are not part of RFC 8259, then you should not refer to your issue as a bug.
**Describe the bug**
A clear and concise description of what the bug is. A bug is a failure to build with normal compiler settings or a misbehaviour: when running the code, you get a result that differs from the expected result from our documentation.
A compiler or static-analyzer warning is not a bug. It is possible with tools such as Visual Studio to require that rarely enabled warnings are considered errors. Do not report such cases as bugs. We do accept pull requests if you want to silence warnings issued by code analyzers, however.
We are committed to providing good documentation. We accept the lack of documentation or a misleading documentation as a bug (a 'documentation bug').
An unexpected poor software performance can be accepted as a bug (a 'performance bug').
We accept the identification of an issue by a sanitizer or some checker tool (e.g., valgrind) as a bug, but you must first ensure that it is not a false positive.
We recommend that you run your tests using different optimization levels. In particular, we recommend your run tests with the simdjson library and you code compiled in debug mode. The simdjson then sets the SIMDJSON_DEVELOPMENT_CHECKS macro to 1, and this triggers additional checks on your code and on the internals of the library. If possible, we recommend that you run tests with sanitizers (e.g., see [No more leaks with sanitize flags in gcc and clang](https://lemire.me/blog/2016/04/20/no-more-leaks-with-sanitize-flags-in-gcc-and-clang/)). You can compile the library with sanitizers for debugging purposes (e.g., set SIMDJSON_SANITIZE to ON using CMake), but you should also turn on sanitizers on your own code. You may also use tools like valgrind or the commercial equivalent.
Before reporting a bug, please ensure that you have read our documentation.
**To Reproduce**
Steps to reproduce the behaviour: provide a code sample if possible. Please provide a complete test with data. Remember that a bug is either a failure to build or an unexpected result when running the code.
If we cannot reproduce the issue, then we cannot address it. Note that a stack trace from your own program is not enough. A sample of your source code is insufficient: please provide a complete test for us to reproduce the issue. Please reduce the issue: use as small and as simple an example of the bug as possible.
It should be possible to trigger the bug by using solely simdjson with our default build setup. If you can only observe the bug within some specific context, with some other software, please reduce the issue first.
**simdjson release**
Unless you plan to contribute to simdjson, you should only work from releases. Please be mindful that our main branch may have additional features, bugs and documentation items.
It is fine to report bugs against our main branch, but if that is what you are doing, please be explicit.
**Configuration (please complete the following information if relevant)**
- OS: [e.g. Ubuntu 16.04.6 LTS]
- Compiler* [e.g. Apple clang version 11.0.3 (clang-1103.0.32.59) x86_64-apple-darwin19.4.0]
- Version [e.g. 22]
- Optimization setting (e.g., -O3)
We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux systems. Please ensure that your configuration is supported before labelling the issue as a bug.
* We do not support unreleased or experimental compilers. If you encounter an issue with a
pre-release version of a compiler, do not report it as a bug to simdjson. However, we always
invite contributions either in the form an analysis or of a code contribution.
Under Windows, we support Visual Studio (both with LLVM and without). We do not support MinGW and other alternate compiler systems. Windows users should be aware that there [is a long-running bug with GCC under Windows](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412).
**Indicate whether you are willing or able to provide a bug fix as a pull request**
If you plan to contribute to simdjson, please read our guide:
* CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017).
We do not make changes to simdjson without clearly identifiable benefits, which typically means either performance improvements, bug fixes or new features. Avoid bike-shedding: we all have opinions about how to write code, but we want to focus on what makes simdjson objectively better.
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
Please provide a clear rationale for the feature. Be advised that simdjson is a community-based project: you should consider providing help.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
**Are you willing to contribute code or documentation toward this new feature?**
If you plan to contribute to simdjson, please read our
* CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017).
We do not make changes to simdjson without clearly identifiable benefits, which typically means either performance improvements, bug fixes or new features. Avoid bike-shedding: we all have opinions about how to write code, but we want to focus on what makes simdjson objectively better.
Is your issue:
1. A bug report? If so, please point at a reproducible test. Indicate whether you are willing or able to provide a bug fix as a pull request. As a matter of policy, we do not consider a compiler warning to be a bug.
2. A build issue? If so, provide all possible details regarding your system configuration. If we cannot reproduce your issue, we cannot fix it.
3. A feature request? Please provide a clear rationale for the feature. Be advised that simdjson is a community-based project: you should consider providing help.
4. A documentation issue? Can you suggest an improvement?
If you plan to contribute to simdjson, please read our
* CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
message(STATUS"Building only the library. Advanced users and contributors may want to turn SIMDJSON_DEVELOPER_MODE to ON, e.g., via -D SIMDJSON_DEVELOPER_MODE=ON.")
The simdjson library is an open project written in C++. Contributions are invited. Contributors
agree to the project's license.
We have an extensive list of issues, and contributions toward any of these issues is invited.
Contributions can take the form of code samples, better documentation or design ideas.
In particular, the following contributions are invited:
- The library is focused on performance. Well-documented performance optimization are invited.
- Fixes to known or newly discovered bugs are always welcome. Typically, a bug fix should come with
a test demonstrating that the bug has been fixed.
- The simdjson library is advanced software and maintainability and flexibility are always a
concern. Specific contributions to improve maintainability and flexibility are invited.
We discourage the following types of contributions:
- Code refactoring. We all have our preferences as to how code should be written, but unnecessary
refactoring can waste time and introduce new bugs. If you believe that refactoring is needed, you
first must explain how it helps in concrete terms. Does it improve the performance?
- Applications of new language features for their own sake. Using advanced C++ language constructs
is actually a negative as it may reduce portability (to old compilers, old standard libraries and
systems) and reduce accessibility (to programmers that have not kept up), so it must be offsetted
by clear gains like performance or maintainability. When in doubt, avoid advanced C++ features
(beyond C++11).
- Style formatting. In general, please abstain from reformatting code just to make it look prettier.
Though code formatting is important, it can also be a waste of time if several contributors try to
tweak the code base toward their own preference. Please do not introduce unneeded white-space
changes.
In short, most code changes should either bring new features or better performance. We want to avoid unmotivated code changes.
Specific rules
----------
We have few hard rules, but we have some:
- Printing to standard output or standard error (`stderr`, `stdout`, `std::cerr`, `std::cout`) in the core library is forbidden. This follows from the [Writing R Extensions](https://cran.r-project.org/doc/manuals/R-exts.html) manual which states that "Compiled code should not write to stdout or stderr".
- Calls to `abort()` are forbidden in the core library. This follows from the [Writing R Extensions](https://cran.r-project.org/doc/manuals/R-exts.html) manual which states that "Under no circumstances should your compiled code ever call abort or exit".
- All source code files (.h, .cpp) must be ASCII.
- All C macros introduced in public headers need to be prefixed with either `SIMDJSON_` or `simdjson_`.
- We avoid trailing white space characters within lines. That is, your lines of code should not terminate with unnecessary spaces. Generally, please avoid making unnecessary changes to white-space characters when contributing code.
Tools, tests and benchmarks are not held to these same strict rules.
General Guidelines
----------
Contributors are encouraged to :
- Document their changes. Though we do not enforce a rule regarding code comments, we prefer that non-trivial algorithms and techniques be somewhat documented in the code.
- Follow as much as possible the existing code style. We do not enforce a specific code style, but we prefer consistency. We avoid contractions (isn't, aren't) in the comments.
- Modify as few lines of code as possible when working on an issue. The more lines you modify, the harder it is for your fellow human beings to understand what is going on.
- Tools may report "problems" with the code, but we never delegate programming to tools: if there is a problem with the code, we need to understand it. Thus we will not "fix" code merely to please a static analyzer.
- Provide tests for any new feature. We will not merge a new feature without tests.
- Run before/after benchmarks so that we can appreciate the effect of the changes on the performance.
Pull Requests
--------------
Pull requests are always invited. However, we ask that you follow these guidelines:
- It is wise to discuss your ideas first as part of an issue before you start coding. If you omit this step and code first, be prepared to have your code receive scrutiny and be dropped.
- Users should provide a rationale for their changes. Does it improve performance? Does it add a feature? Does it improve maintainability? Does it fix a bug? This must be explicitly stated as part of the pull request. Do not propose changes based on taste or intuition. We do not delegate programming to tools: that some tool suggested a code change is not reason enough to change the code.
1. When your code improves performance, please document the gains with a benchmark using hard numbers.
2. If your code fixes a bug, please either fix a failing test, or propose a new test.
3. Other types of changes must be clearly motivated. We openly discourage changes with no identifiable benefits.
- Changes should be focused and minimal. You should change as few lines of code as possible. Please do not reformat or touch files needlessly.
- New features must be accompanied by new tests, in general.
- Your code should pass our continuous-integration tests. It is your responsibility to ensure that your proposal pass the tests. We do not merge pull requests that would break our build.
- An exception to this would be changes to non-code files, such as documentation and assets, or trivial changes to code, such as comments, where it is encouraged to explicitly ask for skipping a CI run using the `[skip ci]` prefix in your Pull Request title **and** in the first line of the most recent commit in a push. Example for such a commit: `[skip ci] Fixed typo in power_of_ten's docs`
This benefits the project in such a way that the CI pipeline is not burdened by running jobs on changes that don't change any behavior in the code, which reduces wait times for other Pull Requests that do change behavior and require testing.
If the benefits of your proposed code remain unclear, we may choose to discard your code: that is not an insult, we frequently discard our own code. We may also consider various alternatives and choose another path. Again, that is not an insult or a sign that you have wasted your time.
Style
-----
Our formatting style is inspired by the LLVM style.
The simdjson library is written using the snake case: when a variable or a function is a phrase, each space is replaced by an underscore character, and the first letter of each word written in lowercase. Compile-time constants are written entirely in uppercase with the same underscore convention.
Code of Conduct
---------------
Though we do not have a formal code of conduct, we will not tolerate bullying, bigotry or
intimidation. Everyone is welcome to contribute. If you have concerns, you can raise them privately with the core team members (e.g., D. Lemire, J. Keiser).
We welcome contributions from women and less represented groups. If you need help, please reach out.
Consider the following points when engaging with the project:
- We discourage arguments from authority: ideas are discussed on their own merits and not based on who stated it.
- Be mindful that what you may view as an aggression is maybe merely a difference of opinion or a misunderstanding.
- Be mindful that a collection of small aggressions, even if mild in isolation, can become harmful.
Getting Started Hacking
-----------------------
An overview of simdjson's directory structure, with pointers to architecture and design
considerations and other helpful notes, can be found at [HACKING.md](HACKING.md).
Here is wisdom about how to build, test and run simdjson from within the repository. This is mostly useful for people who plan to contribute simdjson, or maybe study the design.
If you plan to contribute to simdjson, please read our [CONTRIBUTING](https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md) guide.
- [Hacking simdjson](#hacking-simdjson)
- [Build Quickstart](#build-quickstart)
- [Design notes](#design-notes)
- [Developer mode](#developer-mode)
- [Directory Structure and Source](#directory-structure-and-source)
- Stage 1. (Find marks) Identifies quickly structure elements, strings, and so forth. We validate UTF-8 encoding at that stage.
- Stage 2. (Structure building) Involves constructing a "tree" of sort (materialized as a tape) to navigate through the data. Strings and numbers are parsed at this stage.
The role of stage 1 is to identify pseudo-structural characters as quickly as possible. A character is pseudo-structural if and only if:
1. Not enclosed in quotes, AND
2. Is a non-whitespace character, AND
3. Its preceding character is either:
(a) a structural character, OR
(b) whitespace OR
(c) the final quote in a string.
This helps as we redefine some new characters as pseudo-structural such as the characters 1, G, n in the following:
> { "foo" : 1.5, "bar" : 1.5 GEOFF_IS_A_DUMMY bla bla , "baz", null }
Stage 1 also does unicode validation.
Stage 2 handles all of the rest: number parsings, recognizing atoms like true, false, null, and so forth.
Developer mode
--------------
Build system targets that are only useful for developers of the simdjson
library are behind the `SIMDJSON_DEVELOPER_MODE` option. Enabling this option
makes tests, examples, benchmarks and other developer targets available. Not
enabling this option means that you are a consumer of simdjson and thus you
only get the library targets and options.
Developer mode is forced to be on when the `CI` environment variable is set to
a value that CMake recognizes as "on", which is set to `true` in all of the CI
workflows used by simdjson.
Directory Structure and Source
------------------------------
simdjson's source structure, from the top level, looks like this:
* **CMakeLists.txt:** The main build system.
* **include:** User-facing declarations and inline definitions (most user-facing functions are inlined).
* simdjson.h: the `simdjson` namespace. A "main include" that includes files from include/simdjson/. This is equivalent to
the distributed simdjson.h.
* simdjson/*.h: Declarations for public simdjson classes and functions.
* simdjson/*-inl.h: Definitions for public simdjson classes and functions.
* simdjson/internal/*.h: the `simdjson::internal` namespace. Private classes and functions used by the rest of simdjson.
* simdjson/dom.h: the `simdjson::dom` namespace. Includes all public DOM classes.
* simdjson/dom/*.h: Declarations/definitions for individual DOM classes.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere.h: `simdjson::<implementation>` namespace. Common implementation-specific tools like number and string parsing, as well as minification.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/*.h: implementation-specific functions such as , etc.
* simdjson/generic/*.h: the bulk of the actual code, written generically and compiled for each implementation, using functions defined in the implementation's .h files.
* simdjson/generic/dependencies.h: dependencies on common, non-implementation-specific simdjson classes. This will be included before including amalgamated.h.
* simdjson/generic/amalgamated.h: all generic ondemand classes for an implementation.
* simdjson/ondemand.h: the `simdjson::ondemand` namespace. Includes all public ondemand classes.
* simdjson/builtin.h: the `simdjson::builtin` namespace. Aliased to the most universal implementation available.
* simdjson/builtin/ondemand.h: the `simdjson::builtin::ondemand` namespace.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/ondemand.h: the `simdjson::<implementation>::ondemand` namespace. On-Demand compiled for the specific implementation.
* simdjson/generic/ondemand/dependencies.h: dependencies on common, non-implementation-specific simdjson classes. This will be included before including amalgamated.h.
* simdjson/generic/ondemand/amalgamated.h: all generic ondemand classes for an implementation.
* **src:** The source files for non-inlined functionality (e.g. the architecture-specific parser
implementations).
* simdjson.cpp: A "main source" that includes all implementation files from src/. This is
equivalent to the distributed simdjson.cpp.
* *.cpp: other misc. implementations, such as `simdjson::implementation` and the minifier.
* generic/*.h: `simdjson::<implementation>` namespace. Generic implementation of the parser, particularly the `dom_parser_implementation`.
* generic/stage1/*.h: `simdjson::<implementation>::stage1` namespace. Generic implementation of the simd-heavy tokenizer/indexer pass of the simdjson parser. Used for the On-Demand interface
* generic/stage2/*.h: `simdjson::<implementation>::stage2` namespace. Generic implementation of the tape creator, which consumes the index from stage 1 and actually parses numbers and string and such. Used for the DOM interface.
Other important files and directories:
***.github/workflows:** Definitions for GitHub Actions (CI).
***singleheader:** Contains generated `simdjson.h` and `simdjson.cpp` that we release. The files `singleheader/simdjson.h` and `singleheader/simdjson.cpp` should never be edited by hand.
***singleheader/amalgamate.py:** Generates `singleheader/simdjson.h` and `singleheader/simdjson.cpp` for release (python script). If you add a new implementation (e.g., rvv), you need to edit this file (IMPLEMENTATIONS).
***benchmark:** This is where we do benchmarking. Benchmarking is core to every change we make; the
cardinal rule is don't regress performance without knowing exactly why, and what you're trading
for it. Many of our benchmarks are microbenchmarks. We are effectively doing controlled scientific experiments for the purpose of understanding what affects our performance. So we simplify as much as possible. We try to avoid irrelevant factors such as page faults, interrupts, unnecessary system calls. We recommend checking the performance as follows:
```bash
mkdir build
cd build
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build . --config Release
benchmark/dom/parse ../jsonexamples/twitter.json
```
The last line becomes `./benchmark/Release/parse.exe ../jsonexample/twitter.json` under Windows. You may also use Google Benchmark:
The last line becomes `./benchmark/Release/bench_parse_call.exe` under Windows. Under Windows, you can also build with the clang compiler by adding `-T ClangCL` to the call to `cmake ..`: `cmake -T ClangCL ..`.
* **fuzz:** The source for fuzz testing. This lets us explore important edge and middle cases
* **fuzz:** The source for fuzz testing. This lets us explore important edge and middle cases
automatically, and is run in CI.
* **jsonchecker:** A set of JSON files used to check different functionality of the parser.
* **pass*.json:** Files that should pass validation.
* **fail*.json:** Files that should fail validation.
* **jsonchecker/minefield/y_*.json:** Files that should pass validation.
* **jsonchecker/minefield/n_*.json:** Files that should fail validation.
* **jsonexamples:** A wide spread of useful, real-world JSON files with different characteristics
and sizes.
* **test:** The tests are here. basictests.cpp and errortests.cpp are the primary ones.
* **tools:** Source for executables that can be distributed with simdjson. Some examples:
* `json2json mydoc.json` parses the document, constructs a model and then dumps back the result to standard output.
* `json2json -d mydoc.json` parses the document, constructs a model and then dumps model (as a tape) to standard output. The tape format is described in the accompanying file `tape.md`.
* `minify mydoc.json` minifies the JSON document, outputting the result to standard output. Minifying means to remove the unneeded white space characters.
* `jsonpointer mydoc.json <jsonpath> <jsonpath> ... <jsonpath>` parses the document, constructs a model and then processes a series of [JSON Pointer paths](https://tools.ietf.org/html/rfc6901). The result is itself a JSON document.
> **Don't modify the files in singleheader/ directly; these are automatically generated.**
While simdjson distributes just two files from the singleheader/ directory, we *maintain* the code in
multiple files under include/ and src/. The files include/simdjson.h and src/simdjson.cpp are the "spine" for
these, and you can include them as if they were the corresponding singleheader/ files.
Runtime Dispatching
--------------------
A key feature of simdjson is the ability to compile different processing kernels, optimized for specific instruction sets, and to select
the most appropriate kernel at runtime. This ensures that users get the very best performance while still enabling simdjson to run everywhere.
This technique is frequently called runtime dispatching. The simdjson achieves runtime dispatching entirely in C++: we do not assume
that the user is building the code using CMake, for example.
To make runtime dispatching work, it is critical that the code be compiled for the lowest supported processor. In particular, you should
not use flags such as -mavx2, /arch:AVX2 and so forth while compiling simdjson. When you do so, you allow the compiler to use advanced
instructions. In turn, these advanced instructions present in the code may cause a runtime failure if the runtime processor does not
support them. Even a simple loop, compiled with these flags, might generate binary code that only run on advanced processors.
So we compile simdjson for a generic processor. Our users should do the same if they want simdjson's runtime dispatch to work. It is important
to understand that if runtime dispatching does not work, then simdjson will cause crashes on older processors. Of course, if a user chooses
to compile their code for a specific instruction set (e.g., AVX2), they are responsible for the failures if they later run their code
on a processor that does not support AVX2. Yet, if we were to entice these users to do so, we would share the blame: thus we carefully instruct
users to compile their code in a generic way without doing anything to enable advanced instructions.
We only use runtime dispatching on x64 (AMD/Intel) platforms, at the moment. On ARM processors, we would need a standard way to query, at runtime,
the processor for its supported features. We do not know how to do so on ARM systems in general. Thankfully it is not yet a concern: 64-bit ARM
processors are fairly uniform as far as the instruction sets they support.
In all cases, simdjson uses advanced instructions by relying on "intrinsic functions": we do not write assembly code. The intrinsic functions
are special functions that the compiler might recognize and translate into fast code. To make runtime dispatching work, we rely on the fact that
the header providing these instructions
(intrin.h under Visual Studio, x86intrin.h elsewhere) defines all of the intrinsic functions, including those that are not supported
processor.
At this point, we are require to use one of two main strategies.
1. On POSIX systems, the main compilers (LLVM clang, GNU gcc) allow us to use any intrinsic function after including the header, but they fail to inline the resulting instruction if the target processor does not support them. Because we compile for a generic processor, we would not be able to use most intrinsic functions. Thankfully, more recent versions of these compilers allow us to flag a region of code with a specific target, so that we can compile only some of the code with support for advanced instructions. Thus in our C++, one might notice macros like `TARGET_HASWELL`. It is then our responsibility, at runtime, to only run the regions of code (that we call kernels) matching the properties of the runtime processor. The benefit of this approach is that the compiler not only let us use intrinsic functions, but it can also optimize the rest of the code in the kernel with advanced instructions we enabled.
2. Under Visual Studio, the problem is somewhat simpler. Visual Studio will not only provide the intrinsic functions, but it will also allow us to use them. They will compile just fine. It is at runtime that they may cause a crash. So we do not need to mark regions of code for compilation toward advanced processors (e.g., with `TARGET_HASWELL` macros). The downside of the Visual Studio approach is that the compiler is not allowed to use advanced instructions others than those we specify. In principle, this means that Visual Studio has weaker optimization opportunities.
We also handle the special case where a user is compiling using LLVM clang under Windows, [using the Visual Studio toolchain](https://devblogs.microsoft.com/cppblog/clang-llvm-support-in-visual-studio/). If you compile with LLVM clang under Visual Studio, then the header files (intrin.h or x86intrin.h) no longer provides the intrinsic functions that are unsupported by the processor. This appears to be deliberate on the part of the LLVM engineers. With a few lines of code, we handle this scenario just like LLVM clang under a POSIX system, but forcing the inclusion of the specific headers, and rolling our own intrinsic function as needed.
Regenerating Single-Header Files
---------------------------------------
The simdjson.h and simdjson.cpp files in the singleheader directory are not always up-to-date with the rest of the code; they are only ever
systematically regenerated on releases. To ensure you have the latest code, you can regenerate them by running this at the top level:
```bash
mkdir build
cd build
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build . # needed, because currently dependencies do not work fully for the amalgamate target
cmake --build . --target amalgamate
```
You need to have python3 installed on your system.
The amalgamator script `amalgamate.py` generates singleheader/simdjson.h by
reading through include/simdjson.h, copy/pasting each header file into the amalgamated file at the
point it gets included (but only once per header). singleheader/simdjson.cpp is generated from
src/simdjson.cpp the same way, except files under generic/ may be included and copy/pasted multiple
times.
## Usage (CMake on 64-bit platforms like Linux, FreeBSD or macOS)
Requirements: In addition to git, we require a recent version of CMake as well as bash.
1. On macOS, the easiest way to install cmake might be to use [brew](https://brew.sh) and then type
```
brew install cmake
```
2. Under Linux, you might be able to install CMake as follows:
```
apt-get update -qq
apt-get install -y cmake
```
3. On FreeBSD, you might be able to install bash and CMake as follows:
```
pkg update -f
pkg install bash
pkg install cmake
```
You need a recent compiler like clang or gcc. We recommend at least GNU GCC/G++ 7 or LLVM clang 6.
Building: While in the project repository, do the following:
```
mkdir build
cd build
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build .
ctest
```
CMake will build a library. By default, it builds a static library (e.g., libsimdjson.a on Linux).
In some cases, you may want to specify your compiler, especially if the default compiler on your system is too old. You need to tell cmake which compiler you wish to use by setting the CC and CXX variables. Under bash, you can do so with commands such as `export CC=gcc-7` and `export CXX=g++-7`. You can also do it as part of the `cmake` command: `cmake -DCMAKE_CXX_COMPILER=g++ ..`. You may proceed as follows:
```
brew install gcc@8
mkdir build
cd build
export CXX=g++-8 CC=gcc-8
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build .
ctest
```
If your compiler does not default on C++11 support or better you may get failing tests. If so, you may be able to exclude the failing tests by replacing `ctest` with `ctest -E "^quickstart$"`.
Note that the name of directory (`build`) is arbitrary, you can name it as you want (e.g., `buildgcc`) and you can have as many different such directories as you would like (one per configuration).
## Usage (CMake on 64-bit Windows using Visual Studio 2019 or better)
Recent versions of Visual Studio support CMake natively, [please refer to the Visual Studio documentation](https://learn.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-170).
We assume you have a common 64-bit Windows PC with at least Visual Studio 2019.
- Grab the simdjson code from GitHub, e.g., by cloning it using [GitHub Desktop](https://desktop.github.com/).
- Install [CMake](https://cmake.org/download/). When you install it, make sure to ask that `cmake` be made available from the command line. Please choose a recent version of cmake.
- Create a subdirectory within simdjson, such as `build`.
- Using a shell, go to this newly created directory. You can start a shell directly from GitHub Desktop (Repository > Open in Command Prompt).
- Type `cmake ..` in the shell while in the `build` repository.
- This last command (`cmake ...`) created a Visual Studio solution file in the newly created directory (e.g., `simdjson.sln`). Open this file in Visual Studio. You should now be able to build the project and run the tests. For example, in the `Solution Explorer` window (available from the `View` menu), right-click `ALL_BUILD` and select `Build`. To test the code, still in the `Solution Explorer` window, select `RUN_TESTS` and select `Build`.
Though having Visual Studio installed is necessary, one can build simdjson using only cmake commands:
- `mkdir build`
- `cd build`
- `cmake ..`
- `cmake --build . --config Release`
Furthermore, if you have installed LLVM clang on Windows, for example as a component of Visual Studio 2019, you can configure and build simdjson using LLVM clang on Windows using cmake:
- `mkdir build`
- `cd build`
- `cmake -T ClangCL ..`
- `cmake --build . --config Release`
## Various References
- [How to implement atoi using SIMD?](https://stackoverflow.com/questions/35127060/how-to-implement-atoi-using-simd)
- [Parsing JSON is a Minefield 💣](http://seriot.ch/parsing_json.php)
1. It creates a useful library for parsing JSON data quickly.
2. It is a demonstration of the use of SIMD and pipelined programming techniques to perform a complex and irregular task.
These techniques include the use of large registers and SIMD instructions to process large amounts of input data at once,
to hold larger entities than can typically be held in a single General Purpose Register (GPR), and to perform operations
that are not cheap to perform without use of a SIMD unit (for example table lookup using permute instructions).
The other key technique is that the system is designed to minimize the number of unpredictable branches that must be taken
to perform the task. Modern architectures are both wide and deep (4-wide pipelines with ~14 stages are commonplace). A
recent Intel Architecture processor, for example, can perform 3 256-bit SIMD operations or 2 512-bit SIMD operations per
cycle as well as other operations on general purpose registers or with the load/store unit. An incorrectly predicted branch
will clear this pipeline. While it is rare that a programmer can achieve the maximum throughput on a machine, a developer
may be missing the opportunity to carry out 56 operations for each branch miss.
Many code-bases make use of SIMD and deeply pipelined, "non-branchy", processing for regular tasks. Numerical problems
(e.g. "matrix multiply") or simple 'bulk search' tasks (e.g. "count all the occurrences of a given character in a text",
"find the first occurrence of the string 'foo' in a text") frequently use this class of techniques. We are demonstrating
that these techniques can be applied to much more complex and less regular tasks.
## Design:
### Stage 1: SIMD over bytes; bit vector processing over bytes.
The first stage of our processing must identify key points in our input: the 'structural characters' of JSON (curly and
square braces, colon, and comma), the start and end of strings as delineated by double quote characters, other JSON 'atoms'
that are not distinguishable by simple characters (constructs such as "true", "false", "null" and numbers), as well as
discovering these characters and atoms in the presence of both quoting conventions and backslash escaping conventions.
As such we follow the broad outline of the construction of a structural index as set forth in the Mison paper [XXX]; first,
the discovery of odd-length sequences of backslash characters (which will cause quote characters immediately following to
be escaped and not serve their quoting role but instead be literal charaters), second, the discovery of quote pairs (which
cause structural characters within the quote pairs to also be merely literal characters and have no function as structural
characters), then finally the discovery of structural characters not contained without the quote pairs.
We depart from the Mison paper in terms of method and overall design. In terms of method, the Mison paper uses iteration
over bit vectors to discover backslash sequences and quote pairs; we introduce branch-free techniques to discover both of
these properties.
We also make use of our ability to quickly detect whitespace in this early stage. We can use another bit-vector based
transformation to discover locations in our data that follow a structural character or quote or whitespace and are not whitespace. Excluding locations within strings, and the structural characters we have already discovered,
these locations are the only place that we can expect to see the starts of the JSON 'atoms'. These locations are thus
treated as 'structural' ('pseudo-structural characters').
This stage involves either SIMD processing over bytes or the manipulation of bit arrays that have 1 bit corresponding
to 1 byte of input. As such, it can be quite inefficient for some inputs - it is possible to observe dozens of operations
taking place to discover that there are in fact no odd-numbered sequences of backslashes or quotes in a given block of
input. However, this inefficiency on such inputs is balanced by the fact that it costs no more to run this code over
complex structured input, and the alternatives would generally involve running a number of unpredictable branches (for
example, the loop branches in Mison that iterate over bit vectors).
### Stage 2: The transition from "SIMD over bytes" to "indices"
Our structural, pseudo-structural and other 'interesting' characters are relatively rare (TODO: quantify in detail -
it's typically about 1 in 10). As such, continuing to process them as bit vectors will involve manipulating data structures
that are relatively large as well as being fairly unpredictably spaced. We must transform these bitvectors of "interesting"
locations into offsets.
Note that we can examine the character at the offset to discover what the original function of the item in the bitvector
was. While the JSON structural characters and quotes are relatively self-explanatory (although working only with one offset
at a time, we have lost the distinction between opening quotes and closing quotes, something that was available in Stage 1),
it is a quirk of JSON that the legal atoms can all be distinguished from each other by their first character - 't' for
'true', 'f' for 'false', 'n' for 'null' and the character class [0-9-] for numerical values.
Thus, the offset suffices, as long as we retain our original input.
Our current implementation involves a straightforward transformation of bitmaps to indices by use of the 'count trailing
zeros' operation and the well-known operation to clear the lowest set bit. Note that this implementation introduces an
unpredictable branch; unless there is a regular pattern in our bitmaps, we would expect to have at least one branch miss
for each bitmap.
### Stage 3: Operation over indices
This now works over a dual structure.
1. The "state machine", whose role it is to validate the sequence of structural characters and ensure that the input is at least generally structured like valid JSON (after this stage, the only errors permissible should be malformed atoms and numbers). If and only if the "state machine" reached all accept states, then,
2. The "tape machine" will have produced valid output. The tape machine works blindly over characters writing records to tapes. These records create a lean but somewhat traversable linked structure that, for valid inputs, should represent what we need to know about the JSON input.
FIXME: a lot more detail is required on the operation of both these machines.
JSON documents are everywhere on the Internet. Servers spend a lot of time parsing these documents. We want to accelerate the parsing of JSON per se using commonly available SIMD instructions as much as possible while doing full validation (including character encoding).
JSON is everywhere on the Internet. Servers spend a *lot* of time parsing it. We need a fresh
approach. The simdjson library uses commonly available SIMD instructions and microparallel algorithms
to parse JSON 4x faster than RapidJSON and 25x faster than JSON for Modern C++.
* **Fast:** Over 4x faster than commonly used production-grade JSON parsers.
* **Record Breaking Features:** Minify JSON at 6 GB/s, validate UTF-8 at 13 GB/s, NDJSON at 3.5 GB/s.
* **Easy:** First-class, easy to use and carefully documented APIs.
* **Strict:** Full JSON and UTF-8 validation, lossless parsing. Performance with no compromises.
* **Automatic:** Selects a CPU-tailored parser at runtime. No configuration needed.
* **Reliable:** From memory allocation to error handling, simdjson's design avoids surprises.
* **Peer Reviewed:** Our research appears in venues like VLDB Journal, Software: Practice and Experience.
This library is part of the [Awesome Modern C++](https://awesomecpp.com) list.
Table of Contents
-----------------
* [Real-world usage](#real-world-usage)
* [Quick Start](#quick-start)
* [Documentation](#documentation)
* [Godbolt](#godbolt)
* [Performance results](#performance-results)
* [Packages](#packages)
* [Bindings and Ports of simdjson](#bindings-and-ports-of-simdjson)
* [About simdjson](#about-simdjson)
* [Funding](#funding)
* [Contributing to simdjson](#contributing-to-simdjson)
* [License](#license)
## Paper
Real-world usage
----------------
A description of the design and implementation of simdjson appears at https://arxiv.org/abs/1902.08318 and an informal blog post providing some background and context is at https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of-json-per-second/.
Some people [enjoy reading our paper](https://arxiv.org/abs/1902.08318):
simdjson uses three-quarters less instructions than state-of-the-art parser RapidJSON and fifty percent less than sajson. To our knowledge, simdjson is the first fully-validating JSON parser to run at gigabytes per second on commodity processors.
<img src="doc/gbps.png" width="90%">
On a Skylake processor, the parsing speeds (in GB/s) of various processors on the twitter.json file are as follows.
| parser | GB/s |
| ------------------------------------- | ---- |
| simdjson | 2.2 |
| RapidJSON encoding-validation | 0.51 |
| RapidJSON encoding-validation, insitu | 0.71 |
| sajson (insitu, dynamic) | 0.70 |
| sajson (insitu, static) | 0.97 |
| dropbox | 0.14 |
| fastjson | 0.26 |
| gason | 0.85 |
| ultrajson | 0.42 |
| jsmn | 0.28 |
| cJSON | 0.34 |
## Requirements
- We support platforms like Linux or macOS, as well as Windows through Visual Studio 2017 or later.
- A processor with AVX2 (i.e., Intel processors starting with the Haswell microarchitecture released 2013 and AMD processors starting with the Zen microarchitecture released 2017).
- A recent C++ compiler (e.g., GNU GCC or LLVM CLANG or Visual Studio 2017), we assume C++17. GNU GCC 7 or better or LLVM's clang 6 or better.
- Some benchmark scripts assume bash and other common utilities, but they are optional.
## License
This code is made available under the Apache License 2.0.
Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it under the liberal (business-friendly) MIT license.
## Code example
```C
#include"simdjson/jsonparser.h"
/...
constchar*filename=...//
// use whatever means you want to get a string (UTF-8) of your JSON document
padded_stringp=get_corpus(filename);
ParsedJsonpj;
pj.allocateCapacity(p.size());// allocate memory for parsing up to p.size() bytes
constintres=json_parse(p,pj);// do the parsing, return 0 on success
// parsing is done!
if(res!=0){
// You can use the "simdjson/simdjson.h" header to access the error message
It is also possible to use a simpler API if you do not mind having the overhead
of memory allocation with each new JSON document:
```C
#include"simdjson/jsonparser.h"
/...
constchar*filename=...//
padding_stringp=get_corpus(filename);
ParsedJsonpj=build_parsed_json(p);// do the parsing
if(!pj.isValid()){
// something went wrong
}
```
Though the `padded_string` class is recommended for best performance, you can call `json_parse` and `build_parsed_json`, passing a standard `std::string` object.
If you are planning to use simdjson in a product, please work from one of our releases.
```C
#include"simdjson/jsonparser.h"
/...
std::stringmystring=...//
ParsedJsonpj;
pj.allocateCapacity(mystring.size());// allocate memory for parsing up to p.size() bytes
// std::string may not overallocate so a copy will be needed
constintres=json_parse(mystring,pj);// do the parsing, return 0 on success
// parsing is done!
if(res!=0){
// You can use the "simdjson/simdjson.h" header to access the error message
The simdjson library is easily consumable with a single .h and .cpp file.
/...
0. Prerequisites: `g++` (version 7 or better) or `clang++` (version 6 or better), and a 64-bit
system with a command-line shell (e.g., Linux, macOS, freeBSD). We also support programming
environments like Visual Studio and Xcode, but different steps are needed. Users of clang++ may need to specify the C++ version (e.g., `c++ -std=c++17`) since clang++ tends to default on C++98.
1. Pull [simdjson.h](singleheader/simdjson.h) and [simdjson.cpp](singleheader/simdjson.cpp) into a
directory, along with the sample file [twitter.json](jsonexamples/twitter.json). You can download them with the `wget` utility:
std::stringmystring=...//
// std::string may not overallocate so a copy will be needed
ParsedJsonpj=build_parsed_json(mystring);// do the parsing
As needed, the `json_parse` and `build_parsed_json` functions copy the input data to a temporary buffer readable up to SIMDJSON_PADDING bytes beyond the end of the data.
## Usage: easy single-header version
See the "singleheader" repository for a single header version. See the included
file "amalgamation_demo.cpp" for usage. This requires no specific build system: just
copy the files in your project in your include path. You can then include them quite simply:
```C
```cpp
#include <iostream>
#include "simdjson.h"
#include"simdjson.cpp"
intmain(intargc,char*argv[]){
constchar*filename=argv[1];
padded_stringp=get_corpus(filename);
ParsedJsonpj=build_parsed_json(p);// do the parsing
* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/7G5qE4sr9)
* [simdjson examples with errors without exceptions](https://godbolt.org/z/e9dWb9E4v)
```
make
make test
```
Performance results
-------------------
To run benchmarks:
The simdjson library uses three-quarters less instructions than state-of-the-art parser [RapidJSON](https://rapidjson.org). To our knowledge, simdjson is the first fully-validating JSON parser
to run at [gigabytes per second](https://en.wikipedia.org/wiki/Gigabyte) (GB/s) on commodity processors. It can parse millions of JSON documents per second on a single core.
```
make parse
./parse jsonexamples/twitter.json
```
The following figure represents parsing speed in GB/s for parsing various files
on an Intel Skylake processor (3.4 GHz) using the GNU GCC 10 compiler (with the -O3 flag).
We compare against the best and fastest C++ libraries on benchmarks that load and process the data.
The simdjson library offers full unicode ([UTF-8](https://en.wikipedia.org/wiki/UTF-8)) validation and exact
number parsing.
Under Linux, the `parse` command gives a detailed analysis of the performance counters.
<img src="doc/rome.png" width="60%">
To run comparative benchmarks (with other parsers):
The simdjson library offers high speed whether it processes tiny files (e.g., 300 bytes)
or larger files (e.g., 3MB). The following plot presents parsing
speed for [synthetic files over various sizes generated with a script](https://github.com/simdjson/simdjson_experiments_vldb2019/blob/master/experiments/growing/gen.py) on a 3.4 GHz Skylake processor (GNU GCC 9, -O3).
```
make benchmark
```
<img src="doc/growing.png" width="60%">
## Usage (CMake on platforms like Linux or macOS)
[All our experiments are reproducible](https://github.com/simdjson/simdjson_experiments_vldb2019).
Requirements: We require a recent version of cmake. On macOS, the easiest way to install cmake might be to use [brew](https://brew.sh) and then type
```
brew install cmake
```
For NDJSON files, we can exceed 3 GB/s with [our multithreaded parsing functions](https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md).
There is an [equivalent brew on Linux which works the same way as well](https://linuxbrew.sh).
You need a recent compiler like clang or gcc. We recommend at least GNU GCC/G++ 7 or LLVM clang 6. For example, you can install a recent compiler with brew:
Optional: You need to tell cmake which compiler you wish to use by setting the CC and CXX variables. Under bash, you can do so with commands such as `export CC=gcc-7` and `export CXX=g++-7`.
Building: While in the project repository, do the following:
```
mkdir build
cd build
cmake ..
make
make test
```
CMake will build a library. By default, it builds a shared library (e.g., libsimdjson.so on Linux).
You can build a static library:
```
mkdir buildstatic
cd buildstatic
cmake -DSIMDJSON_BUILD_STATIC=ON ..
make
make test
```
In some cases, you may want to specify your compiler, especially if the default compiler on your system is too old. You may proceed as follows:
```
brew install gcc@8
mkdir build
cd build
export CXX=g++-8 CC=gcc-8
cmake ..
make
make test
```
## Usage (CMake on Windows using Visual Studio)
We assume you have a common Windows PC with at least Visual Studio 2017 and an x64 processor with AVX2 support (2013 Intel Haswell or later).
- Grab the simdjson code from GitHub, e.g., by cloning it using [GitHub Desktop](https://desktop.github.com/).
- Install [CMake](https://cmake.org/download/). When you install it, make sure to ask that `cmake` be made available from the command line. Please choose a recent version of cmake.
- Create a subdirectory within simdjson, such as `VisualStudio`.
- Using a shell, go to this newly created directory.
- Type `cmake -DCMAKE_GENERATOR_PLATFORM=x64 ..` in the shell while in the `VisualStudio` repository. (Alternatively, if you want to build a DLL, you may use the command line `cmake -DCMAKE_GENERATOR_PLATFORM=x64 -DSIMDJSON_BUILD_STATIC=OFF ..`.)
- This last command created a Visual Studio solution file in the newly created directory (e.g., `simdjson.sln`). Open this file in Visual Studio. You should now be able to build the project and run the tests. For example, in the `Solution Explorer` window (available from the `View` menu), right-click `ALL_BUILD` and select `Build`. To test the code, still in the `Solution Explorer` window, select `RUN_TESTS` and select `Build`.
## Usage (Using `vcpkg` on Windows, Linux and MacOS)
[vcpkg](https://github.com/Microsoft/vcpkg) users on Windows, Linux and MacOS can download and install `simdjson` with one single command from their favorite shell.
On Linux and MacOS:
```
$ ./vcpkg install simdjson
```
will build and install `simdjson` as a static library.
On Windows (64-bit):
```
.\vcpkg.exe install simdjson:x64-windows
```
will build and install `simdjson` as a shared library.
```
.\vcpkg.exe install simdjson:x64-windows-static
```
will build and install `simdjson` as a static library.
These commands will also print out instructions on how to use the library from MSBuild or CMake-based projects.
If you find the version of `simdjson` shipped with `vcpkg` is out-of-date, feel free to report it to `vcpkg` community either by submiting an issue or by creating a PR.
## Tools
-`json2json mydoc.json` parses the document, constructs a model and then dumps back the result to standard output.
-`json2json -d mydoc.json` parses the document, constructs a model and then dumps model (as a tape) to standard output. The tape format is described in the accompanying file `tape.md`.
-`minify mydoc.json` minifies the JSON document, outputting the result to standard output. Minifying means to remove the unneeded white space characters.
## Scope
We provide a fast parser, that fully validates an input according to various specifications.
The parser builds a useful immutable (read-only) DOM (document-object model) which can be later accessed.
To simplify the engineering, we make some assumptions.
- We support UTF-8 (and thus ASCII), nothing else (no Latin, no UTF-16). We do not believe this is a genuine limitation, because we do not think there is any serious application that needs to process JSON data without an ASCII or UTF-8 encoding.
- All strings in the JSON document may have up to 4294967295 bytes in UTF-8 (4GB). To enforce this constraint, we refuse to parse a document that contains more than 4294967295 bytes (4GB). This should accommodate most JSON documents.
- We assume AVX2 support, which is available in all recent mainstream x86 processors produced by AMD and Intel. No support for non-x86 processors is included, though it can be done. We plan to support ARM processors (help is invited).
- In cases of failure, we report a failure without any indication to the nature of the problem. (This can be easily improved without affecting performance.)
- As allowed by the specification, we allow repeated keys within an object (other parsers like sajson do the same).
- Performance is optimized for JSON documents spanning at least a tens kilobytes up to many megabytes: the performance issues with having to parse many tiny JSON documents or one truly enormous JSON document are different.
_We do not aim to provide a general-purpose JSON library._ A library like RapidJSON offers much more than just parsing, it helps you generate JSON and offers various other convenient functions. We merely parse the document.
## Features
- The input string is unmodified. (Parsers like sajson and RapidJSON use the input string as a buffer.)
- We parse integers and floating-point numbers as separate types which allows us to support large 64-bit integers in [-9223372036854775808,9223372036854775808), like a Java `long` or a C/C++ `long long`. Among the parsers that differentiate between integers and floating-point numbers, not all support 64-bit integers. (For example, sajson rejects JSON files with integers larger than or equal to 2147483648. RapidJSON will parse a file containing an overly long integer like 18446744073709551616 as a floating-point number.) When we cannot represent exactly an integer as a signed 64-bit value, we reject the JSON document.
- We do full UTF-8 validation as part of the parsing. (Parsers like fastjson, gason and dropbox json11 do not do UTF-8 validation.)
- We fully validate the numbers. (Parsers like gason and ultranjson will accept `[0e+]` as valid JSON.)
- We validate string content for unescaped characters. (Parsers like fastjson and ultrajson accept unescaped line breaks and tabs in strings.)
## Architecture
The parser works in two stages:
- Stage 1. (Find marks) Identifies quickly structure elements, strings, and so forth. We validate UTF-8 encoding at that stage.
- Stage 2. (Structure building) Involves constructing a "tree" of sort (materialized as a tape) to navigate through the data. Strings and numbers are parsed at this stage.
## Navigating the parsed document
Here is a code sample to dump back the parsed JSON to a string:
```c
ParsedJson::iteratorpjh(pj);
if(!pjh.isOk()){
std::cerr<<" Could not iterate parsed result. "<<std::endl;
returnEXIT_FAILURE;
}
compute_dump(pj);
//
// where compute_dump is :
voidcompute_dump(ParsedJson::iterator&pjh){
if(pjh.is_object()){
std::cout<<"{";
if(pjh.down()){
pjh.print(std::cout);// must be a string
std::cout<<":";
pjh.next();
compute_dump(pjh);// let us recurse
while(pjh.next()){
std::cout<<",";
pjh.print(std::cout);
std::cout<<":";
pjh.next();
compute_dump(pjh);// let us recurse
}
pjh.up();
}
std::cout<<"}";
}elseif(pjh.is_array()){
std::cout<<"[";
if(pjh.down()){
compute_dump(pjh);// let us recurse
while(pjh.next()){
std::cout<<",";
compute_dump(pjh);// let us recurse
}
pjh.up();
}
std::cout<<"]";
}else{
pjh.print(std::cout);// just print the lone value
}
}
```
The following function will find all user.id integers:
If you want to see how a wide range of parsers validate a given JSON file:
```
make allparserscheckfile
./allparserscheckfile myfile.json
```
For performance comparisons:
```
make parsingcompetition
./parsingcompetition myfile.json
```
For broader comparisons:
```
make allparsingcompetition
./allparsingcompetition myfile.json
```
## Other programming languages
Bindings and Ports of simdjson
------------------------------
We distinguish between "bindings" (which just wrap the C++ code) and a port to another programming language (which reimplements everything).
- [ZippyJSON](https://github.com/michaeleisel/zippyjson): Swift bindings for the simdjson project.
- [libpy_simdjson](https://github.com/gerrymanoim/libpy_simdjson/): high-speed Python bindings for simdjson using [libpy](https://github.com/quantopian/libpy).
- [pysimdjson](https://github.com/TkTech/pysimdjson): Python bindings for the simdjson project.
- [simdjson-rs](https://github.com/Licenser/simdjson-rs): Rust port
- [SimdJsonSharp](https://github.com/EgorBo/SimdJsonSharp): C# version for .NET Core (bindings and full port)
- [cysimdjson](https://github.com/TeskaLabs/cysimdjson): high-speed Python bindings for the simdjson project.
-John Keiser, Daniel Lemire, [On-Demand JSON: A Better Way to Parse Documents?](http://arxiv.org/abs/2312.17149), Software: Practice and Experience 54 (6), 2024.
Validating UTF-8 takes no more than 0.7 cycles per byte:
Some people [enjoy reading the first (2019) simdjson paper](https://arxiv.org/abs/1902.08318): A description of the design
and implementation of simdjson is in our research article:
- Geoff Langdale, Daniel Lemire, [Parsing Gigabytes of JSON per Second](https://arxiv.org/abs/1902.08318), VLDB Journal 28 (6), 2019.
We have an in-depth paper focused on the UTF-8 validation:
## Remarks on JSON parsing
- John Keiser, Daniel Lemire, [Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090), Software: Practice & Experience 51 (5), 2021.
- The JSON spec defines what a JSON parser is:
> A JSON parser transforms a JSON text into another representation. A JSON parser MUST accept all texts that conform to the JSON grammar. A JSON parser MAY accept non-JSON forms or extensions. An implementation may set limits on the size of texts that it accepts. An implementation may set limits on the maximum depth of nesting. An implementation may set limits on the range and precision of numbers. An implementation may set limits on the length and character contents of strings.
We also have an informal [blog post providing some background and context](https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of-json-per-second/).
* JSON is not JavaScript:
For the video inclined, <br />
[](http://www.youtube.com/watch?v=wlvKAT7SZIQ)<br />
(It was the best voted talk, we're kinda proud of it.)
> All JSON is Javascript but NOT all Javascript is JSON. So {property:1} is invalid because property does not have double quotes around it. {'property':1} is also invalid, because it's single quoted while the only thing that can placate the JSON specification is double quoting. JSON is even fussy enough that {"property":.1} is invalid too, because you should have of course written {"property":0.1}. Also, don't even think about having comments or semicolons, you guessed it: they're invalid. (credit:https://github.com/elzr/vim-json)
Funding
-------
* The structural characters are:
begin-array = [ left square bracket
begin-object = { left curly bracket
end-array = ] right square bracket
end-object = } right curly bracket
name-separator = : colon
value-separator = , comma
### Pseudo-structural elements
A character is pseudo-structural if and only if:
1. Not enclosed in quotes, AND
2. Is a non-whitespace character, AND
3. Its preceding character is either:
(a) a structural character, OR
(b) whitespace.
This helps as we redefine some new characters as pseudo-structural such as the characters 1, G, n in the following:
> { "foo" : 1.5, "bar" : 1.5 GEOFF_IS_A_DUMMY bla bla , "baz", null }
## Academic References
- T.Mühlbauer, W.Rödiger, R.Seilbeck, A.Reiser, A.Kemper, and T.Neumann. Instant loading for main memory databases. PVLDB, 6(14):1702–1713, 2013. (SIMD-based CSV parsing)
- Lu, Yifan, et al. "Tree structured data processing on GPUs." Cloud Computing, Data Science & Engineering-Confluence, 2017 7th International Conference on. IEEE, 2017.
- Sidhu, Reetinder. "High throughput, tree automata based XML processing using FPGAs." Field-Programmable Technology (FPT), 2013 International Conference on. IEEE, 2013.
- Dai, Zefu, Nick Ni, and Jianwen Zhu. "A 1 cycle-per-byte XML parsing accelerator." Proceedings of the 18th annual ACM/SIGDA international symposium on Field programmable gate arrays. ACM, 2010.
- Lin, Dan, et al. "Parabix: Boosting the efficiency of text processing on commodity processors." High Performance Computer Architecture (HPCA), 2012 IEEE 18th International Symposium on. IEEE, 2012. http://parabix.costar.sfu.ca/export/1783/docs/HPCA2012/final_ieee/final.pdf
- Deshmukh, V. M., and G. R. Bamnote. "An empirical evaluation of optimization parameters in XML parsing for performance enhancement." Computer, Communication and Control (IC4), 2015 International Conference on. IEEE, 2015.
- Moussalli, Roger, et al. "Efficient XML Path Filtering Using GPUs." ADMS@ VLDB. 2011.
- Jianliang, Ma, et al. "Parallel speculative dom-based XML parser." High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems (HPCC-ICESS), 2012 IEEE 14th International Conference on. IEEE, 2012.
- Li, Y., Katsipoulakis, N.R., Chandramouli, B., Goldstein, J. and Kossmann, D., 2017. Mison: a fast JSON parser for data analytics. Proceedings of the VLDB Endowment, 10(10), pp.1118-1129. http://www.vldb.org/pvldb/vol10/p1118-li.pdf
- Cameron, Robert D., et al. "Parallel scanning with bitstream addition: An xml case study." European Conference on Parallel Processing. Springer, Berlin, Heidelberg, 2011.
- Cameron, Robert D., Kenneth S. Herdy, and Dan Lin. "High performance XML parsing using parallel bit stream technology." Proceedings of the 2008 conference of the center for advanced studies on collaborative research: meeting of minds. ACM, 2008.
- Shah, Bhavik, et al. "A data parallel algorithm for XML DOM parsing." International XML Database Symposium. Springer, Berlin, Heidelberg, 2009.
- Cameron, Robert D., and Dan Lin. "Architectural support for SWAR text processing with parallel bit streams: the inductive doubling principle." ACM Sigplan Notices. Vol. 44. No. 3. ACM, 2009.
- Amagasa, Toshiyuki, Mana Seino, and Hiroyuki Kitagawa. "Energy-Efficient XML Stream Processing through Element-Skipping Parsing." Database and Expert Systems Applications (DEXA), 2013 24th International Workshop on. IEEE, 2013.
- Medforth, Nigel Woodland. "icXML: Accelerating Xerces-C 3.1. 1 using the Parabix Framework." (2013).
- Zhang, Qiang Scott. Embedding Parallel Bit Stream Technology Into Expat. Diss. Simon Fraser University, 2010.
- Cameron, Robert D., et al. "Fast Regular Expression Matching with Bit-parallel Data Streams."
- Lin, Dan. Bits filter: a high-performance multiple string pattern matching algorithm for malware detection. Diss. School of Computing Science-Simon Fraser University, 2010.
- Yang, Shiyang. Validation of XML Document Based on Parallel Bit Stream Technology. Diss. Applied Sciences: School of Computing Science, 2013.
- N. Nakasato, "Implementation of a parallel tree method on a GPU", Journal of Computational Science, vol. 3, no. 3, pp. 132-141, 2012.
The work is supported by the Natural Sciences and Engineering Research Council of Canada under grants
Head over to [CONTRIBUTING.md](CONTRIBUTING.md) for information on contributing to simdjson, and
[HACKING.md](HACKING.md) for information on source, building, and architecture/design.
Stars
------
[](https://www.star-history.com/#simdjson/simdjson&Date)
License
-------
This code is made available under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) as well as under the MIT License. As a user, you can pick the license you prefer.
Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it is under the liberal (business-friendly) MIT license.
For compilers that do not support [C++17](https://en.wikipedia.org/wiki/C%2B%2B17), we bundle the string-view library which is published under the [Boost license](http://www.boost.org/LICENSE_1_0.txt). Like the Apache license, the Boost license is a permissive license allowing commercial redistribution.
For efficient number serialization, we bundle Florian Loitsch's implementation of the Grisu2 algorithm for binary to decimal floating-point numbers. The implementation was slightly modified by JSON for Modern C++ library. Both Florian Loitsch's implementation and JSON for Modern C++ are provided under the MIT license.
For runtime dispatching, we use some code from the PyTorch project licensed under 3-clause BSD.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Daniel Lemire