Two fixes to the ablation logic:
1. Fix incorrect branch hint in capacity_check() overflow detection
(json_string_builder-inl.h:365): simdjson_likely was used on the
overflow check `position + upcoming_bytes < position`, but overflow
is rare so this should be simdjson_unlikely. The comment itself says
"most of the time there is no overflow". This was causing the branch
predictor to optimize for the wrong path.
2. Add missing ABLATION_NO_CONSTEVAL guard in extract_from()
(json_builder.h:332): consteval_to_quoted_escaped was being used
without an ablation guard, for consistency with all other call sites.
Updated ablation results with both fixes:
| Configuration | CITM (MB/s) | Impact | Twitter (MB/s) | Impact |
|------------------|-------------|--------|----------------|--------|
| Baseline | 3001.10 | - | 5651.89 | - |
| NO_BRANCH_HINTS | 2443.67 | -19% | 4651.78 | -18% |
| NO_SIMD_ESCAPING | 2904.34 | -3% | 1403.36 | -75% |
| NO_CONSTEVAL | 1796.05 | -40% | 3542.77 | -37% |
The ablation study was showing inconsistent results for the consteval
optimization (-9% CITM, -1% Twitter) because the ABLATION_NO_CONSTEVAL
guards were missing from two critical code paths:
1. atom() for structs (line 94-102): The consteval optimization for
pre-computing escaped/quoted field names was not being disabled.
2. atom() for enums (line 136-150): The consteval optimization for
pre-computing escaped/quoted enum string values was not being disabled.
With these guards added, the ablation study now correctly shows the full
impact of the consteval optimization:
- CITM: -47% (was -9%)
- Twitter: -38% (was -1%)
This confirms the consteval optimization provides ~2x performance
improvement by pre-computing escaped/quoted strings at compile time
via std::define_static_string and consteval_to_quoted_escaped.
* perf(serialization): SIMD-accelerated string escape position finding
Profiling revealed that string serialization was performing redundant
scanning: first calling fast_needs_escaping() (SIMD scan to check IF
escape needed), then find_next_json_quotable_character() (scalar
byte-by-byte scan to find WHERE).
Profile data from Twitter benchmark showed:
- 54.76% time in atom<std::string> (string serialization)
- 24.85% time in find_next_json_quotable_character (scalar position finding)
This optimization unifies both operations into a single SIMD pass that
directly locates the first quotable character position:
- NEON (ARM64): Uses vceqq_u8/vcltq_u8 for character detection, then
extracts position via __builtin_ctzll on 64-bit vector lanes
- SSE2 (x86-64): Uses _mm_cmpeq_epi8/_mm_subs_epu8 for detection, then
_mm_movemask_epi8 + __builtin_ctz for position extraction
The write_string_escaped function now uses the position finder directly,
eliminating the separate fast_needs_escaping check.
Benchmark results (ARM64, Apple Silicon via Docker with p2996 clang):
- Twitter (string-heavy): 4330 -> 5723 MB/s (+32%)
- CITM (numeric-heavy): ~neutral (expected, few strings)
* perf(serialization): batch integer formatting with 4-digit processing
Profiling with perf annotate revealed that the integer-to-string
conversion loop was a significant hotspot in numeric-heavy workloads.
The original implementation processed 2 digits per iteration:
while (pv >= 100) {
memcpy(write_pointer - 1, &decimal_table[(pv % 100) * 2], 2);
write_pointer -= 2;
pv /= 100;
}
Profile data from CITM benchmark showed:
- 31.20% time in atom<unsigned long> (integer formatting)
- 39.07% of integer formatting time in the 2-byte store instruction
(sturh on ARM64)
- CITM integers average 8.8 digits, meaning 4+ store operations per number
This optimization processes 4 digits per iteration, reducing both store
operations and division count by approximately half for large numbers:
while (pv >= 10000) {
q = pv / 10000;
r = pv % 10000;
r_hi = r / 100; // High 2 digits
r_lo = r % 100; // Low 2 digits
memcpy(write_pointer - 1, &decimal_table[r_lo * 2], 2);
memcpy(write_pointer - 3, &decimal_table[r_hi * 2], 2);
write_pointer -= 4;
pv = q;
}
The division by 10000 compiles to an efficient multiply-high instruction
(umulh on ARM64). Applied to both unsigned and signed integer paths.
Benchmark results (ARM64, Apple Silicon via Docker with p2996 clang):
- Twitter: ~neutral (few integers)
- CITM (numeric-heavy): 2912 -> 3086 MB/s (+6%)
* fix(build): add MSVC compatibility for bit manipulation intrinsics
MSVC does not have __builtin_ctz/__builtin_ctzll. Use _BitScanForward
and _BitScanForward64 from <intrin.h> on MSVC instead.
This fixes the build on all Windows configurations (x64, ARM64, Win32).
* refactor: clean up comments to be implementation-focused
Remove references to specific benchmarks and previous implementations
from code comments. Comments now describe what the code does rather
than historical context.
* Add FracturedJson formatting support for DOM serialization
Implements FracturedJson formatting as requested in issue #2576.
FracturedJson produces human-readable yet compact JSON output by
intelligently choosing between different layout strategies based on
content complexity, length, and structure similarity.
Key features:
- Four layout modes: inline, compact multiline, table, and expanded
- Structure analysis pass to compute metrics before formatting
- Table formatting for arrays of similar objects with column alignment
- Configurable options for line length, indentation, padding, etc.
New files:
- fractured_json.h: Public API with fractured_json_options struct
- fractured_json-inl.h: Implementation (~1000 lines)
- json_structure_analyzer.h: Structure analysis for layout decisions
- fractured_formatter.h: Formatter class using CRTP pattern
Usage:
dom::parser parser;
element doc = parser.parse(json_string);
std::cout << fractured_json(doc) << std::endl;
// Or with custom options:
fractured_json_options opts;
opts.indent_spaces = 2;
std::cout << fractured_json(doc, opts) << std::endl;
// Or format any JSON string (useful with reflection API):
auto formatted = fractured_json_string(minified_json);
Resolves#2576
* Add comprehensive tests for FracturedJson formatter
Adds 27 test cases covering all aspects of the FracturedJson formatter:
Core functionality tests (13):
- Roundtrip parsing verification
- Inline formatting for simple arrays and objects
- Expanded formatting for complex nested structures
- Compact multiline arrays with configurable items per line
- Table formatting for uniform arrays of objects
- Empty container handling
- All scalar types (string, int, uint, double, bool, null)
- String escaping (quotes, backslashes, control characters)
- Custom indentation options
- Deep nesting (10+ levels)
- Mixed type arrays
Edge case tests (11):
- Unicode strings (Chinese, emoji, Arabic, Russian, accented chars)
- Boundary numbers (INT64_MIN/MAX, UINT64_MAX, DBL_MIN/MAX)
- Nested arrays (arrays of arrays)
- Empty string values
- Keys with special characters (spaces, quotes, colons, etc.)
- Non-uniform arrays (should not trigger table mode)
- Very long strings (500+ chars)
- Large arrays (100 elements)
- Reflection API workflow simulation
- Control characters (tab, newline, CR, null)
- Single element containers
Option tests (3):
- Disable compact multiline mode
- Disable table format mode
- Disable all padding options
* Add FracturedJson integration with builder/reflection API
Extends FracturedJson to work seamlessly with the builder API, enabling
formatted output directly from C++ structs using static reflection.
New functions:
- to_fractured_json_string(obj, opts) - serialize struct to formatted JSON
- to_fractured_json(obj, output, opts) - same with output parameter
- extract_fractured_json<fields...>(obj, opts) - format only specific fields
These functions combine the builder's reflection-based serialization with
FracturedJson formatting in a single convenient call:
struct User { int id; std::string name; bool active; };
User user{1, "Alice", true};
// Minified output (existing):
auto minified = to_json_string(user);
// {"id":1,"name":"Alice","active":true}
// Formatted output (new):
auto formatted = to_fractured_json_string(user);
// { "id": 1, "name": "Alice", "active": true }
// Partial extraction with formatting:
auto partial = extract_fractured_json<"id", "name">(user);
// { "id": 1, "name": "Alice" }
New files:
- generic/builder/fractured_json_builder.h - builder integration
- tests/builder/static_reflection_fractured_json_tests.cpp - 7 tests
* Fix INT64_MIN overflow and implement table_similarity_threshold
- Fix undefined behavior when negating INT64_MIN in estimate_number_length()
and measure_value_length() by returning 20 (the exact length of the
string representation) directly
- Actually use table_similarity_threshold in check_array_uniformity() by
calling compute_object_similarity() to compare objects against the first
object in the array
* Fix -Werror=effc++ member initialization warnings
Initialize all member variables in member initialization lists to
satisfy GCC's -Werror=effc++ flag:
- element_metrics::common_keys - add {} default initializer
- structure_analyzer - add default constructor with member init list
- fractured_formatter - add column_widths_{} to constructor
- fractured_string_builder - add analyzer_{} to constructor
* Add Rule of Five to structure_analyzer class
The class has a pointer member (current_opts_) which triggers
-Werror=effc++ requiring explicit copy/move operations. Delete
copy operations (class shouldn't be copied due to cache) and
default move operations.
* Fix Windows build: wrap std::max to avoid macro conflict
Windows.h defines max/min macros that interfere with std::max/std::min.
Wrapping in parentheses as (std::max)(...) prevents macro expansion.
* Fix GCC 15 false positive -Wfree-nonheap-object warning
GCC 15 on MINGW64 gives a false positive warning in parser_moving_parser()
when the std::vector<std::string> goes out of scope. Suppress this
specific warning with a pragma for GCC builds.
* Fix metrics cache key bug by passing metrics through recursion
The cache was using element addresses as keys, but dom::element objects
are lightweight wrappers that get copied during iteration, causing
different addresses between analysis and formatting phases. This resulted
in cache misses and fallback to empty metrics.
Solution: Store child metrics in the element_metrics struct and pass
them through recursive calls, eliminating the need for address-based
caching entirely.
Changes:
- Add children vector to element_metrics for hierarchical metrics
- Remove metrics_cache_ and related get_metrics/has_metrics methods
- Update all format functions to accept and pass child metrics
- Add public analyze_array/analyze_object overloads for standalone use
* Add ignore patterns for Node.js, Rust, and generated files
Add entries for node_modules, package-lock.json, Rust target
directories, local ablation artifacts, and generated documentation
files.
* Refactor: extract analyze_scalar helper to reduce code duplication
Extract common scalar type handling (STRING, INT64, UINT64, DOUBLE,
BOOL, NULL_VALUE) into a dedicated analyze_scalar method. Each scalar
type shares the same initialization pattern for complexity, child_count,
can_inline, and recommended_layout.
Also simplify boolean formatting in format_scalar to use ternary operator.
* Fix formatting and duplicate error message in amalgamate.py
Reformat cramped is_amalgamator condition to multi-line for readability.
Fix duplicate error message text in _included_filename_root and use
correct variable name (relative_root instead of root).
* Refactor: add count_newlines helper in fractured_json tests
Extract repeated newline counting loop into a reusable static helper
function, used by inline_array_test, inline_object_test, and
expanded_test.
* Revert "Add ignore patterns for Node.js, Rust, and generated files"
This reverts commit 4760ea7cd0.
* various minor changes
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* work on the ondemand iterators
* guarding two SIMDJSON_ASSUME
* simplify following @jkeiser's comment
* adding safety rails to the iterators
* silencing a warning.
* updating the amalgamation files
If you provide a custom serializer for range types it is currently never used due to the requires clause for string_builder::append with ranges is overly broad
* Adding compile time json path
* using string_view
* Adding support for compile-time json pointer as well.
* Removing unnecessary comment
* Tests now working, still will re-review.
* Adding documentation on the compile-time json path/pointer parsing feature.
* Adding benchmark showing the significant performance advantage of using compiled paths whenever you have them a priori.
* going for JSONPath (correct wording).
* minor update (mostly doc)
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Resolves a build failure on the loong64 architecture caused by a narrowing conversion error.
The compiler, with the -Werror=narrowing flag, was flagging the implicit conversion from 'int' (the return
type of to_bitmask()) to 'uint64_t'.
This is fixed by adding an explicit static_cast to uint64_t in include/simdjson/lsx/stringparsing_defs.h.
Signed-off-by: Zhou Qiankang <wszqkzqk@qq.com>
* Using iterators instead of subscript operators and size. This helps us work with a broader range of containers.
* Adding list test
* Using std::ranges::input_range<T> as suggested by moisrex
* modifying simdjson::from to avoid exceptions when needed.
* moved the function
* moving the strings.
* more moving around
* updating cmake version in ci
* This is a small reorg of the new convert code so that we only expose 'simdjson::from'.
This can be changed in a future release, but we don't want our users to start depending
on code that we might need to change.
* marking simdjson::from as experimental
* updating tests to match recent changes
* improving the documentation of raw json access
* minor fix
* documentation update
* guarding for exceptions
* fixing exception issue
* fix test
* update.
* saving comments
* more technical fixes
* correcting path in ci test
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* introducing a thread-local parser
* adding functionality to release the memory
* some more documentation.
* fixing build
* adding benchmarks for 'from'
* generalizing the code somewhat.
* adding tests, fixing the benchmark (now with arrays and streams), and a
minor update to document_stream
* adding missing files (I forgot to check them).
* We cannot use [[nodiscard]] without guarding it, it is C++17
* fixing the cmake
* marking it as experimental
* removing ranges support (it is too experimental)
* putting back documentation.
* guarding SIMDJSON_CONSTEVAL more carefully.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* clang format
* added `chars()` method
* implemented vector with small buffer instead of `std::vector`
* added missing <utility> header
* minor fixes
---------
Co-authored-by: Pavel Novikov <dev-ape@yandex.ru>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
The test was failing in CI with g++-13 because it only handled the
exception case. However, the array() method is marked noexcept and
returns a simdjson_result that may contain an error code instead of
throwing an exception.
This fix checks for both cases:
1. If array_result.error() is not SUCCESS, verify it's INCORRECT_TYPE
2. If no error is returned initially, the exception may be thrown when
iterating over the result
This ensures the test passes regardless of whether the error is
reported via error code or exception.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added (void)position; to suppress unused parameter warning when compiling with SIMDJSON_CLANG_VISUAL_STUDIO defined, where the position parameter isn't used in the SIMDJSON_ASSUME statements.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Resolved conflicts by regenerating the amalgamated single-header
files (simdjson.h, simdjson.cpp, and singleheader.zip) using the
amalgamate.py script after merging latest changes from master.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit addresses multiple issues:
1. Fixed -Werror=effc++ warnings by using #pragma to disable the
warning for constructors that cannot initialize all members in
the member initialization list due to error handling requirements.
2. Added proper error tracking (m_error member) to handle cases where
document initialization fails, preventing segfaults when using
invalid documents.
3. Fixed lifetime issues in tests where temporary auto_parser objects
were being used, causing dangling references. Tests now properly
store the parser object before using it.
4. Simplified range adaptor tests that were expecting features not
yet implemented in simdjson's ondemand API.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The issue was that we were trying to initialize ondemand::document
directly from simdjson_result<ondemand::document> in the member
initializer list. This caused a segfault in C++20 builds.
The fix explicitly handles the simdjson_result in the constructor
body, checking for errors and using value_unsafe() to extract the
document. This avoids potential issues with implicit conversions
and ensures proper error handling.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The compiler was warning about member initialization order mismatch.
C++ initializes members in the order they are declared in the class,
not the order they appear in the initializer list.
Fixed by reordering member declarations to match the initialization
order needed: m_doc must be initialized before m_parser since we
need to call parser.iterate() before moving the parser.
This fixes the -Werror=reorder compilation error in CI.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The issue was that we were calling m_parser.iterate() after moving
the parser, which could leave it in an invalid state. In C++20,
this might behave differently than C++17.
Fixed by reordering the member initializer list to call
parser.iterate() BEFORE moving the parser into m_parser.
This ensures the document is created while the parser is still valid.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The issue might be related to how brace initialization vs parentheses
initialization handles implicit conversion from simdjson_result<document>
to document. This could be compiler-specific behavior.
Using parentheses initialization to ensure the conversion operator
is called properly.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The issue was that the auto_parser constructor was using implicit
conversion from simdjson_result<document> to document, which could
cause issues with certain implementations (particularly fallback).
Changed to use value_unsafe() to explicitly extract the document
after the parser is fully initialized. This ensures the document
is in a valid state for subsequent operations.
This fixes the ondemand_convert_tests failure in CI with clang++-16.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove constexpr from functions that call non-constexpr methods
- The no_errors and to<T> adaptors were marked constexpr but call
simdjson_result methods that are not constexpr in C++20
- This was causing compilation failures in CI for C++20 builds
- Tests now compile and pass with both C++17 and C++20
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Instead of disabling the feature, provide C++20-compatible implementation
of the pipe operators for ranges support. The range_adaptor_closure is
C++23-only, so we implement our own pipe operators for C++20.
This preserves the core functionality of the PR while ensuring
compatibility across different compiler versions.
The test_no_errors() and to_clean_array() tests depend on the C++23
ranges features that we disabled. This commit conditionally compiles
these tests out when ranges support is disabled.
The ranges features were causing compatibility issues across different
compilers and platforms. Disabling them for now until C++23 support
is more widespread.
This should fix the remaining Ubuntu and Windows CI failures.
Add defined() check before comparing the value to avoid preprocessor
errors in compilers where this macro doesn't exist (like g++-13
with certain configurations).
- Fix deprecated reflect_value warning by using reflect_constant
- Fix std::const_iterator C++23 requirement by using auto_iterator
- Fix C++23 std::ranges::range_adaptor_closure availability check
- Add convert.h to main simdjson.h includes
These changes ensure compatibility across different C++ standards
and compiler versions, fixing the Ubuntu CI failures.
Co-Authored-By: Claude <noreply@anthropic.com>
* Adding type validation, enhancing optional type support and adding test a few more tests.
* Adding support for string-based enum serlalization and deserialization.
* Removing unintentional endline.
* Removing trailing whitespace.
* Adding simpler api as suggested by moisrex.
* Removing explicit optiona<int> and optional<std::string> references and using concepts instead! Credit goes to Lemire for pointing this out and suggesting a concepts based approach here.
* Removing tests that are not relevant for this branch.
* Removing api related changes. That will be done by moisrex.
* Removing unnecessary new endlines.
* Removing tests related to api changes and cleaning-up irrelevant tests.
* removing broken reference
* Removing trailing whitespace
* Initial work on JSON builder
* moving the files back to ondemand for now.
* tweak
* more later
* update
* minor edits
* dropping vs arm (missing support)
* adding tests. we still specialized write_string_escaped
* tweaking
* fix typo
* tweaking the approach
* minor fix
* missing store
* another missing store
* Attempt at fixing failing serialization tests. (#2292)
* Fixing appeand_float typo (#2294)
* applying a couple of fixes
* updating single header
* fix for pre C++17 if constexpr
* Fixing unused argument problem and updating the singleheader file
* various pedantic fixes
* Sketch of builder
* reordering.
* simplify
* Adding draft of static reflection based deserialization
* Updating simdjson singleheader
* patching the automated deserialization.
* automated
* Adding support for smart pointers of user defined types.
* Adding specialization for smart pointers for basic types. I think it is highly likely that this can be done in a more generic way.
* Referncing a later version of rapidjson that fixed the issue related with assignment attempt of a const variable for GenericStringRef class.
* guarding the tests
* adding documentation for string_builder
* saving
* rename to 'append'
* saving
* non-functional benchmarks (#2342)
* non-functional benchmarks
* Fix typo
* various fixes
* tweaking
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
* tuning
* various minor fixes
* minor tweak
* minor simplification
* updating amal
* adding a cast
* update
* fancy casting
* removing dead code
* Pushing latest changes. CITM benchmark is still not working.
* Still not working, but now I am getting only 10 errors.
* add static reflection benchmark to 'large random' benchmark and allows (#2349)
deserialization (with static reflection) from objects and arrays.
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Removing std::map from CitmCatalog definition, since that is not currently supported.
* Added free to rust bench, segfault is still happening..
* The syntax changed: ^E became ^^E. (#2350)
* The syntax changed: ^E became ^^E.
* guarding
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Adding support for string_view_keyed_map types.
* Adding concepts as a conditional include.
* updating single-header
* Adding concepts to ondemand deps
* rust benchmark is finally working
* Fixing small typo in docs.
* adding docker config and instructions so that our users can test the static reflection (#2358)
* adding docker config and instructions so that our users can test the
static reflection
* completing the instructions
* pruning white spaces
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* minor optimizations on the JSON builder branch
* avoiding undef behaviour
* saving
* somewhat nicer builder
* make it possible to run just one benchmark
* adding linux perf
* fixing minor issue
* updating swar
* Adding real world compilation benchmark (#2379)
* Adding compilation benchmark for json parsing with and without reflection
* Moving it to the benchmark folder, also reducing a bit the number of iterations.
* Removing script from root folder.
* Reducing number of iterations
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Making the script more customizable and also test whether the compiler being used supports reflection before actually running the benchmark
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Using define_static_string from https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p3491r2.html (#2389)
* Applying changes needed after latest reflection paper updates.
* Working, but no template for yet.
* Updating single-header to incldue the use of define_static_string.
* copying over master
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
* We run on practically all systems.
* Update doc/basics.md
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
---------
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
* tag_invoke based custom types (#2219)
* tag_invoke based custom types
Now you can use tag_invoke to add a custom type or a group of custom types.
* Fixing macro usage + Fixing noexcept
* Fixing the usage of #include
We don't need <concepts> at all seems like it
* Fixing tag_invoke impl for MSVC
* Making `tag_invoke` to support `ondemand::document` as well + docs (#2228)
* Making `tag_invoke` to support `ondemand::document` as well + docs
* Fix typos and doc update by @lemire
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Better docs by @lemire
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Preserving the old, disallowing in the new
I'm disabling `document::get() &&` if the user has provided a `tag_invoke`d version; otherwise, we retain the compatibility.
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* fix: correct small issues with deserialize (#2232)
* Extending the deserialization code with more defaults + docs (#2233)
* Make custom types easier with some predefined cases + docs
* missing include
* adding Ubuntu 24 CXX 20
* using concepts all the way
* minor tweak
* tiny tweak
* tweaks
* more tweaking
* saving
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Making `tag_invoke` a "put" as opposed to a "get" (#2256)
* fix: add tests related to issue 2227 (#2229)
* fix: add tests related to issue 2227
* avoiding name clash
* pedantic fix
* deprecate rvalue get on document
* selectively deprecating
* Fix ndjson spec link (#2234)
* fix ndjson spec link
The link in the readme of parse_many links to a casino spam site
* fix link
* [no-ci] Update README.md
* Make simdjson compile again
* Enable SIMDJSON_SINGLEHEADER=OFF in VS Code
With singleheader on, clangd can't find the right
include files.
* Add missing include directives to static build targets of simdjson. (#2240)
* adding a warning
* adding warning regarding SIMDJSON_BUILD_STATIC_LIB
* release candidate
* pedantic viable size
* Making tag_invoke a feeder instead of a producer
* adding missing undef silencer (#2253)
* Ignore pragma once when amalgamating source files (#2248)
With gcc it causes an error in `simdjson.cpp`:
```
simdjson.cpp:548:9: warning: #pragma once in main file
548 | #pragma once
| ^~~~
```
It had previously been commented out in:
https://github.com/simdjson/simdjson/commit/6ef555e6fb79363fae057a9a46b52cd208d9e305
However, this was lost in an upgrade:
https://github.com/simdjson/simdjson/commit/2a4ff7346813b120f2b5b40e95d69352b593cc9c
* Update CI (#2254)
* adding missing undef silencer
* Updating CI
* more fixes
* fix
* big endian fix
* Moving to the new tag_invoke signature
* Fix nlohmann ambiguity on C++23-enabled clang
* Revert "Merge branch 'master' of https://github.com/simdjson/simdjson into builder_development_branch_extra"
This reverts commit 3eeecbab34, reversing
changes made to 6858b208b4.
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>
* update CI on the builder_development_branch (no code change) (#2262)
* typo
* General madness simpler, no simpler!!! (#2267)
* Minimal tag_invokes for STL types
* simpler madness
* adding a comment
* missing file
* minor tweaks to style
* fixing incorrect max/min usage
* updating single
* simplify
* validating the idea
* putting back the concept
* moving the include
* guarding
* Cheap General Madness (#2268)
* Some General Concepts and their deserializations
* Resolving ambiguity
* Add missing #include
* C++20 custom deserializer: better documentation (#2269)
* mostly a documentation update.
* missing cpp
* [no-ci] fix comment
* various minor fixes
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
---------
Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: M. Bahoosh <moisrex@gmail.com>
* minor update
* More documentation regarding builder (#2270)
* minor update
* more improvment to our documentation (builder branch)
* putting back missing functions
* merge candidate
---------
Co-authored-by: M. Bahoosh <moisrex@gmail.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>
Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
* Add support for JSONPath with '$' prefix and integrate into dom::at_path (#2266)
- Updated `json_path_to_pointer_conversion` to support JSONPath starting with the '$' prefix, while maintaining compatibility with the existing implementation.
- Moved `json_path_to_pointer_conversion` to a separate header file for better modularity and to support JSONPath queries in `dom` mode.
- Implemented `at_path` functionality in `dom` mode to enable querying JSON using JSONPath.
- Added unit tests to validate the new JSONPath support in `dom` mode and ensure compatibility with both standard and existing JSONPath formats.
* two minor fixes
* more tests and documentation
* damn compiler warnings
* more documentation fixes
---------
Co-authored-by: Zhengguo Yang <yangzhgg@gmail.com>
* Make null-like value test consistent with docs
According to the docs: `INCORRECT_TYPE If the JSON value begins with 'n'
and is not 'null'.`
* Match document::is_null behavior to documentation
* Test other token beginning with n with is_null
* This PR does the following:
1. Upgrade cxxopts.
2. Allows field::unescape_key to take in a string parameter (syntaxic sugar).
3. Adds a dev. check to detect a string buffer overflow (indicating broken code). Note that this is unrecoverable and indicates bad code.
* tweak
* marking a few trivial functions as pure
* adding other marks
* additional marks
* vs will issue warnings, so don't use [[gnu::pure]] when __clang__ or __GNUC__ is not defined
1. Allows processing inclomplete, damaged, corrupted json to some extent.
2. Pariity with the Presto Java functionality.
3. Protected with SIMDJSON_EXPERIMENTAL_ALLOW_INCOMPLETE_JSON define.
4. Does not interfere with the normal path (can co-exist).
5. Tested in production forkflow.
```
src/implementation.cpp:193:20: error: no template named 'is_trivially_destructible' in namespace 'std'; did you mean 'is_trivially_move_constructible'?
static_assert(std::is_trivially_destructible<detect_best_supported_implementation_on_first_use>::value, "detect_best_supported_implementation_on_first_use should be trivially destructible");
~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
is_trivially_move_constructible
```
* In some cases, clang might try to cast an ondemand::document to an
ondemand::document, instead of calling the move constructor. So we
can disable the template cast.
* In some cases, clang might get confused when constructing an ondemand
document in a constructor: instead of calling the move constructor, it
somehow ends up trying to cast a document to a document. We can easily
disallow this behavior with std::enable_if.
* making compatible with C++11
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* This documents the big-int feature, and adds a few tests.
* moving check_if_integer
* trimming the example.
* More trimming.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Documenting our support of JSON Path and JSON Pointer vs. Unicode characters
* using more standard terminology (nitpicking)
* specifying UTF-8 encoding
* allow char8_t when compiling as C++20
* casting
* minor doc corrections
* First implementation of the conversion logic.
- Not yet tested
- Not yet compiled
- Not yet used in ondemand array/object/etc...
* Adding at_path to ondemand arrays
* Adding at_path to array, document and value.
* Adding at_path to ondemand object as well.
Pending:
- Building
- Adding tests for each of the on-demand classes usage of at_path
- Properly documenting the subset of json path that is currently supported.
* Adding a simple json_path test to on_demand
* Still trying to compile the ondemand_readme_examples test with the at_path() call
* Fixing linking issues with array::at_path
* Fixing issues with the path -> pointer conversion and removing comments
* Adding a clone of json_pointer to test json_path extensively
* Adding ondemand_json_path_tests (all tests passing)
* Adding documentation for at_path()
* Removing some newlines
* Removing simdjson_result from the return type of the string conversion function. Now json_path_to_pointer_conversion returns a std::string.
* Fixing typo
* Making string concatenation explicit to avoid ubuntu gcc12 issue with -O3 flag.
* Addressing latest reviews
* Addressing latest reviews
* documenting the support for custom types
* fixes
* getting around -Weffc++
* calling get_raw_json_string()
* documenting the support for custom types
* fixes
* getting around -Weffc++
* calling get_raw_json_string()
* Allowing users to write directly to std::optional<std::string>
* better fallback
* do not force the cast to std::string
* missing header
* removing abort
* Standard compatibility fixes
* missing commit
* Should work.
* Fix.
* Fix.
* Should work now.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Add comma separated value parsing
* Fix failing tests
* Make tests work for exceptions
* Fix test
* Fix try catch making test fail
---------
Co-authored-by: Yong Xiang Ng <yxng@drwholdings.com>
* Adding CXX 20 to CI
* side-stepping new CXX 20 guard.
* Going another way
* Saving.
* Explicit.
* Saving...
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Add info and error logging
* Add tests for error logging
* Add logging for missing field
* Update docs for logging usage
* Fix style and pass by ref for string format args
* Make log_level explicit and simplify get log level from env
* Make log level int32_t
* Format enum class
* Fix ci
* Move enum to header
* Fix compilation for noexception build in test case
* Disable warnings and putenv
---------
Co-authored-by: Yong Xiang Ng <yxng@drwholdings.com>
* Adding support for AVX-512 on macOS.
* Fix.
* Fix.
* Minor fix
* Setting the variable to zero.
* Fixing include
* Checking if the OS supports AVX-512
* Tweaking.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
This fixes a bug that caused simdjsonTargets.cmake not to be included in
CPack-generated packages, which—unlike `cmake --install`—does not
pick up this mislabeled install component.
I git-grepped through the code base, after this change all components
are either `simdjson_Development` or `simdjson_Runtime`.
* This adds SIMDJSON_DEVELOPMENT_CHECKS to the DOM API to help users
in the scenario of issue 1914.
* More documentation and warnings.
* Updating following comments by Tyson
Related to #1904
Users of the simdjson library will see json2msgpack as an available
example of how to recursively process json with the ondemand parser,
and checking for trailing tokens in a document is one part of json validation.
These checks shouldn't affect benchmark results performance.
The benchmark is run on the 631KB twitter.json file.
Make it less likely to accidentally introduce tabs, trailing whitespace,
carriage returns, non-utf8 in files, or files without trailing newlines.
https://editorconfig.org/ has plugins for various editors/IDEs and is
enabled by default in some IDEs.
* Fixing issue 1898 Preserve sign for number with underflowing exponent (#1900)
Before this commit, simdjson parsed "-1e-999" and "-0e-999" and "-1e-342"
as 0.0.
After this commit, those JSON strings get parsed as -0.0.
(https://en.wikipedia.org/wiki/Signed_zero)
The old behavior was inconsistent with the way simdjson parsed "-0.0" as -0.0.
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Co-authored-by: Tyson Andre <tysonandre775@hotmail.com>
Load 2 bytes and compare the 2 bytes against `"\u"`
Compilers with optimizations turned on will turn this into a 16-bit load
then 16-bit compare on supported platforms
(with smaller compiled code size).
Make it obvious to the compiler that it's reading two
consecutive bytes of the same pointer
Add parse_surrogate_pairs to show the difference exists.
See discussion in #1896
Closes#1894
Reject low surrogates outside of the range U+DC00—U+DFFF
Related to https://unicodebook.readthedocs.io/unicode_encodings.html#utf-16-surrogate-pairs
A surrogate pair should consist of a high surrogate and low surrogate.
They're used to represent 0x010000-0x10FFFF in the JSON spec because
the JavaScript specification originally only supported `\uXXXX`.
Previously, simdjson would accept some combinations of valid high
surrogates and invalid low surrogates due to a bug in the check.
(e.g. `\uD888\u1234` was accepted)
U+D800—U+DBFF (1,024 code points): high surrogates
U+DC00—U+DFFF (1,024 code points): low surrogates
* build: add pkg-config support
The CMake build script now generates a simple pkg-config files that can
be easily used by non-CMake users.
The file is generated from a template file that gets filled in at
configure time.
As CMake doesn't have anything similar to Meson's pkg-config generator
the file is quite static, i.e. new simdjson public defines/dependencies
won't be picked up automatically.
This approach also suffers from one minor issue, mentioned in
[jtojnar/cmake-snips][]; in short, it doesn't work well when users
specify CMAKE_INSTALL_INCLUDEDIR and similar as absolute paths. It's not
a big deal, and it will easily fixable once you'll require CMake >=3.20.
Fixes#1763
[jtojnar/cmake-snips]: https://github.com/jtojnar/cmake-snips#concatenating-paths-when-building-pkg-config-files
* build: handle absolute paths in .pc generation
As mentioned in the previous commit message, correct concatenation of
paths is only available in CMake >=3.20, so handling absolute paths in
pkg-config file generation requires using jtojnar's JoinPaths module.
* ci: add debian job
This new jobs compiles simdjson on Debian Testing, a semi-rolling
release, so that new compilers are always tested.
This job also tests the pkg-config file introduced in commit
1096c3b299
* Rename simdjson_really_inline -> simdjson_inline
I want to change the simdjson_really_inline macro to sometimes not force
inlining. After that upcoming change, the name simdjson_really_inline
will no longer makes sense.
Rename simdjson_really_inline to simdjson_inline. This patch should not
change semantics; simdjson_inline still forces inlining as before.
Some functions still need to be really inlined for ABI reasons.
(GCC's -Wpsabi complains otherwise.) Leave those functions marked as
simdjson_really_inline.
* Improve build times for debug builds
simdjson_inline is used for most simdjson functions. It forces inlining.
In unoptimized/debug builds, this can lead to a lot of machine code
being generated (especially with Address Sanitizer), causing slow
compilation.
Change simdjson_inline to force inlining only for optimized builds.
Sometimes, the programmer might want a slightly-optimized build and want
fast compilation (e.g. GCC's -Og mode). Allow simdjson users to define
the simdjson_inline macro themselves (e.g. on the command line:
-Dsimdjson_inline=inline) in cases where the default behavior is
undesired.
This patch reduced build times by over 75% for ondemand_object_tests.cpp
with GCC 9.4.0 and CMAKE_BUILD_TYPE=Debug on my AMD 5950X:
Before: 6.885 6.683 6.971 6.957 6.949 seconds (5 samples)
After: 1.492 1.551 1.494 1.490 1.531 seconds (5 samples)
* Patch for possible AVX-512 overflow.
* Updating the test for new padding.
* Preparing new version.
* replace binary integer literals with hex literals for C++11 compatibility (#1855)
Binary integer literals are a C++14 feature, so those are not supported
in C++11 and should be replaced by hexadecimal literals instead.
Fixes#1854.
Co-authored-by: Dirk Stolle <striezel-dev@web.de>
* Let us time minify
* Making AVX-512 available by default.
* Silencing some maybe-uninitialized warning under GCC (warning appears in the standard library).
* Making the Python amalgamation script a bit more Windows friendly.
* We do not try to silence -Wmaybe-uninitialized under clang.
* Optimized the arm64 implementation of simd8x64::compress
This is ~35% faster on the fast_minify benchmarks on Apple M1
* Return byte-count from simd8x64::compress
This avoids a redundant popcount on ARM, for ~3% faster minify
on Apple M1
* This exposes 'rewind' for object and array instances.
* Putting really_inline back to count_elements()
* Update array.h
* Adding empty array rewind.
* Adds "is_empty" method to arrays.
* More fragmentation.
* Tweaking implementation.
* Fixing issue with get_value() on document instances.
* Changing the name of the new rewind functions to reset.
Including <iostream> has two problems:
* Compile times are worse because of over-inclusion
* Binary sizes are worse when statically linking libstdc++ because
iostreams cannot be dead-code-stripped
simdjson only needs std::ostream. Include the header declaring only what
we need (<ostream>), omitting stuff we don't need (std::cout and its
initialization, for example).
This commit should not change behavior, but it might break users who
assume that including <simdjson/simdjson.h> will make std::cout
available (such as many of simdjson's own files).
* Adding test.
* Verifies and fix issue 1668. This commit updates the previous behavior of the
On Demand stream support by return a value type (document_reference) instead
of a reference to a document. This allows us to bridge with the usually simdjson
error system, with its simdjson_result types.
* Minor reformat.
* Adds a test with initial tests passing.
* Adding an example.
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
* Naive implementation for doubles in string.
* Add double from string in atom doc.
* Simplification (removed all *_from_string())
* Add int and uint parsing in string.
* Make duplicates instead.
* Make tests exceptionless.
* Add missing declarations.
* Add more tests (errors, JSON pointer).
* Add crypto json tests.
* Update doc.
* Update doc after review.
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
* Add document_stream constructors and iterate_many
* Attempt to implement streaming.
* Kind of fixed next() for getting next document
* Temporary save.
* Putting in working order.
* Add working doc_index and add function next_document()
* Attempt to implement streaming.
* Re-anchoring json_iterator after a call to stage 1
* I am convinced it should be a 'while'.
* Add source() with test.
* Add truncated_bytes().
* Fix casting issues.
* Fix old style cast.
* Fix privacy issue.
* Fix privacy issues.
* Again
* .
* Add more tests. Add error() for iterator class.
* Fix source() to not included whitespaces between documents.
* Fixing CI.
* Fix source() for multiple batches. Add new tests.
* Fix batch_start when document has leading spaces. Add new tests for that.
* Add new tests.
* Temporary save.
* Working hacky multithread version.
* Small fix in header files.
* Correct version (not working).
* Adding a move assignment to ondemand::parser.
* Fix attempt by changing std::swap.
* Moving DEFAULT_BATCH_SIZE and MINIMAL_BATCH_SIZE.
* Update doc and readme tests.
* Update basics.md
* Update readme_examples tests.
* Fix exceptions in test.
* Partial setup for amazon_cellphones.
* Benchmark with vectors.
* Benchmark with maps
* With vectors again.
* Fix for weighted average.
* DOM benchmark.
* Fix typos. Add On Demand benchmark.
* Add large amazon_cellphones benchmark for DOM
* Add benchmark for On demand.
* Fix broken read_me test.
* Add parser.threaded to enable/disable thread usage.
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Changing the name of the function to 'to_json_string' from 'to_string' to avoid confusion.
* Moving to a fast string_view model
* Making it exception-safe.
* Tweaking.
* Workaround for exceptions.
* more robust to_json_string (#1651)
* WIP.
* Fuzzing timeout (bug fix) (#1650)
* prove pull request #1648 introduces an infinite loop
* Interesting bug!
* Tweak.
Co-authored-by: Paul Dreik <github@pauldreik.se>
* It should now work.
* Moving car examples to exception mode
* Simplifying somewhat.
* I forgot to abandon. Let us do that.
* Adding more tests.
* WIP.
* It should now work.
* Moving car examples to exception mode
* Simplifying somewhat.
* I forgot to abandon. Let us do that.
* Adding more tests.
Co-authored-by: Paul Dreik <github@pauldreik.se>
Co-authored-by: Paul Dreik <github@pauldreik.se>
* upload corpus to https://www.pauldreik.se/ from the x64 github action job (keep the github action cache)
* drop the github action cache (which was not working anyway) for power fuzzer and download the fuzz corpus from https://www.pauldreik.se/ instead
* resurrect arm64 fuzzing on drone CI, downloading the fuzz corpus from https://www.pauldreik.se/
* update the fuzzing documentation
* Update basic.md to document JSON pointer for On Demand.
* Add automatic rewind for at_pointer
* Remove DOM examples in basics.md and update documentation reflecting addition of at_pointer automatic rewinding.
* Review
* Add test
Co-authored-by: Daniel Lemire <lemire@gmail.com>
* Add working JSON pointer for array of atoms.
* Add working JSON pointer for object with key-atom pairs.
* Add first version of JSON pointer.
* Update tests (2 tests).
* Make tests exceptionless.
* Fix builing issues.
* Add more tests. Add json_pointer validation in array-inl.h and object-inl.h and empty json_pointer in document-inl.h.
* Fix errors in tests.
* Review.
* Add missing comment.
* Adds compile-test for Visual Studio + ARM and turn developer mode throughout CI.
* Correcting YAML error.
* Disabling google benchmarks under Windows ARM.
* Turning off exceptions under ARM.
* First try at implementing max_capacity for simdjson_ondemand.
* Add max_capacity check.
* Update doc.
* Add one more example in doc for fixed capacity.
* Make allocate() public.
* Remove whitespace
* Found culprit whitespace.
* Duplicating variable.
* Adding 'count_elements' method.
* Actually reporting errors.
* removing white space.
* Removing white space again.
* Adding an extra example.
* Prettier.
* Making the functionality more error-proof.
* Avoiding exceptions.
* Various fixes including extending count_elements to value types.
* Various fixes.
* Minor fixes.
* Correcting comment.
* Trimming white spaces.
* Add first working version of rapidjson_sax for partial tweets.
* Add cleaner and faster rapidjson_sax
* Add nlohmann_json_sax.
* Replace array of bool by bitsets.
* Replace strdup to copy string in rapidjson_sax.
* Change std::string_view assignment in rapidjson_sax.
* Add rapidjson_sax.h and fix typo in rapidjson.h
* Add nlohmann_json_sax.h and add user key check for screen_name in rapidjson_sax
* Change std::string_view assignement for text and screen_name.
* Add rapidjson_sax.h .
* Add nlohmann_json_sax.h . Fix typos distinct_user_id/nlohmann_json_sax.h, find_tweet/rapidjson.h and find_tweet/rapidjson_sax.h .
* Add extra check for id key when looking for find_id.
When find_field_unordered is used on an empty object, it calls
json_iterator::reenter_child. reenter_child asserts that it doesn't
rewind too far back by consulting parser->start_positions.
When the On Demand parser sees an empty object, it fails to update
parser->start_positions. This means that the assertion in
json_iterator::reenter_child reads stale data, or potentially
uninitialized memory. Reading uninitialized memory can cause spurious
assertion failures and Valgrind memcheck reports:
Running missing_keys_for_empty_top_level_object ...
==170679== Conditional jump or move depends on uninitialised value(s)
==170679== at 0x4943D7: reenter_child (json_iterator-inl.h:208)
==170679== by 0x4943D7: find_field_unordered_raw (value_iterator-inl.h:197)
==170679== by 0x4943D7: find_field_unordered (object-inl.h:13)
==170679== by 0x4943D7: find_field_unordered (object-inl.h:96)
==170679== by 0x4943D7: find_field_unordered (value-inl.h:110)
==170679== by 0x4943D7: find_field_unordered (document-inl.h:105)
==170679== by 0x4943D7: object_tests::missing_keys_for_empty_top_level_object() (ondemand_object_tests.cpp:117)
==170679== by 0x4CA761: object_tests::run() (ondemand_object_tests.cpp:1085)
==170679== by 0x8BA314: int test_main<bool ()>(int, char**, bool ( const&)()) (test_ondemand.h:81)
==170679== by 0x4CA9C8: main (ondemand_object_tests.cpp:1119)
==170679==
Fix the read of uninitialized or stale memory by updating
parser->start_positions regardless of whether we see an empty object or
an object with some keys.
This commit only affects builds where development checks
(SIMDJSON_DEVELOPMENT_CHECKS) are enabled. Builds where development
checks are disabled are unaffected by this bug.
* Verifies bug with missing keys.
* Allowing search from any key.
* Workaround for buggy msys
* Restricting how we can end key searches.
* Adding a few tests.
* Under ARM, it is slightly better to reverse the word once and then extract the bits.
* Guarding the zero_leading_bit call to avoid sanitizer warnings.
* This moves all DOM (benchmark + test) files to a subdir
* Missing file.
* CMake + DLL is not pretty.
* Capitalizing AND
* Fixing mismatch endif
* Flipping the order.
* onedemand => ondemand
* Remove CMP0025 policy
This policy is already set to NEW by the minimum required version.
* Use HOMEPAGE_URL in the project call
* Use VERSION in the project call
* Detect if this is the top project
* Port simdjson-user-cmakecache to a CMake script
* Create a developer mode
The SIMDJSON_DEVELOPER_MODE option set to ON will enable targets that
are only useful for developers of simdjson.
* Consolidate root CML commands into logical sections
* Warn about intended use of developer mode
* Prettify the just_ascii test
* Remove redundant CMake variables
* Inline CML contents from include and src
* Raise minimum CMake requirement to 3.14
* Define proper install rules
* Restore thread support variable
* Add BUILD_SHARED_LIBS as a top level only option
* Force developer mode to be on in CI
* Include flags earlier in developer mode
* Set CMAKE_BUILD_TYPE conditionally
CMAKE_BUILD_TYPE is used only by single configuration generators and is
otherwise completely ignored.
* Remove useless static/shared options
simdjson now uses the CMake builtin BUILD_SHARED_LIBS to switch the
built artifact's type.
* Remove unused CMAKE_MODULE_PATH variable
* Refactor implementation switching into a module
* Factor exception option out into a module
* Reformat simdjson-flags.cmake
* Rename simdjson-flags to developer-options
* Accumulate properties into an include module
This is done this way to avoid using utility targets that must be
exported and installed, which could potentially be misused by users of
the library.
* Port impl definitions to props
* Port exception options to props
* Lift normal options to the top
* Port developer options to props
* Remove simdjson-flags from benchmark
* Document the developer mode in HACKING
* Fix include path in installed config file
* Fix formatting of prop commands
* Fix tests that include .cpp files
* Change GCC AVX fixes back to compile options
* Deprecate SIMDJSON_BUILD_STATIC
* Always link fuzz targets to simdjson
* Install CMake from simdjson's debian repo
* Add gnupg for apt-key
* Make sure ASan link flags come first
* Pass CI env variable to cmake invocation
* Install package for apt-add-repository
* Remove return() from flush macro
* Use directory level commands instead of props
* Restore the github repository variable
* Set developer mode unconditionally for checkperf
The CI env variable is only set in the CI and this target is always run
in developer mode.
* Attempt to fix ODR violation in parsing checks
These tests were compiling the simdjson.cpp file again and linking to
the simdjson library target causes ODR violations.
Instead of linking to the target, just inherit its props.
* Move variables before the source dir
* Mark props to be flushed after adding more
* Use props for every command for the library
* Use keyword form for linking libs
* Handle deprecation of SIMDJSON_JUST_LIBRARY
* Handle deprecations in a separate module
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Truncate final unclosed string.
* Adding more precise remarks.
* Better documentation and more robust code.
* ARM + PPC corrections.
* Patching ARM implementation with new stage1_mode parameter.
* Fixed most problems.
* Correcting white spaces and adding a remark.
* This adds the truncated_bytes() method to the stream instances.
* This implementations string serialization for On Demand instances.
* Adding more documentation.
* Another remark.
* Marking the new functions as inline.
* casts apparently do not work.
* Upgrading the API.
* Making the code really free from exceptions.
* At another fix for exceptionless.
* Modify to_chars so that it does not pad integers with '.0'.
* Negative 0 cannot be expressed as an integer.
* Again, accomodating exceptionless usage.
* Using x <= -0 does not allow you to determine the sign since 0 <= -0. I am not sure where
this bug comes from.
This eliminates the possibility of inlining target failures for ondemand
Also makes it so we always compile common architectures needed by simdjson.cpp in simdjson.h, since amalgamation has no way to reason about whether to include / exclude it.
* This gives the CMake install the necessarily information (and flags) to know
whether we have a Windows DLL and in such cases how to handle the linkage.
This avoids a very unlikely buffer overrun that can occur in a particular kind of invalid JSON:
- the document is invalid with an unclosed top level array or object
- the last thing in the document is a number that ends at EOF
- the padding is filled entirely with numeric digits
* This adds a little test to see if we can compiler with very strict flags.
* Trimming a leftover old-style cast.
* More cleaning.
* A few more pedantic casts.
* Fixing issue 1243
* The tie must go.
* Having std::pair be a protected inheritance breaks on demand.
* Putting it back.
* You really want to use emplace.
* Fixing one botched test.
* Prettier test.
* Using safer code.
* Fixing unsafe code.
* Simplifying the fuzzer.
* Trying another way.
* Ok. It should work without exceptions.
* Removing trailing spaces.
* Minor edits regarding the On Demand documentation.
* Adding more instructions for CMake
* Tweaking.
* Adding changes requested by John.
* Bringing back detailed explanations of -march=native.
as described in the description for the allocate_padded_buffer function: // The caller is responsible to free the memory (e.g., delete [] (...)).
but your code used the function free.
I propose to fix this error.
* Entering a new UTF-8 test
* Maybe *I* had a bug in the tests.
* Replacing nulls with 1s.
* Let us try to be more verbose.
* Return 0.
* Fixing issue.
* Adding puzzler scenario.
* Fixing PPC64
Co-authored-by: Daniel Lemire <dlemire@rcs-power9-talos>
* Reenabling the optimized kernels (main branch).
* Defining SIMDJSON_CAN_ALWAYS_RUN_PPC64 and SIMDJSON_CAN_ALWAYS_RUN_ARM64
* Adding the bad UTF8 string from the fuzzer.
* Taking into account John's comments.
* Bumping the lib version.
* Update CMakeLists.txt
* first try
* use ubuntu 20.04, do the fuzzing
* new try at power fuzz
* hard code clang version
* setting env variables does not seem to work
* use fuzzer-no-link
* switch to Debian Buster for power fuzz
* use non-sanitizer build for power
* me not like yaml
* fix bad syntax
* add ndjson fuzzer
* reproduce #1310 in the newly added unit test
Had to replace the input, because:
1)
the fuzzer uses the first part of the input to determine
the batch_size to use, so that has to be cut off
2)
the master now protects against low values of batch_size
I also made the test not return early, so the error is triggered.
* bump boost.json and see if it works in simdjson CI
* enable boost json
* clean up
* add boost json to deps
* use boost if std::string_view is available
* add build with c++20
* use docker image which has the proper libc++ installed
Forks that would like to contribute via PRs from feature branches
needlessly run CI on those branches on top of the PRs.
This is a waste of resources.
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Fix Cirrus CI
CMake is not installed at this point yet.
* Add caching in Circle CI
* Add caching to the MinGW Github workflows
* Fix Circle CI config
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Bump minimum CMake version
* Remove unnecessary git checks
* Move benchmark options where they are used
* Declare helper functions for dependencies
The custom solution here is tailored for fast configure times, but only
works for dependencies on Github.
* Import dependencies using the declared commands
* Remove git submodules
* Call target_link_libraries properly
target_link_libraries must not be called without a requirement
specifier.
* Fix includes for competition
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Updating main branch for legacy libc++ support
* Adopting
* Removing unnecessary math header.
* Updating the single-header files so we can pass the new tests.
* Portable infinite-value detection is hard.
* Working toward disabling boost json selectively.
* Selectively disabling Boost JSON
* More work toward selectively disabling boost json.
Under compilers like MSVC the `#pragma message` about portability will be issued for each translation module; regardless of whether or not `SIMDJSON_PORTABILITY_H` has already been defined. As such a new define `SIMDJSON_NO_PORTABILITY_WARNING`, can be defined prior to the inclusion of `<simdjson.h>` to silence it.
Regardless of the compiler, Windows targets do not support -fPIC,
as position independent code is already implicitly enabled. Compiling
simdjson with Clang on Windows will error because -fPIC is an
unsupported option for target 'x86_64-pc-windows-msvc'.
fix uninteded early return in ondemand unit test loops
go through and fix warnings appearing in qtcreator,
qualify with std::, add const, abort on error
get rid of ulp_distance, not needed anymore when parsing is exact
Introduce cmake option SIMDJSON_DISABLE_DEPRECATED_API (default Off)
which turns off deprecated simdjson api functions by setting the macro
SIMDJSON_DISABLE_DEPRECATED_API.
For non-cmake users, users will have to set SIMDJSON_DISABLE_DEPRECATED_API
by some other means to disable the api.
Closes#1264
This builds the CI fuzzers with the intended clang version. It also allows users to set the clang version locally,
in case they need to.
It also switches the CI fuzzers to use an optimized sanitizer build, to do something oss-fuzz doesn't and get more done in the short time the CI fuzzer runs.
* Add script for CMake PPA
* Call the CMake PPA script in Drone CI
"apt-get update -qq" can be omitted, as that command is already called
by the script to pull in necessary packages for the CMake GPG keys.
* Remove sudo calls in the CMake PPA script
This script is intended to be run in Docker images, where the default
user is already root.
* Use echo instead of printf
* Use /etc/os-release instead of lsb_release
lsd_release could be installed, but os-release is just more convenient
to grab the version code from at this point.
* On Debian images grab CMake from buster-backports
It's not wise to mix Ubuntu PPAs with Debian and buster-backports has
CMake 3.16, which is recent enough for our purposes.
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Initial PPC64 support
* Add travis CI
* Fix outdated cmake version for travis
* Fix indendtation
* Try another workaround for outdated cmake in travis
* Try beta cmake
* Add dash before beta
* Use builtin snaps
* Use cmake as rocksdb
* Test cmake on bionic
* Remove unnecessary things from travis
* Remove unnecessary things from travis
* Another try of compiler install
* Add all major compilers
* Add all major compilers
* Add all major compilers
* Tweak travis a bit
* Typo
* More robust travis
* Typos typos typos
* Add fewer compilers, add non specific build for clang and gcc, should be the final config
* CMAKE_FLAGS is in incorrect place
* Remove default implementation
* Limit build thread number
* Fall back prefix_xor to a usual implementation, no performance boost is noticed
* Test for power9 as it is the main architecture for OpenPOWER right now
* Add to documentation to build with power9 as the implementation is compatible but compiler optimizations is not
* Replace ARM with PPC in the comment
Projects that link simdjson from MSVC with exceptions off will
include simdjson headers which transitively include STL headers.
The MSVC STL stipulates that _HAS_EXCEPTIONS=0 be defined or code
requiring exceptions will be enabled. This change adds a new job
to the appveyor build matrix to verify the build and tests with
exceptions disabled, and disables exceptions at the compiler level
when SIMDJSON_EXCEPTIONS is specified to OFF.
* Adding a distinct user id benchmark
* reenabling everything
* Removing an unnecessary "value()".
* Better tests of the examples and some fixes.
* Guarding exception code.
* Reenable the on-demand tests and allows us to convert a raw string into a C++ string.
* Fixing a 1-byte buffer overrun.
* More documentation.
* Adding more tests.
* Enabling the new tests
* Committing a nicer example.
* Not yet happy but this should fix our failures.
* Duh.
* Ok. Making it easier to get string_view instances from field instances.
* It is a struct.
* Trying to satisfy VS.
* Adopting John's name.
* add definitions for is_number and tie (by lemire)
* add fuzzer for element
* update fuzz documentation
* fix UB in creating an empty padded string
* don't bother null terminating padded_string, it is done by the std::memset already
* refactor fuzz data splitting into a separate class
* This would allow users to find out what builtin is.
* Trying another approach.
* Added instructions.
* Cleaning up the printout.
* Let us be less invasive.
* Adding a comment.
* This adds new tests regarding ordering.
* Updating the documentation with more examples.
* Adding compilation tests.
* Pruning code for exceptions.
* Guarding exceptionless.
* Remove our dependency on strtod_l by bundling our own slow path.
* Ok. Let us drop strtod entirely.
* Trimming down the powers to -342.
* Removing useless line.
* Many more comments.
* Adding some DLL exports.
* Let the gods help those who rely on windows+gcc.
* Marking the subnormals as unlikely. This is pretty much "performance neutral", but it might help just a bit with twitter.json.
* initial try at adding boost json to the benchmark
* clean up
* qualify memcpy etc. with std::
* clang format
* extra space
* update benchmark with help from Vinnie Falco from Boost.json
* add missing separators
This refactors the dynamic check of which implementations are supported at runtime.
It also reduces duplicated effort in the CI fuzzing job, the differential fuzzers don't need to run with different values of SIMDJSON_FORCE_IMPLEMENTATION.
There is also a convenience script to run the fuzzers locally, to quickly check that the fuzzers still build, run and no easy to find bugs are there. It should be handy not only when developing the fuzzers, but also when modifying simdjson.
- Allow user to specify SIMDJSON_BUILTIN_IMPLEMENTATION
- Make cmake -DSIMDJSON_IMPLEMENTATION=haswell *only* specify haswell
- Move negative implementation selection to
-DSIMDJSON_EXCLUDE_IMPLEMENTATION
- Automatically select SIMDJSON_BUILTIN_IMPLEMENTATION if
SIMDJSON_IMPLEMENTATION is set
- Move implementation enablement mostly to implementation files
- Make implementation enablement and selection simpler and more robust
- Fix bug where programs linked against simdjson were not passed
SIMDJSON_XXX_IMPLEMENTATION or SIMDJSON_EXCEPTIONS
* Make it possible to check that an implementation is supported at runtime.
* add CI fuzzing on arm 64 bit
This adds fuzzing on drone.io arm64
For some reason, leak detection had to be disabled. If it is enabled, the fuzzer falsely reports a crash at the end of fuzzing.
Closes: #1188
* Guarding the implementation accesses.
* Better doc.
* Updating cxxopts.
* Make it possible to check that an implementation is supported at runtime.
* Guarding the implementation accesses.
* Better doc.
* Updating cxxopts.
* We need to accomodate cxxopts
Co-authored-by: Paul Dreik <github@pauldreik.se>
This adds fuzzing on drone.io arm64
For some reason, leak detection had to be disabled. If it is enabled, the fuzzer falsely reports a crash at the end of fuzzing.
Closes: #1188
This adds a minifier fuzzer. There is also an utf-8 fuzzer, but it is disabled until #1187 is fixed.
Run all fuzzers bug the utf-8 one in the github CI fuzz.
* Adding new files.
* Better.
* Fixing minifier and adding tests.
* Adding benchmarks.
* Including the array header.
* Replacing old stream-based code by the new code.
* Doubling up the itoa.
* Hidden away to_chars in internal namespace.
* Removing the repetitions.
* Documented the atoi functions.
* Tuning the escape sequences.
* Moving the operators off the main namespace.
* Added more tests.
* Tweaking the implementation so that it works with and without exp.
* The string_builder template and mini_formatter class
are not part of our public API and are subject to change
at any time!
* Adding a benchmark and some optimization.
* Cleaning.
* Strictly speaking, this header is needed.
This adds a fuzzer for at_pointer() which recently had a bug.
The #1142 bug had been found with this fuzzer
Also, it polishes the github action job:
cross pollinate the fuzzer corpora (lets fuzzers reuse results from other fuzzers)
use github action syntax instead of bash checks
only run on push if on master
* This avoids locale-dependent number parsing at the standard library level.
* Adding missing cast.
* Inserting the missing "endif"
* Trial and error.
* Another attempt.
* Another tweak.
* Another fix.
* Restricting it even more.
* Tweaking our symbol checks.
* Somewhat smarter tests.
* Nice comments.
* Minor simplification.
* Adding cerr.
This adds a fuzzer which parses the same input using all the available implementations (haswell, westmere, fallback on x64).
This should get the otherwise uncovered sourcefiles (mostly fallback) to show up in the fuzz coverage.
For instance, the fallback directory has only one line covered.
As of the 20200909 report, 1866 lines are covered out of 4478.
Also, it will detect if the implementations behave differently:
by making sure they all succeed, or all error
turning the parsed data into text again, should produce equal results
While at it, I corrected some minor things:
clean up building too many variants, run with forced implementation (closes#815 )
always store crashes as artefacts, good in case the fuzzer finds something
return value of the fuzzer function should always be 0
reduce log spam
introduce max size for the seed corpus and the CI fuzzer
* Removes 5 KB of tables at the expense, and a load, at the expense
of a multiplication and a shift. I have not benchmarked this new
code, but my expectation is that it should be largely performance
neutral. The motivation is to reduce the size of the library slightly.
There is also a matter of elegance.
* Adding test.
* Saving.
* With exceptions.
* Added extensive tests.
* Better documentation.
* Tweaking CI
* Cleaning.
* Do not assume make.
* Let us make the build verbose
* Reorg
* I do not understand how circle ci works.
* Breaking it up.
* Better syntax.
* Specification is not followed.
* Fixes.
* Do not pass string_view by reference.
* Better documentation.
* The example is written for exceptions.
* Better documentation.
* Updating with deprecation.
* Updating example.
* Updating example.
* This allows the users to disable threading.
* This would disable bash scripts under FreeBSD. (#1118)
* This would disable bash scripts under FreeBSD.
* Let us also disable GIT.
* Let us try to just disable GIT
* Nope. We must have both bash and git disabled.
* This allows the users to disable threading.
* This would disable bash scripts under FreeBSD.
* Let us also disable GIT.
* Let us try to just disable GIT
* Nope. We must have both bash and git disabled.
The jobs were executed in powershell using the globally installed cmake.
This makes things actually run in a MSYS2 shell.
This also removes the msys/cygwin job because it doesn't build
(it complains about undeclared posix_memalign)
C++ 20 adds a new feature called "ranges", which provides components for dealing
with sequences of values: https://en.cppreference.com/w/cpp/ranges.
A range is like a normal object containing `begin` and `end`, except there are
also composable operations like maps, filters, joins, etc.
The iterator objects returned by a range's `begin` and `end` require a more
strict set of operations than is needed for a range-for loop.
This PR adds the extra operations needed to support turning `dom::array` and
`dom::object` into a range.
This PR does not depend on any C++ 20 behavior, the added operators are all
valid C++ 11, and are already part of the LegacyIterator concepts.
This PR adds extra code behind: `#if defined(__cpp_lib_ranges)` guards, which is
the new C++ 20 specified feature test macro for ranges support. When ranges
support is detected, extra compile time checks are added to ensure that
`dom::array` and `dom::object` satisfy the range concept. No runtime tests have
been added yet because these compile time checks should be sufficient.
If desired, the `static_assert` code could be moved out of the actual code
headers and put into a test file.
* lookup4
* Self-document lookup4 and clean up extra bits
* Maintenance, to match against upcoming PR.
Co-authored-by: Daniel Lemire <lemire@gmai.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
* The initial motivation behind basictests was for a quick set of sanity tests to check whether your code made sense. It
was not meant for thorough testing to find corner cases. However, over time, it grew to include such expensive tests.
This PR takes them out. It also allows us to bring back basictests to MinGW tests, since it is now cheap.
This is not an exercise in software engineering and making things prettier. This is a pragmatic change to improve our
test coverage and quality of life.
* Adds many more cheap tests.
Co-authored-by: Daniel Lemire <lemire@gmai.com>
According to https://gitlab.kitware.com/cmake/cmake/-/issues/17976,
CMAKE_VS_PLATFORM_TOOLSET is set only when using a Visual Studio generator.
When we use Ninja as the generator, CMAKE_VS_PLATFORM_TOOLSET will be empty.
As a result:
if(${CMAKE_VS_PLATFORM_TOOLSET} STREQUAL "v140")
will be treated as:
if( STREQUAL "v140")
We may also quote it like this:
if("${CMAKE_VS_PLATFORM_TOOLSET}" STREQUAL "v140")
but that won't make the warnings disappeared in VS2015.
Links to other files need to be either relative to themselves (doc/performance.md -> performance.md) or absolute (doc/performance.md -> /doc/performance.md). This change fixes the documentation when read on GitHub.
* Testing with GCC 10 and clang 10
* Fixing spurious space
* gcc10 does not need the cmake installation.
* We don't want to run the perf test on ARM. I ignore them systematically. ARM performance
should be assessed manually.
* Switching to GCC 10 and Clang 10
* Disabling some tests under sanitizers when they involve rapidjson or other parsers.
Co-authored-by: Daniel Lemire <lemire@gmai.com>
In the parse_many function, we have one thread doing the stage 1, while the main thread does stage 2. So if stage 1 and stage 2 take half the time, the parse_many could run at twice the speed. It is unlikely to do so. Still, we see benefits of about 40% due to threading.
To achieve this interleaving, we load the data in batches (blocks) of some size. In the current code (master), we create a new thread for each batch. Thread creation is expensive so our approach only works over sizeable batches. This PR improves things and makes parse_many faster when using small batches.
This fixes our parse_stream benchmark which is just busted.
This replaces the one-thread per batch routine by a worker object that reuses the same thread. In benchmarks, this allows us to get the same maximal speed, but with smaller processing blocks. It does not help much with larger blocks because the cost of the thread create gets amortized efficiently.
This PR makes parse_many beneficial over small datasets. It also makes us less dependent on the thread creation time.
Unfortunately, it is going to be difficult to say anything definitive in general. The cost of creating a thread varies widely depending on the OS. On some systems, it might be cheap, in others very expensive. It should be expected that the new code will depend less drastically on the performances of the underlying system, since we create juste one thread.
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Daniel Lemire <lemire@gmai.com>
This regresses performance and is ONLY here because the next
two commits are here; this lets us see the impact of removing
parser.error separately from the impact of the next commit.
When using Ninja as generator, it failed to build:
$ cmake .. -G Ninja
$ ninja
> ninja: error: '/jsonexamples/generated/miss-templates/*.txt', needed by 'jsonexamples/generated/utf-8.json', missing and no known rule to make it
Although it passes user-defined options, if the project is build in Debug mode or with Clang (since
CXX defaults to gcc on Linux) results can flactuate
If linked against Threads::Threads target while building static libraries, cmake cannot find the
threads library while trying to use the installed target afterwards
* This will change the default of the parse benchmark so that it work over hot buffers
by default, thus omitting memory allocation as part of the benchmark.
* Everyone should be using '-H' from now on.
amalgamation.sh shouldn't change contents of src/simdjson.cpp by forcing dmalloc.h that didn't exist in non-amalgamated version and shouldn't change order of includes by placing simdjson.h at the top
fixes#739
* Added bitexact implementations of _BitScanForward64 and _BitScanReverse64 for VS2019 32-bit builds
* Added bitexact implementations of _umul128 for VS2019 x86, arm, arm64 builds
* Implement mul_overflow for VS2019 arm64 builds
+ implement mul_overflow using __umulh (msvc/clang results: https://godbolt.org/z/smRwA7)
* Added Win32 for VS2019 to .appveyor.yml
* Update amalgamated headers (fix x86 builds with VS2019)
* This is an implementation of "size()" for arrays and objects.
* Adding benchmark
* Adding a size() remark in the documentation.
* Extending size() to result types.
To avoid using data belonging to a temporary, the parse functions are ref qualified to get a compile error if used on an rvalue. See https://github.com/simdjson/simdjson/issues/696
Compilation tests are also added, to make sure bad usage fails to compile.
Reviewed by jkeiser.
* Trying to correct the documentation so that it actually describes how the code behaves.
* tweaking the wording.
* Improving.
* Removing confusing sentence.
* Fixing formatting.
* Now with working example, tested.
* Added a smaller piece of code
* It is inconvenient to be unable to print a padded_string.
* Allows us to print the padded_string even when it is embedded in result object when exceptions are enabled.
* move from deprecated interface in fuzz dump raw tape
* update fuzz_dump to the non deprecated replacement
* replace use of deprecated api
* hopefully fix windows build
This enables the minify fuzzer, which has been disabled because it did not pass the oss-fuzz instrumentation test. Now it does, after changes in simdjson (https://github.com/lemire/simdjson/issues/186).
* get minify running (api change)
* disable benchmarks when compiling fuzzers
* catch exceptions from the minify fuzzer
* enable repeated corpus creation without recursive inclusion of zip
* remove leftover comment
* This is a first attempt at fixing issue 660. The hard part is not the fix by itself, the hard part is to make sure we never get caught with our pants down like that again.
I expect the CI tests will fail. Further commits will solve the issues.
* Setting the rpath properly.
* For the sanitize tests, we want verify the installation.
* Some users are interested, as a metric, in the number of documents parsed per second.
Obviously, this means reusing the same parser again and again.
* Adding a sentence
* This update the parsingcompetition benchmark so that it displays the number of documents parsed per second.
* Fallback should use our scalar code.
* parse should have a nicer error message.
* Making it so that "minify" can use different architectures.
* Let us change the minifier competition so that it tests all implementations.
* Documenting the untaken optimization opportunity.
Co-authored-by: John Keiser <john@johnkeiser.com>
* Currently, document::stream contains an attribute that is a reference:
```
document::parser &parser;
```
Yet we try to have it default on the move operator:
```
stream &operator=(document::stream &&other) = default;
stream &operator=(const document::stream &) = delete; // Disallow copying
```
```
stream(document::stream &&other) = default;
stream(const document::stream &) = delete; // Disallow copying
```
I am not sure what the move is supposed to do with the reference.
I cannot find where we test the copy constructor and assignment. This has been concerned that it is either dead code or buggy code.
* Remove non-working, unnecessary move constructors
* We still want to disallow copies.
Co-authored-by: John Keiser <john@johnkeiser.com>
* If we are going to have a google benchmark flag, we better make sure that we test it out minimal (it should build).
* Fix bench_dom_api
Co-authored-by: John Keiser <john@johnkeiser.com>
* Make architecture implementations virtual functions
- Easier to add new architectures (add implementation to implementation.cpp)
- Easier to add new algorithms / functions to architecture selection
(add to implementation.h, implement)
- Automatically select best implementation in static initialization
- Allow user to explicitly select implementation with a string (i.e.
parameter)
- Allow user to inspect current implementation name/description
- Allow user to list available implementations
- Eliminate architecture enum and architecture-based templating
- Add noexcept in non-inline functions
* Move implementation static methods to their own classes
* Detect best supported implementation on first use
* available_implementationsI() -> available_implementations
This creates a "document" class with only user-facing document state (no parser internals).
- document: user-facing document state
- document::iterator: iterator (equivalent of ParsedJsonIterator)
- document::parser: parser state plus a "docked" document we parse into (equivalent of ParsedJson)
Usage:
```c++
auto doc = simdjson::document::parse(buf, len); // less efficient but simplest
```
```c++
simdjson::document::parser parser; // reusable parser
parser.allocate_capacity(len);
simdjson::document* doc = parser.parse(buf, len); // pointer to doc inside parser
doc = parser.parse(buf2, len); // reuses all buffers and overwrites doc; more efficient
```
* Fix issue472: make JsonStream a template.
* Adding missing include.
* Tweaking headers and some minor formatting.
* Removing file from aggregation.
* Moving jsoncharutils
* Adding new header.
* Trying another header.
* Let us try to route around Visual Studio's nonesense.
* Fix for issue467
* Updating single-header
* Let us make it so that JsonStream is constructed from a padded_string which will avoid dangerous overruns.
* Fixing parse_stream
* Updating documentation.
* This revert the code back to how it was prior to the silly "run two stages" routine and instead
adds an option to benchmark the code over hot buffers. It turns out that it can be expensive,
when the files are large, to allocate the pages.
* Instead of emulating the whole parsing as stage 1 + stage 2, let us
benchmark the real thing.
* Adding explicit constructor.
* Adding warning to the benchmark user.
* Making re-running optional.
* dirent portable latest version
* improved
std::string argument passed by const reference
ctor added with std::string_view argument
`allocate_padded_buffer()` moved here with **optional** check on `length < 1`
* allocate_padded_buffer moved to padded_string.h
* string literal + integer means unintended and incorrect pointer arithmetic
fixes a clang warning. it could not be triggered, because it can only be
triggered if the string given to getopt is not covered among the
cases in the switch.
* handle review comment
* I think we can align the numbers better (so it is prettier).
* Remove space before %, align third line better
Co-authored-by: John Keiser <john@johnkeiser.com>
It's only a coincidence that it works in current uses: it doesn't do
what the name says. Particularly, if the high bit is 1 it will yield
0 even if the lower 4 bits would yield something else.
Only the simdjson library should optionally depend on threads,
the executables that link to simdjson will get the dependency
indirectly.
* add option for controlling threads (default is on)
* add CI testing with threading on/off for msvc, gcc and clang
* fix an unrelated copy paste comment error in the cirlce ci build conf
* JsonStream threaded prototype
* JsonStream Threaded version working. Still supporting non-threaded version.
* Fix where invalid files would enter infinite loop.
* SingleHeader update
* I will remove -pthread in cmake for now.
* Attempt at resolving the -pthread issue
* rough prototype working. Needs more test and fine tuning.
* prototype working on large files.
* prototype working on large files.
* Adding benchmarks
* jsonstream API adjustment
* type
* minor fixes and cleaning.
* minor fixes and cleaning.
* removing warnings
* removing some copies
* runtime dispatch error fix
* makefile linking src/jsonstream.cpp
* fixing arm stage 1 headers
* fixing stage 2 headers
* fixing stage 1 arm header
* making jsonstream portable
* cleaning imports
* including <algorithms> for windows compiler
* cleaning benchmark imports
* adding jsonstream to amalgamation
* merged main into branch
* bug fix where JsonStream would bug on rare cases.
* Addind a JsonStream Demo to Amalgamation
* Fix for https://github.com/lemire/simdjson/issues/345
* Follow up test and fix for https://github.com/lemire/simdjson/issues/345 (#347)
* Final (?) fix for https://github.com/lemire/simdjson/issues/345
* Verbose basictest
* Being more forgiving of powers of ten.
* Let us zero the tail end.
* add basic fuzzers (#348)
* add basic fuzzing using libFuzzer
* let cmake respect cflags, otherwise the fuzzer flags go unnoticed
also, integrates badly with oss-fuzz
* add new fuzzer for minification, simplify the old one
* add fuzzer for the dump example
* clang format
* adding Paul Dreik
* rough prototype working. Needs more test and fine tuning.
* prototype working on large files.
* prototype working on large files.
* Adding benchmarks
* jsonstream API adjustment
* type
* minor fixes and cleaning.
* Fixing issue 351 (#352)
* Fixing issues 351 and 353
* minor fixes and cleaning.
* removing warnings
* removing some copies
* Fix ARM compile errors on g++ 7.4 (#354)
* Fix ARM compilation errors
* Update singleheader
* runtime dispatch error fix
* makefile linking src/jsonstream.cpp
* fixing arm stage 1 headers
* fixing stage 2 headers
* fixing stage 1 arm header
* fix integer overflow in subnormal_power10 (#355)
detected by oss-fuzz
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18714
* Adding new test file, following https://github.com/lemire/simdjson/pull/355
* making jsonstream portable
* cleaning imports
* including <algorithms> for windows compiler
* cleaning benchmark imports
* adding jsonstream to amalgamation
* merged main into branch
* bug fix where JsonStream would bug on rare cases.
* Addind a JsonStream Demo to Amalgamation
* merging main
* rough prototype working. Needs more test and fine tuning.
* prototype working on large files.
* prototype working on large files.
* Adding benchmarks
* jsonstream API adjustment
* minor fixes and cleaning.
* minor fixes and cleaning.
* removing warnings
* removing some copies
* runtime dispatch error fix
* makefile linking src/jsonstream.cpp
* fixing arm stage 1 headers
* fixing stage 2 headers
* fixing stage 1 arm header
* making jsonstream portable
* cleaning imports
* including <algorithms> for windows compiler
* cleaning benchmark imports
* adding jsonstream to amalgamation
* bug fix where JsonStream would bug on rare cases.
* Addind a JsonStream Demo to Amalgamation
* rough prototype working. Needs more test and fine tuning.
* minor fixes and cleaning.
* adding jsonstream to amalgamation
* merged main into branch
* Addind a JsonStream Demo to Amalgamation
* merging main
* merging main
* make file fix
* initial oss-fuzz friendly build
parts taken from libfmt, which I wrote and have the copyright to
* fix build error
* add script for building a corpus zip
see https://google.github.io/oss-fuzz/getting-started/new-project-guide/#seed-corpus
* fix zip command
* drop setting the C++ standard
* disable the minify fuzzer, does not pass oss-fuzz check-build test
* fix integer overflow in subnormal_power10
detected by oss-fuzz
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18714
* invoke the build like oss fuzz does
* document what the scripts are for and how to use them
* add a page about fuzzing
* add basic fuzzing using libFuzzer
* let cmake respect cflags, otherwise the fuzzer flags go unnoticed
also, integrates badly with oss-fuzz
* add new fuzzer for minification, simplify the old one
* add fuzzer for the dump example
* clang format
- Removes templating from simd_input, utf8_checker, and parse_string
- Make drone gcc run a lot faster
- Make drone clang run a little faster (NOTE:
https://hub.docker.com/r/silkeh/clang helps even more, but I wasn't sure
whether we wanted to trust that)
- Make drone arm run in parallel to get results quicker
* Allow -f
* Support parse -s (force sse)
* Simplify flatten_bits
- Add directly to base instead of storing variable
- Don't modify base_ptr after beginning of function
- Eliminate base variable and increment base_ptr instead
* De-unroll the flatten_bits loops
* Decrease dependencies in stage 1
- Do all finalize_structurals work before computing the quote mask; mask
out the quote mask later
- Join find_whitespace_and_structurals and finalize_structurals into
single find_structurals call, to reduce variable leakage
- Rework pseudo_pred algorithm to refer to "primitive" for clarity and some
dependency reduction
- Rename quote_mask to in_string to describe what we're trying to
achieve ("mask" could mean many things)
- Break up find_quote_mask_and_bits into find_quote_mask and
invalid_string_bytes to reduce data leakage (i.e. don't expose quote bits
or odd_ends at all to find_structural_bits)
- Genericize overflow methods "follows" and "follows_odd_sequence" for
descriptiveness and possible lifting into a generic simd parsing library
* Mark branches as likely/unlikely
* Reorder and unroll+interleave stage 1 loop
* Nest the cnt > 16 branch inside cnt > 8
* Use generic each/reduce in simdutf8check
* Remove macros from generic simd_input uses
* Use array instead of members to store simd registers
* Default local checkperf to clone from .
* handle uint64 value in JSON
* Add integer_tests
* Add get_unsigned_integer() on ParsedJson::BasicIterator
* Write 'u' to tape when the value seems unsigned
* Add to handle 'u' element
* Brush up integer_tests.cpp
* Append tests/integer_tests in .gitignore
* Add comments to is_integer and is_unsigned_integer
* handle uint64 value in JSON
* Add integer_tests
* Add get_unsigned_integer() on ParsedJson::BasicIterator
* Write 'u' to tape when the value seems unsigned
* Add to handle 'u' element
* Brush up integer_tests.cpp
* Append tests/integer_tests in .gitignore
* Add comments to is_integer and is_unsigned_integer
* Get rid of dynamic allocation in ParsedJson::Iterator.
* Implement copy assignment operator for ParsedJson::Iterator.
* ParsedJson::Iterator is now a template class.
* Add -n and -w arguments
* Add Dockerfile that compares perf against master
* Add checkperf to .drone.yml
* Clone from github instead of .git since CI doesn't have .git
* stage1 compiles without macros
* cleaning
* amalgation is weird but works
* macros are removed from stringparsing
* amalgation fixed
* Huge macros are removed.
* clang-format
* Hiding the runtime dispatch pointer in a source file so it is not an exported symbol
* Disabling hard failure on style check.
* Fixes https://github.com/lemire/simdjson/issues/250
* Attempt 1 - fn targeting
GCC won't work with templates with different targets, need to specialize all the way up the call stack.
* Compiles properly with cmake. Does not with the Makefile.
* Compilation works with Makefile
* instruction_set changes to architecture
* some aesthetic changes
* fix amalgation and tests + aesthetic changes
* This now compiles and passes tests under CLANG
* Minor correction.
* Trying to make it work on ARM
* Adding missing namespace
* Missing bracket
* Fixing minor compilation issues.
* Getting parse to use runtime dispatch
* Fixing amalgamation script.
* Making sure that NEON is supported.
* Fixing typo
* Merging https://github.com/lemire/simdjson/pull/229
* Manual merge of
https://github.com/lemire/simdjson/pull/229
by @jkeiser (second part)
* Trying another way.
* Removing the paral.
* Fixing the make file
* Let us make the practice run long enough.
* Resolved the awful slowness.
* Cleaning the README.md
* With runtime dispatching, we should not need flags anymore.
* Changing isa detection file's name + fixing typos.
* Checks for issue 150. We run through the test files with sanitizers on.
* Fix for issue 150: the remaining issues were an overrun on the depth capacity and an "off-by-1" overrun on tape capacity.
* Improving makefile.
* Safer git submodule command.
* Getting get 'git' on circleci
* Improving portability.
* Revisiting faulty logic regarding same-page overruns.
* Disabling same-page overruns under VS.
* Clarifying the documentation
* Fix for issue 131 + being more explicit regarding memory realloc.
* Fix for issue 137.
* removing "using namespace std" throughout. Fix for 50
* Introducing typed malloc/free.
* Introducing a custom class (padded_string) that solves several minor usability issues.
* Updating amalgamation for testing.
* Making sure we can run with the sanitizers on.
* Minor code simplification in the number parsing.
* Following @EmilGedda 's recommendations regarding the makefile.
* Reference to blog post.
* Adding link to https://johnnylee-sde.github.io/Fast-numeric-string-to-int/
* Better hex parsing.
* Allow passing additional compiler flags through command line
* Simplify branching for compiler flags
* Optimize for debug while debugging or sanitizing specified
* Update CMakeLists.txt
Adds support for CPack so that you can make .deb and .rpm packages.
* Update CMakeLists.txt
* Update .travis.yml
* Update .travis.yml
Speedup compiling
* Update .travis.yml
* Update .drone.yml
Speedup compiling
* Update .travis.yml
Remove `-j2` flag because we probably run out of memory when running 2 jobs in parallel and that's why the compilation fails.
* Minor change to benchmark cmake
* Moved ParsedJson and its Iterator to separate .cpp files
* Uncommented functions, that has nothing to do with this pr
* Removed really_inline comments
* Reinstated some inline functions to restore previous performance
* Re-merged iterator in ParsedJson
* Uncommented some WARN_UNUSED
# Forward slash is used because this is used in CMake as is
simdjson_DEPENDENCY_CACHE_DIR:C:/dependencies
matrix:
- job_name:VS2019
CMAKE_ARGS:-A %Platform%
- job_name:VS2019ARM
CMAKE_ARGS:-A ARM64 -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_CROSSCOMPILING=1 -D SIMDJSON_GOOGLE_BENCHMARKS=OFF# Does Google Benchmark builds under VS ARM?
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017). If you wish to support features that are not part of RFC 8259, then you should not refer to your issue as a bug.
**Describe the bug**
A clear and concise description of what the bug is. A bug is a failure to build with normal compiler settings or a misbehaviour: when running the code, you get a result that differs from the expected result from our documentation.
A compiler or static-analyzer warning is not a bug. It is possible with tools such as Visual Studio to require that rarely enabled warnings are considered errors. Do not report such cases as bugs. We do accept pull requests if you want to silence warnings issued by code analyzers, however.
We are committed to providing good documentation. We accept the lack of documentation or a misleading documentation as a bug (a 'documentation bug').
An unexpected poor software performance can be accepted as a bug (a 'performance bug').
We accept the identification of an issue by a sanitizer or some checker tool (e.g., valgrind) as a bug, but you must first ensure that it is not a false positive.
We recommend that you run your tests using different optimization levels. In particular, we recommend your run tests with the simdjson library and you code compiled in debug mode. The simdjson then sets the SIMDJSON_DEVELOPMENT_CHECKS macro to 1, and this triggers additional checks on your code and on the internals of the library. If possible, we recommend that you run tests with sanitizers (e.g., see [No more leaks with sanitize flags in gcc and clang](https://lemire.me/blog/2016/04/20/no-more-leaks-with-sanitize-flags-in-gcc-and-clang/)). You can compile the library with sanitizers for debugging purposes (e.g., set SIMDJSON_SANITIZE to ON using CMake), but you should also turn on sanitizers on your own code. You may also use tools like valgrind or the commercial equivalent.
Before reporting a bug, please ensure that you have read our documentation.
**To Reproduce**
Steps to reproduce the behaviour: provide a code sample if possible. Please provide a complete test with data. Remember that a bug is either a failure to build or an unexpected result when running the code.
If we cannot reproduce the issue, then we cannot address it. Note that a stack trace from your own program is not enough. A sample of your source code is insufficient: please provide a complete test for us to reproduce the issue. Please reduce the issue: use as small and as simple an example of the bug as possible.
It should be possible to trigger the bug by using solely simdjson with our default build setup. If you can only observe the bug within some specific context, with some other software, please reduce the issue first.
**simdjson release**
Unless you plan to contribute to simdjson, you should only work from releases. Please be mindful that our main branch may have additional features, bugs and documentation items.
It is fine to report bugs against our main branch, but if that is what you are doing, please be explicit.
**Configuration (please complete the following information if relevant)**
- OS: [e.g. Ubuntu 16.04.6 LTS]
- Compiler* [e.g. Apple clang version 11.0.3 (clang-1103.0.32.59) x86_64-apple-darwin19.4.0]
- Version [e.g. 22]
- Optimization setting (e.g., -O3)
We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux systems. Please ensure that your configuration is supported before labelling the issue as a bug.
* We do not support unreleased or experimental compilers. If you encounter an issue with a
pre-release version of a compiler, do not report it as a bug to simdjson. However, we always
invite contributions either in the form an analysis or of a code contribution.
Under Windows, we support Visual Studio (both with LLVM and without). We do not support MinGW and other alternate compiler systems. Windows users should be aware that there [is a long-running bug with GCC under Windows](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412).
**Indicate whether you are willing or able to provide a bug fix as a pull request**
If you plan to contribute to simdjson, please read our guide:
* CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017).
We do not make changes to simdjson without clearly identifiable benefits, which typically means either performance improvements, bug fixes or new features. Avoid bike-shedding: we all have opinions about how to write code, but we want to focus on what makes simdjson objectively better.
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
Please provide a clear rationale for the feature. Be advised that simdjson is a community-based project: you should consider providing help.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
**Are you willing to contribute code or documentation toward this new feature?**
If you plan to contribute to simdjson, please read our
* CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017).
We do not make changes to simdjson without clearly identifiable benefits, which typically means either performance improvements, bug fixes or new features. Avoid bike-shedding: we all have opinions about how to write code, but we want to focus on what makes simdjson objectively better.
Is your issue:
1. A bug report? If so, please point at a reproducible test. Indicate whether you are willing or able to provide a bug fix as a pull request. As a matter of policy, we do not consider a compiler warning to be a bug.
2. A build issue? If so, provide all possible details regarding your system configuration. If we cannot reproduce your issue, we cannot fix it.
3. A feature request? Please provide a clear rationale for the feature. Be advised that simdjson is a community-based project: you should consider providing help.
4. A documentation issue? Can you suggest an improvement?
If you plan to contribute to simdjson, please read our
* CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
message(STATUS"Building only the library. Advanced users and contributors may want to turn SIMDJSON_DEVELOPER_MODE to ON, e.g., via -D SIMDJSON_DEVELOPER_MODE=ON.")
The simdjson library is an open project written in C++. Contributions are invited. Contributors
agree to the project's license.
We have an extensive list of issues, and contributions toward any of these issues is invited.
Contributions can take the form of code samples, better documentation or design ideas.
In particular, the following contributions are invited:
- The library is focused on performance. Well-documented performance optimization are invited.
- Fixes to known or newly discovered bugs are always welcome. Typically, a bug fix should come with
a test demonstrating that the bug has been fixed.
- The simdjson library is advanced software and maintainability and flexibility are always a
concern. Specific contributions to improve maintainability and flexibility are invited.
We discourage the following types of contributions:
- Code refactoring. We all have our preferences as to how code should be written, but unnecessary
refactoring can waste time and introduce new bugs. If you believe that refactoring is needed, you
first must explain how it helps in concrete terms. Does it improve the performance?
- Applications of new language features for their own sake. Using advanced C++ language constructs
is actually a negative as it may reduce portability (to old compilers, old standard libraries and
systems) and reduce accessibility (to programmers that have not kept up), so it must be offsetted
by clear gains like performance or maintainability. When in doubt, avoid advanced C++ features
(beyond C++11).
- Style formatting. In general, please abstain from reformatting code just to make it look prettier.
Though code formatting is important, it can also be a waste of time if several contributors try to
tweak the code base toward their own preference. Please do not introduce unneeded white-space
changes.
In short, most code changes should either bring new features or better performance. We want to avoid unmotivated code changes.
Specific rules
----------
We have few hard rules, but we have some:
- Printing to standard output or standard error (`stderr`, `stdout`, `std::cerr`, `std::cout`) in the core library is forbidden. This follows from the [Writing R Extensions](https://cran.r-project.org/doc/manuals/R-exts.html) manual which states that "Compiled code should not write to stdout or stderr".
- Calls to `abort()` are forbidden in the core library. This follows from the [Writing R Extensions](https://cran.r-project.org/doc/manuals/R-exts.html) manual which states that "Under no circumstances should your compiled code ever call abort or exit".
- All source code files (.h, .cpp) must be ASCII.
- All C macros introduced in public headers need to be prefixed with either `SIMDJSON_` or `simdjson_`.
- We avoid trailing white space characters within lines. That is, your lines of code should not terminate with unnecessary spaces. Generally, please avoid making unnecessary changes to white-space characters when contributing code.
Tools, tests and benchmarks are not held to these same strict rules.
General Guidelines
----------
Contributors are encouraged to :
- Document their changes. Though we do not enforce a rule regarding code comments, we prefer that non-trivial algorithms and techniques be somewhat documented in the code.
- Follow as much as possible the existing code style. We do not enforce a specific code style, but we prefer consistency. We avoid contractions (isn't, aren't) in the comments.
- Modify as few lines of code as possible when working on an issue. The more lines you modify, the harder it is for your fellow human beings to understand what is going on.
- Tools may report "problems" with the code, but we never delegate programming to tools: if there is a problem with the code, we need to understand it. Thus we will not "fix" code merely to please a static analyzer.
- Provide tests for any new feature. We will not merge a new feature without tests.
- Run before/after benchmarks so that we can appreciate the effect of the changes on the performance.
Pull Requests
--------------
Pull requests are always invited. However, we ask that you follow these guidelines:
- It is wise to discuss your ideas first as part of an issue before you start coding. If you omit this step and code first, be prepared to have your code receive scrutiny and be dropped.
- Users should provide a rationale for their changes. Does it improve performance? Does it add a feature? Does it improve maintainability? Does it fix a bug? This must be explicitly stated as part of the pull request. Do not propose changes based on taste or intuition. We do not delegate programming to tools: that some tool suggested a code change is not reason enough to change the code.
1. When your code improves performance, please document the gains with a benchmark using hard numbers.
2. If your code fixes a bug, please either fix a failing test, or propose a new test.
3. Other types of changes must be clearly motivated. We openly discourage changes with no identifiable benefits.
- Changes should be focused and minimal. You should change as few lines of code as possible. Please do not reformat or touch files needlessly.
- New features must be accompanied by new tests, in general.
- Your code should pass our continuous-integration tests. It is your responsibility to ensure that your proposal pass the tests. We do not merge pull requests that would break our build.
- An exception to this would be changes to non-code files, such as documentation and assets, or trivial changes to code, such as comments, where it is encouraged to explicitly ask for skipping a CI run using the `[skip ci]` prefix in your Pull Request title **and** in the first line of the most recent commit in a push. Example for such a commit: `[skip ci] Fixed typo in power_of_ten's docs`
This benefits the project in such a way that the CI pipeline is not burdened by running jobs on changes that don't change any behavior in the code, which reduces wait times for other Pull Requests that do change behavior and require testing.
If the benefits of your proposed code remain unclear, we may choose to discard your code: that is not an insult, we frequently discard our own code. We may also consider various alternatives and choose another path. Again, that is not an insult or a sign that you have wasted your time.
Style
-----
Our formatting style is inspired by the LLVM style.
The simdjson library is written using the snake case: when a variable or a function is a phrase, each space is replaced by an underscore character, and the first letter of each word written in lowercase. Compile-time constants are written entirely in uppercase with the same underscore convention.
Code of Conduct
---------------
Though we do not have a formal code of conduct, we will not tolerate bullying, bigotry or
intimidation. Everyone is welcome to contribute. If you have concerns, you can raise them privately with the core team members (e.g., D. Lemire, J. Keiser).
We welcome contributions from women and less represented groups. If you need help, please reach out.
Consider the following points when engaging with the project:
- We discourage arguments from authority: ideas are discussed on their own merits and not based on who stated it.
- Be mindful that what you may view as an aggression is maybe merely a difference of opinion or a misunderstanding.
- Be mindful that a collection of small aggressions, even if mild in isolation, can become harmful.
Getting Started Hacking
-----------------------
An overview of simdjson's directory structure, with pointers to architecture and design
considerations and other helpful notes, can be found at [HACKING.md](HACKING.md).
Here is wisdom about how to build, test and run simdjson from within the repository. This is mostly useful for people who plan to contribute simdjson, or maybe study the design.
If you plan to contribute to simdjson, please read our [CONTRIBUTING](https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md) guide.
- [Hacking simdjson](#hacking-simdjson)
- [Build Quickstart](#build-quickstart)
- [Design notes](#design-notes)
- [Developer mode](#developer-mode)
- [Directory Structure and Source](#directory-structure-and-source)
- Stage 1. (Find marks) Identifies quickly structure elements, strings, and so forth. We validate UTF-8 encoding at that stage.
- Stage 2. (Structure building) Involves constructing a "tree" of sort (materialized as a tape) to navigate through the data. Strings and numbers are parsed at this stage.
The role of stage 1 is to identify pseudo-structural characters as quickly as possible. A character is pseudo-structural if and only if:
1. Not enclosed in quotes, AND
2. Is a non-whitespace character, AND
3. Its preceding character is either:
(a) a structural character, OR
(b) whitespace OR
(c) the final quote in a string.
This helps as we redefine some new characters as pseudo-structural such as the characters 1, G, n in the following:
> { "foo" : 1.5, "bar" : 1.5 GEOFF_IS_A_DUMMY bla bla , "baz", null }
Stage 1 also does unicode validation.
Stage 2 handles all of the rest: number parsings, recognizing atoms like true, false, null, and so forth.
Developer mode
--------------
Build system targets that are only useful for developers of the simdjson
library are behind the `SIMDJSON_DEVELOPER_MODE` option. Enabling this option
makes tests, examples, benchmarks and other developer targets available. Not
enabling this option means that you are a consumer of simdjson and thus you
only get the library targets and options.
Developer mode is forced to be on when the `CI` environment variable is set to
a value that CMake recognizes as "on", which is set to `true` in all of the CI
workflows used by simdjson.
Directory Structure and Source
------------------------------
Before diving into the directory structure, here are key concepts used in the codebase:
- **Amalgamated File**: A file that is conditionally included in the amalgamation process. These are wrapped in `#ifndef SIMDJSON_CONDITIONAL_INCLUDE` blocks and are included based on the target implementation (e.g., ARM64, x86). They include implementation-specific files (e.g., `arm64.h`) and generic files (e.g., under `generic/`). Amalgamated files have associated dependency files (`dependencies.h`) to track includes.
- **Amalgamator File**: A file that orchestrates the inclusion of amalgamated files. Examples: `arm64.h`, `arm64/implementation.h`, `generic/amalgamated.h`. These are not themselves amalgamated but control conditional inclusions.
- **Free Dependency File**: A top-level header that is always included unconditionally. These do not have dependency files and represent the public API (e.g., main headers).
- **Implementation-Specific File**: A file tied to a specific CPU architecture or instruction set (e.g., `arm64/`, `haswell/`). These must be amalgamated.
- **Generic File**: A shared file (under `generic/` or `simdjson/generic/`) that contains common code included once per implementation.
- **Builtin File**: Special files under `simdjson/builtin/` that handle the builtin implementation, a fallback/default implementation used when no optimized implementation is available.
- **Conditional Include Block**: A section wrapped in `#ifndef SIMDJSON_CONDITIONAL_INCLUDE` for editor-only or implementation-specific content.
The script `singleheader/amalgation_helper.py` will generate an HTML report which you can use to visualize the status of each file.
simdjson's source structure, from the top level, looks like this:
* **CMakeLists.txt:** The main build system.
* **include:** User-facing declarations and inline definitions (most user-facing functions are inlined).
* simdjson.h: the `simdjson` namespace. A "main include" that includes files from include/simdjson/. This is equivalent to
the distributed simdjson.h.
* simdjson/*.h: Declarations for public simdjson classes and functions.
* simdjson/*-inl.h: Definitions for public simdjson classes and functions.
* simdjson/internal/*.h: the `simdjson::internal` namespace. Private classes and functions used by the rest of simdjson.
* simdjson/dom.h: the `simdjson::dom` namespace. Includes all public DOM classes.
* simdjson/dom/*.h: Declarations/definitions for individual DOM classes.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere.h: `simdjson::<implementation>` namespace. Common implementation-specific tools like number and string parsing, as well as minification.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/*.h: implementation-specific functions such as , etc.
* simdjson/generic/*.h: the bulk of the actual code, written generically and compiled for each implementation, using functions defined in the implementation's .h files.
* simdjson/generic/dependencies.h: dependencies on common, non-implementation-specific simdjson classes. This will be included before including amalgamated.h.
* simdjson/generic/amalgamated.h: all generic ondemand classes for an implementation.
* simdjson/ondemand.h: the `simdjson::ondemand` namespace. Includes all public ondemand classes.
* simdjson/builtin.h: the `simdjson::builtin` namespace. Aliased to the most universal implementation available.
* simdjson/builtin/ondemand.h: the `simdjson::builtin::ondemand` namespace.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/ondemand.h: the `simdjson::<implementation>::ondemand` namespace. On-Demand compiled for the specific implementation.
* simdjson/generic/ondemand/dependencies.h: dependencies on common, non-implementation-specific simdjson classes. This will be included before including amalgamated.h.
* simdjson/generic/ondemand/amalgamated.h: all generic ondemand classes for an implementation.
* simdjson/builder.h: the `simdjson::builder` namespace. Includes all public builder classes.
* simdjson/builtin/builder.h: the `simdjson::builtin::builder` namespace.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/builder.h: the `simdjson::<implementation>::builder` namespace. Builder compiled for the specific implementation.
* simdjson/generic/builder/dependencies.h: dependencies on common, non-implementation-specific simdjson classes. This will be included before including amalgamated.h.
* simdjson/generic/builder/amalgamated.h: all generic builder classes for an implementation.
* **src:** The source files for non-inlined functionality (e.g. the architecture-specific parser
implementations).
* simdjson.cpp: A "main source" that includes all implementation files from src/. This is
equivalent to the distributed simdjson.cpp.
* *.cpp: other misc. implementations, such as `simdjson::implementation` and the minifier.
* generic/*.h: `simdjson::<implementation>` namespace. Generic implementation of the parser, particularly the `dom_parser_implementation`.
* generic/stage1/*.h: `simdjson::<implementation>::stage1` namespace. Generic implementation of the simd-heavy tokenizer/indexer pass of the simdjson parser. Used for the On-Demand interface
* generic/stage2/*.h: `simdjson::<implementation>::stage2` namespace. Generic implementation of the tape creator, which consumes the index from stage 1 and actually parses numbers and string and such. Used for the DOM interface.
Other important files and directories:
***.github/workflows:** Definitions for GitHub Actions (CI).
***singleheader:** Contains generated `simdjson.h` and `simdjson.cpp` that we release. The files `singleheader/simdjson.h` and `singleheader/simdjson.cpp` should never be edited by hand.
***singleheader/amalgamate.py:** Generates `singleheader/simdjson.h` and `singleheader/simdjson.cpp` for release (python script). If you add a new implementation (e.g., rvv), you need to edit this file (IMPLEMENTATIONS).
***singleheader/amalgation_helper.py:** Generates and `amalgamation_report.html` that helps you understand the status of each file.
***benchmark:** This is where we do benchmarking. Benchmarking is core to every change we make; the
cardinal rule is don't regress performance without knowing exactly why, and what you're trading
for it. Many of our benchmarks are microbenchmarks. We are effectively doing controlled scientific experiments for the purpose of understanding what affects our performance. So we simplify as much as possible. We try to avoid irrelevant factors such as page faults, interrupts, unnecessary system calls. We recommend checking the performance as follows:
```bash
mkdir build
cd build
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build . --config Release
benchmark/dom/parse ../jsonexamples/twitter.json
```
The last line becomes `./benchmark/Release/parse.exe ../jsonexample/twitter.json` under Windows. You may also use Google Benchmark:
The last line becomes `./benchmark/Release/bench_parse_call.exe` under Windows. Under Windows, you can also build with the clang compiler by adding `-T ClangCL` to the call to `cmake ..`: `cmake -T ClangCL ..`.
* **fuzz:** The source for fuzz testing. This lets us explore important edge and middle cases
* **fuzz:** The source for fuzz testing. This lets us explore important edge and middle cases
automatically, and is run in CI.
* **jsonchecker:** A set of JSON files used to check different functionality of the parser.
* **pass*.json:** Files that should pass validation.
* **fail*.json:** Files that should fail validation.
* **jsonchecker/minefield/y_*.json:** Files that should pass validation.
* **jsonchecker/minefield/n_*.json:** Files that should fail validation.
* **jsonexamples:** A wide spread of useful, real-world JSON files with different characteristics
and sizes.
* **test:** The tests are here. basictests.cpp and errortests.cpp are the primary ones.
* **tools:** Source for executables that can be distributed with simdjson. Some examples:
* `json2json mydoc.json` parses the document, constructs a model and then dumps back the result to standard output.
* `json2json -d mydoc.json` parses the document, constructs a model and then dumps model (as a tape) to standard output. The tape format is described in the accompanying file `tape.md`.
* `minify mydoc.json` minifies the JSON document, outputting the result to standard output. Minifying means to remove the unneeded white space characters.
* `jsonpointer mydoc.json <jsonpath> <jsonpath> ... <jsonpath>` parses the document, constructs a model and then processes a series of [JSON Pointer paths](https://tools.ietf.org/html/rfc6901). The result is itself a JSON document.
> **Don't modify the files in singleheader/ directly; these are automatically generated.**
While simdjson distributes just two files from the singleheader/ directory, we *maintain* the code in
multiple files under include/ and src/. The files include/simdjson.h and src/simdjson.cpp are the "spine" for
these, and you can include them as if they were the corresponding singleheader/ files.
Runtime Dispatching
--------------------
A key feature of simdjson is the ability to compile different processing kernels, optimized for specific instruction sets, and to select
the most appropriate kernel at runtime. This ensures that users get the very best performance while still enabling simdjson to run everywhere.
This technique is frequently called runtime dispatching. The simdjson achieves runtime dispatching entirely in C++: we do not assume
that the user is building the code using CMake, for example.
To make runtime dispatching work, it is critical that the code be compiled for the lowest supported processor. In particular, you should
not use flags such as -mavx2, /arch:AVX2 and so forth while compiling simdjson. When you do so, you allow the compiler to use advanced
instructions. In turn, these advanced instructions present in the code may cause a runtime failure if the runtime processor does not
support them. Even a simple loop, compiled with these flags, might generate binary code that only run on advanced processors.
So we compile simdjson for a generic processor. Our users should do the same if they want simdjson's runtime dispatch to work. It is important
to understand that if runtime dispatching does not work, then simdjson will cause crashes on older processors. Of course, if a user chooses
to compile their code for a specific instruction set (e.g., AVX2), they are responsible for the failures if they later run their code
on a processor that does not support AVX2. Yet, if we were to entice these users to do so, we would share the blame: thus we carefully instruct
users to compile their code in a generic way without doing anything to enable advanced instructions.
We only use runtime dispatching on x64 (AMD/Intel) platforms, at the moment. On ARM processors, we would need a standard way to query, at runtime,
the processor for its supported features. We do not know how to do so on ARM systems in general. Thankfully it is not yet a concern: 64-bit ARM
processors are fairly uniform as far as the instruction sets they support.
In all cases, simdjson uses advanced instructions by relying on "intrinsic functions": we do not write assembly code. The intrinsic functions
are special functions that the compiler might recognize and translate into fast code. To make runtime dispatching work, we rely on the fact that
the header providing these instructions
(intrin.h under Visual Studio, x86intrin.h elsewhere) defines all of the intrinsic functions, including those that are not supported
processor.
At this point, we are require to use one of two main strategies.
1. On POSIX systems, the main compilers (LLVM clang, GNU gcc) allow us to use any intrinsic function after including the header, but they fail to inline the resulting instruction if the target processor does not support them. Because we compile for a generic processor, we would not be able to use most intrinsic functions. Thankfully, more recent versions of these compilers allow us to flag a region of code with a specific target, so that we can compile only some of the code with support for advanced instructions. Thus in our C++, one might notice macros like `TARGET_HASWELL`. It is then our responsibility, at runtime, to only run the regions of code (that we call kernels) matching the properties of the runtime processor. The benefit of this approach is that the compiler not only let us use intrinsic functions, but it can also optimize the rest of the code in the kernel with advanced instructions we enabled.
2. Under Visual Studio, the problem is somewhat simpler. Visual Studio will not only provide the intrinsic functions, but it will also allow us to use them. They will compile just fine. It is at runtime that they may cause a crash. So we do not need to mark regions of code for compilation toward advanced processors (e.g., with `TARGET_HASWELL` macros). The downside of the Visual Studio approach is that the compiler is not allowed to use advanced instructions others than those we specify. In principle, this means that Visual Studio has weaker optimization opportunities.
We also handle the special case where a user is compiling using LLVM clang under Windows, [using the Visual Studio toolchain](https://devblogs.microsoft.com/cppblog/clang-llvm-support-in-visual-studio/). If you compile with LLVM clang under Visual Studio, then the header files (intrin.h or x86intrin.h) no longer provides the intrinsic functions that are unsupported by the processor. This appears to be deliberate on the part of the LLVM engineers. With a few lines of code, we handle this scenario just like LLVM clang under a POSIX system, but forcing the inclusion of the specific headers, and rolling our own intrinsic function as needed.
Regenerating Single-Header Files
---------------------------------------
The simdjson.h and simdjson.cpp files in the singleheader directory are not always up-to-date with the rest of the code; they are only ever
systematically regenerated on releases. To ensure you have the latest code, you can regenerate them by running this at the top level:
```bash
mkdir build
cd build
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build . # needed, because currently dependencies do not work fully for the amalgamate target
cmake --build . --target amalgamate
```
You need to have python3 installed on your system.
The amalgamator script `amalgamate.py` generates singleheader/simdjson.h by
reading through include/simdjson.h, copy/pasting each header file into the amalgamated file at the
point it gets included (but only once per header). singleheader/simdjson.cpp is generated from
src/simdjson.cpp the same way, except files under generic/ may be included and copy/pasted multiple
times.
## Usage (CMake on 64-bit platforms like Linux, FreeBSD or macOS)
Requirements: In addition to git, we require a recent version of CMake as well as bash.
1. On macOS, the easiest way to install cmake might be to use [brew](https://brew.sh) and then type
```
brew install cmake
```
2. Under Linux, you might be able to install CMake as follows:
```
apt-get update -qq
apt-get install -y cmake
```
3. On FreeBSD, you might be able to install bash and CMake as follows:
```
pkg update -f
pkg install bash
pkg install cmake
```
You need a recent compiler like clang or gcc. We recommend at least GNU GCC/G++ 7 or LLVM clang 6.
Building: While in the project repository, do the following:
```
mkdir build
cd build
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build .
ctest
```
CMake will build a library. By default, it builds a static library (e.g., libsimdjson.a on Linux).
In some cases, you may want to specify your compiler, especially if the default compiler on your system is too old. You need to tell cmake which compiler you wish to use by setting the CC and CXX variables. Under bash, you can do so with commands such as `export CC=gcc-7` and `export CXX=g++-7`. You can also do it as part of the `cmake` command: `cmake -DCMAKE_CXX_COMPILER=g++ ..`. You may proceed as follows:
```
brew install gcc@8
mkdir build
cd build
export CXX=g++-8 CC=gcc-8
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build .
ctest
```
If your compiler does not default on C++11 support or better you may get failing tests. If so, you may be able to exclude the failing tests by replacing `ctest` with `ctest -E "^quickstart$"`.
Note that the name of directory (`build`) is arbitrary, you can name it as you want (e.g., `buildgcc`) and you can have as many different such directories as you would like (one per configuration).
## Usage (CMake on 64-bit Windows using Visual Studio 2019 or better)
Recent versions of Visual Studio support CMake natively, [please refer to the Visual Studio documentation](https://learn.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-170).
We assume you have a common 64-bit Windows PC with at least Visual Studio 2019.
- Grab the simdjson code from GitHub, e.g., by cloning it using [GitHub Desktop](https://desktop.github.com/).
- Install [CMake](https://cmake.org/download/). When you install it, make sure to ask that `cmake` be made available from the command line. Please choose a recent version of cmake.
- Create a subdirectory within simdjson, such as `build`.
- Using a shell, go to this newly created directory. You can start a shell directly from GitHub Desktop (Repository > Open in Command Prompt).
- Type `cmake ..` in the shell while in the `build` repository.
- This last command (`cmake ...`) created a Visual Studio solution file in the newly created directory (e.g., `simdjson.sln`). Open this file in Visual Studio. You should now be able to build the project and run the tests. For example, in the `Solution Explorer` window (available from the `View` menu), right-click `ALL_BUILD` and select `Build`. To test the code, still in the `Solution Explorer` window, select `RUN_TESTS` and select `Build`.
Though having Visual Studio installed is necessary, one can build simdjson using only cmake commands:
- `mkdir build`
- `cd build`
- `cmake ..`
- `cmake --build . --config Release`
Furthermore, if you have installed LLVM clang on Windows, for example as a component of Visual Studio 2019, you can configure and build simdjson using LLVM clang on Windows using cmake:
- `mkdir build`
- `cd build`
- `cmake -T ClangCL ..`
- `cmake --build . --config Release`
## Various References
- [How to implement atoi using SIMD?](https://stackoverflow.com/questions/35127060/how-to-implement-atoi-using-simd)
- [Parsing JSON is a Minefield 💣](http://seriot.ch/parsing_json.php)
If you are planning to use simdjson in a product, please work from one of our releases.
Quick Start
-----------
The simdjson library is easily consumable with a single .h and .cpp file.
0. Prerequisites: `g++` (version 7 or better) or `clang++` (version 6 or better), and a 64-bit
system with a command-line shell (e.g., Linux, macOS, freeBSD). We also support programming
environments like Visual Studio and Xcode, but different steps are needed. Users of clang++ may need to specify the C++ version (e.g., `c++ -std=c++17`) since clang++ tends to default on C++98.
1. Pull [simdjson.h](singleheader/simdjson.h) and [simdjson.cpp](singleheader/simdjson.cpp) into a
directory, along with the sample file [twitter.json](jsonexamples/twitter.json). You can download them with the `wget` utility:
* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/7G5qE4sr9)
* [simdjson examples with errors without exceptions](https://godbolt.org/z/e9dWb9E4v)
Performance results
-------------------
The simdjson library uses three-quarters less instructions than state-of-the-art parser [RapidJSON](https://rapidjson.org). To our knowledge, simdjson is the first fully-validating JSON parser
to run at [gigabytes per second](https://en.wikipedia.org/wiki/Gigabyte) (GB/s) on commodity processors. It can parse millions of JSON documents per second on a single core.
The following figure represents parsing speed in GB/s for parsing various files
on an Intel Skylake processor (3.4 GHz) using the GNU GCC 10 compiler (with the -O3 flag).
We compare against the best and fastest C++ libraries on benchmarks that load and process the data.
The simdjson library offers full unicode ([UTF-8](https://en.wikipedia.org/wiki/UTF-8)) validation and exact
number parsing.
<img src="doc/rome.png" width="60%">
The simdjson library offers high speed whether it processes tiny files (e.g., 300 bytes)
or larger files (e.g., 3MB). The following plot presents parsing
speed for [synthetic files over various sizes generated with a script](https://github.com/simdjson/simdjson_experiments_vldb2019/blob/master/experiments/growing/gen.py) on a 3.4 GHz Skylake processor (GNU GCC 9, -O3).
<img src="doc/growing.png" width="60%">
[All our experiments are reproducible](https://github.com/simdjson/simdjson_experiments_vldb2019).
For NDJSON files, we can exceed 3 GB/s with [our multithreaded parsing functions](https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md).
We distinguish between "bindings" (which just wrap the C++ code) and a port to another programming language (which reimplements everything).
- [ZippyJSON](https://github.com/michaeleisel/zippyjson): Swift bindings for the simdjson project.
- [libpy_simdjson](https://github.com/gerrymanoim/libpy_simdjson/): high-speed Python bindings for simdjson using [libpy](https://github.com/quantopian/libpy).
- [pysimdjson](https://github.com/TkTech/pysimdjson): Python bindings for the simdjson project.
- [cysimdjson](https://github.com/TeskaLabs/cysimdjson): high-speed Python bindings for the simdjson project.
The simdjson library takes advantage of modern microarchitectures, parallelizing with SIMD vector
instructions, reducing branch misprediction, and reducing data dependency to take advantage of each
CPU's multiple execution cores.
Our default front-end is called On-Demand, and we wrote a paper about it:
- John Keiser, Daniel Lemire, [On-Demand JSON: A Better Way to Parse Documents?](http://arxiv.org/abs/2312.17149), Software: Practice and Experience 54 (6), 2024.
Some people [enjoy reading the first (2019) simdjson paper](https://arxiv.org/abs/1902.08318): A description of the design
and implementation of simdjson is in our research article:
- Geoff Langdale, Daniel Lemire, [Parsing Gigabytes of JSON per Second](https://arxiv.org/abs/1902.08318), VLDB Journal 28 (6), 2019.
We have an in-depth paper focused on the UTF-8 validation:
- John Keiser, Daniel Lemire, [Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090), Software: Practice & Experience 51 (5), 2021.
We also have an informal [blog post providing some background and context](https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of-json-per-second/).
For the video inclined, <br />
[](http://www.youtube.com/watch?v=wlvKAT7SZIQ)<br />
(It was the best voted talk, we're kinda proud of it.)
Citing this work
-----------------
If you use simdjson in published research, please cite the software library. A suitable BibTeX entry is:
```bibtex
@misc{simdjson,
title={{The simdjson library: Parsing Gigabytes of JSON per Second}},
author={Daniel Lemire and Geoff Langdale and John Keiser and Paul Dreik and Francisco Thiesen and others},
year={2019},
howpublished={Software library},
note={https://github.com/simdjson/simdjson}
}
```
Funding
-------
The work is supported by the Natural Sciences and Engineering Research Council of Canada under grants
Head over to [CONTRIBUTING.md](CONTRIBUTING.md) for information on contributing to simdjson, and
[HACKING.md](HACKING.md) for information on source, building, and architecture/design.
Stars
------
[](https://www.star-history.com/#simdjson/simdjson&Date)
License
-------
This code is made available under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) as well as under the MIT License. As a user, you can pick the license you prefer.
Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it is under the liberal (business-friendly) MIT license.
For compilers that do not support [C++17](https://en.wikipedia.org/wiki/C%2B%2B17), we bundle the string-view library which is published under the [Boost license](http://www.boost.org/LICENSE_1_0.txt). Like the Apache license, the Boost license is a permissive license allowing commercial redistribution.
For efficient number serialization, we bundle Florian Loitsch's implementation of the Grisu2 algorithm for binary to decimal floating-point numbers. The implementation was slightly modified by JSON for Modern C++ library. Both Florian Loitsch's implementation and JSON for Modern C++ are provided under the MIT license.
For runtime dispatching, we use some code from the PyTorch project licensed under 3-clause BSD.
<divclass="line"><aid="l00002"name="l00002"></a><spanclass="lineno"> 2</span><spanclass="preprocessor">#error simdjson/generic/dependencies.h must be included before simdjson/generic/amalgamated.h!</span></div>
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<trid="row_0_"class="even"><tdclass="entry"><spanstyle="width:0px;display:inline-block;"> </span><spanid="arr_0_"class="arrow"onclick="toggleFolder('0_')">▼</span><spanclass="icona"><spanclass="icon">N</span></span><aclass="el"href="namespacesimdjson.html"target="_self">simdjson</a></td><tdclass="desc">The top level simdjson namespace, containing everything the library provides </td></tr>
<trid="row_0_0_"class="odd"><tdclass="entry"><spanstyle="width:16px;display:inline-block;"> </span><spanid="arr_0_0_"class="arrow"onclick="toggleFolder('0_0_')">▼</span><spanclass="icona"><spanclass="icon">N</span></span><aclass="el"href="namespacesimdjson_1_1dom.html"target="_self">dom</a></td><tdclass="desc">A DOM API on top of the simdjson parser </td></tr>
<trid="row_0_0_2_"class="odd"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanid="arr_0_0_2_"class="arrow"onclick="toggleFolder('0_0_2_')">▼</span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1dom_1_1document__stream.html"target="_self">document_stream</a></td><tdclass="desc">A forward-only stream of documents </td></tr>
<trid="row_0_0_2_0_"class="even"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1dom_1_1document__stream_1_1iterator.html"target="_self">iterator</a></td><tdclass="desc">An iterator through a forward-only stream of documents </td></tr>
<trid="row_0_0_3_"class="odd"><tdclass="entry"><spanstyle="width:48px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1dom_1_1element.html"target="_self">element</a></td><tdclass="desc">A JSON element </td></tr>
<trid="row_0_0_4_"class="even"><tdclass="entry"><spanstyle="width:48px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1dom_1_1key__value__pair.html"target="_self">key_value_pair</a></td><tdclass="desc">Key/value pair in an object </td></tr>
<trid="row_0_1_0_0_"class="even"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1builder_1_1string__builder.html"target="_self">string_builder</a></td><tdclass="desc">A builder for JSON strings representing documents </td></tr>
<trid="row_0_1_1_"class="odd"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanid="arr_0_1_1_"class="arrow"onclick="toggleFolder('0_1_1_')">▼</span><spanclass="icona"><spanclass="icon">N</span></span><aclass="el"href="namespacesimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand.html"target="_self">ondemand</a></td><tdclass="desc">A fast, simple, DOM-like interface that parses JSON as you use it </td></tr>
<trid="row_0_1_1_3_"class="odd"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1document__reference.html"target="_self">document_reference</a></td><tdclass="desc">A <aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1document__reference.html"title="A document_reference is a thin wrapper around a document reference instance.">document_reference</a> is a thin wrapper around a document reference instance </td></tr>
<trid="row_0_1_1_4_"class="even"><tdclass="entry"><spanstyle="width:48px;display:inline-block;"> </span><spanid="arr_0_1_1_4_"class="arrow"onclick="toggleFolder('0_1_1_4_')">▼</span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1document__stream.html"target="_self">document_stream</a></td><tdclass="desc">A forward-only stream of documents </td></tr>
<trid="row_0_1_1_5_"class="even"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1field.html"target="_self">field</a></td><tdclass="desc">A JSON field (key/value pair) in an object </td></tr>
<trid="row_0_1_1_6_"class="odd"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="structsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1number.html"target="_self">number</a></td><tdclass="desc">A type representing a JSON number </td></tr>
<trid="row_0_1_1_7_"class="even"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1object.html"target="_self">object</a></td><tdclass="desc">A forward-only JSON object field iterator </td></tr>
<trid="row_0_1_1_9_"class="even"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1parser.html"target="_self">parser</a></td><tdclass="desc">A JSON fragment iterator </td></tr>
<trid="row_0_1_1_10_"class="odd"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1raw__json__string.html"target="_self">raw_json_string</a></td><tdclass="desc">A string escaped per JSON rules, terminated with quote (") </td></tr>
<trid="row_0_1_1_11_"class="even"><tdclass="entry"><spanstyle="width:64px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1value.html"target="_self">value</a></td><tdclass="desc">An ephemeral JSON value returned during iteration </td></tr>
<trid="row_0_1_3_"class="even"><tdclass="entry"><spanstyle="width:48px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="structsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1implementation__simdjson__result__base.html"target="_self">implementation_simdjson_result_base</a></td><tdclass="desc">The result of a simdjson operation that could fail </td></tr>
<trid="row_0_4_"class="even"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1implementation.html"target="_self">implementation</a></td><tdclass="desc">An implementation of simdjson for a particular CPU architecture </td></tr>
<trid="row_0_5_"class="odd"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1padded__memory__map.html"target="_self">padded_memory_map</a></td><tdclass="desc">A class representing a memory-mapped file with padding </td></tr>
<trid="row_0_6_"class="even"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="structsimdjson_1_1padded__string.html"target="_self">padded_string</a></td><tdclass="desc">String with extra allocation for ease of use with parser::parse() </td></tr>
<trid="row_0_7_"class="odd"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1padded__string__builder.html"target="_self">padded_string_builder</a></td><tdclass="desc">Builder for constructing <aclass="el"href="structsimdjson_1_1padded__string.html"title="String with extra allocation for ease of use with parser::parse()">padded_string</a> incrementally </td></tr>
<trid="row_0_8_"class="even"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="classsimdjson_1_1padded__string__view.html"target="_self">padded_string_view</a></td><tdclass="desc">User-provided string that promises it has extra padded bytes at the end for use with parser::parse() </td></tr>
<trid="row_0_9_"class="odd"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="structsimdjson_1_1simdjson__error.html"target="_self">simdjson_error</a></td><tdclass="desc">Exception thrown when an exception-supporting simdjson method is called </td></tr>
<trid="row_0_10_"class="even"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="structsimdjson_1_1simdjson__result.html"target="_self">simdjson_result</a></td><tdclass="desc">The result of a simdjson operation that could fail </td></tr>
<trid="row_0_11_"class="odd"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="structsimdjson_1_1simdjson__result_3_01dom_1_1array_01_4.html"target="_self">simdjson_result< dom::array ></a></td><tdclass="desc">The result of a JSON conversion that may fail </td></tr>
<trid="row_0_13_"class="odd"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="structsimdjson_1_1simdjson__result_3_01dom_1_1element_01_4.html"target="_self">simdjson_result< dom::element ></a></td><tdclass="desc">The result of a JSON navigation that may fail </td></tr>
<trid="row_0_14_"class="even"><tdclass="entry"><spanstyle="width:32px;display:inline-block;"> </span><spanclass="icona"><spanclass="icon">C</span></span><aclass="el"href="structsimdjson_1_1simdjson__result_3_01dom_1_1object_01_4.html"target="_self">simdjson_result< dom::object ></a></td><tdclass="desc">The result of a JSON conversion that may fail </td></tr>
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<ul>
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00013"name="l00013"></a><spanclass="lineno"> 13</span><spanclass="comment">// We sometimes call trailing_zero on inputs that are zero,</span></div>
<divclass="line"><aid="l00014"name="l00014"></a><spanclass="lineno"> 14</span><spanclass="comment">// but the algorithms do not end up using the returned value.</span></div>
<divclass="line"><aid="l00015"name="l00015"></a><spanclass="lineno"> 15</span><spanclass="comment">// Sadly, sanitizers are not smart enough to figure it out.</span></div>
<divclass="line"><aid="l00017"name="l00017"></a><spanclass="lineno"> 17</span><spanclass="comment">// This function can be used safely even if not all bytes have been</span></div>
<divclass="line"><aid="l00019"name="l00019"></a><spanclass="lineno"> 19</span><spanclass="comment">// See issue https://github.com/simdjson/simdjson/issues/1965</span></div>
<divclass="line"><aid="l00024"name="l00024"></a><spanclass="lineno"> 24</span><spanclass="comment">// Search the mask data from least significant bit (LSB)</span></div>
<divclass="line"><aid="l00025"name="l00025"></a><spanclass="lineno"> 25</span><spanclass="comment">// to the most significant bit (MSB) for a set bit (1).</span></div>
<divclass="line"><aid="l00033"name="l00033"></a><spanclass="lineno"> 33</span><spanclass="comment">/* result might be undefined when input_num is zero */</span></div>
<divclass="line"><aid="l00035"name="l00035"></a><spanclass="lineno"> 35</span><spanclass="keywordflow">return</span> input_num & (input_num-1);</div>
<divclass="line"><aid="l00038"name="l00038"></a><spanclass="lineno"> 38</span><spanclass="comment">// We sometimes call leading_zeroes on inputs that are zero,</span></div>
<divclass="line"><aid="l00039"name="l00039"></a><spanclass="lineno"> 39</span><spanclass="comment">// but the algorithms do not end up using the returned value.</span></div>
<divclass="line"><aid="l00040"name="l00040"></a><spanclass="lineno"> 40</span><spanclass="comment">// Sadly, sanitizers are not smart enough to figure it out.</span></div>
<divclass="line"><aid="l00041"name="l00041"></a><spanclass="lineno"> 41</span><spanclass="comment">// Applies only when SIMDJSON_PREFER_REVERSE_BITS is defined and true.</span></div>
<divclass="line"><aid="l00042"name="l00042"></a><spanclass="lineno"> 42</span><spanclass="comment">// (See below.)</span></div>
<divclass="line"><aid="l00044"name="l00044"></a><spanclass="lineno"> 44</span><spanclass="comment">/* result might be undefined when input_num is zero */</span></div>
<divclass="line"><aid="l00048"name="l00048"></a><spanclass="lineno"> 48</span><spanclass="comment">// Search the mask data from most significant bit (MSB)</span></div>
<divclass="line"><aid="l00049"name="l00049"></a><spanclass="lineno"> 49</span><spanclass="comment">// to least significant bit (LSB) for a set bit (1).</span></div>
<divclass="line"><aid="l00059"name="l00059"></a><spanclass="lineno"> 59</span><spanclass="comment">/* result might be undefined when input_num is zero */</span></div>
<divclass="line"><aid="l00073"name="l00073"></a><spanclass="lineno"> 73</span><spanclass="comment"> * We use SIMDJSON_PREFER_REVERSE_BITS as a hint that algorithms that</span></div>
<divclass="line"><aid="l00074"name="l00074"></a><spanclass="lineno"> 74</span><spanclass="comment"> * work well with bit reversal may use it.</span></div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00013"name="l00013"></a><spanclass="lineno"> 13</span><spanclass="comment">// Perform a "cumulative bitwise xor," flipping bits each time a 1 is encountered.</span></div>
<divclass="line"><aid="l00015"name="l00015"></a><spanclass="lineno"> 15</span><spanclass="comment">// For example, prefix_xor(00100100) == 00011100</span></div>
<divclass="line"><aid="l00019"name="l00019"></a><spanclass="lineno"> 19</span><spanclass="comment">// We could do this with PMULL, but it is apparently slow.</span></div>
<divclass="line"><aid="l00021"name="l00021"></a><spanclass="lineno"> 21</span><spanclass="comment">//#ifdef __ARM_FEATURE_CRYPTO // some ARM processors lack this extension</span></div>
<divclass="line"><aid="l00024"name="l00024"></a><spanclass="lineno"> 24</span><spanclass="comment">// Analysis by @sebpop:</span></div>
<divclass="line"><aid="l00025"name="l00025"></a><spanclass="lineno"> 25</span><spanclass="comment">// When diffing the assembly for src/stage1_find_marks.cpp I see that the eors are all spread out</span></div>
<divclass="line"><aid="l00026"name="l00026"></a><spanclass="lineno"> 26</span><spanclass="comment">// in between other vector code, so effectively the extra cycles of the sequence do not matter</span></div>
<divclass="line"><aid="l00027"name="l00027"></a><spanclass="lineno"> 27</span><spanclass="comment">// because the GPR units are idle otherwise and the critical path is on the FP side.</span></div>
<divclass="line"><aid="l00028"name="l00028"></a><spanclass="lineno"> 28</span><spanclass="comment">// Also the PMULL requires two extra fmovs: GPR->FP (3 cycles in N1, 5 cycles in A72 )</span></div>
<divclass="line"><aid="l00029"name="l00029"></a><spanclass="lineno"> 29</span><spanclass="comment">// and FP->GPR (2 cycles on N1 and 5 cycles on A72.)</span></div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="ttc"id="aclasssimdjson_1_1implementation_html"><divclass="ttname"><ahref="classsimdjson_1_1implementation.html">simdjson::implementation</a></div><divclass="ttdoc">An implementation of simdjson for a particular CPU architecture.</div><divclass="ttdef"><b>Definition</b><ahref="implementation_8h_source.html#l00045">implementation.h:45</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_a7b735a3a50ba79e3f7f14df5f77d8da9"><divclass="ttname"><ahref="namespacesimdjson.html#a7b735a3a50ba79e3f7f14df5f77d8da9">simdjson::error_code</a></div><divclass="ttdeci">error_code</div><divclass="ttdoc">All possible errors returned by simdjson.</div><divclass="ttdef"><b>Definition</b><ahref="error_8h_source.html#l00019">error.h:19</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_aeb4ef5cab43d52da3fdd99cb689aff2c"><divclass="ttname"><ahref="namespacesimdjson.html#aeb4ef5cab43d52da3fdd99cb689aff2c">simdjson::minify</a></div><divclass="ttdeci">std::string minify(T x)</div><divclass="ttdoc">Minifies a JSON element or document, printing the smallest possible valid JSON.</div><divclass="ttdef"><b>Definition</b><ahref="dom_2serialization_8h_source.html#l00274">serialization.h:274</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00008"name="l00008"></a><spanclass="lineno"> 8</span><spanclass="comment">// This should be the correct header whether</span></div>
<divclass="line"><aid="l00009"name="l00009"></a><spanclass="lineno"> 9</span><spanclass="comment">// you use visual studio or other compilers.</span></div>
<divclass="ttc"id="anamespacesimdjson_html_aecdd750132f0eb123a6d61113b4197bf"><divclass="ttname"><ahref="namespacesimdjson.html#aecdd750132f0eb123a6d61113b4197bf">simdjson::SIMDJSON_PADDING</a></div><divclass="ttdeci">constexpr size_t SIMDJSON_PADDING</div><divclass="ttdoc">The amount of padding needed in a buffer to parse JSON.</div><divclass="ttdef"><b>Definition</b><ahref="base_8h_source.html#l00033">base.h:33</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00021"name="l00021"></a><spanclass="lineno"> 21</span><spanclass="comment">// we don't have SSE, so let us use a scalar function</span></div>
<divclass="line"><aid="l00036"name="l00036"></a><spanclass="lineno"> 36</span><spanclass="comment">// ARM64 has native support for 64-bit multiplications, no need to emultate</span></div>
<divclass="line"><aid="l00040"name="l00040"></a><spanclass="lineno"> 40</span> answer.low = _umul128(value1, value2, &answer.high); <spanclass="comment">// _umul128 not available on ARM64</span></div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00017"name="l00017"></a><spanclass="lineno"> 17</span><spanclass="comment">// Start of private section with Visual Studio workaround</span></div>
<divclass="line"><aid="l00068"name="l00068"></a><spanclass="lineno"> 68</span><spanclass="comment">// End of private section with Visual Studio workaround</span></div>
<divclass="line"><aid="l00077"name="l00077"></a><spanclass="lineno"> 77</span><spanclass="comment">// Base class of simd8<uint8_t> and simd8<bool>, both of which use uint8x16_t internally.</span></div>
<divclass="line"><aid="l00107"name="l00107"></a><spanclass="lineno"> 107</span><spanclass="comment">// SIMD byte mask type (returned by things like eq and gt)</span></div>
<divclass="line"><aid="l00121"name="l00121"></a><spanclass="lineno"> 121</span><spanclass="comment">// We return uint32_t instead of uint16_t because that seems to be more efficient for most</span></div>
<divclass="line"><aid="l00122"name="l00122"></a><spanclass="lineno"> 122</span><spanclass="comment">// purposes (cutting it down to uint16_t costs performance in some compilers).</span></div>
<divclass="line"><aid="l00137"name="l00137"></a><spanclass="lineno"> 137</span><spanclass="comment">// Returns 4-bit out of each byte, alternating between the high 4 bits and low</span></div>
<divclass="line"><aid="l00138"name="l00138"></a><spanclass="lineno"> 138</span><spanclass="comment">// bits result it is 64 bit.</span></div>
<divclass="line"><aid="l00179"name="l00179"></a><spanclass="lineno"> 179</span><spanclass="comment">// Repeat 16 values as many times as necessary (usually for lookup tables)</span></div>
<divclass="line"><aid="l00197"name="l00197"></a><spanclass="lineno"> 197</span><spanclass="comment">// Addition/subtraction are the same for signed and unsigned</span></div>
<divclass="line"><aid="l00212"name="l00212"></a><spanclass="lineno"> 212</span><spanclass="comment">// Same as >, but instead of guaranteeing all 1's == true, false = 0 and true = nonzero. For ARM, returns all 1's.</span></div>
<divclass="line"><aid="l00214"name="l00214"></a><spanclass="lineno"> 214</span><spanclass="comment">// Same as <, but instead of guaranteeing all 1's == true, false = 0 and true = nonzero. For ARM, returns all 1's.</span></div>
<divclass="line"><aid="l00226"name="l00226"></a><spanclass="lineno"> 226</span><spanclass="comment">// Perform a lookup assuming the value is between 0 and 16 (undefined behavior for out of range values)</span></div>
<divclass="line"><aid="l00232"name="l00232"></a><spanclass="lineno"> 232</span><spanclass="comment">// Returns 4-bit out of each byte, alternating between the high 4 bits and low</span></div>
<divclass="line"><aid="l00233"name="l00233"></a><spanclass="lineno"> 233</span><spanclass="comment">// bits result it is 64 bit.</span></div>
<divclass="line"><aid="l00238"name="l00238"></a><spanclass="lineno"> 238</span><spanclass="comment">// Copies to 'output" all bytes corresponding to a 0 in the mask (interpreted as a bitset).</span></div>
<divclass="line"><aid="l00239"name="l00239"></a><spanclass="lineno"> 239</span><spanclass="comment">// Passing a 0 value for mask would be equivalent to writing out every byte to output.</span></div>
<divclass="line"><aid="l00240"name="l00240"></a><spanclass="lineno"> 240</span><spanclass="comment">// Only the first 16 - count_ones(mask) bytes of the result are significant but 16 bytes</span></div>
<divclass="line"><aid="l00241"name="l00241"></a><spanclass="lineno"> 241</span><spanclass="comment">// get written.</span></div>
<divclass="line"><aid="l00242"name="l00242"></a><spanclass="lineno"> 242</span><spanclass="comment">// Design consideration: it seems like a function with the</span></div>
<divclass="line"><aid="l00243"name="l00243"></a><spanclass="lineno"> 243</span><spanclass="comment">// signature simd8<L> compress(uint16_t mask) would be</span></div>
<divclass="line"><aid="l00244"name="l00244"></a><spanclass="lineno"> 244</span><spanclass="comment">// sensible, but the AVX ISA makes this kind of approach difficult.</span></div>
<divclass="line"><aid="l00250"name="l00250"></a><spanclass="lineno"> 250</span><spanclass="comment">// this particular implementation was inspired by work done by @animetosho</span></div>
<divclass="line"><aid="l00251"name="l00251"></a><spanclass="lineno"> 251</span><spanclass="comment">// we do it in two steps, first 8 bytes and then second 8 bytes</span></div>
<divclass="line"><aid="l00254"name="l00254"></a><spanclass="lineno"> 254</span><spanclass="comment">// next line just loads the 64-bit values thintable_epi8[mask1] and</span></div>
<divclass="line"><aid="l00255"name="l00255"></a><spanclass="lineno"> 255</span><spanclass="comment">// thintable_epi8[mask2] into a 128-bit register, using only</span></div>
<divclass="line"><aid="l00256"name="l00256"></a><spanclass="lineno"> 256</span><spanclass="comment">// two instructions on most compilers.</span></div>
<divclass="line"><aid="l00259"name="l00259"></a><spanclass="lineno"> 259</span><spanclass="comment">// we increment by 0x08 the second half of the mask</span></div>
<divclass="line"><aid="l00266"name="l00266"></a><spanclass="lineno"> 266</span><spanclass="comment">// this is the version "nearly pruned"</span></div>
<divclass="line"><aid="l00268"name="l00268"></a><spanclass="lineno"> 268</span><spanclass="comment">// we still need to put the two halves together.</span></div>
<divclass="line"><aid="l00269"name="l00269"></a><spanclass="lineno"> 269</span><spanclass="comment">// we compute the popcount of the first half:</span></div>
<divclass="line"><aid="l00271"name="l00271"></a><spanclass="lineno"> 271</span><spanclass="comment">// then load the corresponding mask, what it does is to write</span></div>
<divclass="line"><aid="l00272"name="l00272"></a><spanclass="lineno"> 272</span><spanclass="comment">// only the first pop1 bytes from the first 8 bytes, and then</span></div>
<divclass="line"><aid="l00273"name="l00273"></a><spanclass="lineno"> 273</span><spanclass="comment">// it fills in with the bytes from the second 8 bytes + some filling</span></div>
<divclass="line"><aid="l00274"name="l00274"></a><spanclass="lineno"> 274</span><spanclass="comment">// at the end.</span></div>
<divclass="line"><aid="l00280"name="l00280"></a><spanclass="lineno"> 280</span><spanclass="comment">// Copies all bytes corresponding to a 0 in the low half of the mask (interpreted as a</span></div>
<divclass="line"><aid="l00281"name="l00281"></a><spanclass="lineno"> 281</span><spanclass="comment">// bitset) to output1, then those corresponding to a 0 in the high half to output2.</span></div>
<divclass="line"><aid="l00289"name="l00289"></a><spanclass="lineno"> 289</span><spanclass="comment">// we increment by 0x08 the second half of the mask</span></div>
<divclass="line"><aid="l00296"name="l00296"></a><spanclass="lineno"> 296</span><spanclass="comment">// store each result (with the second store possibly overlapping the first)</span></div>
<divclass="line"><aid="l00303"name="l00303"></a><spanclass="lineno"> 303</span> L replace0, L replace1, L replace2, L replace3,</div>
<divclass="line"><aid="l00304"name="l00304"></a><spanclass="lineno"> 304</span> L replace4, L replace5, L replace6, L replace7,</div>
<divclass="line"><aid="l00305"name="l00305"></a><spanclass="lineno"> 305</span> L replace8, L replace9, L replace10, L replace11,</div>
<divclass="line"><aid="l00306"name="l00306"></a><spanclass="lineno"> 306</span> L replace12, L replace13, L replace14, L replace15)<spanclass="keyword"> const </span>{</div>
<divclass="line"><aid="l00359"name="l00359"></a><spanclass="lineno"> 359</span><spanclass="comment">// Repeat 16 values as many times as necessary (usually for lookup tables)</span></div>
<divclass="line"><aid="l00375"name="l00375"></a><spanclass="lineno"> 375</span><spanclass="comment">// Under Visual Studio/ARM64 uint8x16_t and int8x16_t are apparently the same type.</span></div>
<divclass="line"><aid="l00376"name="l00376"></a><spanclass="lineno"> 376</span><spanclass="comment">// In theory, we could check this occurrence with std::same_as and std::enabled_if but it is C++14</span></div>
<divclass="line"><aid="l00377"name="l00377"></a><spanclass="lineno"> 377</span><spanclass="comment">// and relatively ugly and hard to read.</span></div>
<divclass="line"><aid="l00401"name="l00401"></a><spanclass="lineno"> 401</span><spanclass="comment">// Perform a lookup assuming no value is larger than 16</span></div>
<divclass="line"><aid="l00408"name="l00408"></a><spanclass="lineno"> 408</span> L replace0, L replace1, L replace2, L replace3,</div>
<divclass="line"><aid="l00409"name="l00409"></a><spanclass="lineno"> 409</span> L replace4, L replace5, L replace6, L replace7,</div>
<divclass="line"><aid="l00410"name="l00410"></a><spanclass="lineno"> 410</span> L replace8, L replace9, L replace10, L replace11,</div>
<divclass="line"><aid="l00411"name="l00411"></a><spanclass="lineno"> 411</span> L replace12, L replace13, L replace14, L replace15)<spanclass="keyword"> const </span>{</div>
<divclass="line"><aid="l00429"name="l00429"></a><spanclass="lineno"> 429</span><spanclass="keyword">static_assert</span>(NUM_CHUNKS == 4, <spanclass="stringliteral">"ARM kernel should use four registers per 64-byte block."</span>);</div>
<divclass="line"><aid="l00454"name="l00454"></a><spanclass="lineno"> 454</span><spanclass="comment">// compute the prefix sum of the popcounts of each byte</span></div>
<divclass="line"><aid="l00475"name="l00475"></a><spanclass="lineno"> 475</span><spanclass="comment">// Add each of the elements next to each other, successively, to stuff each 8 byte mask into one.</span></div>
<divclass="line"><aid="l00476"name="l00476"></a><spanclass="lineno"> 476</span> uint8x16_t sum0 = vpaddq_u8(this->chunks[0] & bit_mask, this->chunks[1] & bit_mask);</div>
<divclass="line"><aid="l00477"name="l00477"></a><spanclass="lineno"> 477</span> uint8x16_t sum1 = vpaddq_u8(this->chunks[2] & bit_mask, this->chunks[3] & bit_mask);</div>
<divclass="ttc"id="anamespacesimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_html_aab771b17058b0c93bdd0a8679c2c4d84"><divclass="ttname"><ahref="namespacesimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand.html#aab771b17058b0c93bdd0a8679c2c4d84">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::operator==</a></div><divclass="ttdeci">simdjson_unused simdjson_inline bool operator==(const raw_json_string &a, std::string_view c) noexcept</div><divclass="ttdoc">Comparisons between raw_json_string and std::string_view instances are potentially unsafe: the user i...</div><divclass="ttdef"><b>Definition</b><ahref="raw__json__string-inl_8h_source.html#l00147">raw_json_string-inl.h:147</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00032"name="l00032"></a><spanclass="lineno"> 32</span><spanclass="comment">// this can read up to 31 bytes beyond the buffer size, but we require</span></div>
<divclass="line"><aid="l00033"name="l00033"></a><spanclass="lineno"> 33</span><spanclass="comment">// SIMDJSON_PADDING of padding</span></div>
<divclass="line"><aid="l00034"name="l00034"></a><spanclass="lineno"> 34</span><spanclass="keyword">static_assert</span>(<aclass="code hl_variable"href="namespacesimdjson.html#aecdd750132f0eb123a6d61113b4197bf">SIMDJSON_PADDING</a>>= (BYTES_PROCESSED - 1), <spanclass="stringliteral">"backslash and quote finder must process fewer than SIMDJSON_PADDING bytes"</span>);</div>
<divclass="line"><aid="l00040"name="l00040"></a><spanclass="lineno"> 40</span><spanclass="comment">// Getting a 64-bit bitmask is much cheaper than multiple 16-bit bitmasks on ARM; therefore, we</span></div>
<divclass="line"><aid="l00041"name="l00041"></a><spanclass="lineno"> 41</span><spanclass="comment">// smash them together into a 64-byte mask and get the bitmask from there.</span></div>
<divclass="line"><aid="l00062"name="l00062"></a><spanclass="lineno"> 62</span><spanclass="keyword">static_assert</span>(<aclass="code hl_variable"href="namespacesimdjson.html#aecdd750132f0eb123a6d61113b4197bf">SIMDJSON_PADDING</a>>= (BYTES_PROCESSED - 1), <spanclass="stringliteral">"escaping finder must process fewer than SIMDJSON_PADDING bytes"</span>);</div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_aecdd750132f0eb123a6d61113b4197bf"><divclass="ttname"><ahref="namespacesimdjson.html#aecdd750132f0eb123a6d61113b4197bf">simdjson::SIMDJSON_PADDING</a></div><divclass="ttdeci">constexpr size_t SIMDJSON_PADDING</div><divclass="ttdoc">The amount of padding needed in a buffer to parse JSON.</div><divclass="ttdef"><b>Definition</b><ahref="base_8h_source.html#l00033">base.h:33</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00038"name="l00038"></a><spanclass="lineno"> 38</span><spanclass="comment">// PERF NOTE this is a safety rail ... users should exit loops as soon as they receive an error, so we'll never get here.</span></div>
<divclass="line"><aid="l00039"name="l00039"></a><spanclass="lineno"> 39</span><spanclass="comment">// However, it does not seem to make a perf difference, so we add it out of an abundance of caution.</span></div>
<divclass="line"><aid="l00080"name="l00080"></a><spanclass="lineno"> 80</span><spanclass="comment">// Clear the error if there is one, so we don't yield it twice</span></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator_html_a1577fb5a95f92a4a1aaa8eee6440d572"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator.html#a1577fb5a95f92a4a1aaa8eee6440d572">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array_iterator::operator*</a></div><divclass="ttdeci">simdjson_inline simdjson_result< value > operator*() noexcept</div><divclass="ttdoc">Get the current element.</div><divclass="ttdef"><b>Definition</b><ahref="array__iterator-inl_8h_source.html#l00019">array_iterator-inl.h:19</a></div></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator_html_a20cd55e63cf5bdc1561f771b4db1ac1b"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator.html#a20cd55e63cf5bdc1561f771b4db1ac1b">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array_iterator::operator==</a></div><divclass="ttdeci">simdjson_inline bool operator==(const array_iterator &) const noexcept</div><divclass="ttdoc">Check if we are at the end of the JSON.</div><divclass="ttdef"><b>Definition</b><ahref="array__iterator-inl_8h_source.html#l00027">array_iterator-inl.h:27</a></div></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator_html_a55e0ae33e5e5835c841fd082edab5937"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator.html#a55e0ae33e5e5835c841fd082edab5937">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array_iterator::operator++</a></div><divclass="ttdeci">simdjson_inline array_iterator & operator++() noexcept</div><divclass="ttdoc">Move to the next element.</div><divclass="ttdef"><b>Definition</b><ahref="array__iterator-inl_8h_source.html#l00033">array_iterator-inl.h:33</a></div></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator_html_aa330b23feb572f33d77516431572b643"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator.html#aa330b23feb572f33d77516431572b643">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array_iterator::array_iterator</a></div><divclass="ttdeci">simdjson_inline array_iterator() noexcept=default</div><divclass="ttdoc">Create a new, invalid array iterator.</div></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator_html_abbca17b48bd6c2b045b659076f1f64ce"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator.html#abbca17b48bd6c2b045b659076f1f64ce">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array_iterator::operator!=</a></div><divclass="ttdeci">simdjson_inline bool operator!=(const array_iterator &) const noexcept</div><divclass="ttdoc">Check if there are more elements in the JSON array.</div><divclass="ttdef"><b>Definition</b><ahref="array__iterator-inl_8h_source.html#l00030">array_iterator-inl.h:30</a></div></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator_html_ae7629ba7c3cec3be3bc61d5d79c154c1"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator.html#ae7629ba7c3cec3be3bc61d5d79c154c1">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array_iterator::at_end</a></div><divclass="ttdeci">simdjson_warn_unused simdjson_inline bool at_end() const noexcept</div><divclass="ttdoc">Check if the array is at the end.</div><divclass="ttdef"><b>Definition</b><ahref="array__iterator-inl_8h_source.html#l00046">array_iterator-inl.h:46</a></div></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1value_html"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1value.html">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::value</a></div><divclass="ttdoc">An ephemeral JSON value returned during iteration.</div><divclass="ttdef"><b>Definition</b><ahref="value_8h_source.html#l00022">value.h:22</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_a7b735a3a50ba79e3f7f14df5f77d8da9"><divclass="ttname"><ahref="namespacesimdjson.html#a7b735a3a50ba79e3f7f14df5f77d8da9">simdjson::error_code</a></div><divclass="ttdeci">error_code</div><divclass="ttdoc">All possible errors returned by simdjson.</div><divclass="ttdef"><b>Definition</b><ahref="error_8h_source.html#l00019">error.h:19</a></div></div>
<divclass="ttc"id="astructsimdjson_1_1simdjson__result_html"><divclass="ttname"><ahref="structsimdjson_1_1simdjson__result.html">simdjson::simdjson_result</a></div><divclass="ttdoc">The result of a simdjson operation that could fail.</div><divclass="ttdef"><b>Definition</b><ahref="error_8h_source.html#l00280">error.h:280</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00045"name="l00045"></a><spanclass="lineno"> 45</span> operator*() noexcept; <spanclass="comment">// MUST ONLY BE CALLED ONCE PER ITERATION.</span></div>
<divclass="line"><aid="l00109"name="l00109"></a><spanclass="lineno"> 109</span> simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> operator*() noexcept; <spanclass="comment">// MUST ONLY BE CALLED ONCE PER ITERATION.</span></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator_html_aa330b23feb572f33d77516431572b643"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator.html#aa330b23feb572f33d77516431572b643">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array_iterator::array_iterator</a></div><divclass="ttdeci">simdjson_inline array_iterator() noexcept=default</div><divclass="ttdoc">Create a new, invalid array iterator.</div></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator_html_ae7629ba7c3cec3be3bc61d5d79c154c1"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1array__iterator.html#ae7629ba7c3cec3be3bc61d5d79c154c1">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array_iterator::at_end</a></div><divclass="ttdeci">simdjson_warn_unused simdjson_inline bool at_end() const noexcept</div><divclass="ttdoc">Check if the array is at the end.</div><divclass="ttdef"><b>Definition</b><ahref="array__iterator-inl_8h_source.html#l00046">array_iterator-inl.h:46</a></div></div>
<divclass="ttc"id="aclasssimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1value_html"><divclass="ttname"><ahref="classsimdjson_1_1_s_i_m_d_j_s_o_n___i_m_p_l_e_m_e_n_t_a_t_i_o_n_1_1ondemand_1_1value.html">simdjson::SIMDJSON_IMPLEMENTATION::ondemand::value</a></div><divclass="ttdoc">An ephemeral JSON value returned during iteration.</div><divclass="ttdef"><b>Definition</b><ahref="value_8h_source.html#l00022">value.h:22</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_a7b735a3a50ba79e3f7f14df5f77d8da9"><divclass="ttname"><ahref="namespacesimdjson.html#a7b735a3a50ba79e3f7f14df5f77d8da9">simdjson::error_code</a></div><divclass="ttdeci">error_code</div><divclass="ttdoc">All possible errors returned by simdjson.</div><divclass="ttdef"><b>Definition</b><ahref="error_8h_source.html#l00019">error.h:19</a></div></div>
<divclass="ttc"id="astructsimdjson_1_1simdjson__result_html"><divclass="ttname"><ahref="structsimdjson_1_1simdjson__result.html">simdjson::simdjson_result</a></div><divclass="ttdoc">The result of a simdjson operation that could fail.</div><divclass="ttdef"><b>Definition</b><ahref="error_8h_source.html#l00280">error.h:280</a></div></div>
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="line"><aid="l00017"name="l00017"></a><spanclass="lineno"> 17</span><spanclass="comment">// The string_to_uint32 is exclusively used to map literal strings to 32-bit values.</span></div>
<divclass="line"><aid="l00018"name="l00018"></a><spanclass="lineno"> 18</span><spanclass="comment">// We use memcpy instead of a pointer cast to avoid undefined behaviors since we cannot</span></div>
<divclass="line"><aid="l00019"name="l00019"></a><spanclass="lineno"> 19</span><spanclass="comment">// be certain that the character pointer will be properly aligned.</span></div>
<divclass="line"><aid="l00020"name="l00020"></a><spanclass="lineno"> 20</span><spanclass="comment">// You might think that using memcpy makes this function expensive, but you'd be wrong.</span></div>
<divclass="line"><aid="l00021"name="l00021"></a><spanclass="lineno"> 21</span><spanclass="comment">// All decent optimizing compilers (GCC, clang, Visual Studio) will compile string_to_uint32("false");</span></div>
<divclass="line"><aid="l00022"name="l00022"></a><spanclass="lineno"> 22</span><spanclass="comment">// to the compile-time constant 1936482662.</span></div>
<divclass="line"><aid="l00026"name="l00026"></a><spanclass="lineno"> 26</span><spanclass="comment">// Again in str4ncmp we use a memcpy to avoid undefined behavior. The memcpy may appear expensive.</span></div>
<divclass="line"><aid="l00027"name="l00027"></a><spanclass="lineno"> 27</span><spanclass="comment">// Yet all decent optimizing compilers will compile memcpy to a single instruction, just about.</span></div>
<divclass="line"><aid="l00030"name="l00030"></a><spanclass="lineno"> 30</span> uint32_t srcval; <spanclass="comment">// we want to avoid unaligned 32-bit loads (undefined in C/C++)</span></div>
<divclass="line"><aid="l00031"name="l00031"></a><spanclass="lineno"> 31</span><spanclass="keyword">static_assert</span>(<spanclass="keyword">sizeof</span>(uint32_t) <= <aclass="code hl_variable"href="namespacesimdjson.html#aecdd750132f0eb123a6d61113b4197bf">SIMDJSON_PADDING</a>, <spanclass="stringliteral">"SIMDJSON_PADDING must be larger than 4 bytes"</span>);</div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_aecdd750132f0eb123a6d61113b4197bf"><divclass="ttname"><ahref="namespacesimdjson.html#aecdd750132f0eb123a6d61113b4197bf">simdjson::SIMDJSON_PADDING</a></div><divclass="ttdeci">constexpr size_t SIMDJSON_PADDING</div><divclass="ttdoc">The amount of padding needed in a buffer to parse JSON.</div><divclass="ttdef"><b>Definition</b><ahref="base_8h_source.html#l00033">base.h:33</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
<divclass="ttc"id="aclasssimdjson_1_1implementation_html"><divclass="ttname"><ahref="classsimdjson_1_1implementation.html">simdjson::implementation</a></div><divclass="ttdoc">An implementation of simdjson for a particular CPU architecture.</div><divclass="ttdef"><b>Definition</b><ahref="implementation_8h_source.html#l00045">implementation.h:45</a></div></div>
<divclass="ttc"id="aclasssimdjson_1_1padded__string__view_html"><divclass="ttname"><ahref="classsimdjson_1_1padded__string__view.html">simdjson::padded_string_view</a></div><divclass="ttdoc">User-provided string that promises it has extra padded bytes at the end for use with parser::parse().</div><divclass="ttdef"><b>Definition</b><ahref="padded__string__view_8h_source.html#l00018">padded_string_view.h:18</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html"><divclass="ttname"><ahref="namespacesimdjson.html">simdjson</a></div><divclass="ttdoc">The top level simdjson namespace, containing everything the library provides.</div><divclass="ttdef"><b>Definition</b><ahref="arm64_2base_8h_source.html#l00008">base.h:8</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_a6df2598eb1d4e1ea669c41831cc7325d"><divclass="ttname"><ahref="namespacesimdjson.html#a6df2598eb1d4e1ea669c41831cc7325d">simdjson::DEFAULT_MAX_DEPTH</a></div><divclass="ttdeci">constexpr size_t DEFAULT_MAX_DEPTH</div><divclass="ttdoc">By default, simdjson supports this many nested objects and arrays.</div><divclass="ttdef"><b>Definition</b><ahref="base_8h_source.html#l00040">base.h:40</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_ad0bad3783275be4012bd5cfd0327875a"><divclass="ttname"><ahref="namespacesimdjson.html#ad0bad3783275be4012bd5cfd0327875a">simdjson::SIMDJSON_MAXSIZE_BYTES</a></div><divclass="ttdeci">SIMDJSON_PUSH_DISABLE_UNUSED_WARNINGS constexpr size_t SIMDJSON_MAXSIZE_BYTES</div><divclass="ttdoc">The maximum document size supported by simdjson.</div><divclass="ttdef"><b>Definition</b><ahref="base_8h_source.html#l00023">base.h:23</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_ae6ec9f0ce23fc51d87116b64fdbeb811"><divclass="ttname"><ahref="namespacesimdjson.html#ae6ec9f0ce23fc51d87116b64fdbeb811">simdjson::stage1_mode</a></div><divclass="ttdeci">stage1_mode</div><divclass="ttdoc">This enum is used with the dom_parser_implementation::stage1 function.</div><divclass="ttdef"><b>Definition</b><ahref="internal_2dom__parser__implementation_8h_source.html#l00022">dom_parser_implementation.h:22</a></div></div>
<divclass="ttc"id="anamespacesimdjson_html_aecdd750132f0eb123a6d61113b4197bf"><divclass="ttname"><ahref="namespacesimdjson.html#aecdd750132f0eb123a6d61113b4197bf">simdjson::SIMDJSON_PADDING</a></div><divclass="ttdeci">constexpr size_t SIMDJSON_PADDING</div><divclass="ttdoc">The amount of padding needed in a buffer to parse JSON.</div><divclass="ttdef"><b>Definition</b><ahref="base_8h_source.html#l00033">base.h:33</a></div></div>
<divclass="ttc"id="astructsimdjson_1_1padded__string_html"><divclass="ttname"><ahref="structsimdjson_1_1padded__string.html">simdjson::padded_string</a></div><divclass="ttdoc">String with extra allocation for ease of use with parser::parse()</div><divclass="ttdef"><b>Definition</b><ahref="padded__string_8h_source.html#l00023">padded_string.h:23</a></div></div>
</div><!-- fragment --></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<divid="nav-path"class="navpath"><!-- id is needed for treeview function! -->
<liclass="footer">Generated by <ahref="https://www.doxygen.org/index.html"><imgclass="footer"src="doxygen.svg"width="104"height="31"alt="doxygen"/></a> 1.9.8 </li>
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Francisco Geiman Thiesen
