Add zero-cost range wrappers (array_range, object_range) that satisfy
std::ranges::input_range, enabling std::views::transform and other
C++20 range adaptors with the On-Demand parser.
Uses direct forwarding via simdjson_inline with no value buffering,
avoiding the per-element overhead (~20%) of the previous approach.
Guarded by SIMDJSON_SUPPORTS_RANGES.
* adding a guard in document::allocate.
Co-authored-by: jmestwa-coder jmestwa@gmail.com
* adding a max depth
Co-authored-by: jmestwa-coder jmestwa@gmail.com
* add get_int32() and get_uint32() to the On Demand API
Add convenience methods that call get_int64()/get_uint64() and
range-check the result, returning NUMBER_OUT_OF_RANGE on overflow.
Added to value, document, document_reference, and their
simdjson_result wrappers.
Closes#1890
* fix document_reference streaming for get_int32/get_uint32
The document_reference getters delegated to document::get_int32/get_uint32
which call get_root_int64/get_root_uint64 with allow_trailing_content=true,
rejecting the next document in a stream as trailing content.
Call get_root_value_iterator().get_root_int64/get_root_uint64(false) directly
to allow trailing content, matching get_int64/get_uint64.
Add streaming tests for both getters.
* add get_int32/get_uint32 to basics.md
* add get_int32/get_uint32 to wrong-type error tests
* add get<uint32_t>/get<int32_t> template specializations
Wire the 32-bit getters into the generic get<T>() and get(T&)
dispatch so that doc.get<uint32_t>() compiles on C++11/14/17
without requiring the C++20 concepts path in std_deserialize.h.
When parser.number_as_string(true) is set and a number exceeds uint64
range, write the raw digits to the string buffer and emit a BIGINT ('Z')
tape tag instead of returning BIGINT_ERROR. Default behavior unchanged.
Changes:
- tape_type.h: add BIGINT = 'Z'
- dom_parser_implementation.h: add _number_as_string flag
- parser.h: add number_as_string() getter/setter
- parser-inl.h: propagate flag before parse
- tape_builder.h: visit_number checks flag on BIGINT_ERROR
- tape_writer.h: add append_bigint helper
- serialization-inl.h: handle BIGINT in serialization (raw digits)
- tape.md: document big integer tape format
Builds on PR #2139 which added BIGINT_ERROR detection.
All 119 existing tests pass — default behavior unchanged.
Closes#167
Co-authored-by: harshagd <harshagd@amazon.com>
#2617 introduced an RVV check that only checks if the compiler supports RVV intrinsics without checking if RVV is enabled at compile time.
This has caused compilation failure of node.js on riscv64. Fix it by appending a check for __riscv_vector.
* perf(serialization): SIMD-accelerated string escape position finding
Profiling revealed that string serialization was performing redundant
scanning: first calling fast_needs_escaping() (SIMD scan to check IF
escape needed), then find_next_json_quotable_character() (scalar
byte-by-byte scan to find WHERE).
Profile data from Twitter benchmark showed:
- 54.76% time in atom<std::string> (string serialization)
- 24.85% time in find_next_json_quotable_character (scalar position finding)
This optimization unifies both operations into a single SIMD pass that
directly locates the first quotable character position:
- NEON (ARM64): Uses vceqq_u8/vcltq_u8 for character detection, then
extracts position via __builtin_ctzll on 64-bit vector lanes
- SSE2 (x86-64): Uses _mm_cmpeq_epi8/_mm_subs_epu8 for detection, then
_mm_movemask_epi8 + __builtin_ctz for position extraction
The write_string_escaped function now uses the position finder directly,
eliminating the separate fast_needs_escaping check.
Benchmark results (ARM64, Apple Silicon via Docker with p2996 clang):
- Twitter (string-heavy): 4330 -> 5723 MB/s (+32%)
- CITM (numeric-heavy): ~neutral (expected, few strings)
* perf(serialization): batch integer formatting with 4-digit processing
Profiling with perf annotate revealed that the integer-to-string
conversion loop was a significant hotspot in numeric-heavy workloads.
The original implementation processed 2 digits per iteration:
while (pv >= 100) {
memcpy(write_pointer - 1, &decimal_table[(pv % 100) * 2], 2);
write_pointer -= 2;
pv /= 100;
}
Profile data from CITM benchmark showed:
- 31.20% time in atom<unsigned long> (integer formatting)
- 39.07% of integer formatting time in the 2-byte store instruction
(sturh on ARM64)
- CITM integers average 8.8 digits, meaning 4+ store operations per number
This optimization processes 4 digits per iteration, reducing both store
operations and division count by approximately half for large numbers:
while (pv >= 10000) {
q = pv / 10000;
r = pv % 10000;
r_hi = r / 100; // High 2 digits
r_lo = r % 100; // Low 2 digits
memcpy(write_pointer - 1, &decimal_table[r_lo * 2], 2);
memcpy(write_pointer - 3, &decimal_table[r_hi * 2], 2);
write_pointer -= 4;
pv = q;
}
The division by 10000 compiles to an efficient multiply-high instruction
(umulh on ARM64). Applied to both unsigned and signed integer paths.
Benchmark results (ARM64, Apple Silicon via Docker with p2996 clang):
- Twitter: ~neutral (few integers)
- CITM (numeric-heavy): 2912 -> 3086 MB/s (+6%)
* fix(build): add MSVC compatibility for bit manipulation intrinsics
MSVC does not have __builtin_ctz/__builtin_ctzll. Use _BitScanForward
and _BitScanForward64 from <intrin.h> on MSVC instead.
This fixes the build on all Windows configurations (x64, ARM64, Win32).
* refactor: clean up comments to be implementation-focused
Remove references to specific benchmarks and previous implementations
from code comments. Comments now describe what the code does rather
than historical context.
* Add FracturedJson formatting support for DOM serialization
Implements FracturedJson formatting as requested in issue #2576.
FracturedJson produces human-readable yet compact JSON output by
intelligently choosing between different layout strategies based on
content complexity, length, and structure similarity.
Key features:
- Four layout modes: inline, compact multiline, table, and expanded
- Structure analysis pass to compute metrics before formatting
- Table formatting for arrays of similar objects with column alignment
- Configurable options for line length, indentation, padding, etc.
New files:
- fractured_json.h: Public API with fractured_json_options struct
- fractured_json-inl.h: Implementation (~1000 lines)
- json_structure_analyzer.h: Structure analysis for layout decisions
- fractured_formatter.h: Formatter class using CRTP pattern
Usage:
dom::parser parser;
element doc = parser.parse(json_string);
std::cout << fractured_json(doc) << std::endl;
// Or with custom options:
fractured_json_options opts;
opts.indent_spaces = 2;
std::cout << fractured_json(doc, opts) << std::endl;
// Or format any JSON string (useful with reflection API):
auto formatted = fractured_json_string(minified_json);
Resolves#2576
* Add comprehensive tests for FracturedJson formatter
Adds 27 test cases covering all aspects of the FracturedJson formatter:
Core functionality tests (13):
- Roundtrip parsing verification
- Inline formatting for simple arrays and objects
- Expanded formatting for complex nested structures
- Compact multiline arrays with configurable items per line
- Table formatting for uniform arrays of objects
- Empty container handling
- All scalar types (string, int, uint, double, bool, null)
- String escaping (quotes, backslashes, control characters)
- Custom indentation options
- Deep nesting (10+ levels)
- Mixed type arrays
Edge case tests (11):
- Unicode strings (Chinese, emoji, Arabic, Russian, accented chars)
- Boundary numbers (INT64_MIN/MAX, UINT64_MAX, DBL_MIN/MAX)
- Nested arrays (arrays of arrays)
- Empty string values
- Keys with special characters (spaces, quotes, colons, etc.)
- Non-uniform arrays (should not trigger table mode)
- Very long strings (500+ chars)
- Large arrays (100 elements)
- Reflection API workflow simulation
- Control characters (tab, newline, CR, null)
- Single element containers
Option tests (3):
- Disable compact multiline mode
- Disable table format mode
- Disable all padding options
* Add FracturedJson integration with builder/reflection API
Extends FracturedJson to work seamlessly with the builder API, enabling
formatted output directly from C++ structs using static reflection.
New functions:
- to_fractured_json_string(obj, opts) - serialize struct to formatted JSON
- to_fractured_json(obj, output, opts) - same with output parameter
- extract_fractured_json<fields...>(obj, opts) - format only specific fields
These functions combine the builder's reflection-based serialization with
FracturedJson formatting in a single convenient call:
struct User { int id; std::string name; bool active; };
User user{1, "Alice", true};
// Minified output (existing):
auto minified = to_json_string(user);
// {"id":1,"name":"Alice","active":true}
// Formatted output (new):
auto formatted = to_fractured_json_string(user);
// { "id": 1, "name": "Alice", "active": true }
// Partial extraction with formatting:
auto partial = extract_fractured_json<"id", "name">(user);
// { "id": 1, "name": "Alice" }
New files:
- generic/builder/fractured_json_builder.h - builder integration
- tests/builder/static_reflection_fractured_json_tests.cpp - 7 tests
* Fix INT64_MIN overflow and implement table_similarity_threshold
- Fix undefined behavior when negating INT64_MIN in estimate_number_length()
and measure_value_length() by returning 20 (the exact length of the
string representation) directly
- Actually use table_similarity_threshold in check_array_uniformity() by
calling compute_object_similarity() to compare objects against the first
object in the array
* Fix -Werror=effc++ member initialization warnings
Initialize all member variables in member initialization lists to
satisfy GCC's -Werror=effc++ flag:
- element_metrics::common_keys - add {} default initializer
- structure_analyzer - add default constructor with member init list
- fractured_formatter - add column_widths_{} to constructor
- fractured_string_builder - add analyzer_{} to constructor
* Add Rule of Five to structure_analyzer class
The class has a pointer member (current_opts_) which triggers
-Werror=effc++ requiring explicit copy/move operations. Delete
copy operations (class shouldn't be copied due to cache) and
default move operations.
* Fix Windows build: wrap std::max to avoid macro conflict
Windows.h defines max/min macros that interfere with std::max/std::min.
Wrapping in parentheses as (std::max)(...) prevents macro expansion.
* Fix GCC 15 false positive -Wfree-nonheap-object warning
GCC 15 on MINGW64 gives a false positive warning in parser_moving_parser()
when the std::vector<std::string> goes out of scope. Suppress this
specific warning with a pragma for GCC builds.
* Fix metrics cache key bug by passing metrics through recursion
The cache was using element addresses as keys, but dom::element objects
are lightweight wrappers that get copied during iteration, causing
different addresses between analysis and formatting phases. This resulted
in cache misses and fallback to empty metrics.
Solution: Store child metrics in the element_metrics struct and pass
them through recursive calls, eliminating the need for address-based
caching entirely.
Changes:
- Add children vector to element_metrics for hierarchical metrics
- Remove metrics_cache_ and related get_metrics/has_metrics methods
- Update all format functions to accept and pass child metrics
- Add public analyze_array/analyze_object overloads for standalone use
* Add ignore patterns for Node.js, Rust, and generated files
Add entries for node_modules, package-lock.json, Rust target
directories, local ablation artifacts, and generated documentation
files.
* Refactor: extract analyze_scalar helper to reduce code duplication
Extract common scalar type handling (STRING, INT64, UINT64, DOUBLE,
BOOL, NULL_VALUE) into a dedicated analyze_scalar method. Each scalar
type shares the same initialization pattern for complexity, child_count,
can_inline, and recommended_layout.
Also simplify boolean formatting in format_scalar to use ternary operator.
* Fix formatting and duplicate error message in amalgamate.py
Reformat cramped is_amalgamator condition to multi-line for readability.
Fix duplicate error message text in _included_filename_root and use
correct variable name (relative_root instead of root).
* Refactor: add count_newlines helper in fractured_json tests
Extract repeated newline counting loop into a reusable static helper
function, used by inline_array_test, inline_object_test, and
expanded_test.
* Revert "Add ignore patterns for Node.js, Rust, and generated files"
This reverts commit 4760ea7cd0.
* various minor changes
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* work on the ondemand iterators
* guarding two SIMDJSON_ASSUME
* simplify following @jkeiser's comment
* adding safety rails to the iterators
* silencing a warning.
* updating the amalgamation files
If you provide a custom serializer for range types it is currently never used due to the requires clause for string_builder::append with ranges is overly broad
Daniel Lemire