When parser.number_as_string(true) is set and a number exceeds uint64
range, write the raw digits to the string buffer and emit a BIGINT ('Z')
tape tag instead of returning BIGINT_ERROR. Default behavior unchanged.
Changes:
- tape_type.h: add BIGINT = 'Z'
- dom_parser_implementation.h: add _number_as_string flag
- parser.h: add number_as_string() getter/setter
- parser-inl.h: propagate flag before parse
- tape_builder.h: visit_number checks flag on BIGINT_ERROR
- tape_writer.h: add append_bigint helper
- serialization-inl.h: handle BIGINT in serialization (raw digits)
- tape.md: document big integer tape format
Builds on PR #2139 which added BIGINT_ERROR detection.
All 119 existing tests pass — default behavior unchanged.
Closes#167
Co-authored-by: harshagd <harshagd@amazon.com>
* Add FracturedJson formatting support for DOM serialization
Implements FracturedJson formatting as requested in issue #2576.
FracturedJson produces human-readable yet compact JSON output by
intelligently choosing between different layout strategies based on
content complexity, length, and structure similarity.
Key features:
- Four layout modes: inline, compact multiline, table, and expanded
- Structure analysis pass to compute metrics before formatting
- Table formatting for arrays of similar objects with column alignment
- Configurable options for line length, indentation, padding, etc.
New files:
- fractured_json.h: Public API with fractured_json_options struct
- fractured_json-inl.h: Implementation (~1000 lines)
- json_structure_analyzer.h: Structure analysis for layout decisions
- fractured_formatter.h: Formatter class using CRTP pattern
Usage:
dom::parser parser;
element doc = parser.parse(json_string);
std::cout << fractured_json(doc) << std::endl;
// Or with custom options:
fractured_json_options opts;
opts.indent_spaces = 2;
std::cout << fractured_json(doc, opts) << std::endl;
// Or format any JSON string (useful with reflection API):
auto formatted = fractured_json_string(minified_json);
Resolves#2576
* Add comprehensive tests for FracturedJson formatter
Adds 27 test cases covering all aspects of the FracturedJson formatter:
Core functionality tests (13):
- Roundtrip parsing verification
- Inline formatting for simple arrays and objects
- Expanded formatting for complex nested structures
- Compact multiline arrays with configurable items per line
- Table formatting for uniform arrays of objects
- Empty container handling
- All scalar types (string, int, uint, double, bool, null)
- String escaping (quotes, backslashes, control characters)
- Custom indentation options
- Deep nesting (10+ levels)
- Mixed type arrays
Edge case tests (11):
- Unicode strings (Chinese, emoji, Arabic, Russian, accented chars)
- Boundary numbers (INT64_MIN/MAX, UINT64_MAX, DBL_MIN/MAX)
- Nested arrays (arrays of arrays)
- Empty string values
- Keys with special characters (spaces, quotes, colons, etc.)
- Non-uniform arrays (should not trigger table mode)
- Very long strings (500+ chars)
- Large arrays (100 elements)
- Reflection API workflow simulation
- Control characters (tab, newline, CR, null)
- Single element containers
Option tests (3):
- Disable compact multiline mode
- Disable table format mode
- Disable all padding options
* Add FracturedJson integration with builder/reflection API
Extends FracturedJson to work seamlessly with the builder API, enabling
formatted output directly from C++ structs using static reflection.
New functions:
- to_fractured_json_string(obj, opts) - serialize struct to formatted JSON
- to_fractured_json(obj, output, opts) - same with output parameter
- extract_fractured_json<fields...>(obj, opts) - format only specific fields
These functions combine the builder's reflection-based serialization with
FracturedJson formatting in a single convenient call:
struct User { int id; std::string name; bool active; };
User user{1, "Alice", true};
// Minified output (existing):
auto minified = to_json_string(user);
// {"id":1,"name":"Alice","active":true}
// Formatted output (new):
auto formatted = to_fractured_json_string(user);
// { "id": 1, "name": "Alice", "active": true }
// Partial extraction with formatting:
auto partial = extract_fractured_json<"id", "name">(user);
// { "id": 1, "name": "Alice" }
New files:
- generic/builder/fractured_json_builder.h - builder integration
- tests/builder/static_reflection_fractured_json_tests.cpp - 7 tests
* Fix INT64_MIN overflow and implement table_similarity_threshold
- Fix undefined behavior when negating INT64_MIN in estimate_number_length()
and measure_value_length() by returning 20 (the exact length of the
string representation) directly
- Actually use table_similarity_threshold in check_array_uniformity() by
calling compute_object_similarity() to compare objects against the first
object in the array
* Fix -Werror=effc++ member initialization warnings
Initialize all member variables in member initialization lists to
satisfy GCC's -Werror=effc++ flag:
- element_metrics::common_keys - add {} default initializer
- structure_analyzer - add default constructor with member init list
- fractured_formatter - add column_widths_{} to constructor
- fractured_string_builder - add analyzer_{} to constructor
* Add Rule of Five to structure_analyzer class
The class has a pointer member (current_opts_) which triggers
-Werror=effc++ requiring explicit copy/move operations. Delete
copy operations (class shouldn't be copied due to cache) and
default move operations.
* Fix Windows build: wrap std::max to avoid macro conflict
Windows.h defines max/min macros that interfere with std::max/std::min.
Wrapping in parentheses as (std::max)(...) prevents macro expansion.
* Fix GCC 15 false positive -Wfree-nonheap-object warning
GCC 15 on MINGW64 gives a false positive warning in parser_moving_parser()
when the std::vector<std::string> goes out of scope. Suppress this
specific warning with a pragma for GCC builds.
* Fix metrics cache key bug by passing metrics through recursion
The cache was using element addresses as keys, but dom::element objects
are lightweight wrappers that get copied during iteration, causing
different addresses between analysis and formatting phases. This resulted
in cache misses and fallback to empty metrics.
Solution: Store child metrics in the element_metrics struct and pass
them through recursive calls, eliminating the need for address-based
caching entirely.
Changes:
- Add children vector to element_metrics for hierarchical metrics
- Remove metrics_cache_ and related get_metrics/has_metrics methods
- Update all format functions to accept and pass child metrics
- Add public analyze_array/analyze_object overloads for standalone use
* Add ignore patterns for Node.js, Rust, and generated files
Add entries for node_modules, package-lock.json, Rust target
directories, local ablation artifacts, and generated documentation
files.
* Refactor: extract analyze_scalar helper to reduce code duplication
Extract common scalar type handling (STRING, INT64, UINT64, DOUBLE,
BOOL, NULL_VALUE) into a dedicated analyze_scalar method. Each scalar
type shares the same initialization pattern for complexity, child_count,
can_inline, and recommended_layout.
Also simplify boolean formatting in format_scalar to use ternary operator.
* Fix formatting and duplicate error message in amalgamate.py
Reformat cramped is_amalgamator condition to multi-line for readability.
Fix duplicate error message text in _included_filename_root and use
correct variable name (relative_root instead of root).
* Refactor: add count_newlines helper in fractured_json tests
Extract repeated newline counting loop into a reusable static helper
function, used by inline_array_test, inline_object_test, and
expanded_test.
* Revert "Add ignore patterns for Node.js, Rust, and generated files"
This reverts commit 4760ea7cd0.
* various minor changes
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* improving the documentation of raw json access
* minor fix
* documentation update
* guarding for exceptions
* fixing exception issue
* fix test
* update.
* saving comments
* more technical fixes
* correcting path in ci test
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* clang format
* added `chars()` method
* implemented vector with small buffer instead of `std::vector`
* added missing <utility> header
* minor fixes
---------
Co-authored-by: Pavel Novikov <dev-ape@yandex.ru>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Initial work on JSON builder
* moving the files back to ondemand for now.
* tweak
* more later
* update
* minor edits
* dropping vs arm (missing support)
* adding tests. we still specialized write_string_escaped
* tweaking
* fix typo
* tweaking the approach
* minor fix
* missing store
* another missing store
* Attempt at fixing failing serialization tests. (#2292)
* Fixing appeand_float typo (#2294)
* applying a couple of fixes
* updating single header
* fix for pre C++17 if constexpr
* Fixing unused argument problem and updating the singleheader file
* various pedantic fixes
* Sketch of builder
* reordering.
* simplify
* Adding draft of static reflection based deserialization
* Updating simdjson singleheader
* patching the automated deserialization.
* automated
* Adding support for smart pointers of user defined types.
* Adding specialization for smart pointers for basic types. I think it is highly likely that this can be done in a more generic way.
* Referncing a later version of rapidjson that fixed the issue related with assignment attempt of a const variable for GenericStringRef class.
* guarding the tests
* adding documentation for string_builder
* saving
* rename to 'append'
* saving
* non-functional benchmarks (#2342)
* non-functional benchmarks
* Fix typo
* various fixes
* tweaking
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
* tuning
* various minor fixes
* minor tweak
* minor simplification
* updating amal
* adding a cast
* update
* fancy casting
* removing dead code
* Pushing latest changes. CITM benchmark is still not working.
* Still not working, but now I am getting only 10 errors.
* add static reflection benchmark to 'large random' benchmark and allows (#2349)
deserialization (with static reflection) from objects and arrays.
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Removing std::map from CitmCatalog definition, since that is not currently supported.
* Added free to rust bench, segfault is still happening..
* The syntax changed: ^E became ^^E. (#2350)
* The syntax changed: ^E became ^^E.
* guarding
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Adding support for string_view_keyed_map types.
* Adding concepts as a conditional include.
* updating single-header
* Adding concepts to ondemand deps
* rust benchmark is finally working
* Fixing small typo in docs.
* adding docker config and instructions so that our users can test the static reflection (#2358)
* adding docker config and instructions so that our users can test the
static reflection
* completing the instructions
* pruning white spaces
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* minor optimizations on the JSON builder branch
* avoiding undef behaviour
* saving
* somewhat nicer builder
* make it possible to run just one benchmark
* adding linux perf
* fixing minor issue
* updating swar
* Adding real world compilation benchmark (#2379)
* Adding compilation benchmark for json parsing with and without reflection
* Moving it to the benchmark folder, also reducing a bit the number of iterations.
* Removing script from root folder.
* Reducing number of iterations
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Update benchmark/benchmark_reflection_usage_compilation.sh
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Making the script more customizable and also test whether the compiler being used supports reflection before actually running the benchmark
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
* Using define_static_string from https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p3491r2.html (#2389)
* Applying changes needed after latest reflection paper updates.
* Working, but no template for yet.
* Updating single-header to incldue the use of define_static_string.
* copying over master
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
* Add support for JSONPath with '$' prefix and integrate into dom::at_path (#2266)
- Updated `json_path_to_pointer_conversion` to support JSONPath starting with the '$' prefix, while maintaining compatibility with the existing implementation.
- Moved `json_path_to_pointer_conversion` to a separate header file for better modularity and to support JSONPath queries in `dom` mode.
- Implemented `at_path` functionality in `dom` mode to enable querying JSON using JSONPath.
- Added unit tests to validate the new JSONPath support in `dom` mode and ensure compatibility with both standard and existing JSONPath formats.
* two minor fixes
* more tests and documentation
* damn compiler warnings
* more documentation fixes
---------
Co-authored-by: Zhengguo Yang <yangzhgg@gmail.com>
* Standard compatibility fixes
* missing commit
* Should work.
* Fix.
* Fix.
* Should work now.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Adding CXX 20 to CI
* side-stepping new CXX 20 guard.
* Going another way
* Saving.
* Explicit.
* Saving...
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
* Adding support for AVX-512 on macOS.
* Fix.
* Fix.
* Minor fix
* Setting the variable to zero.
* Fixing include
* Checking if the OS supports AVX-512
* Tweaking.
---------
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Jaël Champagne Gareau