mirror of
https://github.com/simdjson/simdjson
synced 2026-06-08 17:27:07 +00:00
fc57c09cf0
* Add FracturedJson formatting support for DOM serialization
Implements FracturedJson formatting as requested in issue #2576.
FracturedJson produces human-readable yet compact JSON output by
intelligently choosing between different layout strategies based on
content complexity, length, and structure similarity.
Key features:
- Four layout modes: inline, compact multiline, table, and expanded
- Structure analysis pass to compute metrics before formatting
- Table formatting for arrays of similar objects with column alignment
- Configurable options for line length, indentation, padding, etc.
New files:
- fractured_json.h: Public API with fractured_json_options struct
- fractured_json-inl.h: Implementation (~1000 lines)
- json_structure_analyzer.h: Structure analysis for layout decisions
- fractured_formatter.h: Formatter class using CRTP pattern
Usage:
dom::parser parser;
element doc = parser.parse(json_string);
std::cout << fractured_json(doc) << std::endl;
// Or with custom options:
fractured_json_options opts;
opts.indent_spaces = 2;
std::cout << fractured_json(doc, opts) << std::endl;
// Or format any JSON string (useful with reflection API):
auto formatted = fractured_json_string(minified_json);
Resolves #2576
* Add comprehensive tests for FracturedJson formatter
Adds 27 test cases covering all aspects of the FracturedJson formatter:
Core functionality tests (13):
- Roundtrip parsing verification
- Inline formatting for simple arrays and objects
- Expanded formatting for complex nested structures
- Compact multiline arrays with configurable items per line
- Table formatting for uniform arrays of objects
- Empty container handling
- All scalar types (string, int, uint, double, bool, null)
- String escaping (quotes, backslashes, control characters)
- Custom indentation options
- Deep nesting (10+ levels)
- Mixed type arrays
Edge case tests (11):
- Unicode strings (Chinese, emoji, Arabic, Russian, accented chars)
- Boundary numbers (INT64_MIN/MAX, UINT64_MAX, DBL_MIN/MAX)
- Nested arrays (arrays of arrays)
- Empty string values
- Keys with special characters (spaces, quotes, colons, etc.)
- Non-uniform arrays (should not trigger table mode)
- Very long strings (500+ chars)
- Large arrays (100 elements)
- Reflection API workflow simulation
- Control characters (tab, newline, CR, null)
- Single element containers
Option tests (3):
- Disable compact multiline mode
- Disable table format mode
- Disable all padding options
* Add FracturedJson integration with builder/reflection API
Extends FracturedJson to work seamlessly with the builder API, enabling
formatted output directly from C++ structs using static reflection.
New functions:
- to_fractured_json_string(obj, opts) - serialize struct to formatted JSON
- to_fractured_json(obj, output, opts) - same with output parameter
- extract_fractured_json<fields...>(obj, opts) - format only specific fields
These functions combine the builder's reflection-based serialization with
FracturedJson formatting in a single convenient call:
struct User { int id; std::string name; bool active; };
User user{1, "Alice", true};
// Minified output (existing):
auto minified = to_json_string(user);
// {"id":1,"name":"Alice","active":true}
// Formatted output (new):
auto formatted = to_fractured_json_string(user);
// { "id": 1, "name": "Alice", "active": true }
// Partial extraction with formatting:
auto partial = extract_fractured_json<"id", "name">(user);
// { "id": 1, "name": "Alice" }
New files:
- generic/builder/fractured_json_builder.h - builder integration
- tests/builder/static_reflection_fractured_json_tests.cpp - 7 tests
* Fix INT64_MIN overflow and implement table_similarity_threshold
- Fix undefined behavior when negating INT64_MIN in estimate_number_length()
and measure_value_length() by returning 20 (the exact length of the
string representation) directly
- Actually use table_similarity_threshold in check_array_uniformity() by
calling compute_object_similarity() to compare objects against the first
object in the array
* Fix -Werror=effc++ member initialization warnings
Initialize all member variables in member initialization lists to
satisfy GCC's -Werror=effc++ flag:
- element_metrics::common_keys - add {} default initializer
- structure_analyzer - add default constructor with member init list
- fractured_formatter - add column_widths_{} to constructor
- fractured_string_builder - add analyzer_{} to constructor
* Add Rule of Five to structure_analyzer class
The class has a pointer member (current_opts_) which triggers
-Werror=effc++ requiring explicit copy/move operations. Delete
copy operations (class shouldn't be copied due to cache) and
default move operations.
* Fix Windows build: wrap std::max to avoid macro conflict
Windows.h defines max/min macros that interfere with std::max/std::min.
Wrapping in parentheses as (std::max)(...) prevents macro expansion.
* Fix GCC 15 false positive -Wfree-nonheap-object warning
GCC 15 on MINGW64 gives a false positive warning in parser_moving_parser()
when the std::vector<std::string> goes out of scope. Suppress this
specific warning with a pragma for GCC builds.
* Fix metrics cache key bug by passing metrics through recursion
The cache was using element addresses as keys, but dom::element objects
are lightweight wrappers that get copied during iteration, causing
different addresses between analysis and formatting phases. This resulted
in cache misses and fallback to empty metrics.
Solution: Store child metrics in the element_metrics struct and pass
them through recursive calls, eliminating the need for address-based
caching entirely.
Changes:
- Add children vector to element_metrics for hierarchical metrics
- Remove metrics_cache_ and related get_metrics/has_metrics methods
- Update all format functions to accept and pass child metrics
- Add public analyze_array/analyze_object overloads for standalone use
* Add ignore patterns for Node.js, Rust, and generated files
Add entries for node_modules, package-lock.json, Rust target
directories, local ablation artifacts, and generated documentation
files.
* Refactor: extract analyze_scalar helper to reduce code duplication
Extract common scalar type handling (STRING, INT64, UINT64, DOUBLE,
BOOL, NULL_VALUE) into a dedicated analyze_scalar method. Each scalar
type shares the same initialization pattern for complexity, child_count,
can_inline, and recommended_layout.
Also simplify boolean formatting in format_scalar to use ternary operator.
* Fix formatting and duplicate error message in amalgamate.py
Reformat cramped is_amalgamator condition to multi-line for readability.
Fix duplicate error message text in _included_filename_root and use
correct variable name (relative_root instead of root).
* Refactor: add count_newlines helper in fractured_json tests
Extract repeated newline counting loop into a reusable static helper
function, used by inline_array_test, inline_object_test, and
expanded_test.
* Revert "Add ignore patterns for Node.js, Rust, and generated files"
This reverts commit 4760ea7cd0.
* various minor changes
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
237 lines
6.2 KiB
C++
237 lines
6.2 KiB
C++
#include "simdjson.h"
|
|
#include "test_builder.h"
|
|
#include <string>
|
|
#include <string_view>
|
|
#include <vector>
|
|
|
|
using namespace simdjson;
|
|
|
|
// Test structures for FracturedJson builder integration
|
|
struct SimpleUser {
|
|
int id;
|
|
std::string name;
|
|
bool active;
|
|
};
|
|
|
|
struct UserWithEmail {
|
|
int id;
|
|
std::string name;
|
|
std::string email;
|
|
bool active;
|
|
};
|
|
|
|
struct NestedData {
|
|
std::string title;
|
|
std::vector<SimpleUser> users;
|
|
int count;
|
|
};
|
|
|
|
struct TableTestData {
|
|
std::vector<SimpleUser> records;
|
|
};
|
|
|
|
namespace fractured_json_builder_tests {
|
|
|
|
// Test basic to_fractured_json_string functionality
|
|
bool basic_fractured_json_test() {
|
|
TEST_START();
|
|
#if SIMDJSON_STATIC_REFLECTION
|
|
SimpleUser user{1, "Alice", true};
|
|
|
|
std::string formatted;
|
|
ASSERT_SUCCESS(simdjson::to_fractured_json_string(user).get(formatted));
|
|
|
|
// Verify it produces valid JSON by re-parsing
|
|
simdjson::dom::parser parser;
|
|
simdjson::dom::element doc;
|
|
ASSERT_SUCCESS(parser.parse(formatted).get(doc));
|
|
|
|
// Verify values are preserved
|
|
int64_t id;
|
|
ASSERT_SUCCESS(doc["id"].get_int64().get(id));
|
|
ASSERT_EQUAL(id, 1);
|
|
|
|
std::string_view name;
|
|
ASSERT_SUCCESS(doc["name"].get_string().get(name));
|
|
ASSERT_EQUAL(name, "Alice");
|
|
|
|
bool active;
|
|
ASSERT_SUCCESS(doc["active"].get_bool().get(active));
|
|
ASSERT_EQUAL(active, true);
|
|
|
|
// Output should be inline since it's simple
|
|
std::cout << "Basic FracturedJson output: " << formatted << std::endl;
|
|
#endif
|
|
TEST_SUCCEED();
|
|
}
|
|
|
|
// Test with custom formatting options
|
|
bool custom_options_test() {
|
|
TEST_START();
|
|
#if SIMDJSON_STATIC_REFLECTION
|
|
NestedData data{"Test", {{1, "Alice", true}, {2, "Bob", false}}, 2};
|
|
|
|
fractured_json_options opts;
|
|
opts.indent_spaces = 2;
|
|
opts.simple_bracket_padding = false;
|
|
|
|
std::string formatted;
|
|
ASSERT_SUCCESS(simdjson::to_fractured_json_string(data, opts).get(formatted));
|
|
|
|
// Verify it produces valid JSON
|
|
simdjson::dom::parser parser;
|
|
simdjson::dom::element doc;
|
|
ASSERT_SUCCESS(parser.parse(formatted).get(doc));
|
|
|
|
std::cout << "Custom options output:\n" << formatted << std::endl;
|
|
#endif
|
|
TEST_SUCCEED();
|
|
}
|
|
|
|
// Test to_fractured_json with output parameter
|
|
bool output_param_test() {
|
|
TEST_START();
|
|
#if SIMDJSON_STATIC_REFLECTION
|
|
SimpleUser user{42, "Charlie", false};
|
|
|
|
std::string output;
|
|
ASSERT_SUCCESS(simdjson::to_fractured_json_string(user).get(output));
|
|
|
|
// Verify it produces valid JSON
|
|
simdjson::dom::parser parser;
|
|
simdjson::dom::element doc;
|
|
ASSERT_SUCCESS(parser.parse(output).get(doc));
|
|
|
|
std::cout << "Output param result: " << output << std::endl;
|
|
#endif
|
|
TEST_SUCCEED();
|
|
}
|
|
|
|
// Test extract_fractured_json for partial serialization
|
|
bool extract_fields_test() {
|
|
TEST_START();
|
|
#if SIMDJSON_STATIC_REFLECTION
|
|
UserWithEmail user{1, "Alice", "alice@example.com", true};
|
|
|
|
// Extract only id and name fields
|
|
std::string formatted;
|
|
ASSERT_SUCCESS(simdjson::extract_fractured_json<"id", "name">(user).get(formatted));
|
|
|
|
// Verify it produces valid JSON
|
|
simdjson::dom::parser parser;
|
|
simdjson::dom::element doc;
|
|
ASSERT_SUCCESS(parser.parse(formatted).get(doc));
|
|
|
|
// Should have id and name but NOT email and active
|
|
int64_t id;
|
|
ASSERT_SUCCESS(doc["id"].get_int64().get(id));
|
|
ASSERT_EQUAL(id, 1);
|
|
|
|
std::string_view name;
|
|
ASSERT_SUCCESS(doc["name"].get_string().get(name));
|
|
ASSERT_EQUAL(name, "Alice");
|
|
|
|
// email should not be present
|
|
simdjson::dom::element email_elem;
|
|
auto err = doc["email"].get(email_elem);
|
|
ASSERT_TRUE(err != SUCCESS); // Should fail - field not present
|
|
|
|
std::cout << "Extract fields output: " << formatted << std::endl;
|
|
#endif
|
|
TEST_SUCCEED();
|
|
}
|
|
|
|
// Test table formatting with array of similar objects
|
|
bool table_format_test() {
|
|
TEST_START();
|
|
#if SIMDJSON_STATIC_REFLECTION
|
|
TableTestData data{
|
|
{{1, "Alice", true}, {2, "Bob", false}, {3, "Carol", true}, {4, "Dave", false}}
|
|
};
|
|
|
|
fractured_json_options opts;
|
|
opts.enable_table_format = true;
|
|
opts.min_table_rows = 3;
|
|
|
|
std::string formatted;
|
|
ASSERT_SUCCESS(simdjson::to_fractured_json_string(data, opts).get(formatted));
|
|
|
|
// Verify it produces valid JSON
|
|
simdjson::dom::parser parser;
|
|
simdjson::dom::element doc;
|
|
ASSERT_SUCCESS(parser.parse(formatted).get(doc));
|
|
|
|
std::cout << "Table format output:\n" << formatted << std::endl;
|
|
#endif
|
|
TEST_SUCCEED();
|
|
}
|
|
|
|
// Test roundtrip: serialize with reflection, parse back
|
|
bool roundtrip_test() {
|
|
TEST_START();
|
|
#if SIMDJSON_STATIC_REFLECTION
|
|
SimpleUser original{123, "TestUser", true};
|
|
|
|
// Serialize with FracturedJson
|
|
std::string formatted;
|
|
ASSERT_SUCCESS(simdjson::to_fractured_json_string(original).get(formatted));
|
|
|
|
// Parse back using ondemand
|
|
simdjson::ondemand::parser parser;
|
|
simdjson::ondemand::document doc;
|
|
ASSERT_SUCCESS(parser.iterate(simdjson::pad(formatted)).get(doc));
|
|
|
|
// Deserialize back to struct
|
|
SimpleUser parsed;
|
|
ASSERT_SUCCESS(doc.get<SimpleUser>().get(parsed));
|
|
|
|
// Verify values match
|
|
ASSERT_EQUAL(parsed.id, original.id);
|
|
ASSERT_EQUAL(parsed.name, original.name);
|
|
ASSERT_EQUAL(parsed.active, original.active);
|
|
|
|
std::cout << "Roundtrip test passed with: " << formatted << std::endl;
|
|
#endif
|
|
TEST_SUCCEED();
|
|
}
|
|
|
|
// Test global namespace functions
|
|
bool global_namespace_test() {
|
|
TEST_START();
|
|
#if SIMDJSON_STATIC_REFLECTION
|
|
SimpleUser user{99, "GlobalTest", false};
|
|
|
|
// Test global to_fractured_json_string
|
|
std::string formatted;
|
|
ASSERT_SUCCESS(simdjson::to_fractured_json_string(user).get(formatted));
|
|
// Verify output
|
|
simdjson::dom::parser parser;
|
|
simdjson::dom::element doc;
|
|
ASSERT_SUCCESS(parser.parse(simdjson::pad(formatted)).get(doc));
|
|
std::cout << "Global namespace output: " << formatted << std::endl;
|
|
#endif
|
|
TEST_SUCCEED();
|
|
}
|
|
|
|
bool run() {
|
|
return basic_fractured_json_test() &&
|
|
custom_options_test() &&
|
|
output_param_test() &&
|
|
extract_fields_test() &&
|
|
table_format_test() &&
|
|
roundtrip_test() &&
|
|
global_namespace_test();
|
|
}
|
|
|
|
} // namespace fractured_json_builder_tests
|
|
|
|
int main(int argc, char *argv[]) {
|
|
std::cout << "Running FracturedJson builder integration tests..." << std::endl;
|
|
#if SIMDJSON_STATIC_REFLECTION
|
|
std::cout << "Static reflection is ENABLED" << std::endl;
|
|
#else
|
|
std::cout << "Static reflection is DISABLED - tests will be skipped" << std::endl;
|
|
#endif
|
|
return test_main(argc, argv, fractured_json_builder_tests::run);
|
|
}
|