mirror of
https://github.com/simdjson/simdjson
synced 2026-06-08 17:27:07 +00:00
fc57c09cf0
* Add FracturedJson formatting support for DOM serialization
Implements FracturedJson formatting as requested in issue #2576.
FracturedJson produces human-readable yet compact JSON output by
intelligently choosing between different layout strategies based on
content complexity, length, and structure similarity.
Key features:
- Four layout modes: inline, compact multiline, table, and expanded
- Structure analysis pass to compute metrics before formatting
- Table formatting for arrays of similar objects with column alignment
- Configurable options for line length, indentation, padding, etc.
New files:
- fractured_json.h: Public API with fractured_json_options struct
- fractured_json-inl.h: Implementation (~1000 lines)
- json_structure_analyzer.h: Structure analysis for layout decisions
- fractured_formatter.h: Formatter class using CRTP pattern
Usage:
dom::parser parser;
element doc = parser.parse(json_string);
std::cout << fractured_json(doc) << std::endl;
// Or with custom options:
fractured_json_options opts;
opts.indent_spaces = 2;
std::cout << fractured_json(doc, opts) << std::endl;
// Or format any JSON string (useful with reflection API):
auto formatted = fractured_json_string(minified_json);
Resolves #2576
* Add comprehensive tests for FracturedJson formatter
Adds 27 test cases covering all aspects of the FracturedJson formatter:
Core functionality tests (13):
- Roundtrip parsing verification
- Inline formatting for simple arrays and objects
- Expanded formatting for complex nested structures
- Compact multiline arrays with configurable items per line
- Table formatting for uniform arrays of objects
- Empty container handling
- All scalar types (string, int, uint, double, bool, null)
- String escaping (quotes, backslashes, control characters)
- Custom indentation options
- Deep nesting (10+ levels)
- Mixed type arrays
Edge case tests (11):
- Unicode strings (Chinese, emoji, Arabic, Russian, accented chars)
- Boundary numbers (INT64_MIN/MAX, UINT64_MAX, DBL_MIN/MAX)
- Nested arrays (arrays of arrays)
- Empty string values
- Keys with special characters (spaces, quotes, colons, etc.)
- Non-uniform arrays (should not trigger table mode)
- Very long strings (500+ chars)
- Large arrays (100 elements)
- Reflection API workflow simulation
- Control characters (tab, newline, CR, null)
- Single element containers
Option tests (3):
- Disable compact multiline mode
- Disable table format mode
- Disable all padding options
* Add FracturedJson integration with builder/reflection API
Extends FracturedJson to work seamlessly with the builder API, enabling
formatted output directly from C++ structs using static reflection.
New functions:
- to_fractured_json_string(obj, opts) - serialize struct to formatted JSON
- to_fractured_json(obj, output, opts) - same with output parameter
- extract_fractured_json<fields...>(obj, opts) - format only specific fields
These functions combine the builder's reflection-based serialization with
FracturedJson formatting in a single convenient call:
struct User { int id; std::string name; bool active; };
User user{1, "Alice", true};
// Minified output (existing):
auto minified = to_json_string(user);
// {"id":1,"name":"Alice","active":true}
// Formatted output (new):
auto formatted = to_fractured_json_string(user);
// { "id": 1, "name": "Alice", "active": true }
// Partial extraction with formatting:
auto partial = extract_fractured_json<"id", "name">(user);
// { "id": 1, "name": "Alice" }
New files:
- generic/builder/fractured_json_builder.h - builder integration
- tests/builder/static_reflection_fractured_json_tests.cpp - 7 tests
* Fix INT64_MIN overflow and implement table_similarity_threshold
- Fix undefined behavior when negating INT64_MIN in estimate_number_length()
and measure_value_length() by returning 20 (the exact length of the
string representation) directly
- Actually use table_similarity_threshold in check_array_uniformity() by
calling compute_object_similarity() to compare objects against the first
object in the array
* Fix -Werror=effc++ member initialization warnings
Initialize all member variables in member initialization lists to
satisfy GCC's -Werror=effc++ flag:
- element_metrics::common_keys - add {} default initializer
- structure_analyzer - add default constructor with member init list
- fractured_formatter - add column_widths_{} to constructor
- fractured_string_builder - add analyzer_{} to constructor
* Add Rule of Five to structure_analyzer class
The class has a pointer member (current_opts_) which triggers
-Werror=effc++ requiring explicit copy/move operations. Delete
copy operations (class shouldn't be copied due to cache) and
default move operations.
* Fix Windows build: wrap std::max to avoid macro conflict
Windows.h defines max/min macros that interfere with std::max/std::min.
Wrapping in parentheses as (std::max)(...) prevents macro expansion.
* Fix GCC 15 false positive -Wfree-nonheap-object warning
GCC 15 on MINGW64 gives a false positive warning in parser_moving_parser()
when the std::vector<std::string> goes out of scope. Suppress this
specific warning with a pragma for GCC builds.
* Fix metrics cache key bug by passing metrics through recursion
The cache was using element addresses as keys, but dom::element objects
are lightweight wrappers that get copied during iteration, causing
different addresses between analysis and formatting phases. This resulted
in cache misses and fallback to empty metrics.
Solution: Store child metrics in the element_metrics struct and pass
them through recursive calls, eliminating the need for address-based
caching entirely.
Changes:
- Add children vector to element_metrics for hierarchical metrics
- Remove metrics_cache_ and related get_metrics/has_metrics methods
- Update all format functions to accept and pass child metrics
- Add public analyze_array/analyze_object overloads for standalone use
* Add ignore patterns for Node.js, Rust, and generated files
Add entries for node_modules, package-lock.json, Rust target
directories, local ablation artifacts, and generated documentation
files.
* Refactor: extract analyze_scalar helper to reduce code duplication
Extract common scalar type handling (STRING, INT64, UINT64, DOUBLE,
BOOL, NULL_VALUE) into a dedicated analyze_scalar method. Each scalar
type shares the same initialization pattern for complexity, child_count,
can_inline, and recommended_layout.
Also simplify boolean formatting in format_scalar to use ternary operator.
* Fix formatting and duplicate error message in amalgamate.py
Reformat cramped is_amalgamator condition to multi-line for readability.
Fix duplicate error message text in _included_filename_root and use
correct variable name (relative_root instead of root).
* Refactor: add count_newlines helper in fractured_json tests
Extract repeated newline counting loop into a reusable static helper
function, used by inline_array_test, inline_object_test, and
expanded_test.
* Revert "Add ignore patterns for Node.js, Rust, and generated files"
This reverts commit 4760ea7cd0.
* various minor changes
---------
Co-authored-by: Daniel Lemire <daniel@lemire.me>
170 lines
5.5 KiB
C++
170 lines
5.5 KiB
C++
#ifndef SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H
|
|
#define SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H
|
|
|
|
#include "simdjson/dom/base.h"
|
|
#include "simdjson/dom/element.h"
|
|
#include "simdjson/dom/array.h"
|
|
#include "simdjson/dom/object.h"
|
|
#include "simdjson/dom/fractured_json.h"
|
|
#include "simdjson/internal/tape_type.h"
|
|
|
|
#include <vector>
|
|
#include <string>
|
|
#include <string_view>
|
|
#include <set>
|
|
|
|
namespace simdjson {
|
|
namespace internal {
|
|
|
|
/**
|
|
* Layout mode for fractured JSON formatting.
|
|
*/
|
|
enum class layout_mode {
|
|
INLINE, // Single line: [1, 2, 3] or {"a": 1}
|
|
COMPACT_MULTILINE, // Multiple items per line with breaks
|
|
TABLE, // Tabular format for arrays of similar objects
|
|
EXPANDED // Traditional multi-line with indentation
|
|
};
|
|
|
|
/**
|
|
* Metrics computed for a JSON element during structure analysis.
|
|
* These metrics drive layout decisions and contain child metrics for recursive formatting.
|
|
*/
|
|
struct element_metrics {
|
|
/** Nesting depth score (0 = scalar, 1 = flat container, etc.) */
|
|
size_t complexity = 0;
|
|
|
|
/** Estimated character length if rendered inline (minified + spaces) */
|
|
size_t estimated_inline_len = 0;
|
|
|
|
/** Number of direct children (0 for scalars) */
|
|
size_t child_count = 0;
|
|
|
|
/** Pre-computed: can this element be rendered inline? */
|
|
bool can_inline = false;
|
|
|
|
/** Is this an array where all elements have similar structure? */
|
|
bool is_uniform_array = false;
|
|
|
|
/** For uniform arrays of objects: the common keys */
|
|
std::vector<std::string> common_keys{};
|
|
|
|
/** Recommended layout mode based on analysis */
|
|
layout_mode recommended_layout = layout_mode::EXPANDED;
|
|
|
|
/** Child metrics for arrays and objects (in order of iteration) */
|
|
std::vector<element_metrics> children{};
|
|
};
|
|
|
|
/**
|
|
* Analyzes JSON structure to compute metrics for formatting decisions.
|
|
*
|
|
* The analyzer performs a single pass over the DOM to compute:
|
|
* - Complexity (nesting depth)
|
|
* - Estimated inline length
|
|
* - Array uniformity for table detection
|
|
*
|
|
* Metrics are stored hierarchically with child metrics embedded in parent metrics,
|
|
* enabling efficient lookup during formatting without address-based caching.
|
|
*/
|
|
class structure_analyzer {
|
|
public:
|
|
/** Default constructor */
|
|
structure_analyzer() : current_opts_(nullptr) {}
|
|
|
|
/** Copy constructor - deleted since class has pointer member */
|
|
structure_analyzer(const structure_analyzer&) = delete;
|
|
|
|
/** Copy assignment - deleted since class has pointer member */
|
|
structure_analyzer& operator=(const structure_analyzer&) = delete;
|
|
|
|
/** Move constructor */
|
|
structure_analyzer(structure_analyzer&&) = default;
|
|
|
|
/** Move assignment */
|
|
structure_analyzer& operator=(structure_analyzer&&) = default;
|
|
|
|
/**
|
|
* Analyze a DOM element and compute metrics.
|
|
* @param elem The element to analyze
|
|
* @param opts Formatting options that affect metric computation
|
|
* @return Metrics for the root element (with child metrics embedded)
|
|
*/
|
|
element_metrics analyze(const dom::element& elem,
|
|
const fractured_json_options& opts);
|
|
|
|
/**
|
|
* Clear state.
|
|
*/
|
|
void clear();
|
|
|
|
/**
|
|
* Analyze an array element directly (for standalone array formatting).
|
|
* @param arr The array to analyze
|
|
* @param opts Formatting options
|
|
* @return Metrics for the array
|
|
*/
|
|
element_metrics analyze_array(const dom::array& arr,
|
|
const fractured_json_options& opts);
|
|
|
|
/**
|
|
* Analyze an object element directly (for standalone object formatting).
|
|
* @param obj The object to analyze
|
|
* @param opts Formatting options
|
|
* @return Metrics for the object
|
|
*/
|
|
element_metrics analyze_object(const dom::object& obj,
|
|
const fractured_json_options& opts);
|
|
|
|
private:
|
|
const fractured_json_options* current_opts_ = nullptr;
|
|
|
|
/** Recursive analysis implementation */
|
|
element_metrics analyze_element(const dom::element& elem, size_t depth);
|
|
|
|
/** Analyze scalar values (strings, numbers, booleans, null) */
|
|
element_metrics analyze_scalar(const dom::element& elem);
|
|
|
|
/** Analyze an array element */
|
|
element_metrics analyze_array(const dom::array& arr, size_t depth);
|
|
|
|
/** Analyze an object element */
|
|
element_metrics analyze_object(const dom::object& obj, size_t depth);
|
|
|
|
/** Estimate inline length for a string (including quotes and escaping) */
|
|
size_t estimate_string_length(std::string_view s) const;
|
|
|
|
/** Estimate inline length for a number */
|
|
size_t estimate_number_length(double d) const;
|
|
size_t estimate_number_length(int64_t i) const;
|
|
size_t estimate_number_length(uint64_t u) const;
|
|
|
|
/**
|
|
* Check if an array contains uniform objects suitable for table formatting.
|
|
* @param arr The array to check
|
|
* @param common_keys Output: keys common to all objects
|
|
* @return true if the array is suitable for table formatting
|
|
*/
|
|
bool check_array_uniformity(const dom::array& arr,
|
|
std::vector<std::string>& common_keys) const;
|
|
|
|
/**
|
|
* Compute similarity between two objects.
|
|
* @return Fraction of keys that are common (0.0 to 1.0)
|
|
*/
|
|
double compute_object_similarity(const dom::object& a,
|
|
const dom::object& b) const;
|
|
|
|
/**
|
|
* Decide the recommended layout mode based on metrics and options.
|
|
*/
|
|
layout_mode decide_layout(const element_metrics& metrics,
|
|
size_t depth,
|
|
size_t available_width) const;
|
|
};
|
|
|
|
} // namespace internal
|
|
} // namespace simdjson
|
|
|
|
#endif // SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H
|