Add FracturedJson formatting support for DOM serialization (#2580)

* Add FracturedJson formatting support for DOM serialization

Implements FracturedJson formatting as requested in issue #2576.
FracturedJson produces human-readable yet compact JSON output by
intelligently choosing between different layout strategies based on
content complexity, length, and structure similarity.

Key features:
- Four layout modes: inline, compact multiline, table, and expanded
- Structure analysis pass to compute metrics before formatting
- Table formatting for arrays of similar objects with column alignment
- Configurable options for line length, indentation, padding, etc.

New files:
- fractured_json.h: Public API with fractured_json_options struct
- fractured_json-inl.h: Implementation (~1000 lines)
- json_structure_analyzer.h: Structure analysis for layout decisions
- fractured_formatter.h: Formatter class using CRTP pattern

Usage:
  dom::parser parser;
  element doc = parser.parse(json_string);
  std::cout << fractured_json(doc) << std::endl;

  // Or with custom options:
  fractured_json_options opts;
  opts.indent_spaces = 2;
  std::cout << fractured_json(doc, opts) << std::endl;

  // Or format any JSON string (useful with reflection API):
  auto formatted = fractured_json_string(minified_json);

Resolves #2576

* Add comprehensive tests for FracturedJson formatter

Adds 27 test cases covering all aspects of the FracturedJson formatter:

Core functionality tests (13):
- Roundtrip parsing verification
- Inline formatting for simple arrays and objects
- Expanded formatting for complex nested structures
- Compact multiline arrays with configurable items per line
- Table formatting for uniform arrays of objects
- Empty container handling
- All scalar types (string, int, uint, double, bool, null)
- String escaping (quotes, backslashes, control characters)
- Custom indentation options
- Deep nesting (10+ levels)
- Mixed type arrays

Edge case tests (11):
- Unicode strings (Chinese, emoji, Arabic, Russian, accented chars)
- Boundary numbers (INT64_MIN/MAX, UINT64_MAX, DBL_MIN/MAX)
- Nested arrays (arrays of arrays)
- Empty string values
- Keys with special characters (spaces, quotes, colons, etc.)
- Non-uniform arrays (should not trigger table mode)
- Very long strings (500+ chars)
- Large arrays (100 elements)
- Reflection API workflow simulation
- Control characters (tab, newline, CR, null)
- Single element containers

Option tests (3):
- Disable compact multiline mode
- Disable table format mode
- Disable all padding options

* Add FracturedJson integration with builder/reflection API

Extends FracturedJson to work seamlessly with the builder API, enabling
formatted output directly from C++ structs using static reflection.

New functions:
- to_fractured_json_string(obj, opts) - serialize struct to formatted JSON
- to_fractured_json(obj, output, opts) - same with output parameter
- extract_fractured_json<fields...>(obj, opts) - format only specific fields

These functions combine the builder's reflection-based serialization with
FracturedJson formatting in a single convenient call:

  struct User { int id; std::string name; bool active; };
  User user{1, "Alice", true};

  // Minified output (existing):
  auto minified = to_json_string(user);
  // {"id":1,"name":"Alice","active":true}

  // Formatted output (new):
  auto formatted = to_fractured_json_string(user);
  // { "id": 1, "name": "Alice", "active": true }

  // Partial extraction with formatting:
  auto partial = extract_fractured_json<"id", "name">(user);
  // { "id": 1, "name": "Alice" }

New files:
- generic/builder/fractured_json_builder.h - builder integration
- tests/builder/static_reflection_fractured_json_tests.cpp - 7 tests

* Fix INT64_MIN overflow and implement table_similarity_threshold

- Fix undefined behavior when negating INT64_MIN in estimate_number_length()
  and measure_value_length() by returning 20 (the exact length of the
  string representation) directly
- Actually use table_similarity_threshold in check_array_uniformity() by
  calling compute_object_similarity() to compare objects against the first
  object in the array

* Fix -Werror=effc++ member initialization warnings

Initialize all member variables in member initialization lists to
satisfy GCC's -Werror=effc++ flag:
- element_metrics::common_keys - add {} default initializer
- structure_analyzer - add default constructor with member init list
- fractured_formatter - add column_widths_{} to constructor
- fractured_string_builder - add analyzer_{} to constructor

* Add Rule of Five to structure_analyzer class

The class has a pointer member (current_opts_) which triggers
-Werror=effc++ requiring explicit copy/move operations. Delete
copy operations (class shouldn't be copied due to cache) and
default move operations.

* Fix Windows build: wrap std::max to avoid macro conflict

Windows.h defines max/min macros that interfere with std::max/std::min.
Wrapping in parentheses as (std::max)(...) prevents macro expansion.

* Fix GCC 15 false positive -Wfree-nonheap-object warning

GCC 15 on MINGW64 gives a false positive warning in parser_moving_parser()
when the std::vector<std::string> goes out of scope. Suppress this
specific warning with a pragma for GCC builds.

* Fix metrics cache key bug by passing metrics through recursion

The cache was using element addresses as keys, but dom::element objects
are lightweight wrappers that get copied during iteration, causing
different addresses between analysis and formatting phases. This resulted
in cache misses and fallback to empty metrics.

Solution: Store child metrics in the element_metrics struct and pass
them through recursive calls, eliminating the need for address-based
caching entirely.

Changes:
- Add children vector to element_metrics for hierarchical metrics
- Remove metrics_cache_ and related get_metrics/has_metrics methods
- Update all format functions to accept and pass child metrics
- Add public analyze_array/analyze_object overloads for standalone use

* Add ignore patterns for Node.js, Rust, and generated files

Add entries for node_modules, package-lock.json, Rust target
directories, local ablation artifacts, and generated documentation
files.

* Refactor: extract analyze_scalar helper to reduce code duplication

Extract common scalar type handling (STRING, INT64, UINT64, DOUBLE,
BOOL, NULL_VALUE) into a dedicated analyze_scalar method. Each scalar
type shares the same initialization pattern for complexity, child_count,
can_inline, and recommended_layout.

Also simplify boolean formatting in format_scalar to use ternary operator.

* Fix formatting and duplicate error message in amalgamate.py

Reformat cramped is_amalgamator condition to multi-line for readability.
Fix duplicate error message text in _included_filename_root and use
correct variable name (relative_root instead of root).

* Refactor: add count_newlines helper in fractured_json tests

Extract repeated newline counting loop into a reusable static helper
function, used by inline_array_test, inline_object_test, and
expanded_test.

* Revert "Add ignore patterns for Node.js, Rust, and generated files"

This reverts commit 4760ea7cd0.

* various minor changes

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>
This commit is contained in:
Francisco Geiman Thiesen
2026-01-20 07:32:24 -08:00
committed by GitHub
parent 2058b47dfe
commit fc57c09cf0
15 changed files with 3011 additions and 2 deletions
+47
View File
@@ -14,6 +14,7 @@ speed and high convenience.
* [C++26 static reflection](#c--26-static-reflection)
+ [Without `string_buffer` instance](#without--string-buffer--instance)
+ [Without `string_buffer` instance but with explicit error handling](#without--string-buffer--instance-but-with-explicit-error-handling)
+ [Pretty formatted (fractured JSON)](#pretty-formatted-fractured-json)
Overview: string_builder
---------------------------
@@ -364,3 +365,49 @@ void tag_invoke(serialize_tag, builder_type &builder, const Car& car) {
} // namespace simdjson
```
### Pretty formatted (fractured JSON)
In some instances, you may want your JSON to be more readable. For this pupose, we also
support the Fractured JSON standard.
```Cpp
TableTestData data{
{{1, "Alice", true}, {2, "Bob", false}, {3, "Carol", true}, {4, "Dave", false}}
};
fractured_json_options opts;
opts.enable_table_format = true;
opts.min_table_rows = 3;
std::string formatted = simdjson::to_fractured_json_string(data, opts);
```
The result might be as follows.
```json
{
"records": [
{ "active": true , "id": 1, "name": "Alice" },
{ "active": false, "id": 2, "name": "Bob" },
{ "active": true , "id": 3, "name": "Carol" },
{ "active": false, "id": 4, "name": "Dave" }
]
}
```
The `fractured_json_options` struct allows you to customize the formatting behavior. It includes the following options:
- `max_total_line_length` (default: 120): Maximum total characters per line. Content exceeding this will be expanded to multiple lines.
- `max_inline_length` (default: 80): Maximum length for inlined elements. Simple arrays/objects shorter than this may be rendered inline.
- `max_inline_complexity` (default: 2): Maximum nesting depth for inline rendering. Elements with complexity exceeding this will be expanded. Complexity 0 = scalar, 1 = flat array/object, 2 = one level of nesting.
- `max_compact_array_complexity` (default: 1): Maximum complexity for compact array formatting. Arrays with elements of this complexity or less may have multiple items per line.
- `indent_spaces` (default: 4): Number of spaces per indentation level.
- `enable_table_format` (default: true): Enable tabular formatting for arrays of similar objects. When enabled, arrays of objects with identical keys are formatted as aligned tables.
- `min_table_rows` (default: 3): Minimum number of rows to trigger table mode.
- `table_similarity_threshold` (default: 0.8): Similarity threshold for table detection. Objects must share at least this fraction of keys to be formatted as a table.
- `enable_compact_multiline` (default: true): Enable compact multiline arrays. When enabled, arrays of simple elements may have multiple items per line.
- `max_items_per_line` (default: 10): Maximum array items per line in compact mode.
- `simple_bracket_padding` (default: true): Add space inside brackets for simple containers. When true: `{ "key": "value" }`, when false: `{"key": "value"}`.
- `colon_padding` (default: true): Add space after colons. When true: `"key": "value"`, when false: `"key":"value"`.
- `comma_padding` (default: true): Add space after commas in inline content. When true: `[1, 2, 3]`, when false: `[1,2,3]`.
+2
View File
@@ -9,6 +9,7 @@
#include "simdjson/dom/object.h"
#include "simdjson/dom/parser.h"
#include "simdjson/dom/serialization.h"
#include "simdjson/dom/fractured_json.h"
// Inline functions
#include "simdjson/dom/array-inl.h"
@@ -19,5 +20,6 @@
#include "simdjson/dom/parser-inl.h"
#include "simdjson/internal/tape_ref-inl.h"
#include "simdjson/dom/serialization-inl.h"
#include "simdjson/dom/fractured_json-inl.h"
#endif // SIMDJSON_DOM_H
File diff suppressed because it is too large Load Diff
+159
View File
@@ -0,0 +1,159 @@
#ifndef SIMDJSON_DOM_FRACTURED_JSON_H
#define SIMDJSON_DOM_FRACTURED_JSON_H
#include "simdjson/dom/base.h"
#include "simdjson/dom/element.h"
namespace simdjson {
/**
* Configuration options for FracturedJson formatting.
*
* FracturedJson intelligently chooses between different layout strategies
* (inline, compact multiline, table, expanded) based on content complexity,
* length, and structure similarity.
*/
struct fractured_json_options {
/**
* Maximum total characters per line (default: 120).
* Content exceeding this will be expanded to multiple lines.
*/
size_t max_total_line_length = 120;
/**
* Maximum length for inlined elements (default: 80).
* Simple arrays/objects shorter than this may be rendered inline.
*/
size_t max_inline_length = 80;
/**
* Maximum nesting depth for inline rendering (default: 2).
* Elements with complexity exceeding this will be expanded.
* Complexity 0 = scalar, 1 = flat array/object, 2 = one level of nesting.
*/
size_t max_inline_complexity = 2;
/**
* Maximum complexity for compact array formatting (default: 1).
* Arrays with elements of this complexity or less may have multiple
* items per line.
*/
size_t max_compact_array_complexity = 1;
/**
* Number of spaces per indentation level (default: 4).
*/
size_t indent_spaces = 4;
/**
* Enable tabular formatting for arrays of similar objects (default: true).
* When enabled, arrays of objects with identical keys are formatted
* as aligned tables.
*/
bool enable_table_format = true;
/**
* Minimum number of rows to trigger table mode (default: 3).
*/
size_t min_table_rows = 3;
/**
* Similarity threshold for table detection (default: 0.8).
* Objects must share at least this fraction of keys to be formatted
* as a table.
*/
double table_similarity_threshold = 0.8;
/**
* Enable compact multiline arrays (default: true).
* When enabled, arrays of simple elements may have multiple items
* per line.
*/
bool enable_compact_multiline = true;
/**
* Maximum array items per line in compact mode (default: 10).
*/
size_t max_items_per_line = 10;
/**
* Add space inside brackets for simple containers (default: true).
* When true: { "key": "value" }
* When false: {"key": "value"}
*/
bool simple_bracket_padding = true;
/**
* Add space after colons (default: true).
* When true: "key": "value"
* When false: "key":"value"
*/
bool colon_padding = true;
/**
* Add space after commas in inline content (default: true).
* When true: [1, 2, 3]
* When false: [1,2,3]
*/
bool comma_padding = true;
};
/**
* Format JSON using FracturedJson formatting with default options.
*
* FracturedJson produces human-readable yet compact output by intelligently
* choosing between inline, compact multiline, table, and expanded layouts.
*
* dom::parser parser;
* element doc = parser.parse(json_string);
* cout << fractured_json(doc) << endl;
*/
template <class T>
std::string fractured_json(T x);
/**
* Format JSON using FracturedJson formatting with custom options.
*
* dom::parser parser;
* element doc = parser.parse(json_string);
* fractured_json_options opts;
* opts.max_total_line_length = 80;
* cout << fractured_json(doc, opts) << endl;
*/
template <class T>
std::string fractured_json(T x, const fractured_json_options& options);
#if SIMDJSON_EXCEPTIONS
template <class T>
std::string fractured_json(simdjson_result<T> x);
template <class T>
std::string fractured_json(simdjson_result<T> x, const fractured_json_options& options);
#endif
/**
* Format a JSON string using FracturedJson formatting.
*
* This is useful for formatting output from the builder/static reflection API
* or any valid JSON string.
*
* // With static reflection
* MyStruct data = {...};
* auto minified = simdjson::to_json_string(data);
* auto formatted = simdjson::fractured_json_string(minified.value());
*
* // Or with any JSON string
* std::string json = R"({"key":"value"})";
* auto formatted = simdjson::fractured_json_string(json);
*/
inline std::string fractured_json_string(std::string_view json_str);
/**
* Format a JSON string using FracturedJson formatting with custom options.
*/
inline std::string fractured_json_string(std::string_view json_str,
const fractured_json_options& options);
} // namespace simdjson
#endif // SIMDJSON_DOM_FRACTURED_JSON_H
@@ -4,6 +4,7 @@
#include "simdjson/generic/builder/json_string_builder.h"
#include "simdjson/generic/builder/json_builder.h"
#include "simdjson/generic/builder/fractured_json_builder.h"
@@ -9,5 +9,6 @@
// All includes not under simdjson/generic/builder must be here!
// Otherwise, amalgamation will fail.
#include "simdjson/concepts.h"
#include "simdjson/dom/fractured_json.h"
#endif // SIMDJSON_GENERIC_BUILDER_DEPENDENCIES_H
@@ -0,0 +1,117 @@
#ifndef SIMDJSON_GENERIC_FRACTURED_JSON_BUILDER_H
#define SIMDJSON_GENERIC_FRACTURED_JSON_BUILDER_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/generic/builder/json_builder.h"
#include "simdjson/dom/fractured_json.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#if SIMDJSON_STATIC_REFLECTION
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace builder {
/**
* Serialize an object to a FracturedJson-formatted string.
*
* FracturedJson produces human-readable yet compact JSON output by intelligently
* choosing between different layout strategies (inline, compact multiline, table,
* expanded) based on content complexity, length, and structure similarity.
*
* This function combines the builder's serialization with FracturedJson formatting:
* 1. Serializes the object to minified JSON using reflection
* 2. Parses and reformats using FracturedJson
*
* Example:
* struct User { int id; std::string name; bool active; };
* User user{1, "Alice", true};
* auto result = to_fractured_json_string(user);
* // result.value() == "{ \"id\": 1, \"name\": \"Alice\", \"active\": true }"
*
* @param obj The object to serialize (must be a reflectable type)
* @param opts FracturedJson formatting options
* @param initial_capacity Initial buffer capacity for serialization
* @return The formatted JSON string, or an error
*/
template <class T>
simdjson_warn_unused simdjson_result<std::string> to_fractured_json_string(
const T& obj,
const fractured_json_options& opts = {},
size_t initial_capacity = string_builder::DEFAULT_INITIAL_CAPACITY) {
// Step 1: Serialize to minified JSON
std::string formatted;
auto error = to_json_string(obj, initial_capacity).get(formatted);
if (error) {
return error;
}
// Step 2: Reformat with FracturedJson
return fractured_json_string(formatted, opts);
}
/**
* Extract specific fields from an object and format with FracturedJson.
*
* Example:
* struct User { int id; std::string name; std::string email; bool active; };
* User user{1, "Alice", "alice@example.com", true};
* auto result = extract_fractured_json<"id", "name">(user);
* // result.value() == "{ \"id\": 1, \"name\": \"Alice\" }"
*
* @param obj The object to serialize
* @param opts FracturedJson formatting options
* @param initial_capacity Initial buffer capacity for serialization
* @return The formatted JSON string containing only the specified fields
*/
template<constevalutil::fixed_string... FieldNames, typename T>
requires(std::is_class_v<T> && (sizeof...(FieldNames) > 0))
simdjson_warn_unused simdjson_result<std::string> extract_fractured_json(
const T& obj,
const fractured_json_options& opts = {},
size_t initial_capacity = string_builder::DEFAULT_INITIAL_CAPACITY) {
// Step 1: Extract fields to minified JSON
std::string formatted;
auto error = extract_from<FieldNames...>(obj, initial_capacity).get(formatted);
if (error) {
return error;
}
// Step 2: Reformat with FracturedJson
return fractured_json_string(formatted, opts);
}
} // namespace builder
} // namespace SIMDJSON_IMPLEMENTATION
// Global namespace convenience functions
/**
* Serialize an object to a FracturedJson-formatted string.
* Global namespace version for convenience.
*/
template <class T>
simdjson_warn_unused simdjson_result<std::string> to_fractured_json_string(
const T& obj,
const fractured_json_options& opts = {},
size_t initial_capacity = SIMDJSON_IMPLEMENTATION::builder::string_builder::DEFAULT_INITIAL_CAPACITY) {
return SIMDJSON_IMPLEMENTATION::builder::to_fractured_json_string(obj, opts, initial_capacity);
}
/**
* Extract specific fields from an object and format with FracturedJson.
* Global namespace version for convenience.
*/
template<constevalutil::fixed_string... FieldNames, typename T>
requires(std::is_class_v<T> && (sizeof...(FieldNames) > 0))
simdjson_warn_unused simdjson_result<std::string> extract_fractured_json(
const T& obj,
const fractured_json_options& opts = {},
size_t initial_capacity = SIMDJSON_IMPLEMENTATION::builder::string_builder::DEFAULT_INITIAL_CAPACITY) {
return SIMDJSON_IMPLEMENTATION::builder::extract_fractured_json<FieldNames...>(obj, opts, initial_capacity);
}
} // namespace simdjson
#endif // SIMDJSON_STATIC_REFLECTION
#endif // SIMDJSON_GENERIC_FRACTURED_JSON_BUILDER_H
@@ -0,0 +1,161 @@
#ifndef SIMDJSON_INTERNAL_FRACTURED_FORMATTER_H
#define SIMDJSON_INTERNAL_FRACTURED_FORMATTER_H
#include "simdjson/dom/serialization.h"
#include "simdjson/dom/fractured_json.h"
#include "simdjson/internal/json_structure_analyzer.h"
namespace simdjson {
namespace internal {
/**
* Fractured JSON formatter using CRTP pattern.
*
* This formatter intelligently chooses between different layout modes
* (inline, compact multiline, table, expanded) based on pre-computed
* structure metrics.
*/
class fractured_formatter : public base_formatter<fractured_formatter> {
public:
explicit fractured_formatter(const fractured_json_options& opts = {});
/** CRTP hook: print newline (context-aware) */
simdjson_inline void print_newline();
/** CRTP hook: print indentation */
simdjson_inline void print_indents(size_t depth);
/** CRTP hook: print space (context-aware) */
simdjson_inline void print_space();
/** Set the current layout mode */
void set_layout_mode(layout_mode mode);
/** Get the current layout mode */
layout_mode get_layout_mode() const;
/** Set current depth for formatting decisions */
void set_depth(size_t depth);
/** Get current depth */
size_t get_depth() const;
/** Track current line length for compact multiline decisions */
void track_line_length(size_t chars);
/** Reset line length (after newline) */
void reset_line_length();
/** Get current line length */
size_t get_line_length() const;
/** Check if we should break to a new line in compact mode */
bool should_break_line(size_t upcoming_length) const;
/** Get the options */
const fractured_json_options& options() const;
// Table formatting support
/** Begin a table row */
void begin_table_row();
/** End a table row */
void end_table_row();
/** Set column widths for table alignment */
void set_column_widths(const std::vector<size_t>& widths);
/** Get current column index in table mode */
size_t get_column_index() const;
/** Advance to next column */
void next_column();
/** Add padding to align with column width */
void align_to_column_width(size_t actual_width);
private:
fractured_json_options options_;
layout_mode current_layout_ = layout_mode::EXPANDED;
size_t current_depth_ = 0;
size_t current_line_length_ = 0;
// Table state
bool in_table_mode_ = false;
std::vector<size_t> column_widths_;
size_t current_column_ = 0;
};
/**
* Specialized string builder for fractured JSON formatting.
*
* This builder performs two passes:
* 1. Analyze the structure to compute metrics
* 2. Format using the metrics to make layout decisions
*/
class fractured_string_builder {
public:
fractured_string_builder(const fractured_json_options& opts = {});
/** Append a DOM element with fractured formatting */
void append(const dom::element& value);
/** Append a DOM array with fractured formatting */
void append(const dom::array& value);
/** Append a DOM object with fractured formatting */
void append(const dom::object& value);
/** Clear the builder */
simdjson_inline void clear();
/** Get the formatted string */
simdjson_inline std::string_view str() const;
private:
fractured_formatter format_;
structure_analyzer analyzer_;
fractured_json_options options_;
/** Format an element using pre-computed metrics */
void format_element(const dom::element& elem, const element_metrics& metrics, size_t depth);
/** Format an array with the appropriate layout */
void format_array(const dom::array& arr, const element_metrics& metrics, size_t depth);
/** Format an array inline: [1, 2, 3] */
void format_array_inline(const dom::array& arr, const element_metrics& metrics);
/** Format an array with compact multiline: multiple items per line */
void format_array_compact_multiline(const dom::array& arr, const element_metrics& metrics, size_t depth);
/** Format an array as a table */
void format_array_as_table(const dom::array& arr, const element_metrics& metrics, size_t depth);
/** Format an array expanded: one item per line */
void format_array_expanded(const dom::array& arr, const element_metrics& metrics, size_t depth);
/** Format an object with the appropriate layout */
void format_object(const dom::object& obj, const element_metrics& metrics, size_t depth);
/** Format an object inline: {"a": 1, "b": 2} */
void format_object_inline(const dom::object& obj, const element_metrics& metrics);
/** Format an object expanded: one key per line */
void format_object_expanded(const dom::object& obj, const element_metrics& metrics, size_t depth);
/** Format a scalar value */
void format_scalar(const dom::element& elem);
/** Calculate column widths for table formatting */
std::vector<size_t> calculate_column_widths(const dom::array& arr,
const std::vector<std::string>& columns) const;
/** Measure the actual formatted length of a value (for alignment) */
size_t measure_value_length(const dom::element& elem) const;
};
} // namespace internal
} // namespace simdjson
#endif // SIMDJSON_INTERNAL_FRACTURED_FORMATTER_H
@@ -0,0 +1,169 @@
#ifndef SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H
#define SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H
#include "simdjson/dom/base.h"
#include "simdjson/dom/element.h"
#include "simdjson/dom/array.h"
#include "simdjson/dom/object.h"
#include "simdjson/dom/fractured_json.h"
#include "simdjson/internal/tape_type.h"
#include <vector>
#include <string>
#include <string_view>
#include <set>
namespace simdjson {
namespace internal {
/**
* Layout mode for fractured JSON formatting.
*/
enum class layout_mode {
INLINE, // Single line: [1, 2, 3] or {"a": 1}
COMPACT_MULTILINE, // Multiple items per line with breaks
TABLE, // Tabular format for arrays of similar objects
EXPANDED // Traditional multi-line with indentation
};
/**
* Metrics computed for a JSON element during structure analysis.
* These metrics drive layout decisions and contain child metrics for recursive formatting.
*/
struct element_metrics {
/** Nesting depth score (0 = scalar, 1 = flat container, etc.) */
size_t complexity = 0;
/** Estimated character length if rendered inline (minified + spaces) */
size_t estimated_inline_len = 0;
/** Number of direct children (0 for scalars) */
size_t child_count = 0;
/** Pre-computed: can this element be rendered inline? */
bool can_inline = false;
/** Is this an array where all elements have similar structure? */
bool is_uniform_array = false;
/** For uniform arrays of objects: the common keys */
std::vector<std::string> common_keys{};
/** Recommended layout mode based on analysis */
layout_mode recommended_layout = layout_mode::EXPANDED;
/** Child metrics for arrays and objects (in order of iteration) */
std::vector<element_metrics> children{};
};
/**
* Analyzes JSON structure to compute metrics for formatting decisions.
*
* The analyzer performs a single pass over the DOM to compute:
* - Complexity (nesting depth)
* - Estimated inline length
* - Array uniformity for table detection
*
* Metrics are stored hierarchically with child metrics embedded in parent metrics,
* enabling efficient lookup during formatting without address-based caching.
*/
class structure_analyzer {
public:
/** Default constructor */
structure_analyzer() : current_opts_(nullptr) {}
/** Copy constructor - deleted since class has pointer member */
structure_analyzer(const structure_analyzer&) = delete;
/** Copy assignment - deleted since class has pointer member */
structure_analyzer& operator=(const structure_analyzer&) = delete;
/** Move constructor */
structure_analyzer(structure_analyzer&&) = default;
/** Move assignment */
structure_analyzer& operator=(structure_analyzer&&) = default;
/**
* Analyze a DOM element and compute metrics.
* @param elem The element to analyze
* @param opts Formatting options that affect metric computation
* @return Metrics for the root element (with child metrics embedded)
*/
element_metrics analyze(const dom::element& elem,
const fractured_json_options& opts);
/**
* Clear state.
*/
void clear();
/**
* Analyze an array element directly (for standalone array formatting).
* @param arr The array to analyze
* @param opts Formatting options
* @return Metrics for the array
*/
element_metrics analyze_array(const dom::array& arr,
const fractured_json_options& opts);
/**
* Analyze an object element directly (for standalone object formatting).
* @param obj The object to analyze
* @param opts Formatting options
* @return Metrics for the object
*/
element_metrics analyze_object(const dom::object& obj,
const fractured_json_options& opts);
private:
const fractured_json_options* current_opts_ = nullptr;
/** Recursive analysis implementation */
element_metrics analyze_element(const dom::element& elem, size_t depth);
/** Analyze scalar values (strings, numbers, booleans, null) */
element_metrics analyze_scalar(const dom::element& elem);
/** Analyze an array element */
element_metrics analyze_array(const dom::array& arr, size_t depth);
/** Analyze an object element */
element_metrics analyze_object(const dom::object& obj, size_t depth);
/** Estimate inline length for a string (including quotes and escaping) */
size_t estimate_string_length(std::string_view s) const;
/** Estimate inline length for a number */
size_t estimate_number_length(double d) const;
size_t estimate_number_length(int64_t i) const;
size_t estimate_number_length(uint64_t u) const;
/**
* Check if an array contains uniform objects suitable for table formatting.
* @param arr The array to check
* @param common_keys Output: keys common to all objects
* @return true if the array is suitable for table formatting
*/
bool check_array_uniformity(const dom::array& arr,
std::vector<std::string>& common_keys) const;
/**
* Compute similarity between two objects.
* @return Fraction of keys that are common (0.0 to 1.0)
*/
double compute_object_similarity(const dom::object& a,
const dom::object& b) const;
/**
* Decide the recommended layout mode based on metrics and options.
*/
layout_mode decide_layout(const element_metrics& metrics,
size_t depth,
size_t available_width) const;
};
} // namespace internal
} // namespace simdjson
#endif // SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H
+6 -2
View File
@@ -231,7 +231,11 @@ class SimdjsonFile:
@property
def is_amalgamator(self):
if self.implementation:
return self.root == 'src' or self.include_dir == 'simdjson' or self.filename == 'ondemand.h'or self.filename == 'builder.h' or self.filename == 'implementation.h'
return (self.root == 'src' or
self.include_dir == 'simdjson' or
self.filename == 'ondemand.h' or
self.filename == 'builder.h' or
self.filename == 'implementation.h')
else:
return self.filename == 'amalgamated.h'
@@ -319,7 +323,7 @@ class SimdjsonRepository:
result = None
for relative_root in self.relative_roots:
if os.path.exists(os.path.join(self.project_path, relative_root, filename)):
assert result is None, f"Error: File '{filename}' exists in both '{result}' and '{root}' directories. Files must be unique across roots to avoid ambiguity. Rename or move the duplicate file.{result}' and '{root}' directories. Files must be unique across roots to avoid ambiguity. Rename or move the duplicate file."
assert result is None, f"Error: File '{filename}' exists in both '{result}' and '{relative_root}' directories. Files must be unique across roots to avoid ambiguity. Rename or move the duplicate file."
result = relative_root
return result
+1
View File
@@ -12,6 +12,7 @@ add_cpp_test(unicode_tests LABELS dom acceptance per_implementation)
add_cpp_test(minify_tests LABELS other acceptance per_implementation)
add_cpp_test(padded_string_tests LABELS other acceptance )
add_cpp_test(prettify_tests LABELS other acceptance per_implementation)
add_cpp_test(fractured_json_tests LABELS other acceptance per_implementation)
if(WIN32 AND BUILD_SHARED_LIBS)
# Copy the simdjson dll into the tests directory
+1
View File
@@ -7,6 +7,7 @@ if(SIMDJSON_STATIC_REFLECTION)
add_cpp_test(static_reflection_comprehensive_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(static_reflection_edge_cases_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(static_reflection_enum_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(static_reflection_fractured_json_tests LABELS ondemand acceptance per_implementation)
endif(SIMDJSON_STATIC_REFLECTION)
# Copy the simdjson dll into the tests directory
if(MSVC AND BUILD_SHARED_LIBS)
@@ -0,0 +1,236 @@
#include "simdjson.h"
#include "test_builder.h"
#include <string>
#include <string_view>
#include <vector>
using namespace simdjson;
// Test structures for FracturedJson builder integration
struct SimpleUser {
int id;
std::string name;
bool active;
};
struct UserWithEmail {
int id;
std::string name;
std::string email;
bool active;
};
struct NestedData {
std::string title;
std::vector<SimpleUser> users;
int count;
};
struct TableTestData {
std::vector<SimpleUser> records;
};
namespace fractured_json_builder_tests {
// Test basic to_fractured_json_string functionality
bool basic_fractured_json_test() {
TEST_START();
#if SIMDJSON_STATIC_REFLECTION
SimpleUser user{1, "Alice", true};
std::string formatted;
ASSERT_SUCCESS(simdjson::to_fractured_json_string(user).get(formatted));
// Verify it produces valid JSON by re-parsing
simdjson::dom::parser parser;
simdjson::dom::element doc;
ASSERT_SUCCESS(parser.parse(formatted).get(doc));
// Verify values are preserved
int64_t id;
ASSERT_SUCCESS(doc["id"].get_int64().get(id));
ASSERT_EQUAL(id, 1);
std::string_view name;
ASSERT_SUCCESS(doc["name"].get_string().get(name));
ASSERT_EQUAL(name, "Alice");
bool active;
ASSERT_SUCCESS(doc["active"].get_bool().get(active));
ASSERT_EQUAL(active, true);
// Output should be inline since it's simple
std::cout << "Basic FracturedJson output: " << formatted << std::endl;
#endif
TEST_SUCCEED();
}
// Test with custom formatting options
bool custom_options_test() {
TEST_START();
#if SIMDJSON_STATIC_REFLECTION
NestedData data{"Test", {{1, "Alice", true}, {2, "Bob", false}}, 2};
fractured_json_options opts;
opts.indent_spaces = 2;
opts.simple_bracket_padding = false;
std::string formatted;
ASSERT_SUCCESS(simdjson::to_fractured_json_string(data, opts).get(formatted));
// Verify it produces valid JSON
simdjson::dom::parser parser;
simdjson::dom::element doc;
ASSERT_SUCCESS(parser.parse(formatted).get(doc));
std::cout << "Custom options output:\n" << formatted << std::endl;
#endif
TEST_SUCCEED();
}
// Test to_fractured_json with output parameter
bool output_param_test() {
TEST_START();
#if SIMDJSON_STATIC_REFLECTION
SimpleUser user{42, "Charlie", false};
std::string output;
ASSERT_SUCCESS(simdjson::to_fractured_json_string(user).get(output));
// Verify it produces valid JSON
simdjson::dom::parser parser;
simdjson::dom::element doc;
ASSERT_SUCCESS(parser.parse(output).get(doc));
std::cout << "Output param result: " << output << std::endl;
#endif
TEST_SUCCEED();
}
// Test extract_fractured_json for partial serialization
bool extract_fields_test() {
TEST_START();
#if SIMDJSON_STATIC_REFLECTION
UserWithEmail user{1, "Alice", "alice@example.com", true};
// Extract only id and name fields
std::string formatted;
ASSERT_SUCCESS(simdjson::extract_fractured_json<"id", "name">(user).get(formatted));
// Verify it produces valid JSON
simdjson::dom::parser parser;
simdjson::dom::element doc;
ASSERT_SUCCESS(parser.parse(formatted).get(doc));
// Should have id and name but NOT email and active
int64_t id;
ASSERT_SUCCESS(doc["id"].get_int64().get(id));
ASSERT_EQUAL(id, 1);
std::string_view name;
ASSERT_SUCCESS(doc["name"].get_string().get(name));
ASSERT_EQUAL(name, "Alice");
// email should not be present
simdjson::dom::element email_elem;
auto err = doc["email"].get(email_elem);
ASSERT_TRUE(err != SUCCESS); // Should fail - field not present
std::cout << "Extract fields output: " << formatted << std::endl;
#endif
TEST_SUCCEED();
}
// Test table formatting with array of similar objects
bool table_format_test() {
TEST_START();
#if SIMDJSON_STATIC_REFLECTION
TableTestData data{
{{1, "Alice", true}, {2, "Bob", false}, {3, "Carol", true}, {4, "Dave", false}}
};
fractured_json_options opts;
opts.enable_table_format = true;
opts.min_table_rows = 3;
std::string formatted;
ASSERT_SUCCESS(simdjson::to_fractured_json_string(data, opts).get(formatted));
// Verify it produces valid JSON
simdjson::dom::parser parser;
simdjson::dom::element doc;
ASSERT_SUCCESS(parser.parse(formatted).get(doc));
std::cout << "Table format output:\n" << formatted << std::endl;
#endif
TEST_SUCCEED();
}
// Test roundtrip: serialize with reflection, parse back
bool roundtrip_test() {
TEST_START();
#if SIMDJSON_STATIC_REFLECTION
SimpleUser original{123, "TestUser", true};
// Serialize with FracturedJson
std::string formatted;
ASSERT_SUCCESS(simdjson::to_fractured_json_string(original).get(formatted));
// Parse back using ondemand
simdjson::ondemand::parser parser;
simdjson::ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(simdjson::pad(formatted)).get(doc));
// Deserialize back to struct
SimpleUser parsed;
ASSERT_SUCCESS(doc.get<SimpleUser>().get(parsed));
// Verify values match
ASSERT_EQUAL(parsed.id, original.id);
ASSERT_EQUAL(parsed.name, original.name);
ASSERT_EQUAL(parsed.active, original.active);
std::cout << "Roundtrip test passed with: " << formatted << std::endl;
#endif
TEST_SUCCEED();
}
// Test global namespace functions
bool global_namespace_test() {
TEST_START();
#if SIMDJSON_STATIC_REFLECTION
SimpleUser user{99, "GlobalTest", false};
// Test global to_fractured_json_string
std::string formatted;
ASSERT_SUCCESS(simdjson::to_fractured_json_string(user).get(formatted));
// Verify output
simdjson::dom::parser parser;
simdjson::dom::element doc;
ASSERT_SUCCESS(parser.parse(simdjson::pad(formatted)).get(doc));
std::cout << "Global namespace output: " << formatted << std::endl;
#endif
TEST_SUCCEED();
}
bool run() {
return basic_fractured_json_test() &&
custom_options_test() &&
output_param_test() &&
extract_fields_test() &&
table_format_test() &&
roundtrip_test() &&
global_namespace_test();
}
} // namespace fractured_json_builder_tests
int main(int argc, char *argv[]) {
std::cout << "Running FracturedJson builder integration tests..." << std::endl;
#if SIMDJSON_STATIC_REFLECTION
std::cout << "Static reflection is ENABLED" << std::endl;
#else
std::cout << "Static reflection is DISABLED - tests will be skipped" << std::endl;
#endif
return test_main(argc, argv, fractured_json_builder_tests::run);
}
+8
View File
@@ -462,6 +462,11 @@ namespace parse_api_tests {
const padded_string BASIC_JSON = "[1,2,3]"_padded;
const padded_string BASIC_NDJSON = "[1,2,3]\n[4,5,6]"_padded;
const padded_string EMPTY_NDJSON = ""_padded;
// GCC 15 gives false positive -Wfree-nonheap-object warning here
#if defined(__GNUC__) && !defined(__clang__)
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wfree-nonheap-object"
#endif
bool parser_moving_parser() {
std::cout << "Running " << __func__ << std::endl;
typedef std::tuple<std::string, std::unique_ptr<parser>,element> simdjson_tuple;
@@ -479,6 +484,9 @@ namespace parse_api_tests {
}
return true;
}
#if defined(__GNUC__) && !defined(__clang__)
#pragma GCC diagnostic pop
#endif
#if SIMDJSON_EXCEPTIONS
bool issue_2375() {
File diff suppressed because it is too large Load Diff