Files
simdjson-simdjson/include/simdjson/internal/json_structure_analyzer.h
Francisco Geiman Thiesen fc57c09cf0 Add FracturedJson formatting support for DOM serialization (#2580)
* Add FracturedJson formatting support for DOM serialization

Implements FracturedJson formatting as requested in issue #2576.
FracturedJson produces human-readable yet compact JSON output by
intelligently choosing between different layout strategies based on
content complexity, length, and structure similarity.

Key features:
- Four layout modes: inline, compact multiline, table, and expanded
- Structure analysis pass to compute metrics before formatting
- Table formatting for arrays of similar objects with column alignment
- Configurable options for line length, indentation, padding, etc.

New files:
- fractured_json.h: Public API with fractured_json_options struct
- fractured_json-inl.h: Implementation (~1000 lines)
- json_structure_analyzer.h: Structure analysis for layout decisions
- fractured_formatter.h: Formatter class using CRTP pattern

Usage:
  dom::parser parser;
  element doc = parser.parse(json_string);
  std::cout << fractured_json(doc) << std::endl;

  // Or with custom options:
  fractured_json_options opts;
  opts.indent_spaces = 2;
  std::cout << fractured_json(doc, opts) << std::endl;

  // Or format any JSON string (useful with reflection API):
  auto formatted = fractured_json_string(minified_json);

Resolves #2576

* Add comprehensive tests for FracturedJson formatter

Adds 27 test cases covering all aspects of the FracturedJson formatter:

Core functionality tests (13):
- Roundtrip parsing verification
- Inline formatting for simple arrays and objects
- Expanded formatting for complex nested structures
- Compact multiline arrays with configurable items per line
- Table formatting for uniform arrays of objects
- Empty container handling
- All scalar types (string, int, uint, double, bool, null)
- String escaping (quotes, backslashes, control characters)
- Custom indentation options
- Deep nesting (10+ levels)
- Mixed type arrays

Edge case tests (11):
- Unicode strings (Chinese, emoji, Arabic, Russian, accented chars)
- Boundary numbers (INT64_MIN/MAX, UINT64_MAX, DBL_MIN/MAX)
- Nested arrays (arrays of arrays)
- Empty string values
- Keys with special characters (spaces, quotes, colons, etc.)
- Non-uniform arrays (should not trigger table mode)
- Very long strings (500+ chars)
- Large arrays (100 elements)
- Reflection API workflow simulation
- Control characters (tab, newline, CR, null)
- Single element containers

Option tests (3):
- Disable compact multiline mode
- Disable table format mode
- Disable all padding options

* Add FracturedJson integration with builder/reflection API

Extends FracturedJson to work seamlessly with the builder API, enabling
formatted output directly from C++ structs using static reflection.

New functions:
- to_fractured_json_string(obj, opts) - serialize struct to formatted JSON
- to_fractured_json(obj, output, opts) - same with output parameter
- extract_fractured_json<fields...>(obj, opts) - format only specific fields

These functions combine the builder's reflection-based serialization with
FracturedJson formatting in a single convenient call:

  struct User { int id; std::string name; bool active; };
  User user{1, "Alice", true};

  // Minified output (existing):
  auto minified = to_json_string(user);
  // {"id":1,"name":"Alice","active":true}

  // Formatted output (new):
  auto formatted = to_fractured_json_string(user);
  // { "id": 1, "name": "Alice", "active": true }

  // Partial extraction with formatting:
  auto partial = extract_fractured_json<"id", "name">(user);
  // { "id": 1, "name": "Alice" }

New files:
- generic/builder/fractured_json_builder.h - builder integration
- tests/builder/static_reflection_fractured_json_tests.cpp - 7 tests

* Fix INT64_MIN overflow and implement table_similarity_threshold

- Fix undefined behavior when negating INT64_MIN in estimate_number_length()
  and measure_value_length() by returning 20 (the exact length of the
  string representation) directly
- Actually use table_similarity_threshold in check_array_uniformity() by
  calling compute_object_similarity() to compare objects against the first
  object in the array

* Fix -Werror=effc++ member initialization warnings

Initialize all member variables in member initialization lists to
satisfy GCC's -Werror=effc++ flag:
- element_metrics::common_keys - add {} default initializer
- structure_analyzer - add default constructor with member init list
- fractured_formatter - add column_widths_{} to constructor
- fractured_string_builder - add analyzer_{} to constructor

* Add Rule of Five to structure_analyzer class

The class has a pointer member (current_opts_) which triggers
-Werror=effc++ requiring explicit copy/move operations. Delete
copy operations (class shouldn't be copied due to cache) and
default move operations.

* Fix Windows build: wrap std::max to avoid macro conflict

Windows.h defines max/min macros that interfere with std::max/std::min.
Wrapping in parentheses as (std::max)(...) prevents macro expansion.

* Fix GCC 15 false positive -Wfree-nonheap-object warning

GCC 15 on MINGW64 gives a false positive warning in parser_moving_parser()
when the std::vector<std::string> goes out of scope. Suppress this
specific warning with a pragma for GCC builds.

* Fix metrics cache key bug by passing metrics through recursion

The cache was using element addresses as keys, but dom::element objects
are lightweight wrappers that get copied during iteration, causing
different addresses between analysis and formatting phases. This resulted
in cache misses and fallback to empty metrics.

Solution: Store child metrics in the element_metrics struct and pass
them through recursive calls, eliminating the need for address-based
caching entirely.

Changes:
- Add children vector to element_metrics for hierarchical metrics
- Remove metrics_cache_ and related get_metrics/has_metrics methods
- Update all format functions to accept and pass child metrics
- Add public analyze_array/analyze_object overloads for standalone use

* Add ignore patterns for Node.js, Rust, and generated files

Add entries for node_modules, package-lock.json, Rust target
directories, local ablation artifacts, and generated documentation
files.

* Refactor: extract analyze_scalar helper to reduce code duplication

Extract common scalar type handling (STRING, INT64, UINT64, DOUBLE,
BOOL, NULL_VALUE) into a dedicated analyze_scalar method. Each scalar
type shares the same initialization pattern for complexity, child_count,
can_inline, and recommended_layout.

Also simplify boolean formatting in format_scalar to use ternary operator.

* Fix formatting and duplicate error message in amalgamate.py

Reformat cramped is_amalgamator condition to multi-line for readability.
Fix duplicate error message text in _included_filename_root and use
correct variable name (relative_root instead of root).

* Refactor: add count_newlines helper in fractured_json tests

Extract repeated newline counting loop into a reusable static helper
function, used by inline_array_test, inline_object_test, and
expanded_test.

* Revert "Add ignore patterns for Node.js, Rust, and generated files"

This reverts commit 4760ea7cd0.

* various minor changes

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>
2026-01-20 10:32:24 -05:00

170 lines
5.5 KiB
C++

#ifndef SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H
#define SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H
#include "simdjson/dom/base.h"
#include "simdjson/dom/element.h"
#include "simdjson/dom/array.h"
#include "simdjson/dom/object.h"
#include "simdjson/dom/fractured_json.h"
#include "simdjson/internal/tape_type.h"
#include <vector>
#include <string>
#include <string_view>
#include <set>
namespace simdjson {
namespace internal {
/**
* Layout mode for fractured JSON formatting.
*/
enum class layout_mode {
INLINE, // Single line: [1, 2, 3] or {"a": 1}
COMPACT_MULTILINE, // Multiple items per line with breaks
TABLE, // Tabular format for arrays of similar objects
EXPANDED // Traditional multi-line with indentation
};
/**
* Metrics computed for a JSON element during structure analysis.
* These metrics drive layout decisions and contain child metrics for recursive formatting.
*/
struct element_metrics {
/** Nesting depth score (0 = scalar, 1 = flat container, etc.) */
size_t complexity = 0;
/** Estimated character length if rendered inline (minified + spaces) */
size_t estimated_inline_len = 0;
/** Number of direct children (0 for scalars) */
size_t child_count = 0;
/** Pre-computed: can this element be rendered inline? */
bool can_inline = false;
/** Is this an array where all elements have similar structure? */
bool is_uniform_array = false;
/** For uniform arrays of objects: the common keys */
std::vector<std::string> common_keys{};
/** Recommended layout mode based on analysis */
layout_mode recommended_layout = layout_mode::EXPANDED;
/** Child metrics for arrays and objects (in order of iteration) */
std::vector<element_metrics> children{};
};
/**
* Analyzes JSON structure to compute metrics for formatting decisions.
*
* The analyzer performs a single pass over the DOM to compute:
* - Complexity (nesting depth)
* - Estimated inline length
* - Array uniformity for table detection
*
* Metrics are stored hierarchically with child metrics embedded in parent metrics,
* enabling efficient lookup during formatting without address-based caching.
*/
class structure_analyzer {
public:
/** Default constructor */
structure_analyzer() : current_opts_(nullptr) {}
/** Copy constructor - deleted since class has pointer member */
structure_analyzer(const structure_analyzer&) = delete;
/** Copy assignment - deleted since class has pointer member */
structure_analyzer& operator=(const structure_analyzer&) = delete;
/** Move constructor */
structure_analyzer(structure_analyzer&&) = default;
/** Move assignment */
structure_analyzer& operator=(structure_analyzer&&) = default;
/**
* Analyze a DOM element and compute metrics.
* @param elem The element to analyze
* @param opts Formatting options that affect metric computation
* @return Metrics for the root element (with child metrics embedded)
*/
element_metrics analyze(const dom::element& elem,
const fractured_json_options& opts);
/**
* Clear state.
*/
void clear();
/**
* Analyze an array element directly (for standalone array formatting).
* @param arr The array to analyze
* @param opts Formatting options
* @return Metrics for the array
*/
element_metrics analyze_array(const dom::array& arr,
const fractured_json_options& opts);
/**
* Analyze an object element directly (for standalone object formatting).
* @param obj The object to analyze
* @param opts Formatting options
* @return Metrics for the object
*/
element_metrics analyze_object(const dom::object& obj,
const fractured_json_options& opts);
private:
const fractured_json_options* current_opts_ = nullptr;
/** Recursive analysis implementation */
element_metrics analyze_element(const dom::element& elem, size_t depth);
/** Analyze scalar values (strings, numbers, booleans, null) */
element_metrics analyze_scalar(const dom::element& elem);
/** Analyze an array element */
element_metrics analyze_array(const dom::array& arr, size_t depth);
/** Analyze an object element */
element_metrics analyze_object(const dom::object& obj, size_t depth);
/** Estimate inline length for a string (including quotes and escaping) */
size_t estimate_string_length(std::string_view s) const;
/** Estimate inline length for a number */
size_t estimate_number_length(double d) const;
size_t estimate_number_length(int64_t i) const;
size_t estimate_number_length(uint64_t u) const;
/**
* Check if an array contains uniform objects suitable for table formatting.
* @param arr The array to check
* @param common_keys Output: keys common to all objects
* @return true if the array is suitable for table formatting
*/
bool check_array_uniformity(const dom::array& arr,
std::vector<std::string>& common_keys) const;
/**
* Compute similarity between two objects.
* @return Fraction of keys that are common (0.0 to 1.0)
*/
double compute_object_similarity(const dom::object& a,
const dom::object& b) const;
/**
* Decide the recommended layout mode based on metrics and options.
*/
layout_mode decide_layout(const element_metrics& metrics,
size_t depth,
size_t available_width) const;
};
} // namespace internal
} // namespace simdjson
#endif // SIMDJSON_INTERNAL_JSON_STRUCTURE_ANALYZER_H