Compare commits

..

523 Commits

Author SHA1 Message Date
Francisco Geiman Thiesen 920573fffd Optimize reflection-based deserialization with hybrid algorithm
- Add threshold-based algorithm selection (N<=8: per-field, N>=9: single-pass)
- Per-field lookup leverages simdjson's SIMD-optimized find_field_unordered
- Single-pass uses bitmask tracking, first-char filter, and early termination
- Add fields_in_order<T> trait for user-defined ordered field optimization

Performance improvement on benchmarks:
- Twitter: +5.3% (4167 -> 4389 MB/s)
- CITM: +14.6% (2600 -> 2994 MB/s)

Threshold=8 is based on crossover analysis: at N=9 fields, per-field lookup
causes ~35% regression due to quadratic scan cost vs single-pass linear cost.
2026-02-19 09:04:05 -08:00
Francisco Geiman Thiesen af2a43610e perf: SIMD string escaping and batch integer formatting optimizations (#2605)
* perf(serialization): SIMD-accelerated string escape position finding

Profiling revealed that string serialization was performing redundant
scanning: first calling fast_needs_escaping() (SIMD scan to check IF
escape needed), then find_next_json_quotable_character() (scalar
byte-by-byte scan to find WHERE).

Profile data from Twitter benchmark showed:
- 54.76% time in atom<std::string> (string serialization)
- 24.85% time in find_next_json_quotable_character (scalar position finding)

This optimization unifies both operations into a single SIMD pass that
directly locates the first quotable character position:

- NEON (ARM64): Uses vceqq_u8/vcltq_u8 for character detection, then
  extracts position via __builtin_ctzll on 64-bit vector lanes
- SSE2 (x86-64): Uses _mm_cmpeq_epi8/_mm_subs_epu8 for detection, then
  _mm_movemask_epi8 + __builtin_ctz for position extraction

The write_string_escaped function now uses the position finder directly,
eliminating the separate fast_needs_escaping check.

Benchmark results (ARM64, Apple Silicon via Docker with p2996 clang):
- Twitter (string-heavy): 4330 -> 5723 MB/s (+32%)
- CITM (numeric-heavy): ~neutral (expected, few strings)

* perf(serialization): batch integer formatting with 4-digit processing

Profiling with perf annotate revealed that the integer-to-string
conversion loop was a significant hotspot in numeric-heavy workloads.
The original implementation processed 2 digits per iteration:

    while (pv >= 100) {
        memcpy(write_pointer - 1, &decimal_table[(pv % 100) * 2], 2);
        write_pointer -= 2;
        pv /= 100;
    }

Profile data from CITM benchmark showed:
- 31.20% time in atom<unsigned long> (integer formatting)
- 39.07% of integer formatting time in the 2-byte store instruction
  (sturh on ARM64)
- CITM integers average 8.8 digits, meaning 4+ store operations per number

This optimization processes 4 digits per iteration, reducing both store
operations and division count by approximately half for large numbers:

    while (pv >= 10000) {
        q = pv / 10000;
        r = pv % 10000;
        r_hi = r / 100;  // High 2 digits
        r_lo = r % 100;  // Low 2 digits
        memcpy(write_pointer - 1, &decimal_table[r_lo * 2], 2);
        memcpy(write_pointer - 3, &decimal_table[r_hi * 2], 2);
        write_pointer -= 4;
        pv = q;
    }

The division by 10000 compiles to an efficient multiply-high instruction
(umulh on ARM64). Applied to both unsigned and signed integer paths.

Benchmark results (ARM64, Apple Silicon via Docker with p2996 clang):
- Twitter: ~neutral (few integers)
- CITM (numeric-heavy): 2912 -> 3086 MB/s (+6%)

* fix(build): add MSVC compatibility for bit manipulation intrinsics

MSVC does not have __builtin_ctz/__builtin_ctzll. Use _BitScanForward
and _BitScanForward64 from <intrin.h> on MSVC instead.

This fixes the build on all Windows configurations (x64, ARM64, Win32).

* refactor: clean up comments to be implementation-focused

Remove references to specific benchmarks and previous implementations
from code comments. Comments now describe what the code does rather
than historical context.
2026-02-16 11:50:27 -05:00
Daniel Lemire e362b18499 saving. 2026-02-09 00:31:31 -05:00
Daniel Lemire 6e60ed8338 updating rust 2026-02-08 23:42:43 -05:00
Daniel Lemire e2ca00c841 reenabling bench_nlohmann_parsing in twitter bench 2026-02-08 22:36:34 -05:00
Daniel Lemire b7124f7c64 inlining get_single_implementation (#2600) 2026-02-06 20:34:00 -05:00
Daniel Lemire e6c6db6493 Update field access section in basics.md
Clarified field access behavior and error handling in documentation.
2026-02-03 14:04:26 -05:00
Daniel Lemire 6fffc99dac Add Ladybird Browser to the list of projects 2026-02-02 21:08:31 -05:00
Daniel Lemire 8c5cc8c443 updating the reflection benchmarks (#2598)
* updating the reflection benchmarks

* removing exception during parsing.

* saving.
2026-02-02 11:28:08 -05:00
Daniel Lemire d8c49a8f25 fix to rust instructions 2026-01-27 20:08:06 -05:00
Daniel Lemire cda98064b8 faster amalgamation script (#2597) 2026-01-27 11:35:58 -05:00
Daniel Lemire e532d61e16 when calling get_string with a mutable string parameter, we want to only use the string buffer (#2595)
as scratch space
2026-01-22 10:23:39 -05:00
Olaf Bernstein db93de2a21 add rvv-vls backend (#2593) 2026-01-21 10:13:50 -05:00
Francisco Geiman Thiesen fc57c09cf0 Add FracturedJson formatting support for DOM serialization (#2580)
* Add FracturedJson formatting support for DOM serialization

Implements FracturedJson formatting as requested in issue #2576.
FracturedJson produces human-readable yet compact JSON output by
intelligently choosing between different layout strategies based on
content complexity, length, and structure similarity.

Key features:
- Four layout modes: inline, compact multiline, table, and expanded
- Structure analysis pass to compute metrics before formatting
- Table formatting for arrays of similar objects with column alignment
- Configurable options for line length, indentation, padding, etc.

New files:
- fractured_json.h: Public API with fractured_json_options struct
- fractured_json-inl.h: Implementation (~1000 lines)
- json_structure_analyzer.h: Structure analysis for layout decisions
- fractured_formatter.h: Formatter class using CRTP pattern

Usage:
  dom::parser parser;
  element doc = parser.parse(json_string);
  std::cout << fractured_json(doc) << std::endl;

  // Or with custom options:
  fractured_json_options opts;
  opts.indent_spaces = 2;
  std::cout << fractured_json(doc, opts) << std::endl;

  // Or format any JSON string (useful with reflection API):
  auto formatted = fractured_json_string(minified_json);

Resolves #2576

* Add comprehensive tests for FracturedJson formatter

Adds 27 test cases covering all aspects of the FracturedJson formatter:

Core functionality tests (13):
- Roundtrip parsing verification
- Inline formatting for simple arrays and objects
- Expanded formatting for complex nested structures
- Compact multiline arrays with configurable items per line
- Table formatting for uniform arrays of objects
- Empty container handling
- All scalar types (string, int, uint, double, bool, null)
- String escaping (quotes, backslashes, control characters)
- Custom indentation options
- Deep nesting (10+ levels)
- Mixed type arrays

Edge case tests (11):
- Unicode strings (Chinese, emoji, Arabic, Russian, accented chars)
- Boundary numbers (INT64_MIN/MAX, UINT64_MAX, DBL_MIN/MAX)
- Nested arrays (arrays of arrays)
- Empty string values
- Keys with special characters (spaces, quotes, colons, etc.)
- Non-uniform arrays (should not trigger table mode)
- Very long strings (500+ chars)
- Large arrays (100 elements)
- Reflection API workflow simulation
- Control characters (tab, newline, CR, null)
- Single element containers

Option tests (3):
- Disable compact multiline mode
- Disable table format mode
- Disable all padding options

* Add FracturedJson integration with builder/reflection API

Extends FracturedJson to work seamlessly with the builder API, enabling
formatted output directly from C++ structs using static reflection.

New functions:
- to_fractured_json_string(obj, opts) - serialize struct to formatted JSON
- to_fractured_json(obj, output, opts) - same with output parameter
- extract_fractured_json<fields...>(obj, opts) - format only specific fields

These functions combine the builder's reflection-based serialization with
FracturedJson formatting in a single convenient call:

  struct User { int id; std::string name; bool active; };
  User user{1, "Alice", true};

  // Minified output (existing):
  auto minified = to_json_string(user);
  // {"id":1,"name":"Alice","active":true}

  // Formatted output (new):
  auto formatted = to_fractured_json_string(user);
  // { "id": 1, "name": "Alice", "active": true }

  // Partial extraction with formatting:
  auto partial = extract_fractured_json<"id", "name">(user);
  // { "id": 1, "name": "Alice" }

New files:
- generic/builder/fractured_json_builder.h - builder integration
- tests/builder/static_reflection_fractured_json_tests.cpp - 7 tests

* Fix INT64_MIN overflow and implement table_similarity_threshold

- Fix undefined behavior when negating INT64_MIN in estimate_number_length()
  and measure_value_length() by returning 20 (the exact length of the
  string representation) directly
- Actually use table_similarity_threshold in check_array_uniformity() by
  calling compute_object_similarity() to compare objects against the first
  object in the array

* Fix -Werror=effc++ member initialization warnings

Initialize all member variables in member initialization lists to
satisfy GCC's -Werror=effc++ flag:
- element_metrics::common_keys - add {} default initializer
- structure_analyzer - add default constructor with member init list
- fractured_formatter - add column_widths_{} to constructor
- fractured_string_builder - add analyzer_{} to constructor

* Add Rule of Five to structure_analyzer class

The class has a pointer member (current_opts_) which triggers
-Werror=effc++ requiring explicit copy/move operations. Delete
copy operations (class shouldn't be copied due to cache) and
default move operations.

* Fix Windows build: wrap std::max to avoid macro conflict

Windows.h defines max/min macros that interfere with std::max/std::min.
Wrapping in parentheses as (std::max)(...) prevents macro expansion.

* Fix GCC 15 false positive -Wfree-nonheap-object warning

GCC 15 on MINGW64 gives a false positive warning in parser_moving_parser()
when the std::vector<std::string> goes out of scope. Suppress this
specific warning with a pragma for GCC builds.

* Fix metrics cache key bug by passing metrics through recursion

The cache was using element addresses as keys, but dom::element objects
are lightweight wrappers that get copied during iteration, causing
different addresses between analysis and formatting phases. This resulted
in cache misses and fallback to empty metrics.

Solution: Store child metrics in the element_metrics struct and pass
them through recursive calls, eliminating the need for address-based
caching entirely.

Changes:
- Add children vector to element_metrics for hierarchical metrics
- Remove metrics_cache_ and related get_metrics/has_metrics methods
- Update all format functions to accept and pass child metrics
- Add public analyze_array/analyze_object overloads for standalone use

* Add ignore patterns for Node.js, Rust, and generated files

Add entries for node_modules, package-lock.json, Rust target
directories, local ablation artifacts, and generated documentation
files.

* Refactor: extract analyze_scalar helper to reduce code duplication

Extract common scalar type handling (STRING, INT64, UINT64, DOUBLE,
BOOL, NULL_VALUE) into a dedicated analyze_scalar method. Each scalar
type shares the same initialization pattern for complexity, child_count,
can_inline, and recommended_layout.

Also simplify boolean formatting in format_scalar to use ternary operator.

* Fix formatting and duplicate error message in amalgamate.py

Reformat cramped is_amalgamator condition to multi-line for readability.
Fix duplicate error message text in _included_filename_root and use
correct variable name (relative_root instead of root).

* Refactor: add count_newlines helper in fractured_json tests

Extract repeated newline counting loop into a reusable static helper
function, used by inline_array_test, inline_object_test, and
expanded_test.

* Revert "Add ignore patterns for Node.js, Rust, and generated files"

This reverts commit 4760ea7cd0.

* various minor changes

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>
2026-01-20 10:32:24 -05:00
Daniel Lemire 2058b47dfe adding padded string builder (#2592)
* adding padded string builder

* minor rename

* deleting copy constructor

* typo

* [no-ci] tuning documentation.
2026-01-18 20:18:19 -05:00
Daniel Lemire b0486c7fa2 saving. 2026-01-18 11:57:55 -05:00
Daniel Lemire feb1e7feb6 work on the ondemand iterators (#2590)
* work on the ondemand iterators

* guarding two SIMDJSON_ASSUME

* simplify following @jkeiser's comment

* adding safety rails to the iterators

* silencing a warning.

* updating the amalgamation files
2026-01-17 21:15:18 -05:00
Eve Silfanus 2c7fbc1538 Build Performance Optimization (#2588) 2026-01-16 11:42:39 -05:00
Daniel Lemire 8b69401d8a moving the builder files in their own directory (#2578) 2026-01-07 18:08:11 -05:00
Daniel Lemire f504e57e7a Add runtime dispatching for loongarch (#2575)
* loongarch runtime dispatching

* remove CMake config for Loongarch.

* make lasx available

* flipping

* moving...

* fixing minor logic error

* minor fixes

* flipping

* adding hackish header

* better comment and reordering

* adding dispatch
2026-01-02 14:28:23 -05:00
Arthur Chan 135c173053 oss-fuzz: Add unit testing build to oss-fuzz build script (#2574) 2026-01-02 14:28:01 -05:00
Daniel Lemire b4ed3a99a9 fixing typos (#2573) 2025-12-30 17:38:03 -05:00
Daniel Lemire d8e1b36c88 just new small tests. (#2571) 2025-12-23 11:14:09 -05:00
Daniel Lemire c249a1b456 Merge branch 'master' of github.com:simdjson/simdjson 2025-12-22 16:08:30 -05:00
Daniel Lemire 9c4b793c90 adding ref 2025-12-22 16:08:19 -05:00
Daniel Lemire dd92a8414c Add optimization option to pull request template 2025-12-19 23:30:13 -05:00
Dirk Stolle 835bdba123 adjust logo image file names (*_simdjason_* -> *_simdjson_*) (#2569) 2025-12-19 23:21:59 -05:00
Dirk Stolle ad3cd71ca2 fix a few typos (#2568) 2025-12-19 21:56:32 -05:00
Daniel Lemire 860f7e0458 Update logo in README.md 2025-12-17 20:36:08 -05:00
Daniel Lemire 980f2ad3af 4.2.4 2025-12-17 20:33:11 -05:00
Daniel Lemire 7ad9fe63a6 fixing issue 2549 (#2567)
* fixing issue 2549

* saving.
2025-12-17 20:32:36 -05:00
Daniel Lemire 7987418b1f adding the official simdjson logo files 2025-12-13 12:14:40 -05:00
Daniel Lemire 5e871f6724 improving slightly the documentation. 2025-12-12 19:04:34 -05:00
Daniel Lemire 5d16fd5f31 4.2.3 2025-12-12 17:51:39 -05:00
Jake S. Del Mastro 4e9ff03af5 Make it possible to provide custom serializers for range types ( (#2550)
If you provide a custom serializer for range types it is currently never used due to the requires clause for string_builder::append with ranges is overly broad
2025-12-12 17:50:52 -05:00
Daniel Lemire aa7489060a Fix typo in bug report template 2025-12-12 15:24:48 -05:00
Daniel Lemire ae32422891 a few additional tests and removing a bad remark in the documentation... 2025-12-03 19:35:18 -05:00
Liqiang TAO 667d0ed3c7 make code branchless (#2546) 2025-11-18 16:57:06 -05:00
Daniel Lemire b1c31b428d update 2025-11-11 14:21:04 -05:00
Daniel Lemire 56ac56ba32 Merge branch 'master' of github.com:simdjson/simdjson 2025-11-11 14:17:08 -05:00
Muhammad Rizal Nurromdhoni 19549c60ec string_builder range-based append fix (#2544)
* Use std::ranges::range_value_t on range

* Add ranges test
2025-11-11 14:15:00 -05:00
Daniel Lemire 16e99f229b Update iterate_many.md for clarity on JSON processing
Clarified the example JSON format and emphasized the need for efficient processing.
2025-11-10 13:49:13 -05:00
Liqiang TAO 21342a4142 Fix some wrong content in doc (#2542) 2025-11-10 11:24:38 -05:00
Daniel Lemire d0e841d3e9 Release Candidate 4.2.2 (#2539)
* adding documentation.

* release candidate
2025-11-06 12:00:21 -05:00
Daniel Lemire a962652ec3 adding documentation. 2025-11-05 11:42:38 -05:00
hiteshmk05 77d73b068a add: windows wstring support for padded_str (#2537)
* add: windows wstring support for padded_str

* add: padded_string::load for wstring windows

* fix: extra space

* change: file path
2025-11-05 11:29:30 -05:00
Daniel Lemire 19ff7a572d adding concept examples to the compile-time JSON. (#2538) 2025-11-04 15:06:40 -05:00
Daniel Lemire 235dbc5369 4.2.1 2025-11-03 11:04:23 -05:00
Daniel Lemire 2b9c8977af using _json for compile-time JSON strings. (#2536) 2025-11-03 11:03:21 -05:00
Daniel Lemire 3d87bd4abc release 4.2.0 2025-11-02 16:19:39 -05:00
hiteshmk05 a60c0d1e39 fix: cmake error when CMAKE_CXX_FLAGS is empty (#2535) 2025-11-02 11:53:56 -05:00
Francisco Geiman Thiesen 86bfbaada7 Merge pull request #2534 from simdjson/francisco/compile-time-parsing_daniel
Compile-time parsing (C++26)
2025-11-01 22:19:37 -07:00
Daniel Lemire ccac6403d9 fixing support for pre-C++17 2025-11-01 17:28:17 -04:00
Daniel Lemire aade58c3dc tweaks 2025-11-01 17:08:03 -04:00
Daniel Lemire 3319815e25 bringing back compatibility with pre-C++17 2025-11-01 16:46:34 -04:00
hiteshmk05 a24f845bd7 Feature/ondemand wildcard support (#2533)
* Add feature for ondemand-wildcard-JSONQueries

* fix wildcard_test

* fix: extra whitespace
2025-11-01 16:46:08 -04:00
Daniel Lemire 212e2d5857 removing unnecessary changes 2025-10-31 18:56:23 -04:00
Daniel Lemire 547e156e33 update 2025-10-31 18:54:04 -04:00
Daniel Lemire 98a45f7229 fix 2025-10-31 18:48:49 -04:00
Daniel Lemire 8589509d1e Merge branch 'master' into francisco/compile-time-parsing_daniel 2025-10-31 18:47:42 -04:00
Daniel Lemire 2fce4a843d update 2025-10-31 18:46:13 -04:00
Daniel Lemire 62803512e4 saving. 2025-10-31 15:58:51 -04:00
Daniel Lemire 76d9dee854 update 2025-10-31 00:13:00 -04:00
Daniel Lemire bf15f21b0b not great but a start. 2025-10-29 20:10:06 -04:00
Daniel Lemire 32b301893c updating toc 2025-10-29 09:53:39 -04:00
Daniel Lemire 7f1531a1f9 removing garbage. 2025-10-29 09:53:13 -04:00
Daniel Lemire 0a3b555ff7 Merge branch 'master' of github.com:simdjson/simdjson 2025-10-29 09:50:15 -04:00
Daniel Lemire 114d45ad54 some garbage 2025-10-29 00:07:48 -04:00
Daniel Lemire 0112be86b0 4.1.0 (#2532) 2025-10-28 00:10:59 -04:00
Daniel Lemire bf52d8198b 4.1.0 2025-10-27 16:56:06 -04:00
Francisco Geiman Thiesen 58c92d6d82 Adding support for compiled json path + json pointer (reflection based) (#2483)
* Adding compile time json path

* using string_view

* Adding support for compile-time json pointer as well.

* Removing unnecessary comment

* Tests now working, still will re-review.

* Adding documentation on the compile-time json path/pointer parsing feature.

* Adding benchmark showing the significant performance advantage of using compiled paths whenever you have them a priori.

* going for JSONPath (correct wording).

* minor update (mostly doc)

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>
2025-10-27 16:52:41 -04:00
Daniel Lemire 781a7d6c89 removing an unnecessary branch (#2530)
* removing an unnecessary branch

* fixing typo
2025-10-27 14:19:58 -04:00
Max Marrone 3d0de709a8 Fix outdated references to JsonStream. (#2531) 2025-10-26 16:30:31 -04:00
kevyang 49b86721b4 add missing OUT_OF_CAPACITY error code to error codes array (#2527)
* add missing error code to DLLIMPORTEXPORT

* fix syntax

---------

Co-authored-by: Kevin Yang <kjy@meta.com>
2025-10-22 22:20:08 -04:00
Daniel Lemire def2b6efd2 still not good 2025-10-19 22:23:43 -04:00
Daniel Lemire 87a186fbf1 removing circleci 2025-10-19 20:03:52 -04:00
Daniel Lemire 81f10a01b7 documentation update 2025-10-17 21:00:52 -04:00
Daniel Lemire 36ed7ab48a saving 2025-10-17 20:59:37 -04:00
Daniel Lemire c3d1d62dfe use cpp 2025-10-17 20:45:57 -04:00
Daniel Lemire 67821cb6fd updating the documentation. 2025-10-17 20:33:42 -04:00
Daniel Lemire 3ac287ba3d update dox 2025-10-17 20:28:08 -04:00
Daniel Lemire ec352430a0 JSONPath is now an RFC (#2517)
* JSONPath is now an RFC

* up
2025-10-17 13:35:04 -04:00
Francisco Geiman Thiesen d326f2ce9f Working! 2025-10-10 21:32:10 -07:00
Francisco Geiman Thiesen ca42a49fba Compile-time support for parsing json objects! 2025-10-10 18:40:09 -07:00
Daniel Lemire 8a9daeb0ad Restore Star History Chart in README
Readded the Star History Chart section to the README.
2025-10-10 09:07:28 -04:00
Daniel Lemire 9c5a88f1f3 Update README with star history chart 2025-10-10 09:06:49 -04:00
0xflotus a7f8fb71c5 chore: fix small error in docs (#2497) 2025-10-03 11:03:17 -04:00
Howard Guo a553db4c67 Update workflow name to Ubuntu aarch64 (GCC 13) (#2484) 2025-10-03 09:56:07 -04:00
Jaël Champagne Gareau 1fa1af8c15 Fix yyjson leaks when running ./bench_ondemand (#2485) 2025-10-03 09:55:32 -04:00
Daniel Lemire 5ed1044056 4.0.7 2025-09-30 11:26:03 -04:00
Francisco Geiman Thiesen 62913867ff Merge pull request #2475 from simdjson/francisco/extract_from
Adding extract_from functionality + unit tests
2025-09-29 20:34:35 -07:00
Francisco Geiman Thiesen 6700d48b57 Merge pull request #2480 from simdjson/francisco/extract_from3
minor tweaks... ;-)
2025-09-29 13:54:05 -07:00
Francisco Geiman Thiesen b5577d5e85 Merge branch 'master' into francisco/extract_from 2025-09-29 12:46:48 -07:00
wszqkzqk b84a4ec2b9 Fix: Correct narrowing conversion in lsx string parsing (#2481)
Resolves a build failure on the loong64 architecture caused by a narrowing conversion error.

The compiler, with the -Werror=narrowing flag, was flagging the implicit conversion from 'int' (the return
type of to_bitmask()) to 'uint64_t'.

This is fixed by adding an explicit static_cast to uint64_t in include/simdjson/lsx/stringparsing_defs.h.

Signed-off-by: Zhou Qiankang <wszqkzqk@qq.com>
2025-09-29 11:57:06 -04:00
Daniel Lemire 1638a185f7 minor tweaks... ;-) 2025-09-29 11:12:37 -04:00
Francisco Geiman Thiesen 6fe450f5ce Updateing single_header 2025-09-29 03:44:29 -07:00
Francisco Geiman Thiesen c7b70de070 Merge branch 'master' into francisco/extract_from 2025-09-29 03:23:02 -07:00
Francisco Geiman Thiesen 3279fbd55b Merge pull request #2474 from simdjson/complete_extract_into
this completes the extract_into work.
2025-09-29 03:12:20 -07:00
Francisco Geiman Thiesen 66e64e0e5f Merge branch 'master' into francisco/extract_from 2025-09-29 03:02:12 -07:00
Daniel Lemire 03f81e66af updating single header 2025-09-27 12:26:04 -04:00
Daniel Lemire e3b7eddb37 fixing off-by-one mistake in the documentation (#2477) 2025-09-27 12:19:44 -04:00
Daniel Lemire 99c4ba6e8f tweak 2025-09-26 23:44:42 -04:00
Francisco Geiman Thiesen 6aa7eea334 Adding extract_from functionality + unit tests 2025-09-26 19:48:16 -07:00
Daniel Lemire 6a47cda07f guarding 2025-09-26 22:34:36 -04:00
Daniel Lemire 617c69e104 completing doc 2025-09-26 21:33:10 -04:00
Daniel Lemire 625adceb24 updating single-header 2025-09-26 21:29:28 -04:00
Daniel Lemire bd0e9c1336 tweak 2025-09-26 21:04:53 -04:00
Daniel Lemire 7bf82b02d5 this completes the extra_into work. 2025-09-26 21:00:46 -04:00
Francisco Geiman Thiesen 88a1b3e83b Merge pull request #2471 from simdjson/francisco/extract_into
Adding extract_into functionality + test (targets simdjson >= 4.0 as it relies on reflection)
2025-09-26 01:29:33 -07:00
Francisco Geiman Thiesen c72954eade Addressing reviews. 2025-09-25 21:08:38 -07:00
Francisco Geiman Thiesen 4456a10469 Francisco/using iterators for containers (#2470)
* Using iterators instead of subscript operators and size. This helps us work with a broader range of containers.

* Adding list test

* Using std::ranges::input_range<T> as suggested by moisrex
2025-09-25 17:30:42 -04:00
Francisco Geiman Thiesen d8ed2417ad Adding coverage for extract_into with types that have custom serialization 2025-09-25 03:35:46 -07:00
Francisco Geiman Thiesen 5517df7aee Adding extract_into functionality + test 2025-09-24 22:58:23 -07:00
evbse 6e618b0805 Improve DOM implementation (#2434) 2025-09-21 11:26:33 -06:00
Pavel Novikov bde288a623 Fixed string_builder::operator std::string() (#2465)
* clang format

* fixed `string_builder::operator std::string()`

* fixed variable shadowing error false positive
2025-09-21 11:25:42 -06:00
Daniel Lemire b2932d1b8f release candidate 4.0.6 (#2464) 2025-09-21 08:13:48 -06:00
Daniel Lemire a7811090ef fixing issue 2458 (#2461) 2025-09-20 22:23:09 -06:00
Daniel Lemire 3320885fac Fixing issue 2462 (#2463)
* fun

* progress

* completing the documentation
2025-09-20 22:22:57 -06:00
Daniel Lemire 786c68b158 release 4.0.5 2025-09-18 15:17:32 -06:00
Daniel Lemire 703ef54bd9 allow string reuse (#2454)
* allow string reuse

* portability fix

* using data and not begin
2025-09-18 15:16:38 -06:00
Daniel Lemire 2526068e2f Add mamba link to README 2025-09-18 09:22:03 -06:00
Daniel Lemire ddc7b8c7dd update 2025-09-17 18:59:06 -06:00
Daniel Lemire d8f90bdd14 modifying simdjson::from to avoid exceptions when needed. (#2452)
* modifying simdjson::from to avoid exceptions when needed.

* moved the function

* moving the strings.

* more moving around

* updating cmake version in ci
2025-09-17 18:58:25 -06:00
Daniel Lemire e38a4923e5 adding keys as templates in builder (#2453)
* adding keys as templates in builder

* more guarding

* cmake update in ci

* guarding.

* guarding
2025-09-17 18:58:05 -06:00
Dirk Stolle e6dfa2e0ed remove trailing whitespace + fix typos (#2451) 2025-09-16 22:41:14 -06:00
Daniel Lemire 1f369ef210 minor patch which allows us to pass mutable strings to simdjson::from… (#2448)
* minor patch which allows us to pass mutable strings to simdjson::from and fix
an issue with ambiguous integrals

* compatibility patch.
2025-09-15 22:22:25 -06:00
Daniel Lemire 22dcdc9f1e Update README.md 2025-09-15 18:52:15 -06:00
Dirk Stolle dda2dafa30 fix some typos (#2446) 2025-09-15 18:23:13 -06:00
Daniel Lemire 72e9d44e10 fixing indent 2025-09-15 17:08:12 -06:00
Daniel Lemire 8aae14931d release candidate 4.0.2 (#2441)
* release candidate 4.0.2

* more fixes

* fixing typos

* adding macro check
2025-09-15 09:17:43 -06:00
Daniel Lemire ef3d1ac25f fixing 2440 2025-09-14 08:09:27 -06:00
Daniel Lemire 9292480a9b removing leftover <experimental/...> (#2438) 2025-09-14 08:00:26 -06:00
Daniel Lemire 611ea97dbd Add C++26 reflection example link to README 2025-09-12 21:22:45 -04:00
Daniel Lemire f249e7e128 patch release 2025-09-12 19:26:58 -04:00
Marian Klymov 44fddaa807 Replace POSITION_INDEPENDENT_CODE with INTERFACE_POSITION_INDEPENDENT_CODE for older CMake (#2437) 2025-09-12 19:26:22 -04:00
Daniel Lemire be16e1ae04 removing space 2025-09-11 20:58:31 -04:00
Daniel Lemire 124a33c160 Update JSON example and add section on special cases
Removed code block formatting for JSON example and added a section header for special cases.
2025-09-11 19:32:38 -04:00
Daniel Lemire a0b870e9c2 Add example for deserializing 'Car' type
Updated the documentation to include an example of deserializing a 'Car' type using simdjson.
2025-09-11 19:31:15 -04:00
Daniel Lemire 68699eb73c release 4.0.0 2025-09-11 19:25:10 -04:00
Daniel Lemire 06453f1637 Remove Fuzzing Status badge from README
Removed Fuzzing Status badge from README.
2025-09-09 18:44:21 -04:00
Daniel Lemire eb828d97ba fixing a few typos 2025-09-09 18:41:00 -04:00
Daniel Lemire 00f84fb448 simplify 2025-09-09 14:21:14 -04:00
Daniel Lemire 5bf0954c6e hop 2025-09-08 19:26:34 -04:00
Daniel Lemire 3056d54fa3 adding instructions 2025-09-08 17:47:14 -04:00
Daniel Lemire 241b9f6ea7 adding bad/good player test 2025-09-08 17:04:52 -04:00
Daniel Lemire 015daad6a9 Merge branch 'master' of github.com:simdjson/simdjson 2025-09-05 19:50:05 -04:00
Daniel Lemire fc3b7766c0 typo fix 2025-09-05 19:49:57 -04:00
Daniel Lemire b5c78288e5 can we run the address sanitizer under VS in CI ? (#2435)
* can we run the address sanitizer under VS in CI ?

* fixed typo
2025-09-05 18:32:15 -04:00
Daniel Lemire bd86bb0cfb adding UTF-8 test 2025-09-05 14:56:02 -04:00
Daniel Lemire 4840a0347c extra documentation. 2025-09-05 14:44:53 -04:00
Daniel Lemire 7f68baec1e minor tweak 2025-09-05 08:01:04 -04:00
Daniel Lemire e2ea5fb8de This PR adds -> and * operators to our error types and it (#2433)
improves slightly the documentation.
2025-09-04 22:35:01 -04:00
Daniel Lemire 7dfd165ecd Merge branch 'master' of github.com:simdjson/simdjson 2025-09-04 09:46:07 -04:00
Daniel Lemire 4dfa0a2407 guarding 2025-09-03 16:45:09 -04:00
Daniel Lemire e252a21f65 removing expand workaround (#2431)
* removing expland workaround

* minor fixes
2025-09-03 13:30:35 -04:00
Daniel Lemire 89f2b634df guarding from benchmark + rvv 2025-09-02 13:06:25 -04:00
Daniel Lemire addca203d5 adding a 'car builder' benchmark (#2428)
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2025-08-26 09:16:18 -04:00
Felipe Monteiro 1825d7e88c Add "node:" prefix on fs import (#2426) 2025-08-25 10:21:13 -04:00
Daniel Lemire f87d6af3e9 fix issue 2424 (#2425) 2025-08-24 16:04:56 -04:00
Daniel Lemire 8fff57578c let us default on developer mode when building with VScode 2025-08-23 20:59:58 -04:00
Daniel Lemire 056d66926a Renames a few macros and extends slightly our basic builder (#2422)
* This PR renames a few macros and extends slightly our basic builder

* minor tuning
2025-08-20 09:01:58 -04:00
Daniel Lemire b434a50681 doing more to discourage value_unsafe(); (#2421) 2025-08-19 16:10:38 -04:00
Daniel Lemire e8ff2fa692 This is a small reorg of the new convert code so that we only expose 'simdjson::from' as experimental (#2418)
* This is a small reorg of the new convert code so that we only expose 'simdjson::from'.
This can be changed in a future release, but we don't want our users to start depending
on code that we might need to change.

* marking simdjson::from as experimental

* updating tests to match recent changes
2025-08-15 15:13:36 -04:00
Daniel Lemire 1ebf115b2b improving the documentation of raw json access (#2416)
* improving the documentation of raw json access

* minor fix

* documentation update

* guarding for exceptions

* fixing exception issue

* fix test

* update.

* saving comments

* more technical fixes

* correcting path in ci test

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2025-08-15 10:26:20 -04:00
Joshua Gawley 172fbe785d Use std::string_view in parser.load function (#2417) 2025-08-15 10:26:03 -04:00
Daniel Lemire a10f096af2 let us just say that it should not work 2025-08-14 11:49:42 -04:00
Daniel Lemire a85e474360 documentation update 2025-08-14 09:33:11 -04:00
Daniel Lemire 05752d6c6f Update ubuntu22-cxx20.yml 2025-08-13 21:17:41 -04:00
Joseph Olabisi fb9a689dff Adding DOM support for json path with wildcard (#2346)
* wip brute-force wildcard for json_path

* wip - bruteforce surface wildcard with result

* partially handle keys with wildcard

* wip - nested paths/pointers on wildcard results

* wip

* wip - handling child properties of wildcard result

* done - handling child properties of wildcard result

* fix key

* add support for wildcard for arrays

* handle array INCORRECT_TYPE

* rename at_path_new to at_path_with_wildcard

* add benchmark

* fix benchmark

* use memcmp

* minor improvements

* refactor to tail recursion

* nit

* approximately 30% improvement in runtime

* nit

* corrected logic

* nit

* cleanup

* add some initial tests

* cleanup 2

* modified:   CMakeLists.txt

* cleanup

* cleanup

* cleanup

* cleanup

* restore examples/quickstart/CMakeLists.txt

* cleanup

* restore quickstart.cpp

* revert array-inl.h

* cleanup array-inl.h

* revert object-inl.h

* cleanup object-inl.h

* cleanup

* nit

* nit

* add test

* address some feedbacks

* address additional feedbacks (copilot)

* final changes based on feedback - reduce string allocations

* fix bug in object-inl.h

* fix logic for wildcards inside arrays

* add test for wildcard in nested array

* refactor and create util for getting key and json path

* minor error handling

* refactor process_json_path_of_child_element from recursion to loop in order to prevent stack overflow

* fix bug with array, add boundary check to get_next_key_and_json_path function

* add more tests

* fix boundary check and unnecessary string allocation

* minor changes

* fix

* fix jsonpathutil

* remove unneccessary string allocation

* some minor fixes

* documentation

* removing printout

* various fixes

* reorg of the CI test file

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>
2025-08-13 21:16:32 -04:00
Daniel Lemire 472f818fe5 trimming repeated code 2025-08-13 20:40:04 -04:00
Daniel Lemire 5c9c133632 improve how we pad strings. (#2415) 2025-08-13 18:41:14 -04:00
Daniel Lemire faf921bc7e introducing a thread-local parser and removing ranges (#2412)
* introducing a thread-local parser

* adding functionality to release the memory

* some more documentation.

* fixing build

* adding benchmarks for 'from'

* generalizing the code somewhat.

* adding tests, fixing the benchmark (now with arrays and streams), and a
minor update to document_stream

* adding missing files (I forgot to check them).

* We cannot use [[nodiscard]] without guarding it, it is C++17

* fixing the cmake

* marking it as experimental

* removing ranges support (it is too experimental)

* putting back documentation.

* guarding SIMDJSON_CONSTEVAL more carefully.

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2025-08-13 18:08:25 -04:00
Daniel Lemire 626eedc3d8 documentation update 2025-08-13 16:24:55 -04:00
Daniel Lemire 294aa64f94 Update bug_report.md 2025-08-13 15:01:53 -04:00
Daniel Lemire 397b40e574 more documentation tuning 2025-08-11 15:39:10 -04:00
Daniel Lemire fdc93528d1 updating documentation. 2025-08-11 13:48:01 -04:00
Francisco Geiman Thiesen 3e00a431a0 Merge pull request #2406 from simdjson/fix-convert-ci-failures
Introducing simplified api with from/to adapters (thanks to @the-moisrex for driving this) and fixing all the CI errors.
2025-08-08 10:42:32 -07:00
Pavel Novikov d3626c6a37 removed now unnecessary <utility> header, (#2411)
disabled copy ctor and assignment for `vector_with_small_buffer`
2025-08-08 08:39:28 -04:00
Daniel Lemire 686a1869c8 updating commit 2025-08-08 08:38:41 -04:00
Daniel Lemire 4c0f86db44 minor fixes 2025-08-07 23:59:30 -04:00
Daniel Lemire 6c4c934457 added a new test. 2025-08-07 18:11:44 -04:00
Daniel Lemire fcae795042 minor fix 2025-08-07 16:05:24 -04:00
Daniel Lemire b7c51156b9 Sped up serialization fix (#2409)
* clang format

* added `chars()` method

* implemented vector with small buffer instead of `std::vector`

* added missing <utility> header

* minor fixes

---------

Co-authored-by: Pavel Novikov <dev-ape@yandex.ru>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2025-08-07 16:04:20 -04:00
Pavel Novikov 662e3d96c7 Sped up serialization by 10..70%-ish (#2408)
* clang format

* added `chars()` method

* implemented vector with small buffer instead of `std::vector`

* added missing <utility> header
2025-08-07 16:03:39 -04:00
Daniel Lemire 64009f7063 more code simplification. 2025-08-07 13:25:18 -04:00
Daniel Lemire 74fb3ecac7 removing another pragma and some code simplification. 2025-08-07 13:23:21 -04:00
Daniel Lemire 26abf1d180 removing macros 2025-08-07 11:39:51 -04:00
Francisco Geiman Thiesen e878dc6be3 Addressing @the-moisrex review. 2025-08-06 03:05:32 +00:00
Francisco Geiman Thiesen efa03f5733 Fix to_bad_array test to handle both exception and non-exception error cases
The test was failing in CI with g++-13 because it only handled the
exception case. However, the array() method is marked noexcept and
returns a simdjson_result that may contain an error code instead of
throwing an exception.

This fix checks for both cases:
1. If array_result.error() is not SUCCESS, verify it's INCORRECT_TYPE
2. If no error is returned initially, the exception may be thrown when
   iterating over the result

This ensures the test passes regardless of whether the error is
reported via error code or exception.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-05 20:51:17 +00:00
Francisco Geiman Thiesen 8519623257 Fix unused parameter warning in json_iterator::assert_valid_position for SIMDJSON_CLANG_VISUAL_STUDIO
Added (void)position; to suppress unused parameter warning when compiling with SIMDJSON_CLANG_VISUAL_STUDIO defined, where the position parameter isn't used in the SIMDJSON_ASSUME statements.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-05 19:27:01 +00:00
Francisco Geiman Thiesen 484a092c31 Adding a few tests for the simdjson::to adapter. 2025-08-05 16:29:59 +00:00
Francisco Geiman Thiesen 8bb520adbf Merge master and regenerate amalgamated files
Resolved conflicts by regenerating the amalgamated single-header
files (simdjson.h, simdjson.cpp, and singleheader.zip) using the
amalgamate.py script after merging latest changes from master.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-03 20:16:47 +00:00
Francisco Geiman Thiesen ec8e6a6758 Fix -Werror=effc++ warnings and test issues in convert.h
This commit addresses multiple issues:

1. Fixed -Werror=effc++ warnings by using #pragma to disable the
   warning for constructors that cannot initialize all members in
   the member initialization list due to error handling requirements.

2. Added proper error tracking (m_error member) to handle cases where
   document initialization fails, preventing segfaults when using
   invalid documents.

3. Fixed lifetime issues in tests where temporary auto_parser objects
   were being used, causing dangling references. Tests now properly
   store the parser object before using it.

4. Simplified range adaptor tests that were expecting features not
   yet implemented in simdjson's ondemand API.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-03 19:26:58 +00:00
Francisco Geiman Thiesen 769528b6c0 Fix segmentation fault in auto_parser constructor for C++20
The issue was that we were trying to initialize ondemand::document
directly from simdjson_result<ondemand::document> in the member
initializer list. This caused a segfault in C++20 builds.

The fix explicitly handles the simdjson_result in the constructor
body, checking for errors and using value_unsafe() to extract the
document. This avoids potential issues with implicit conversions
and ensures proper error handling.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-03 19:01:42 +00:00
Francisco Geiman Thiesen 9fbc577be7 Fix member initialization order warning in auto_parser
The compiler was warning about member initialization order mismatch.
C++ initializes members in the order they are declared in the class,
not the order they appear in the initializer list.

Fixed by reordering member declarations to match the initialization
order needed: m_doc must be initialized before m_parser since we
need to call parser.iterate() before moving the parser.

This fixes the -Werror=reorder compilation error in CI.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-03 18:48:07 +00:00
Francisco Geiman Thiesen a7c95e9cc8 Fix initialization order in auto_parser constructor
The issue was that we were calling m_parser.iterate() after moving
the parser, which could leave it in an invalid state. In C++20,
this might behave differently than C++17.

Fixed by reordering the member initializer list to call
parser.iterate() BEFORE moving the parser into m_parser.

This ensures the document is created while the parser is still valid.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-03 18:36:14 +00:00
Francisco Geiman Thiesen 248d4eb2aa Try using parentheses instead of braces for document initialization
The issue might be related to how brace initialization vs parentheses
initialization handles implicit conversion from simdjson_result<document>
to document. This could be compiler-specific behavior.

Using parentheses initialization to ensure the conversion operator
is called properly.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-03 18:12:50 +00:00
Francisco Geiman Thiesen 559493e2bf Fix auto_parser constructor to handle document initialization correctly
The issue was that the auto_parser constructor was using implicit
conversion from simdjson_result<document> to document, which could
cause issues with certain implementations (particularly fallback).

Changed to use value_unsafe() to explicitly extract the document
after the parser is fully initialized. This ensures the document
is in a valid state for subsequent operations.

This fixes the ondemand_convert_tests failure in CI with clang++-16.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-03 15:28:40 +00:00
Francisco Geiman Thiesen 75acae5c46 Fix C++20 compatibility issues in convert.h
- Remove constexpr from functions that call non-constexpr methods
- The no_errors and to<T> adaptors were marked constexpr but call
  simdjson_result methods that are not constexpr in C++20
- This was causing compilation failures in CI for C++20 builds
- Tests now compile and pass with both C++17 and C++20

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-03 06:01:50 +00:00
Francisco Geiman Thiesen bd761ef573 Fix ranges support for C++20 compatibility
Instead of disabling the feature, provide C++20-compatible implementation
of the pipe operators for ranges support. The range_adaptor_closure is
C++23-only, so we implement our own pipe operators for C++20.

This preserves the core functionality of the PR while ensuring
compatibility across different compiler versions.
2025-08-03 05:39:10 +00:00
Francisco Geiman Thiesen 76ed73f07f Disable ranges-dependent tests to fix compilation
The test_no_errors() and to_clean_array() tests depend on the C++23
ranges features that we disabled. This commit conditionally compiles
these tests out when ranges support is disabled.
2025-08-03 05:34:34 +00:00
Francisco Geiman Thiesen 1b59b38de8 Disable C++23 ranges features to fix CI compatibility
The ranges features were causing compatibility issues across different
compilers and platforms. Disabling them for now until C++23 support
is more widespread.

This should fix the remaining Ubuntu and Windows CI failures.
2025-08-03 05:11:31 +00:00
Francisco Geiman Thiesen 4794b5d936 Update amalgamated files with convert.h fixes
Regenerate singleheader/simdjson.cpp and singleheader/simdjson.h
to include all the fixes for CI compatibility issues.
2025-08-02 17:06:45 +00:00
Francisco Geiman Thiesen 114f14924c Simplify ranges feature detection for C++23
Only enable range_adaptor_closure features when compiling with C++23
or later, as this feature is not available in C++20 implementations.
2025-08-02 16:04:58 +00:00
Francisco Geiman Thiesen e94825c9a8 Fix preprocessor check for __cpp_lib_ranges_zip
Add defined() check before comparing the value to avoid preprocessor
errors in compilers where this macro doesn't exist (like g++-13
with certain configurations).
2025-08-02 15:43:31 +00:00
Francisco Geiman Thiesen 333fd72f98 Fix CI failures in convert.h implementation
- Fix deprecated reflect_value warning by using reflect_constant
- Fix std::const_iterator C++23 requirement by using auto_iterator
- Fix C++23 std::ranges::range_adaptor_closure availability check
- Add convert.h to main simdjson.h includes

These changes ensure compatibility across different C++ standards
and compiler versions, fixing the Ubuntu CI failures.

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-02 15:04:25 +00:00
Daniel Lemire dd4d02617f removing unnecessary hack 2025-07-31 18:40:54 -04:00
Daniel Lemire e28d63334e fix 2025-07-31 13:07:29 -04:00
Daniel Lemire 3d1ab87ecb Merge branch 'master' into release_candidate_4_0_0 2025-07-31 10:24:54 -04:00
Borislav Stanimirov d365cfb4a1 remove cmake_policy (#2404)
* properly gitignore Visual Studio artifacts

* remove cmake_policy
2025-07-31 10:18:28 -04:00
Daniel Lemire b9e308a727 marking them as 'really inline' 2025-07-28 11:47:09 -04:00
Brad Bramble aa6817d5aa Fix linker errors for downstream users caused by non-inline symbols in header (#2403) 2025-07-28 11:44:32 -04:00
Francisco Geiman Thiesen 90e4a66c93 Bringing a bit more use-cases for reflection based serializations (optional). Also adding string-based enum handling as requested on X. (#2395)
* Adding type validation, enhancing optional type support and adding test a few more tests.

* Adding support for string-based enum serlalization and deserialization.

* Removing unintentional endline.

* Removing trailing whitespace.

* Adding simpler api as suggested by moisrex.

* Removing explicit optiona<int> and optional<std::string> references and using concepts instead! Credit goes to Lemire for pointing this out and suggesting a concepts based approach here.

* Removing tests that are not relevant for this branch.

* Removing api related changes. That will be done by moisrex.

* Removing unnecessary new endlines.

* Removing tests related to api changes and cleaning-up irrelevant tests.

* removing broken reference

* Removing trailing whitespace
2025-07-23 09:24:05 +02:00
evbse e6240f18c4 Add version to amalgamated files (#2400) 2025-07-23 09:20:06 +02:00
M. Bahoosh d43fb6ff84 test for no_errors 2025-07-21 05:24:12 -10:00
M. Bahoosh 5eec29a6db Moving iterator's storage to auto_parser 2025-07-21 05:18:13 -10:00
M. Bahoosh 228501f786 From/To adaptors 2025-07-21 03:22:20 -10:00
M. Bahoosh 9d9f2427c5 Make auto_parser and auto_iterator comply with ranges. 2025-07-20 03:22:07 -10:00
M. Bahoosh 8d53840253 Removing unneeded code 2025-07-19 08:37:04 -10:00
M. Bahoosh 0a82fb110f Auto Iterator 2025-07-19 08:35:49 -10:00
M. Bahoosh 11f273c580 Moving ondemand::document into auto_parser 2025-07-19 04:37:13 -10:00
M. Bahoosh 31662531ac Basic Auto Parser 2025-07-19 03:58:21 -10:00
Daniel Lemire 97eb557388 Merge branch 'release_candidate_4_0_0' of github.com:simdjson/simdjson into release_candidate_4_0_0 2025-07-16 12:08:03 -04:00
Daniel Lemire 41ced28821 adding a space 2025-07-16 12:07:49 -04:00
Daniel Lemire 1078eb4034 minor update to the release candidate (#2394) 2025-07-16 12:05:09 -04:00
evbse 6029af75b3 Improve fallback implementation (#2393) 2025-07-15 14:39:09 -04:00
Daniel Lemire 0cd774097f [skip ci] doc fixes 2025-07-14 23:46:30 -04:00
Daniel Lemire b5e27af4da release candidate 4.0.0 2025-07-14 15:55:00 -04:00
Daniel Lemire c806e955c4 C++26 static reflection (#2282)
* Initial work on JSON builder

* moving the files back to ondemand for now.

* tweak

* more later

* update

* minor edits

* dropping vs arm (missing support)

* adding tests. we still specialized write_string_escaped

* tweaking

* fix typo

* tweaking the approach

* minor fix

* missing store

* another missing store

* Attempt at fixing failing serialization tests. (#2292)

* Fixing appeand_float typo (#2294)

* applying a couple of fixes

* updating single header

* fix for pre C++17 if constexpr

* Fixing unused argument problem and updating the singleheader file

* various pedantic fixes

* Sketch of builder

* reordering.

* simplify

* Adding draft of static reflection based deserialization

* Updating simdjson singleheader

* patching the automated deserialization.

* automated

* Adding support for smart pointers of user defined types.

* Adding specialization for smart pointers for basic types. I think it is highly likely that this can be done in a more generic way.

* Referncing a later version of rapidjson that fixed the issue related with assignment attempt of a const variable for GenericStringRef class.

* guarding the tests

* adding documentation for string_builder

* saving

* rename to 'append'

* saving

* non-functional benchmarks (#2342)

* non-functional benchmarks

* Fix typo

* various fixes

* tweaking

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>

* tuning

* various minor fixes

* minor tweak

* minor simplification

* updating amal

* adding a cast

* update

* fancy casting

* removing dead code

* Pushing latest changes. CITM benchmark is still not working.

* Still not working, but now I am getting only 10 errors.

* add static reflection benchmark to 'large random' benchmark and allows (#2349)

deserialization (with static reflection) from objects and arrays.

Co-authored-by: Daniel Lemire <dlemire@lemire.me>

* Removing std::map from CitmCatalog definition, since that is not currently supported.

* Added free to rust bench, segfault is still happening..

* The syntax changed: ^E became ^^E. (#2350)

* The syntax changed: ^E became ^^E.

* guarding

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>

* Adding support for string_view_keyed_map types.

* Adding concepts as a conditional include.

* updating single-header

* Adding concepts to ondemand deps

* rust benchmark is finally working

* Fixing small typo in docs.

* adding docker config and instructions so that our users can test the static reflection (#2358)

* adding docker config and instructions so that our users can test the
static reflection

* completing the instructions

* pruning white spaces

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>

* minor optimizations on the JSON builder branch

* avoiding undef behaviour

* saving

* somewhat nicer builder

* make it possible to run just one benchmark

* adding linux perf

* fixing minor issue

* updating swar

* Adding real world compilation benchmark (#2379)

* Adding compilation benchmark for json parsing with and without reflection

* Moving it to the benchmark folder, also reducing a bit the number of iterations.

* Removing script from root folder.

* Reducing number of iterations

* Update benchmark/benchmark_reflection_usage_compilation.sh

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Update benchmark/benchmark_reflection_usage_compilation.sh

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Update benchmark/benchmark_reflection_usage_compilation.sh

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Making the script more customizable and also test whether the compiler being used supports reflection before actually running the benchmark

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Using define_static_string from  https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p3491r2.html (#2389)

* Applying changes needed after latest reflection paper updates.

* Working, but no template for yet.

* Updating single-header to incldue the use of define_static_string.

* copying over master

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
2025-07-14 15:43:52 -04:00
Daniel Lemire 4acaf2ea1c improving DOM ranges test and silencing a warning (#2385)
* improving DOM ranges test and silencing a warning

* moving test_main to macros (even though it is not a macro)

* moving test_main
2025-07-04 15:46:50 -04:00
Daniel Lemire 86adda06d0 clarifying NDEBUG usage (#2388)
* clarifying NDEBUG usage

* init
2025-07-04 15:46:37 -04:00
Daniel Lemire 8df5f96eba guarding the POSITION_INDEPENDENT_CODE (#2386) 2025-06-30 16:41:49 -04:00
xkszltl 937667796c Replace -fPIC with POSITION_INDEPENDENT_CODE. (#2383) 2025-06-30 11:26:27 -04:00
Cuda Chen 7d86b04247 Add document and tests of std::ranges support of DOM API (#2381)
* Add document and tests of std::ranges support of DOM API

* Alter lambda

* Fix require of range
2025-06-24 15:13:44 -04:00
Daniel Lemire 265022ad8f Fixing MSYS2 issue (#2380)
* Google Benchmarks does not support MSYS2.

* narrowing it down
2025-06-21 21:26:52 -04:00
Daniel Lemire 0c0ce1bd48 3.13.0 2025-06-04 00:46:03 -04:00
Daniel Lemire f64c004cb7 making it easier to convert to std::string (#2378)
* making it easier to convert to std::string

* typo

* guarding C++20

* bad semi-colon
2025-06-04 00:22:10 -04:00
Daniel Lemire d7d19db6ae Emscripten (#2377)
* emscripten

* cmake_policy

* space
2025-06-03 09:10:59 -04:00
Daniel Lemire a260c967ed adding test for issue 2375 (#2376)
* adding test for issue 2375

* update

* removing space
2025-06-02 09:24:40 -04:00
Daniel Lemire 16e390d81a fix for issue 2373 (#2374)
* fix for issue 2373

* bumping google bench

* removing json11 since dropbox retired it

* removing json11 as a dependency
2025-05-31 00:56:19 -04:00
Daniel Lemire 8d34e14000 Update dom.md 2025-05-31 00:27:28 -04:00
shuiyisong b717136fd9 chore: add greptimedb in real world usage (#2372) 2025-05-07 16:33:24 -04:00
Daniel Lemire f3ac74caf8 slightly better documemntation of the C++20 features (#2371)
* slightly better documemntation of the C++20 features

* adding another remark
2025-04-23 18:32:51 -04:00
Daniel Lemire 2369927661 Moving from Ubuntu 20 to 24 (#2366) 2025-04-18 17:56:46 -04:00
Daniel Lemire 818c0491a1 Treat -0 as -0.0 when SIMDJSON_MINUS_ZERO_AS_FLOAT is set (#2364)
* In the DOM API, treat -0 as -0.0

* documenting...

* adding it to On-Demand
2025-04-18 13:56:02 -04:00
Daniel Lemire 9f14f90a64 [documentation] We run on practically all systems. (#2365)
* We run on practically all systems.

* Update doc/basics.md

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

---------

Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2025-04-18 13:48:13 -04:00
Daniel Lemire 00843d2711 better documentation 2025-04-16 11:45:52 -04:00
Daniel Lemire 2887a17bab bumping up nlohmann/json 2025-04-13 14:43:30 -04:00
huangqinjin d84c934768 do not try to include <string_view> without C++17 for MSVC (#2360)
Doing that will emit warning at https://github.com/microsoft/STL/blob/vs-2022-17.13/stl/inc/string_view#L12.
2025-03-28 19:23:15 -04:00
huangqinjin 7cec7c7ae4 fix VS2017 exception detection (#2359)
MSVC option /EHsc defines __cpp_exceptions only in VS2019 and above,
_CPPUNWIND should be used before VS2019.

https://learn.microsoft.com/en-us/cpp/preprocessor/predefined-macros#microsoft-specific-predefined-macros
2025-03-28 19:22:56 -04:00
Daniel Lemire c52b010a57 missing string_view include (#2361) 2025-03-28 19:22:38 -04:00
Daniel Lemire ee301599b1 Update basics.md 2025-03-28 11:53:07 -04:00
Daniel Lemire 7382dc2be8 preparing patch release 3.12.3 (#2357) 2025-03-28 11:26:19 -04:00
Daniel Lemire 8c14e0c56f fixing issue 2354: get_bool() on document with trailing spaces (#2356) 2025-03-27 15:01:10 -04:00
Daniel Lemire a9a62feb75 Map proto (#2353)
* std::map support

* completed

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2025-03-23 16:32:48 -04:00
Ezequiel Ramis Folberg b9228b4d3c Update README.md (#2352) 2025-03-22 15:49:02 -04:00
Dirk Stolle 726c3eb611 fix several typos (#2348) 2025-03-17 09:57:08 -04:00
Daniel Lemire 4cdc4f18ef documenting fatal errors 2025-03-13 13:26:50 -04:00
Dirk Eddelbuettel f3b034ac38 Address 'whitespace in literal' decreation warning from clang++-20 2025-03-06 14:28:01 -06:00
Daniel Lemire 9c2e8a8f39 upgrading uraimo/run-on-arch-action 2025-03-01 14:32:13 -05:00
yun dfa43f6cdd doc: replace ndjson[dot]org with ndjson spec (#2340)
the `ndjson[dot]org` expired, and point to incorrect website with malware.
more in https://github.com/ndjson/ndjson.github.io/issues/24

follow up #2234
2025-02-21 09:07:39 +00:00
Daniel Lemire 797e61742c release bump 2025-02-14 16:12:28 -05:00
Daniel Lemire f289412e0a fixing performance issue under Zen 4 processors (#2335)
* fixing performance issue under Zen 4 processors

* trying something else

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2025-02-14 16:11:36 -05:00
Daniel Lemire 7bd79b4445 Update README.md 2025-02-14 16:03:01 -05:00
Daniel Lemire 078e2c9073 patch release 3.12.1 2025-02-11 13:46:03 -05:00
Daniel Lemire d7b6b20511 creating zip files. 2025-02-11 13:43:34 -05:00
Daniel Lemire dbea3bbd62 Update README.md 2025-01-29 11:48:36 -05:00
Daniel Lemire e422933414 release 3.12.0 2025-01-27 20:35:57 -05:00
Daniel Lemire de4d69b367 Introducing dual licensing (#2328)
* Introducing dual licensing

* adding missing file
2025-01-27 20:34:35 -05:00
Paul Dreik b8675a7f7b fix fuzzing when running on icelake (#2327) 2025-01-26 05:36:33 +01:00
Paul Dreik 5642bb93a4 fix fuzzer CI job (#2324) 2025-01-16 13:33:53 -05:00
Daniel Lemire 1b23a77e03 3.11.6 2025-01-14 21:33:29 -05:00
Daniel Lemire 57699bfed8 release candidate (#2323) 2025-01-09 11:01:48 -05:00
Daniel Lemire 648303b26a typo 2025-01-08 21:29:11 -05:00
Daniel Lemire 9008960e36 fixed comment 2025-01-08 20:35:30 -05:00
Daniel Lemire 8a9e8a1792 fixing issue 2322 2025-01-08 20:28:25 -05:00
Daniel Lemire ba33e9e78f release 2025-01-06 19:33:53 -05:00
Daniel Lemire d98b351eef let us be more careful with concepts, we require __cpp_concepts >= 201907L and check for old versions of Apple clang (#2321)
* let us be more careful with concepts, we require __cpp_concepts >= 202002L

* checking Apple version

* let us be more specific

* removing extra endif
2025-01-06 19:23:09 -05:00
Daniel Lemire 5488dca126 Accidentally, it appears that PPC64 was always disabled by default. (#2320) 2025-01-03 11:01:04 -05:00
Daniel Lemire 7712ecf164 adding char bit check (pedantic) (#2316) 2024-12-31 14:34:13 -05:00
Paul Dreik 2803ca3093 use clang 19 in fuzzer job (#2315)
* use clang 19 in fuzzer job

* fix syntax error in msys2-clang.yml
2024-12-22 15:00:58 -05:00
Daniel Lemire e7f2463920 some tweaks 2024-12-17 16:27:31 -05:00
Daniel Lemire 5bfa0b098c patch release 3.11.3 (#2313)
* preparing patch release 3.11.3
2024-12-12 13:38:22 -05:00
Daniel Lemire f7ba9cb11b Update basics.md 2024-12-10 11:47:21 -05:00
Daniel Lemire d4bf0cc7ec sync 2024-12-09 21:36:43 -05:00
Daniel Lemire 2fbbea0b15 Merge branch 'master' of github.com:simdjson/simdjson 2024-12-09 21:36:12 -05:00
Daniel Lemire c16486f702 release candidate 3.11.2 2024-12-09 21:35:21 -05:00
zhanglistar f615112093 Disabling memory sanitizer with parse_string (#2310) 2024-12-09 10:20:33 -05:00
Daniel Lemire 6bbcbfbb95 version 3.11.1 2024-12-07 11:13:00 -05:00
Valery Mironov e6578fea39 Fix missing override (#2305) 2024-12-07 11:12:25 -05:00
Joyee Cheung 6fb050d04e fix: fix deprecation from -Wdeprecated-literal-operator (#2307)
Otherwise simdjson doesn't build with V8's Node.js fork, which
uses -Werror,-Wdeprecated-literal-operator and latest version
of clang.
2024-12-07 11:12:14 -05:00
Daniel Lemire b4242d3b4f release candidate (#2304) 2024-12-05 22:05:39 -05:00
Daniel Lemire cbfe2d4a21 Update vs17-arm-ci.yml 2024-12-02 20:24:24 -05:00
halx99 79126ca323 fix: processor checks inconsistently problem (#2299) 2024-12-02 20:23:28 -05:00
Daniel Lemire 3f87f958d7 documentation 2024-12-02 00:06:20 -05:00
Daniel Lemire cab383e1de Update README.md 2024-11-30 20:43:20 -05:00
Daniel Lemire 9b314922a1 additional documentation 2024-11-08 11:53:34 -05:00
Daniel Lemire 587beae307 prevent int to pointer implicit conversion for operator[](const char *) (#2286) 2024-11-01 18:11:43 -04:00
Daniel Lemire f53981d945 Update dom.md 2024-11-01 12:23:35 -04:00
Daniel Lemire 41b4ffd6aa Update basics.md 2024-11-01 12:15:40 -04:00
Daniel Lemire 6cd8a6d776 adding pad function (#2283) 2024-10-29 21:26:01 -04:00
Daniel Lemire 3c0d032ded Candidate for C++20 deserialization features (#2276)
* tag_invoke based custom types (#2219)

* tag_invoke based custom types

Now you can use tag_invoke to add a custom type or a group of custom types.

* Fixing macro usage + Fixing noexcept

* Fixing the usage of #include

We don't need <concepts> at all seems like it

* Fixing tag_invoke impl for MSVC

* Making `tag_invoke` to support `ondemand::document` as well + docs (#2228)

* Making `tag_invoke` to support `ondemand::document` as well + docs

* Fix typos and doc update by @lemire

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Better docs by @lemire

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Preserving the old, disallowing in the new

I'm disabling `document::get() &&` if the user has provided a `tag_invoke`d version; otherwise, we retain the compatibility.

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* fix: correct small issues with deserialize (#2232)

* Extending the deserialization code with more defaults + docs (#2233)

* Make custom types easier with some predefined cases + docs

* missing include

* adding Ubuntu 24 CXX 20

* using concepts all the way

* minor tweak

* tiny tweak

* tweaks

* more tweaking

* saving

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>

* Making `tag_invoke` a "put" as opposed to a "get" (#2256)

* fix: add tests related to issue 2227 (#2229)

* fix: add tests related to issue 2227

* avoiding name clash

* pedantic fix

* deprecate rvalue get on document

* selectively deprecating

* Fix ndjson spec link (#2234)

* fix ndjson spec link

The link in the readme of parse_many links to a casino spam site

* fix link

* [no-ci] Update README.md

* Make simdjson compile again

* Enable SIMDJSON_SINGLEHEADER=OFF in VS Code

With singleheader on, clangd can't find the right
include files.

* Add missing include directives to static build targets of simdjson. (#2240)

* adding a warning

* adding warning regarding SIMDJSON_BUILD_STATIC_LIB

* release candidate

* pedantic viable size

* Making tag_invoke a feeder instead of a producer

* adding missing undef silencer (#2253)

* Ignore pragma once when amalgamating source files (#2248)

With gcc it causes an error in `simdjson.cpp`:
```
simdjson.cpp:548:9: warning: #pragma once in main file
  548 | #pragma once
      |         ^~~~
```

It had previously been commented out in:
https://github.com/simdjson/simdjson/commit/6ef555e6fb79363fae057a9a46b52cd208d9e305

However, this was lost in an upgrade:
https://github.com/simdjson/simdjson/commit/2a4ff7346813b120f2b5b40e95d69352b593cc9c

* Update CI (#2254)

* adding missing undef silencer

* Updating CI

* more fixes

* fix

* big endian fix

* Moving to the new tag_invoke signature

* Fix nlohmann ambiguity on C++23-enabled clang

* Revert "Merge branch 'master' of https://github.com/simdjson/simdjson into builder_development_branch_extra"

This reverts commit 3eeecbab34, reversing
changes made to 6858b208b4.

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>

* update CI on the builder_development_branch (no code change) (#2262)

* typo

* General madness simpler, no simpler!!! (#2267)

* Minimal tag_invokes for STL types

* simpler madness

* adding a comment

* missing file

* minor tweaks to style

* fixing incorrect max/min usage

* updating single

* simplify

* validating the idea

* putting back the concept

* moving the include

* guarding

* Cheap General Madness (#2268)

* Some General Concepts and their deserializations

* Resolving ambiguity

* Add missing #include

* C++20 custom deserializer: better documentation (#2269)

* mostly a documentation update.

* missing cpp

* [no-ci] fix comment

* various minor fixes

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>

---------

Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: M. Bahoosh <moisrex@gmail.com>

* minor update

* More documentation regarding builder (#2270)

* minor update

* more improvment to our documentation (builder branch)

* putting back missing functions

* merge candidate

---------

Co-authored-by: M. Bahoosh <moisrex@gmail.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>
Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
2024-10-18 20:37:38 -04:00
Daniel Lemire 36f5dbcb75 documentation for threading... (#2277)
* documentation for threading...

* remove const

* doc
2024-10-11 12:34:11 -04:00
Daniel Lemire 49901fb254 doc 2024-10-10 21:32:51 -04:00
Daniel Lemire c066b5421b Jsonpath (#2272)
* Add support for JSONPath with '$' prefix and integrate into dom::at_path (#2266)

- Updated `json_path_to_pointer_conversion` to support JSONPath starting with the '$' prefix, while maintaining compatibility with the existing implementation.
- Moved `json_path_to_pointer_conversion` to a separate header file for better modularity and to support JSONPath queries in `dom` mode.
- Implemented `at_path` functionality in `dom` mode to enable querying JSON using JSONPath.
- Added unit tests to validate the new JSONPath support in `dom` mode and ensure compatibility with both standard and existing JSONPath formats.

* two minor fixes

* more tests and documentation

* damn compiler warnings

* more documentation fixes

---------

Co-authored-by: Zhengguo Yang <yangzhgg@gmail.com>
2024-10-08 10:37:59 -04:00
Daniel Lemire ec7550a70e fixes issue 2271 (#2273) 2024-10-08 10:36:12 -04:00
Daniel Lemire 3ef3078e51 documentation for issue 2259 (clarifying) (#2260) 2024-09-21 09:10:35 -04:00
tobil4sk fd06782c97 Make document::is_null behavior consistent with documentation (#2258)
* Make null-like value test consistent with docs

According to the docs: `INCORRECT_TYPE If the JSON value begins with 'n'
and is not 'null'.`

* Match document::is_null behavior to documentation

* Test other token beginning with n with is_null
2024-09-20 20:58:41 -04:00
Daniel Lemire 9303efbd0c [no-ci] Update basics.md 2024-09-20 20:56:11 -04:00
Daniel Lemire 6c979f15cc deps: switch to CPM (#2257)
* deps: switch to CPM

* missing file

* setting the dependencies on URL download

* using fixed version of cxxopts
2024-09-20 13:29:47 -04:00
Daniel Lemire 09ccabbe6c enable BMI, but only when LLVM is the compiler (#2255) 2024-09-20 10:59:54 -04:00
Daniel Lemire e00cc8c6dc Update CI (#2254)
* adding missing undef silencer

* Updating CI

* more fixes

* fix

* big endian fix
2024-09-18 09:09:11 -04:00
tobil4sk c10b32d463 Ignore pragma once when amalgamating source files (#2248)
With gcc it causes an error in `simdjson.cpp`:
```
simdjson.cpp:548:9: warning: #pragma once in main file
  548 | #pragma once
      |         ^~~~
```

It had previously been commented out in:
https://github.com/simdjson/simdjson/commit/6ef555e6fb79363fae057a9a46b52cd208d9e305

However, this was lost in an upgrade:
https://github.com/simdjson/simdjson/commit/2a4ff7346813b120f2b5b40e95d69352b593cc9c
2024-09-17 15:41:12 -04:00
Daniel Lemire 025a44348a adding missing undef silencer (#2253) 2024-09-16 21:04:09 -04:00
Daniel Lemire 70a68da941 pedantic viable size 2024-09-06 01:19:47 -04:00
Daniel Lemire e341c8b438 release candidate 2024-08-26 09:37:33 -04:00
Daniel Lemire 0ac0a80e28 Merge branch 'master' of github.com:simdjson/simdjson 2024-08-26 09:37:03 -04:00
Daniel Lemire 6b9117c029 adding warning regarding SIMDJSON_BUILD_STATIC_LIB 2024-08-26 09:36:44 -04:00
Daniel Lemire dd92151971 adding a warning 2024-08-26 09:34:39 -04:00
Tan Li Boon 0679c247f4 Add missing include directives to static build targets of simdjson. (#2240) 2024-08-26 09:24:01 -04:00
John Keiser 615218a3ad Merge pull request #2237 from simdjson/jkeiser/simdjson-vscode
Reduce errors in vscode
2024-08-18 22:04:20 -07:00
John Keiser 4c1b0a41d8 Enable SIMDJSON_SINGLEHEADER=OFF in VS Code
With singleheader on, clangd can't find the right
include files.
2024-08-18 14:21:00 -07:00
John Keiser ef563a4b09 Make simdjson compile again 2024-08-18 11:21:26 -07:00
Daniel Lemire 7a9ff93388 [no-ci] Update README.md 2024-08-15 12:52:47 -04:00
Sasha Lopoukhine fc61d7c7ba Fix ndjson spec link (#2234)
* fix ndjson spec link

The link in the readme of parse_many links to a casino spam site

* fix link
2024-08-10 10:11:05 -04:00
Daniel Lemire d506af0a79 fix: add tests related to issue 2227 (#2229)
* fix: add tests related to issue 2227

* avoiding name clash

* pedantic fix

* deprecate rvalue get on document

* selectively deprecating
2024-08-07 20:15:06 -04:00
Daniel Lemire ccf8694510 v3.10.0 2024-08-01 09:32:54 -04:00
Daniel Lemire 9b67497ed0 Allows field::unescape_key to take in a string parameter + additional dev. checks for string overflow (#2224)
* This PR does the following:

1. Upgrade cxxopts.
2. Allows field::unescape_key to take in a string parameter (syntaxic sugar).
3. Adds a dev. check to detect a string buffer overflow (indicating broken code). Note that this is unrecoverable and indicates bad code.

* tweak
2024-08-01 09:31:50 -04:00
Daniel Lemire 0336684df7 [no-ci] Update basics.md 2024-07-31 11:10:01 -04:00
Daniel Lemire c19320dd6e making it more precise 2024-07-31 10:08:04 -04:00
Daniel Lemire 412a5680e8 update 2024-07-31 10:06:47 -04:00
Daniel Lemire a05a56856d fix: use On-Demand throughout. (#2222) 2024-07-29 15:54:21 -04:00
didarpin 58173a6a1f Added the functionality to convert dom::object and dom::array to dom::element. (#2221)
Co-authored-by: didarpin <didarpin@163.com>
2024-07-25 22:26:20 -04:00
Francisco Geiman Thiesen 1721032cfd Merge pull request #2220 from simdjson/adding_macros_for_cpp20_cpp23
fix: add macros to detected C++20 and C++23
2024-07-25 17:44:36 -07:00
Daniel Lemire b73877f95e Update compiler_check.h 2024-07-23 14:25:27 -04:00
Daniel Lemire 09723897e9 fix: add macros to detected C++20 and C++23 2024-07-23 14:23:00 -04:00
Daniel Lemire 49e231b634 [no-ci] Update README.md 2024-07-16 16:05:32 -04:00
Daniel Lemire acdbbab916 adding an example of value capture with std::string_view (#2216)
* adding an example of value capture with std::string_view

* minor fix

* minor fix
2024-07-16 15:51:43 -04:00
Daniel Lemire 5090247c34 adding test (#2214) 2024-07-13 11:40:13 -04:00
Daniel Lemire 4180e05730 [no-ci] fix 'null_ptr' written as 'null_nullptrptr' in the comments 2024-07-11 08:25:27 -04:00
Daniel Lemire feea2bce2c Create config.yml 2024-07-04 16:48:09 -04:00
Daniel Lemire 692f43cd84 Update standard-issue-template.md 2024-07-04 16:46:34 -04:00
Daniel Lemire 3240d55bcc chore: add RelWithDebInfo to our CI tests (#2209)
* chore: add RelWithDebInfo to our tests

* fix

* chore: add various build types to ubuntu ci
2024-07-04 16:26:22 -04:00
Daniel Lemire 66fd28fc00 Marking a few functions as pure (no side-effect) (#2210)
* marking a few trivial functions as pure

* adding other marks

* additional marks

* vs will issue warnings, so don't use [[gnu::pure]] when __clang__ or __GNUC__ is not defined
2024-07-04 16:26:11 -04:00
Daniel Lemire 5f638951c6 Update basics.md 2024-06-27 14:49:57 -04:00
Daniel Lemire 3e94eea939 adding another example (#2206)
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2024-06-27 13:22:44 -04:00
Tunghohin 0e8311f812 reset moved object's viable_size to 0 in the move ctor of padded_string (#2204) 2024-06-25 18:06:32 -04:00
Daniel Lemire 3620e9d151 version bump 2024-06-11 15:37:22 -04:00
Daniel Lemire d017cd7ca4 Merge branch 'master' of github.com:simdjson/simdjson 2024-06-11 14:08:20 -04:00
Daniel Lemire d9d1ff5856 chore: remove unneeded gcc13 ci tests 2024-06-11 14:07:56 -04:00
Daniel Lemire eb8f2bce14 fix: add test for issue 2199 (#2200)
* fix: add test for issue 2199

* added ubuntu 24 workflow + silencing a warning
2024-06-11 13:55:55 -04:00
Dirk Stolle 2a4ff73468 update string_view lite to version 1.8.0 (#2197)
This is the header as seen for the tag v1.8.0,
commit a47222b9855dd6e6d1eac38acaa495822e2caa69, on
<https://github.com/martinmoene/string-view-lite>.
2024-06-10 10:29:25 -04:00
Daniel Lemire 77fc2b8447 doc: explaining the page trick (#2196)
* doc: explaining the page trick

* simplify

* did as john said

* trying something else

* flipping order

* trying some other order

* hmmm
2024-06-07 22:12:08 -04:00
Daniel Lemire ba8b66a633 Update basics.md 2024-06-07 12:49:32 -04:00
Daniel Lemire 66eec5feaf Update basics.md 2024-06-05 08:55:12 -04:00
Janeczko Jakub 3964f3e5d2 pull size_t from the std namespace (#2191) 2024-06-03 14:08:23 -04:00
Daniel Lemire ee8515122d version bump 2024-05-30 10:53:40 -04:00
halx99 5d35e7ca1f Fix compile error on llvm-19 (#2187) 2024-05-30 10:52:38 -04:00
spershin deefc88b9c Adding path for parsing incomplete json. (#2189)
1. Allows processing inclomplete, damaged, corrupted json to some extent.
2. Pariity with the Presto Java functionality.
3. Protected with SIMDJSON_EXPERIMENTAL_ALLOW_INCOMPLETE_JSON define.
4. Does not interfere with the normal path (can co-exist).
5. Tested in production forkflow.
2024-05-30 10:52:09 -04:00
Yuriy Chernyshov c80dda7c58 Fix building simdjson against libc++ with _LIBCPP_REMOVE_TRANSITIVE_INCLUDES defined (#2184)
```
src/implementation.cpp:193:20: error: no template named 'is_trivially_destructible' in namespace 'std'; did you mean 'is_trivially_move_constructible'?
static_assert(std::is_trivially_destructible<detect_best_supported_implementation_on_first_use>::value, "detect_best_supported_implementation_on_first_use should be trivially destructible");
              ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
                   is_trivially_move_constructible
```
2024-05-23 16:54:23 -04:00
Daniel Lemire d2954ef68b Update ubuntu22-gcc13.yml 2024-05-23 16:53:48 -04:00
Daniel Lemire ac719827ff fix: solve issue 2181 (#2182) 2024-05-11 20:44:38 -04:00
Daniel Lemire 6ea77392a7 Update basics.md 2024-05-10 12:21:26 -04:00
pnck e2f879751c fix: issue #2154 (#2178) 2024-05-10 00:33:09 -04:00
Daniel Lemire bf7834179c version bump 2024-05-07 19:15:43 -04:00
Daniel Lemire 69ab8848bf fix: satisfy a C++20 compiler warning (#2177) 2024-05-07 18:04:59 -04:00
Daniel Lemire 7288323e36 Update basics.md 2024-05-06 21:09:56 -04:00
Daniel Lemire 3f381cf3ee Update README.md 2024-05-06 11:11:34 -04:00
Daniel Lemire 52406402ed fix: replace vm*vq_u8 by vm*vq_u32 for better performance under some systems. 2024-05-01 14:53:48 -04:00
Daniel Lemire 799d8e3fc9 fix ref 2024-04-26 14:09:42 -04:00
Daniel Lemire 6c647d9f3b Update README.md 2024-04-24 16:23:45 -04:00
Daniel Lemire 8519e24f12 issue 2170 (#2172) 2024-04-24 01:41:40 -04:00
Daniel Lemire fffb62743c removing power fuzzer. 2024-04-23 23:50:49 -04:00
Daniel Lemire c888075f8d makes implementations trivially destructible (#2171) 2024-04-23 23:49:20 -04:00
Daniel Lemire 9b8aac5f22 [skip ci] updating documentation 2024-04-22 09:18:49 -04:00
Javier Blazquez 40b414d184 add Windows ARM64EC build and CI support (#2168)
* add Windows ARM64EC support

* add ARM64EC to vs17-arm-ci workflow
2024-04-09 14:56:58 -04:00
Daniel Lemire c85e8a7db1 version bump 2024-04-05 15:19:02 -04:00
Daniel Lemire 83f390f8e7 fix: adopt recommendation from issue 2163 (#2164) 2024-04-05 15:17:57 -04:00
Daniel Lemire 82563dcf70 correcting version bump 2024-04-04 14:08:25 -04:00
Daniel Lemire 646bd3f2c3 version bump 2024-04-04 12:24:07 -04:00
Daniel Lemire 4b0a908bb2 [skip ci] adding documentation regarding new hardware support 2024-04-04 12:22:03 -04:00
Jinyang He 4c98e51c53 Add LoongArch LSX and LASX support (#2159)
* Add LoongArch SX support

* Add LoongArch ASX support
2024-04-04 12:09:36 -04:00
Daniel Lemire 58e3d5d9cc fix: fix path to cross-compiler 2024-04-03 22:37:46 -04:00
Daniel Lemire a22a92dc72 trying to add loong to our ci tests (#2160)
* trying to add loong to our ci tests

* tuning loongarch

* updating the loong toolset
2024-04-03 15:18:45 -04:00
Daniel Lemire c4654553dd [skip ci] windows and unicode 2024-03-26 09:31:50 -04:00
Daniel Lemire 5997f82c37 Update basics.md 2024-03-20 14:26:11 -04:00
Daniel Lemire bee410b315 comment 2024-03-20 01:52:42 -04:00
Daniel Lemire b115e3ef61 some updates to our CI (#2151) 2024-03-17 11:59:50 -04:00
Daniel Lemire 669229df5d more documentation. 2024-03-17 10:29:41 -04:00
Daniel Lemire b6a2286e62 adding some documentation for escaped_key(). 2024-03-17 10:27:13 -04:00
Twice 3a97c57773 Add escaped_key method to ondemand::field (#2150) 2024-03-17 10:10:31 -04:00
Daniel Lemire e1c6a778f8 3.8.0 2024-03-10 13:25:53 -04:00
Daniel Lemire d413dd5b0e v3.7.1 2024-03-10 13:24:07 -04:00
Daniel Lemire a33fdc2770 Unconfusing clang (#2147)
* In some cases, clang might try to cast an ondemand::document to an
ondemand::document, instead of calling the move constructor. So we
can disable the template cast.

* In some cases, clang might get confused when constructing an ondemand
document in a constructor: instead of calling the move constructor, it
somehow ends up trying to cast a document to a document. We can easily
disallow this behavior with std::enable_if.

* making compatible with C++11

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2024-03-10 13:20:39 -04:00
Daniel Lemire 87f5f7a250 added some documentation in the code related to custom types (#2146) 2024-03-09 18:45:05 -05:00
Daniel Lemire 9923d27994 upper case 2024-03-06 16:01:21 -05:00
Piotr Rżysko 876be23545 Add link to simdjson-java (#2145) 2024-03-06 15:55:16 -05:00
Daniel Lemire 2028bdaeac This documents the big-int feature, and adds a few tests. (#2142)
* This documents the big-int feature, and adds a few tests.

* moving check_if_integer

* trimming the example.

* More trimming.

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2024-02-26 20:39:20 -05:00
Daniel Lemire 1ac3a00ca9 verify issue 2137 (#2138)
* fix issue 2137

* updating the single header

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2024-02-26 20:39:08 -05:00
Daniel Lemire 6d7cc2c7c1 fixing build 2024-02-26 20:00:17 -05:00
Daniel Lemire 61a52287fc Fixing apple builds 2024-02-26 19:50:22 -05:00
Daniel Lemire 37a9aaad48 [no ci] tweaking the wording 2024-02-26 14:05:12 -05:00
Serge Aleynikov 78325888ea Add functionality to detect big integers (#2139) 2024-02-26 14:00:33 -05:00
Shi Pujin a90cb5d900 add loongarch64 (#2141) 2024-02-26 13:54:49 -05:00
Daniel Lemire 34a01ec3af removing unnecessary spaces 2024-02-23 19:21:31 -05:00
Daniel Lemire e510ed2774 Release candidate (3.7.0) (#2136)
* Release candidate (3.7.0)

* some extra comments.
2024-02-23 16:50:55 -05:00
Dirk Stolle dabac13dde Add missing word in documentation (#2133) 2024-02-18 11:50:12 -05:00
Daniel Lemire aef7111c05 Documenting Unicode support in JSONPath and JSON Pointer (#2132)
* Documenting our support of JSON Path and JSON Pointer vs. Unicode characters

* using more standard terminology (nitpicking)

* specifying UTF-8 encoding

* allow char8_t when compiling as C++20

* casting

* minor doc corrections
2024-02-15 20:28:38 -05:00
Daniel Lemire 0c08894a54 adopting string literals (#2128)
* adopting string literals

* String suffixes are C++14
2024-02-15 10:40:11 -05:00
Daniel Lemire 2fe3a1467d This PR adds a convenience method (is_string) as well as a new method (#2131)
key_raw_json_token() to get the raw key token.
2024-02-15 10:39:30 -05:00
Daniel Lemire 167a46a0e7 adding remark 2024-02-15 09:56:05 -05:00
Francisco Geiman Thiesen 2b532489cc Adding on-demand support for a subset of json path. (#2127)
* First implementation of the conversion logic.

- Not yet tested
- Not yet compiled
- Not yet used in ondemand array/object/etc...

* Adding at_path to ondemand arrays

* Adding at_path to array, document and value.

* Adding at_path to ondemand object as well.

Pending:
- Building
- Adding tests for each of the on-demand classes usage of at_path
- Properly documenting the subset of json path that is currently supported.

* Adding a simple json_path test to on_demand

* Still trying to compile the ondemand_readme_examples test with the at_path() call

* Fixing linking issues with array::at_path

* Fixing issues with the path -> pointer conversion and removing comments

* Adding a clone of json_pointer to test json_path extensively

* Adding ondemand_json_path_tests (all tests passing)

* Adding documentation for at_path()

* Removing some newlines

* Removing simdjson_result from the return type of the string conversion function. Now json_path_to_pointer_conversion returns a std::string.

* Fixing typo

* Making string concatenation explicit to avoid ubuntu gcc12 issue with -O3 flag.

* Addressing latest reviews

* Addressing latest reviews
2024-02-15 09:47:25 -05:00
Daniel Lemire 85163df88c Update ondemand_design.md 2024-02-14 23:36:25 -05:00
Daniel Lemire 6a5be0f0db Create SECURITY.md 2024-02-09 17:16:54 -05:00
Daniel Lemire 710a3c7241 Update implementation-selection.md 2024-02-09 11:46:22 -05:00
Daniel Lemire c93e02e172 Update implementation-selection.md 2024-02-09 11:45:15 -05:00
Daniel Lemire 517143aec4 Update implementation-selection.md 2024-02-09 11:36:38 -05:00
Francisco Geiman Thiesen 9feda5e703 Update HACKING.md (#2122)
Updating windows command
2024-02-09 10:29:07 -05:00
Daniel Lemire cda198b7b9 Update ondemand_design.md 2024-02-09 10:22:44 -05:00
Daniel Lemire aad17a6a8d Update basics.md 2024-02-09 10:20:42 -05:00
Daniel Lemire 32d072a208 fixing issue2120 (#2121) 2024-02-08 11:34:36 -05:00
Daniel Lemire 5eab555053 documenting the support for custom types (#2098)
* documenting the support for custom types

* fixes

* getting around -Weffc++

* calling get_raw_json_string()

* documenting the support for custom types

* fixes

* getting around -Weffc++

* calling get_raw_json_string()
2024-02-06 13:20:35 -05:00
Daniel Lemire 6430fbd39a Better documentation and support for the case where you have a single implementation. (#2118) 2024-02-06 13:20:23 -05:00
Daniel Lemire 4d2fc31b27 removes simdjson.cpp include from the amalgated demo (#2119)
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2024-02-05 19:34:50 -05:00
Bo Anderson 33610b59eb Allow some iterators to be used in standard C++ iterator functions (#2106) 2024-02-05 17:52:01 -05:00
Daniel Lemire 23290ae48d retiring deprecated DOM APIs (#2109) 2024-02-04 21:58:48 -05:00
Daniel Lemire eb3a085e0f Bash check is unneeded. (#2112) 2024-02-04 21:58:36 -05:00
Daniel Lemire 4c8ec35a48 fixing the order of the apple counters 2024-01-31 14:30:44 -05:00
Daniel Lemire 88bf5cb6d9 version bump 2024-01-29 10:43:12 -05:00
Niles Salter 9b0435d864 [utf8-validator] eliminate unnecessary comparison from must_be_2_3_continuation (#2113)
* [utf8-validator] eliminate unnecessary comparison from must_be_2_3_continuation

* Fix comment in
2024-01-28 12:42:59 -05:00
Daniel Lemire ddecadde86 Update README.md 2024-01-23 09:35:12 -05:00
Bo Anderson 02bee7d1b9 Add load_into_document to simdjson::dom::parser (#2103) 2024-01-08 20:30:26 -05:00
Daniel Lemire 6ef555e6fb This PR updates the backup string_view implementation. It also adds a few comments to our README.md file. (#2105)
* This PR updates the backup string_view implementation. It also adds a few comments to our README.md file.

* removing trailing space
2024-01-08 14:52:00 -05:00
Daniel Lemire 6952f8dfce adding gcc 13 (#2101) 2024-01-04 18:57:44 -05:00
Daniel Lemire 03d0a4731a Update README.md 2024-01-03 17:49:43 -05:00
Daniel Lemire 185ff4e8fe Update basics.md 2023-12-29 09:35:35 -05:00
Daniel Lemire 5bf6d9f3f0 Update README.md 2023-12-29 09:30:01 -05:00
Daniel Lemire 2fa729922e fix for issue 2099 (#2100)
* fix for issue 2099

* avoid exceptions.

* fix for exception-less code
2023-12-19 12:15:59 -05:00
Daniel Lemire ebd09cb2a3 Update basics.md 2023-12-15 08:29:16 -05:00
Dirk Stolle 3978b9f159 update actions/checkout in GitHub Actions to v4 (#2095) 2023-12-09 23:21:28 -05:00
Dirk Stolle edc36a87ab fix a small typo (#2096) 2023-12-09 23:20:51 -05:00
Daniel Lemire 6060be2fdf version bump 2023-12-08 12:09:45 -05:00
Daniel Lemire 3251f61c9d fix for issue 2093 (#2094) 2023-12-07 12:42:28 -05:00
Ikraduya Edian 561ffcd519 Add benchmarks for boost json (#2092) 2023-12-04 10:33:53 -05:00
Daniel Lemire 5541e78402 version bump 2023-12-01 13:56:12 -05:00
Daniel Lemire 9a9ed0e2da Fix Visual Studio 64-bit ARM regression (#2091)
* fix for ARM64 kernel under Visual Studio (bug introduced in release 3.6.1)

* revert comment

* updating amal.
2023-12-01 13:55:28 -05:00
Daniel Lemire bca8d3e277 version bump. 2023-11-30 18:13:02 -05:00
Daniel Lemire 97196fa969 Fixing warnings issued by clang (Node issue 50930) (#2090)
* Fixing warnings issued by clang (Node issue 50930)

* removing space

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-11-30 18:06:50 -05:00
John Keiser 23fab307ca Merge pull request #2088 from luhenry/upstream-riscv64
Make simdjson riscv64 aware
2023-11-30 11:06:43 -08:00
Daniel Lemire 0a14e05ec7 Update README.md 2023-11-30 13:42:00 -05:00
Daniel Lemire b88f831dbd Update basics.md 2023-11-30 11:52:24 -05:00
Daniel Lemire 71a73c2c47 Update basics.md 2023-11-30 11:51:35 -05:00
Daniel Lemire 07603aa7f7 fix space 2023-11-29 18:57:37 -05:00
Ludovic Henry b8a6b889c7 review: Revert changes in include/simdjson/implementation_detection.h 2023-11-29 22:09:25 +00:00
Ludovic Henry 7898234bb3 Make simdjson riscv64 aware 2023-11-29 19:28:14 +00:00
Daniel Lemire f52533538a Update basics.md 2023-11-28 23:49:40 -05:00
Daniel Lemire 3bb2f0a186 Return a better error when a user tries to cast a document that has already been accessed to a value (#2085)
* Fix.

* adding a test, cleaning some comments, adding documentation

* Correcting the JSON input.

* adding another test
2023-11-26 14:13:05 -05:00
Daniel Lemire d6551bf278 Update README.md 2023-11-22 11:36:33 -05:00
Daniel Lemire 0173eaaad7 Macro-based workaround for Visual Studio ARM (#2082) 2023-11-20 09:58:50 -05:00
Daniel Lemire 5459dbd6c6 typos. 2023-11-13 21:48:21 -05:00
Saint 9c45f1f292 Added link to Nim bindings (#2081) 2023-11-11 04:06:32 -05:00
int mian 665cbba159 fix typo (#2080) 2023-11-06 07:34:12 -05:00
Daniel Lemire a26f664907 removing space 2023-10-30 12:56:26 -04:00
Daniel Lemire 17cb457ffd version bump 2023-10-30 12:52:49 -04:00
Daniel Lemire 8b2af0473a skip UTF-8 BOM if present. (#2079) 2023-10-30 12:51:02 -04:00
Daniel Lemire f1bfa43385 Update ubuntu20-sani.yml 2023-10-27 20:58:54 -04:00
Daniel Lemire 29fbbee852 version bump 2023-10-27 20:57:30 -04:00
Daniel Lemire baa7d961ce Allowing users to write directly to std::optional<std::string> (#2078)
* Allowing users to write directly to std::optional<std::string>

* better fallback

* do not force the cast to std::string

* missing header

* removing abort
2023-10-27 20:56:12 -04:00
Yagiz Nizipli 6412b27c7e Merge pull request #2077 from simdjson/raw_json_on_values 2023-10-27 19:11:01 -04:00
Daniel Lemire 503da5ec83 raw_json() on values 2023-10-27 16:22:22 -04:00
Daniel Lemire 9dd6e9c959 version bump 2023-10-25 19:35:51 -04:00
Daniel Lemire 6db330c1a2 Automating string reallocation when it is possible (#2076)
* Automating string reallocation

* Typo
2023-10-25 19:34:52 -04:00
Daniel Lemire 993ac4b87c Adding the ability to write directly to an std::string. It is mostly syntaxic sugar (#2075) 2023-10-25 19:32:12 -04:00
Daniel Lemire 35e87896f0 minor tweak 2023-10-25 10:50:27 -04:00
Daniel Lemire f7e281cadc removing noisy warning 2023-10-20 15:45:40 -04:00
Daniel Lemire 13405afd4b This provides verbose static_assert messages (#2074)
* This provides verbose static_assert messages

* minor fix
2023-10-20 15:42:18 -04:00
Daniel Lemire 24b44309fb Removing commented code 2023-10-15 14:56:32 -04:00
Momtchil Momtchev e11ad58aad add a total order for elements (#2072) 2023-10-09 15:41:57 -04:00
Daniel Lemire 1875ed6550 Version bump. 2023-09-20 10:13:47 -04:00
Daniel Lemire 80d26298a0 Redesigning visit_primitive so that it is optimized for strings and (#2060)
numbers.

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-09-20 10:09:56 -04:00
Antoine Prouvost 25b5015c09 Add simdjson_static lib (#2068)
* Add simdjson_static lib

* Make simdjson_static a separate optional export file
2023-09-20 10:09:27 -04:00
Antoine Prouvost 26f8c566c7 Fix issue template bold text (#2067) 2023-09-19 11:15:54 -04:00
Daniel Lemire ff77ac801e Additional documentation based on issue 2063 (#2065)
* Additional documentation based on issue 2063

* Update basics.md
2023-09-11 10:01:35 -04:00
Daniel Lemire e0699994ef Testing macOS shared library support (#2062) 2023-09-09 12:48:42 -04:00
Daniel Lemire 68ba9a1b2a Adding a few spaces (minor formatting) (#2059) 2023-08-30 20:23:58 -04:00
Daniel Lemire 6fed6bd29b Update README.md 2023-08-30 19:26:34 -04:00
Piotr Rżysko cc24bb4114 Removed number parsing fallback (#2056) 2023-08-29 18:28:21 -04:00
John Keiser 14ec0ca8f4 Merge pull request #2052 from simdjson/jkeiser/structural_indexer_step
Write out 4 structural indexes at a time instead of 8
2023-08-29 15:24:54 -07:00
John Keiser 8dabd02c3a Default to step = 4 2023-08-29 15:48:52 -04:00
Daniel Lemire c9692005fd This allows us to pass SIMDJSON_STRUCTURAL_INDEXER_STEP as a command-line option. (#2055) 2023-08-25 13:47:20 -04:00
Daniel Lemire 2e0a9397c9 Merge branch 'master' into jkeiser/structural_indexer_step 2023-08-25 11:34:13 -04:00
Daniel Lemire f7922075c3 Adding support for Apple event counters (#2054) 2023-08-25 11:33:24 -04:00
Daniel Lemire b2e20e04c9 Standard compatibility fixes (#2053)
* Standard compatibility fixes

* missing commit

* Should work.

* Fix.

* Fix.

* Should work now.

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-08-25 10:18:02 -04:00
John Keiser adc9d18efd Change step for structural_indexes to 2 2023-08-23 19:02:05 -07:00
Daniel Lemire bf849e3619 Preparing release 2023-08-22 14:05:14 -04:00
Daniel Lemire c09e8dc225 Icelake processes 64 bytes at a time when processing strings. (#2050)
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-08-18 14:37:10 -04:00
Daniel Lemire 5bf585ef02 Fixing failing thread sanitizer (#2051) 2023-08-18 14:35:12 -04:00
Daniel Lemire caff09cafc Update iterate_many.md 2023-08-14 12:18:09 -04:00
Daniel Lemire f043db4dca Preparing release. 2023-08-02 16:01:39 -04:00
Daniel Lemire e3bd0828d1 Fix for issue 2045. (#2046)
* Fix for issue 2045.

* Retauring single-header code.
2023-08-02 16:00:45 -04:00
John Keiser 9408298e0e Merge pull request #2042 from simdjson/jkeiser/escape3
New escape algorithm with significantly less data dependency
2023-08-02 10:38:09 -07:00
John Keiser 573bbac2d2 Reamalgamate again 2023-08-02 08:28:35 -07:00
John Keiser ac78c625df simdjson_inline -> simdjson_really_inline 2023-08-02 08:28:05 -07:00
John Keiser 390a66c6e8 Make next_structurals inline 2023-08-02 08:28:05 -07:00
John Keiser 210e0a56a1 Reamalgamate 2023-08-02 08:28:05 -07:00
John Keiser 084f662f32 Don't use C++20 designated initializers 2023-08-02 08:28:05 -07:00
John Keiser c878ff2500 Fix compiler warnings 2023-08-02 08:28:05 -07:00
John Keiser 06afe9ecb3 New escape algorithm with significantly less data dependency 2023-08-02 08:28:05 -07:00
Daniel Lemire e658502adb We are having issues with llvm.sh (#2044) 2023-08-02 11:24:28 -04:00
Daniel Lemire 6e7d415040 These lines should not be necessary. (#2043) 2023-08-01 19:33:27 -04:00
John Keiser 2ccd5b9c97 Merge pull request #2039 from simdjson/jkeiser/more-headers
Disallow including headers from implementation files
2023-07-21 12:09:57 -07:00
Daniel Lemire 1cdcbf79b3 We do actually support legacy 32-bit systems, although not with fast kernels. (#2041) 2023-07-21 13:53:10 -04:00
Calum McConnell 7013ed98cc Peek(0) is an error on 32-bit systems, where pointers are indistinguishable from int32_t (#2040) 2023-07-21 13:52:53 -04:00
John Keiser b383c717d7 Have amalgamate.py verify it's actually amalgamating all the files 2023-07-20 15:11:52 -07:00
John Keiser b8743bced0 Fix amalgamate.py under python 3.7 2023-07-20 14:14:44 -07:00
John Keiser ae9ba2bed2 Rename SIMDJSON_AMALGAMATED to SIMDJSON_CONDITIONAL_INCLUDE 2023-07-20 11:43:19 -07:00
Daniel Lemire 939844d79f Adding tests for issue 2037. (#2038)
* Adding tests for issue 2037.

* Fixed typo

* Simplifying.
2023-07-20 14:11:33 -04:00
John Keiser 4f4e81668e Change all include paths to <> instead of "" to avoid relative path includes 2023-07-20 10:48:25 -07:00
John Keiser bb54946b78 Disallow including headers from implementation files 2023-07-20 10:01:22 -07:00
John Keiser 4dec6ed5a7 Merge pull request #2036 from simdjson/jkeiser/more-header-fixes
A few more header fixes
2023-07-18 10:26:40 -07:00
John Keiser 1dfd48195d Merge pull request #2035 from simdjson/jkeiser/sanitizer-ci
Run sanitizers in parallel
2023-07-17 23:37:34 -07:00
John Keiser 23afc6b695 Merge branch 'master' into jkeiser/sanitizer-ci 2023-07-17 16:09:20 -07:00
John Keiser 80bc9e5051 Run sanitizers in parallel 2023-07-17 12:57:50 -07:00
593 changed files with 252662 additions and 122620 deletions
-316
View File
@@ -1,316 +0,0 @@
version: 2.1
# We constantly run out of memory so please do not use parallelism (-j, -j4).
# Reusable image / compiler definitions
executors:
gcc8:
docker:
- image: conanio/gcc8
environment:
CXX: g++-8
CC: gcc-8
CMAKE_BUILD_FLAGS:
CTEST_FLAGS: --output-on-failure
gcc9:
docker:
- image: conanio/gcc9
environment:
CXX: g++-9
CC: gcc-9
CMAKE_BUILD_FLAGS:
CTEST_FLAGS: --output-on-failure
gcc10:
docker:
- image: conanio/gcc10
environment:
CXX: g++-10
CC: gcc-10
CMAKE_BUILD_FLAGS:
CTEST_FLAGS: --output-on-failure
clang10:
docker:
- image: conanio/clang10
environment:
CXX: clang++-10
CC: clang-10
CMAKE_BUILD_FLAGS:
CTEST_FLAGS: --output-on-failure
clang9:
docker:
- image: conanio/clang9
environment:
CXX: clang++-9
CC: clang-9
CMAKE_BUILD_FLAGS:
CTEST_FLAGS: --output-on-failure
clang6:
docker:
- image: conanio/clang60
environment:
CXX: clang++-6.0
CC: clang-6.0
CMAKE_BUILD_FLAGS:
CTEST_FLAGS: --output-on-failure
# Reusable test commands (and initializer for clang 6)
commands:
dependency_restore:
steps:
- restore_cache:
keys:
- cmake-cache-{{ checksum "dependencies/CMakeLists.txt" }}
dependency_cache:
steps:
- save_cache:
key: cmake-cache-{{ checksum "dependencies/CMakeLists.txt" }}
paths:
- dependencies/.cache
install_cmake:
steps:
- run: apt-get update -qq
- run: apt-get install -y cmake
cmake_prep:
steps:
- checkout
- run: mkdir -p build
cmake_build_cache:
steps:
- cmake_prep
- dependency_restore
- run: cmake -DSIMDJSON_DEVELOPER_MODE=ON $CMAKE_FLAGS -DCMAKE_INSTALL_PREFIX:PATH=destination -B build .
- dependency_cache # dependencies are produced in the configure step
cmake_build:
steps:
- cmake_build_cache
- run: cmake --build build
cmake_test:
steps:
- cmake_build
- run: |
cd build &&
tools/json2json -h &&
ctest $CTEST_FLAGS -L acceptance &&
ctest $CTEST_FLAGS -LE acceptance -LE explicitonly
cmake_assert_test:
steps:
- run: |
cd build &&
tools/json2json -h &&
ctest $CTEST_FLAGS -L assert
cmake_test_all:
steps:
- cmake_build
- run: |
cd build &&
tools/json2json -h &&
ctest $CTEST_FLAGS -DSIMDJSON_IMPLEMENTATION="haswell;westmere;fallback" -L acceptance -LE per_implementation &&
SIMDJSON_FORCE_IMPLEMENTATION=haswell ctest $CTEST_FLAGS -L per_implementation -LE explicitonly &&
SIMDJSON_FORCE_IMPLEMENTATION=westmere ctest $CTEST_FLAGS -L per_implementation -LE explicitonly &&
SIMDJSON_FORCE_IMPLEMENTATION=fallback ctest $CTEST_FLAGS -L per_implementation -LE explicitonly &&
ctest $CTEST_FLAGS -LE "acceptance|per_implementation" # Everything we haven't run yet, run now.
cmake_perftest:
steps:
- cmake_build_cache
- run: |
cmake -DSIMDJSON_ENABLE_DOM_CHECKPERF=ON --build build --target checkperf &&
cd build &&
ctest --output-on-failure -R checkperf
# we not only want cmake to build and run tests, but we want also a successful installation from which we can build, link and run programs
cmake_install_test: # this version builds, install, test and then verify from the installation
steps:
- run: cd build && make install
- run: echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Ibuild/destination/include -Lbuild/destination/lib -std=c++17 -Wl,-rpath,build/destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json
cmake_installed_test_cxx20: # assuming that it was installed, this tries to build using C++20
steps:
- run: echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Ibuild/destination/include -Lbuild/destination/lib -std=c++20 -Wl,-rpath,build/destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json
jobs:
# static
justlib-gcc10:
description: Build just the library, install it and do a basic test
executor: gcc10
environment: { CMAKE_FLAGS: -DSIMDJSON_JUST_LIBRARY=ON }
steps: [ cmake_build, cmake_install_test, cmake_installed_test_cxx20 ]
assert-gcc10:
description: Build the library with asserts on, install it and run tests
executor: gcc10
environment: { CMAKE_FLAGS: -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DCMAKE_CXX_FLAGS_RELEASE=-O3 }
steps: [ cmake_test, cmake_assert_test ]
assert-clang10:
description: Build just the library, install it and do a basic test
executor: clang10
environment: { CMAKE_FLAGS: -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DCMAKE_CXX_FLAGS_RELEASE=-O3 }
steps: [ cmake_test, cmake_assert_test ]
gcc10-perftest:
description: Build and run performance tests on GCC 10 and AVX 2 with a cmake static build, this test performance regression
executor: gcc10
environment: { CMAKE_FLAGS: -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DBUILD_SHARED_LIBS=OFF }
steps: [ cmake_perftest ]
gcc10:
description: Build and run tests on GCC 10 and AVX 2 with a cmake static build
executor: gcc10
environment: { CMAKE_FLAGS: -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF }
steps: [ cmake_test, cmake_install_test, cmake_installed_test_cxx20 ]
clang6:
description: Build and run tests on clang 6 and AVX 2 with a cmake static build
executor: clang6
environment: { CMAKE_FLAGS: -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF }
steps: [ cmake_test, cmake_install_test ]
clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake static build
executor: clang10
environment: { CMAKE_FLAGS: -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF }
steps: [ cmake_test, cmake_install_test, cmake_installed_test_cxx20 ]
# libcpp
libcpp-clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake static build and libc++
executor: clang10
environment: { CMAKE_FLAGS: -DSIMDJSON_USE_LIBCPP=ON -DBUILD_SHARED_LIBS=OFF }
steps: [ cmake_test, cmake_install_test, cmake_installed_test_cxx20 ]
# sanitize
sanitize-gcc10:
description: Build and run tests on GCC 10 and AVX 2 with a cmake sanitize build
executor: gcc10
environment: { CMAKE_FLAGS: -DCMAKE_BUILD_TYPE=Debug -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
sanitize-clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake sanitize build
executor: clang10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
threadsanitize-gcc10:
description: Build and run tests on GCC 10 and AVX 2 with a cmake sanitize build
executor: gcc10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE_THREADS=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
threadsanitize-clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake sanitize build
executor: clang10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE_THREADS=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
# dynamic
dynamic-gcc10:
description: Build and run tests on GCC 10 and AVX 2 with a cmake dynamic build
executor: gcc10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON }
steps: [ cmake_test, cmake_install_test ]
dynamic-clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake dynamic build
executor: clang10
environment: { CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON }
steps: [ cmake_test, cmake_install_test ]
# unthreaded
unthreaded-gcc10:
description: Build and run tests on GCC 10 and AVX 2 *without* threads
executor: gcc10
environment: { CMAKE_FLAGS: -DSIMDJSON_ENABLE_THREADS=OFF }
steps: [ cmake_test, cmake_install_test ]
unthreaded-clang10:
description: Build and run tests on Clang 10 and AVX 2 *without* threads
executor: clang10
environment: { CMAKE_FLAGS: -DSIMDJSON_ENABLE_THREADS=OFF }
steps: [ cmake_test, cmake_install_test ]
# noexcept
noexcept-gcc10:
description: Build and run tests on GCC 10 and AVX 2 with exceptions off
executor: gcc10
environment: { CMAKE_FLAGS: -DSIMDJSON_EXCEPTIONS=OFF }
steps: [ cmake_test, cmake_install_test ]
noexcept-clang10:
description: Build and run tests on Clang 10 and AVX 2 with exceptions off
executor: clang10
environment: { CMAKE_FLAGS: -DSIMDJSON_EXCEPTIONS=OFF }
steps: [ cmake_test, cmake_install_test ]
#
# Misc.
#
# make (test and checkperf)
arch-haswell-gcc10:
description: Build, run tests and check performance on GCC 10 with -march=haswell
executor: gcc10
environment: { CXXFLAGS: -march=haswell }
steps: [ cmake_test ]
arch-nehalem-gcc10:
description: Build, run tests and check performance on GCC 10 with -march=nehalem
executor: gcc10
environment: { CXXFLAGS: -march=nehalem }
steps: [ cmake_test ]
sanitize-haswell-gcc10:
description: Build and run tests on GCC 10 and AVX 2 with a cmake sanitize build
executor: gcc10
environment: { CXXFLAGS: -march=haswell, CMAKE_FLAGS: -DCMAKE_BUILD_TYPE=Debug -DBUILD_SHARED_LIBS=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
sanitize-haswell-clang10:
description: Build and run tests on clang 10 and AVX 2 with a cmake sanitize build
executor: clang10
environment: { CXXFLAGS: -march=haswell, CMAKE_FLAGS: -DBUILD_SHARED_LIBS=ON -DSIMDJSON_NO_FORCE_INLINING=ON -DSIMDJSON_SANITIZE=ON, CTEST_FLAGS: --output-on-failure -LE explicitonly }
steps: [ cmake_test ]
workflows:
version: 2.1
build_and_test:
jobs:
# full multi-implementation tests
#- gcc7 tested on GitHub actions
- gcc10 # do not delete this as it tests our performance
- clang6
#- clang10 # this gets tested a lot below
# libc++
- libcpp-clang10
# full single-implementation tests
- sanitize-gcc10
- sanitize-clang10
- threadsanitize-gcc10
- threadsanitize-clang10
- dynamic-gcc10
- dynamic-clang10
- unthreaded-gcc10
- unthreaded-clang10
# no exceptions
- noexcept-gcc10
- noexcept-clang10
# quicker make single-implementation tests
- arch-haswell-gcc10
- arch-nehalem-gcc10
# sanitized single-implementation tests
- sanitize-haswell-gcc10
- sanitize-haswell-clang10
# testing "just the library"
- justlib-gcc10
# testing asserts
- assert-gcc10
- assert-clang10
# TODO add windows: https://circleci.com/docs/2.0/configuration-reference/#windows
+23 -1
View File
@@ -1,5 +1,8 @@
CompileFlags:
CompilationDatabase: build
Add:
- -Werror -Wall -Wextra -Weffc++ -Wsign-compare -Wshadow -Wwrite-strings -Wpointer-arith -Winit-self -Wconversion -Wno-sign-conversion
- -Wundefined-inline
Diagnostics:
Suppress:
- misc-unused-alias-decls
@@ -16,10 +19,29 @@ If:
CompileFlags:
Add:
- -Wno-unneeded-internal-declaration
- -Wno-undefined-inline # TODO fix and remove these violations
- -Wno-undefined-internal # TODO fix and remove these violations
- -Wno-unused-function
- -Wno-unused-const-variable
Diagnostics:
Suppress:
- pp_including_mainfile_in_preamble
- unused-includes
---
# Amalgamated files that require or partly define an implementation
If:
PathMatch:
- .*/(arm64|fallback|haswell|icelake|ppc64|westmere)/begin.h
- .*/generic/.*
Diagnostics:
Suppress:
- pragma_attribute_no_pop_eof
---
# clang has a bad time detecting the push/pop together in src/ for some reason
If:
PathMatch:
- include/simdjson/.*/end.h
- src/(arm64|fallback|haswell|icelake|ppc64|westmere).cpp
Diagnostics:
Suppress:
- pragma_attribute_no_pop_eof
- pragma_attribute_stack_mismatch
+11 -4
View File
@@ -9,7 +9,8 @@ assignees: ''
Before submitting an issue, please ensure that you have read the documentation:
* Basics is an overview of how to use simdjson and its APIs: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
* Basics is an overview of how to use simdjson and its APIs to parse JSON: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
* Builder is an overview of how to use simdjson to generate JSON: https://github.com/simdjson/simdjson/blob/master/doc/builder.md
* Performance shows some more advanced scenarios and how to tune for them: https://github.com/simdjson/simdjson/blob/master/doc/performance.md
* Contributing: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017). If you wish to support features that are not part of RFC 8259, then you should not refer to your issue as a bug.
@@ -37,7 +38,7 @@ If we cannot reproduce the issue, then we cannot address it. Note that a stack t
It should be possible to trigger the bug by using solely simdjson with our default build setup. If you can only observe the bug within some specific context, with some other software, please reduce the issue first.
**simjson release**
**simdjson release**
Unless you plan to contribute to simdjson, you should only work from releases. Please be mindful that our main branch may have additional features, bugs and documentation items.
@@ -45,11 +46,17 @@ It is fine to report bugs against our main branch, but if that is what you are d
**Configuration (please complete the following information if relevant)**
- OS: [e.g. Ubuntu 16.04.6 LTS]
- Compiler [e.g. Apple clang version 11.0.3 (clang-1103.0.32.59) x86_64-apple-darwin19.4.0]
- Compiler* [e.g. Apple clang version 11.0.3 (clang-1103.0.32.59) x86_64-apple-darwin19.4.0]
- Version [e.g. 22]
- Optimization setting (e.g., -O3)
We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux systems. Please ensure that your configuration is supported before labelling the issue as a bug. In particular, we do not support legacy 32-bit systems.
We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux systems. Please ensure that your configuration is supported before labelling the issue as a bug.
* We do not support unreleased or experimental compilers. If you encounter an issue with a
pre-release version of a compiler, do not report it as a bug to simdjson. However, we always
invite contributions either in the form an analysis or of a code contribution.
Under Windows, we support Visual Studio (both with LLVM and without). We do not support MinGW and other alternate compiler systems. Windows users should be aware that there [is a long-running bug with GCC under Windows](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412).
**Indicate whether you are willing or able to provide a bug fix as a pull request**
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: false
+3 -2
View File
@@ -9,7 +9,8 @@ assignees: ''
Before submitting an issue, please ensure that you have read the documentation:
* Basics is an overview of how to use simdjson and its APIs: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
* Basics is an overview of how to use simdjson and its APIs to parse JSON: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
* Builder is an overview of how to use simdjson to generate JSON: https://github.com/simdjson/simdjson/blob/master/doc/builder.md
* Performance shows some more advanced scenarios and how to tune for them: https://github.com/simdjson/simdjson/blob/master/doc/performance.md
* Contributing: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017).
@@ -31,7 +32,7 @@ A clear and concise description of any alternative solutions or features you've
**Additional context**
Add any other context or screenshots about the feature request here.
** Are you willing to contribute code or documentation toward this new feature? **
**Are you willing to contribute code or documentation toward this new feature?**
If you plan to contribute to simdjson, please read our
* CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
* HACKING guide: https://github.com/simdjson/simdjson/blob/master/HACKING.md
@@ -9,7 +9,8 @@ assignees: ''
Before submitting an issue, please ensure that you have read the documentation:
* Basics is an overview of how to use simdjson and its APIs: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
* Basics is an overview of how to use simdjson and its APIs to parse JSON: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
* Builder is an overview of how to use simdjson to generate JSON: https://github.com/simdjson/simdjson/blob/master/doc/builder.md
* Performance shows some more advanced scenarios and how to tune for them: https://github.com/simdjson/simdjson/blob/master/doc/performance.md
* Contributing: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017).
@@ -18,7 +19,7 @@ We do not make changes to simdjson without clearly identifiable benefits, which
Is your issue:
1. A bug report? If so, please point at a reproducible test. Indicate whether you are willing or able to provide a bug fix as a pull request.
1. A bug report? If so, please point at a reproducible test. Indicate whether you are willing or able to provide a bug fix as a pull request. As a matter of policy, we do not consider a compiler warning to be a bug.
2. A build issue? If so, provide all possible details regarding your system configuration. If we cannot reproduce your issue, we cannot fix it.
+50 -4
View File
@@ -1,8 +1,54 @@
Short title (summary):
Description
- What did you change and why? (1-3 sentences)
- Issue reproduced / related issue: link the issue if relevant (e.g. #123)
Type of change
- [ ] Bug fix
- [ ] Optimization
- [ ] New feature
- [ ] Refactor / cleanup
- [ ] Documentation / tests
- [ ] Other (please describe):
How to verify / test
- Add additional tests to verify bugs or new features.
- If you claim performance gains, you should provide benchmark numbers using high quality benchmarking code.
Our tests check whether you have introduced trailing white space. If such a test fails, please check the "artifacts button" above, which if you click it gives a link to a downloadable file to help you identify the issue. You can also run scripts/remove_trailing_whitespace.sh locally if you have a bash shell and the sed command available on your system.
Please read before contributing:
- CONTRIBUTING: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md
- HACKING: https://github.com/simdjson/simdjson/blob/master/HACKING.md
If you plan to contribute to simdjson, please read our
CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
HACKING guide: https://github.com/simdjson/simdjson/blob/master/HACKING.md
If you can, we recommend running our tests with the sanitizers turned on.
For non-Visual Studio users, it is as easy as doing:
```bash
cmake -B build -D SIMDJSON_SANITIZE=ON -D SIMDJSON_DEVELOPER_MODE=ON
cmake --build build
ctest --test-dir build
```
Our CI checks, among other things, for trailing whitespace. If a test fails for that reason,
use the "artifacts" button to download the artifact and inspect the problematic lines,
or run `scripts/remove_trailing_whitespace.sh` locally if you have a bash shell and `sed`.
Checklist before submitting
- [ ] I added/updated tests covering my change (if applicable)
- [ ] Code builds locally and passes my check
- [ ] Documentation / README updated if needed
- [ ] Commits are atomic and messages are clear
- [ ] I linked the related issue (if applicable)
Final notes
- For large PRs, prefer smaller incremental PRs or request staged review.
Thanks for the contribution!
+29
View File
@@ -0,0 +1,29 @@
name: Ubuntu aarch64 (GCC 13)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: uraimo/run-on-arch-action@v3
name: Test
id: runcmd
with:
arch: aarch64
distro: ubuntu_latest
githubToken: ${{ github.token }}
install: |
apt-get update -q -y
apt-get install -y cmake make g++
run: |
cmake -DSIMDJSON_SANITIZE_UNDEFINED=ON -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
cmake --build build -j=2
ctest --output-on-failure --test-dir build
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+1 -1
View File
@@ -17,7 +17,7 @@ jobs:
fuzz-seconds: 600
dry-run: false
- name: Upload Crash
uses: actions/upload-artifact@v1
uses: actions/upload-artifact@v4
if: failure() && steps.build.outcome == 'success'
with:
name: artifacts
+1 -1
View File
@@ -16,7 +16,7 @@ jobs:
image: debian:testing
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Install dependencies
run: |
+9 -4
View File
@@ -1,9 +1,14 @@
name: Doxygen GitHub Pages
on:
release:
# Trigger when a release object is created and when it's published.
# Some GitHub flows create a release object then publish it later; include both.
types: [created, published]
# Also trigger on tag creation pushes so releasing via Git tags still runs the workflow
push:
branches:
- master
tags:
- "v*" # common release tag pattern like v1.2.3
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
@@ -19,7 +24,7 @@ jobs:
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Install Doxygen
run: sudo apt-get install doxygen graphviz -y
- run: mkdir docs
@@ -28,7 +33,7 @@ jobs:
- name: Generate Doxygen Documentation
run: doxygen
- name: Deploy to GitHub Pages
uses: peaceiris/actions-gh-pages@v3
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: doc/api/html
+17
View File
@@ -0,0 +1,17 @@
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
- uses: mymindstorm/setup-emsdk@6ab9eb1bda2574c4ddb79809fc9247783eaf9021 # v14
- name: Verify
run: emcc -v
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v3.6.0
- name: Configure
run: emcmake cmake -B build
- name: Build # We build but do not test
run: cmake --build build
@@ -4,9 +4,9 @@ on: [push, pull_request]
jobs:
whitespace:
runs-on: ubuntu-20.04
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Remove whitespace and check the diff
run: |
set -eu
@@ -24,7 +24,7 @@ jobs:
echo "no trailing whitespace found, good!"
fi
- name: Archive whitespace patch
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
if: always()
with:
name: whitespace-patch
+9 -8
View File
@@ -20,11 +20,11 @@ jobs:
# fuzzers that change behaviour with SIMDJSON_FORCE_IMPLEMENTATION
defaultimplfuzzers: atpointer dump dump_raw_tape element minify parser print_json
# fuzzers that loop over the implementations themselves, or don't need to switch.
implfuzzers: implementations minifyimpl ndjson ondemand padded utf8 # TODO add singlestage
implfuzzers: implementations minifyimpl ndjson ondemand padded utf8
implementations: haswell westmere fallback
UBSAN_OPTIONS: halt_on_error=1
MAXLEN: -max_len=4000
CLANGVERSION: 15
CLANGVERSION: 19
# which optimization level to use for the sanitizer build (see build_fuzzer.variants.sh)
OPTLEVEL: -O3
@@ -34,17 +34,18 @@ jobs:
sudo apt update
sudo apt-get install --quiet ninja-build valgrind zip unzip lsb-release wget software-properties-common gnupg
wget https://apt.llvm.org/llvm.sh
sudo apt-get purge --auto-remove llvm python3-lldb-15 llvm-15
chmod +x llvm.sh
sudo ./llvm.sh $CLANGVERSION
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: actions/cache@v3
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- uses: actions/cache@v3
- uses: actions/cache@v4
id: cache-corpus
with:
path: out/
@@ -124,7 +125,7 @@ jobs:
done
- name: Save the corpus as a github artifact
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: corpus
path: corpus.tar
@@ -147,7 +148,7 @@ jobs:
run: tar cf valgrind.tar valgrind-*.txt
- name: Save valgrind output as a github artifact
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
if: always()
with:
name: valgrindresults
@@ -155,7 +156,7 @@ jobs:
if-no-files-found: ignore
- name: Archive any crashes as an artifact
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
if: always()
with:
name: crashes
+50
View File
@@ -0,0 +1,50 @@
name: LoongArch64-CI
on: [push, pull_request]
jobs:
loongarch64:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
platform:
- { toolchain-version: 2023.08.08 }
steps:
- uses: actions/checkout@v4
- name: Install build requirements
run: |
sudo apt-get update -y
sudo apt-get install -y --no-install-recommends cmake
- uses: actions/cache/restore@v4
id: restore-cache
with:
path: /opt/cross-tools
key: loongarch64-${{ matrix.platform.toolchain-version }}
- name: Download LoongArch64 gcc+glibc toolchain
if: ${{ !steps.restore-cache.outputs.cache-hit }}
run: |
url="https://github.com/loongson/build-tools/releases/download/${{ matrix.platform.toolchain-version }}/x86_64-cross-tools-loongarch64-gcc-libc.tar.xz"
wget "$url" -O /tmp/toolchain.tar.xz
mkdir -p /opt
tar -C /opt -x -f /tmp/toolchain.tar.xz
- uses: actions/cache/save@v3
if: ${{ !steps.restore-cache.outputs.cache-hit }}
with:
path: /opt/cross-tools
key: loongarch64-${{ matrix.platform.toolchain-version }}
- name: setup Loongarch64 build environment
run: |
echo "/opt/cross-tools/bin" >> $GITHUB_PATH
echo "CC=loongarch64-unknown-linux-gnu-gcc" >> $GITHUB_ENV
echo "CXX=loongarch64-unknown-linux-gnu-g++" >> $GITHUB_ENV
- name: configure
run: cmake -B build -DCMAKE_SYSTEM_PROCESSOR=loongarch64 -DARCH=lonngarch64 -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_C_COMPILER=loongarch64-unknown-linux-gnu-gcc -DCMAKE_CXX_COMPILER=loongarch64-unknown-linux-gnu-g++
- name: build
run: cmake --build build
+13 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
@@ -31,3 +31,14 @@ jobs:
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
cd ../tests/installation_tests/find &&
mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../build/destination .. && cmake --build .
- name: Use cmake (shared)
run: |
mkdir buildshared &&
cd buildshared &&
cmake -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
cd ../tests/installation_tests/find &&
mkdir buildshared && cd buildshared && cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../buildshared/destination .. && cmake --build .
+5 -2
View File
@@ -20,12 +20,15 @@ jobs:
- msystem: "MINGW64"
install: mingw-w64-x86_64-libxml2 mingw-w64-x86_64-cmake mingw-w64-x86_64-ninja mingw-w64-x86_64-clang
type: Debug
- msystem: "MINGW64"
install: mingw-w64-x86_64-libxml2 mingw-w64-x86_64-cmake mingw-w64-x86_64-ninja mingw-w64-x86_64-clang
type: RelWithDebInfo
env:
CMAKE_GENERATOR: Ninja
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+5 -2
View File
@@ -22,12 +22,15 @@ jobs:
- msystem: "MINGW64"
install: mingw-w64-x86_64-cmake mingw-w64-x86_64-ninja mingw-w64-x86_64-gcc
type: Debug
- msystem: "MINGW64"
install: mingw-w64-x86_64-cmake mingw-w64-x86_64-ninja mingw-w64-x86_64-gcc
type: RelWithDebInfo
env:
CMAKE_GENERATOR: Ninja
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
-71
View File
@@ -1,71 +0,0 @@
name: short fuzz on the power arch
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
armv7_job:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
# The host should always be Linux
runs-on: ubuntu-20.04
name: Build on ubuntu-20.04 ppc64le
steps:
- uses: actions/checkout@v3
- uses: uraimo/run-on-arch-action@v2.0.5
name: Run commands
id: runcmd
env:
DEBIAN_FRONTEND: noninteractive
with:
arch: ppc64le
distro: buster
# Not required, but speeds up builds by storing container images in
# a GitHub package registry.
githubToken: ${{ github.token }}
run: |
export CLANGSUFFIX="-7"
apt-get -qq update
apt-get install -q -y clang-7 libfuzzer-7-dev git wget zip ninja-build gnupg software-properties-common
wget -q -O - "https://raw.githubusercontent.com/simdjson/debian-ppa/master/key.gpg" | apt-key add -
apt-add-repository "deb https://raw.githubusercontent.com/simdjson/debian-ppa/master simdjson main"
apt-get -qq update
apt-get purge cmake cmake-data
apt-get -t simdjson -y install cmake
mkdir -p build ; cd build
cmake .. -GNinja \
-DCMAKE_CXX_COMPILER=clang++$CLANGSUFFIX \
-DCMAKE_C_COMPILER=clang$CLANGSUFFIX \
-DBUILD_SHARED_LIBS=OFF \
-DSIMDJSON_DEVELOPER_MODE=ON \
-DSIMDJSON_ENABLE_FUZZING=On \
-DSIMDJSON_COMPETITION=OFF \
-DSIMDJSON_GOOGLE_BENCHMARKS=OFF \
-DSIMDJSON_DISABLE_DEPRECATED_API=On \
-DSIMDJSON_FUZZ_LDFLAGS=-lFuzzer \
-DCMAKE_CXX_FLAGS="-fsanitize=fuzzer-no-link -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION=" \
-DCMAKE_C_FLAGS="-fsanitize=fuzzer-no-link" \
-DCMAKE_BUILD_TYPE=Release \
-DSIMDJSON_FUZZ_LINKMAIN=Off
cd ..
builddir=build
cmake --build $builddir
wget -O corpus.tar.gz https://readonly:readonly@www.pauldreik.se/fuzzdata/index.php?project=simdjson
tar xf corpus.tar.gz
fuzzernames=$(cmake --build $builddir --target print_all_fuzzernames |tail -n1)
for fuzzer in $fuzzernames ; do
exe=$builddir/fuzz/$fuzzer
shortname=$(echo $fuzzer |cut -f2- -d_)
echo found fuzzer $shortname with executable $exe
mkdir -p out/$shortname
others=$(find out -type d -not -name $shortname -not -name out -not -name cmin)
$exe -max_total_time=20 -max_len=4000 out/$shortname $others
echo "*************************************************************************"
done
echo "all is good, no errors found in any of these fuzzers: $fuzzernames"
+3 -3
View File
@@ -12,8 +12,8 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: uraimo/run-on-arch-action@v2
- uses: actions/checkout@v4
- uses: uraimo/run-on-arch-action@v3
name: Test
id: runcmd
with:
@@ -24,6 +24,6 @@ jobs:
apt-get update -q -y
apt-get install -y cmake make g++
run: |
cmake -DCMAKE_BUILD_TYPE=Release -B build
cmake -DCMAKE_BUILD_TYPE=Release -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
cmake --build build -j=2
ctest --output-on-failure --test-dir build
+29
View File
@@ -0,0 +1,29 @@
name: Ubuntu riscv64 (GCC 11)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: uraimo/run-on-arch-action@v3
name: Test
id: runcmd
with:
arch: riscv64
distro: ubuntu_latest
githubToken: ${{ github.token }}
install: |
apt-get update -q -y
apt-get install -y cmake make g++
run: |
cmake -DCMAKE_BUILD_TYPE=Release -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
cmake --build build -j=2
ctest --output-on-failure --test-dir build
+29
View File
@@ -0,0 +1,29 @@
name: Ubuntu rvv VLEN=1024 (clang 18)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Install packages
run: |
sudo apt-get update -q -y
sudo apt-get install -y cmake make g++-riscv64-linux-gnu qemu-user-static clang-18
- name: Build
run: |
CC=clang-18 CXX=clang++-18 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv_zvbb" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --build build/ -j$(nproc) --config Release
- name: Test VLEN=1024
run: |
QEMU_LD_PREFIX="/usr/riscv64-linux-gnu" \
QEMU_CPU="rv64,v=on,zvbb=on,vlen=1024,rvv_ta_all_1s=on,rvv_ma_all_1s=on" \
ctest --timeout 1800 --output-on-failure --test-dir build -j $(nproc)
+24
View File
@@ -0,0 +1,24 @@
name: Ubuntu rvv VLEN=128 (clang 17)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Install packages
run: |
sudo apt-get update -q -y
sudo apt-get install -y cmake make g++-riscv64-linux-gnu qemu-user-static clang-17
- name: Build
run: |
CC=clang-17 CXX=clang++-17 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --build build/ -j$(nproc) --config Release
+39
View File
@@ -0,0 +1,39 @@
name: Ubuntu rvv VLEN=128 (clang 20)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Install packages
run: |
sudo apt-get update -q -y
sudo apt-get install -y cmake make g++-riscv64-linux-gnu qemu-user-static clang-20
- name: Build
run: |
CC=clang-20 CXX=clang++-20 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --build build/ -j$(nproc) --config Release
- name: Test VLEN=128
run: |
QEMU_LD_PREFIX="/usr/riscv64-linux-gnu" \
QEMU_CPU="rv64,v=on,vlen=128,rvv_ta_all_1s=on,rvv_ma_all_1s=on" \
ctest --timeout 1800 --output-on-failure --test-dir build -j $(nproc)
- name: Build VLS
run: |
CC=clang-20 CXX=clang++-20 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv_zvl128b_zba_zbb_zbc -mrvv-vector-bits=zvl" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build-vls
cmake --build build-vls/ -j$(nproc) --config Release
- name: Test VLEN=128 VLS
run: |
QEMU_LD_PREFIX="/usr/riscv64-linux-gnu" \
QEMU_CPU="rv64,v=on,zba=on,zbb=on,zbc=on,vlen=128,rvv_ta_all_1s=on,rvv_ma_all_1s=on" \
ctest --timeout 1800 --output-on-failure --test-dir build-vls -j $(nproc)
+39
View File
@@ -0,0 +1,39 @@
name: Ubuntu rvv VLEN=256 (gcc 14)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Install packages
run: |
sudo apt-get update -q -y
sudo apt-get install -y cmake make g++-14-riscv64-linux-gnu qemu-user-static
- name: Build
run: |
CC=riscv64-linux-gnu-gcc-14 CXX=riscv64-linux-gnu-g++-14 CFLAGS=-march=rv64gcv CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --build build/ -j$(nproc) --config Release
- name: Test VLEN=256
run: |
QEMU_LD_PREFIX="/usr/riscv64-linux-gnu" \
QEMU_CPU="rv64,v=on,zvbb=on,vlen=256,rvv_ta_all_1s=on,rvv_ma_all_1s=on" \
ctest --timeout 1800 --output-on-failure --test-dir build -j $(nproc)
- name: Build VLS
run: |
CC=riscv64-linux-gnu-gcc-14 CXX=riscv64-linux-gnu-g++-14 CFLAGS="-march=rv64gcv_zvl256b -mrvv-vector-bits=zvl" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build-vls
cmake --build build-vls/ -j$(nproc) --config Release
- name: Test VLEN=256 VLS
run: |
QEMU_LD_PREFIX="/usr/riscv64-linux-gnu" \
QEMU_CPU="rv64,v=on,zvbb=on,vlen=256,rvv_ta_all_1s=on,rvv_ma_all_1s=on" \
ctest --timeout 1800 --output-on-failure --test-dir build-vls -j $(nproc)
+39
View File
@@ -0,0 +1,39 @@
name: Ubuntu rvv VLEN=512 (clang 19)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Install packages
run: |
sudo apt-get update -q -y
sudo apt-get install -y cmake make g++-riscv64-linux-gnu qemu-user-static clang-19
- name: Build
run: |
CC=clang-19 CXX=clang++-19 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --build build/ -j$(nproc) --config Release
- name: Test VLEN=512
run: |
QEMU_LD_PREFIX="/usr/riscv64-linux-gnu" \
QEMU_CPU="rv64,v=on,vlen=512,rvv_ta_all_1s=on,rvv_ma_all_1s=on" \
ctest --timeout 1800 --output-on-failure --test-dir build -j $(nproc)
- name: Build VLS
run: |
CC=clang-19 CXX=clang++-19 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv_zvl512b_zba_zbb_zbc -mrvv-vector-bits=zvl" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build-vls
cmake --build build-vls/ -j$(nproc) --config Release
- name: Test VLEN=512 VLS
run: |
QEMU_LD_PREFIX="/usr/riscv64-linux-gnu" \
QEMU_CPU="rv64,v=on,zba=on,zbb=on,zbc=on,vlen=512,rvv_ta_all_1s=on,rvv_ma_all_1s=on" \
ctest --timeout 1800 --output-on-failure --test-dir build-vls -j $(nproc)
+3 -3
View File
@@ -12,8 +12,8 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: uraimo/run-on-arch-action@v2
- uses: actions/checkout@v4
- uses: uraimo/run-on-arch-action@v3
name: Test
id: runcmd
with:
@@ -24,6 +24,6 @@ jobs:
apt-get update -q -y
apt-get install -y cmake make g++
run: |
cmake -DCMAKE_BUILD_TYPE=Release -B build
cmake -DCMAKE_BUILD_TYPE=Release -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
cmake --build build -j=2
ctest --output-on-failure --test-dir build
-38
View File
@@ -1,38 +0,0 @@
name: Ubuntu 20.04 CI (GCC 8)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
env:
CXX: g++-8
CC: gcc-8
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install GCC 8
run: sudo apt-get install -y g++-8
- name: Use cmake
run: |
mkdir builddebug &&
cd builddebug &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cd .. &&
mkdir build &&
cd build &&
cmake -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
cd ../tests/installation_tests/find &&
mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../build/destination .. && cmake --build .
-33
View File
@@ -1,33 +0,0 @@
name: Ubuntu 20.04 CI (GCC 9)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake
run: |
mkdir builddebug &&
cd builddebug &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cd .. &&
mkdir build &&
cd build &&
cmake -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
cd ../tests/installation_tests/find &&
mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../build/destination .. && cmake --build .
+2 -4
View File
@@ -9,13 +9,11 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install clang++-13
run: sudo apt-get install -y clang++-13
- name: Use cmake
run: |
mkdir build &&
+2 -4
View File
@@ -9,13 +9,11 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install clang++-14
run: sudo apt-get install -y clang++-14
- name: Use cmake
run: |
mkdir build &&
+50 -18
View File
@@ -9,25 +9,57 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake
- name: Configure Debug Build
run: |
mkdir builddebug &&
cd builddebug &&
CXX=g++-12 cmake -DSIMDJSON_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cd .. &&
mkdir build &&
cd build &&
CXX=g++-12 cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
cd ../tests/installation_tests/find &&
mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../build/destination .. && cmake --build .
CXX=g++-12 cmake -DSIMDJSON_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -B builddebug
- name: Compile Debug Build
run: |
cmake --build builddebug
- name: Test Debug Build
run: |
ctest --output-on-failure -LE explicitonly -j --test-dir builddebug
- name: Configure Release Build
run: |
CXX=g++-12 cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination -B build
- name: Compile Release Build
run: |
cmake --build build
- name: Test Release Build
run: |
ctest --output-on-failure -LE explicitonly -j --test-dir build
- name: Install Release Build
run: |
cmake --install build
- name: Generate Example Code
run: |
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp
- name: Compile Example Code
run: |
c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson
- name: Run Example Code
run: |
./linkandrun jsonexamples/twitter.json
- name: Configure Find Tests
run: |
cd tests/installation_tests/find && \
cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../destination -B build
- name: Compile Find Tests
run: |
cd tests/installation_tests/find && cmake --build build
+2 -4
View File
@@ -9,13 +9,11 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install gcc12
run: sudo apt-get install -y g++-12
- name: Use cmake
run: |
mkdir build &&
@@ -8,8 +8,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
@@ -1,4 +1,4 @@
name: Ubuntu 20.04 CI (GCC 9) with Thread Sanitizer
name: Ubuntu 22.04 CI (GCC 12) with Thread Sanitizer
on: [push, pull_request]
@@ -7,10 +7,10 @@ jobs:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
@@ -18,7 +18,7 @@ jobs:
run: |
mkdir build &&
cd build &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE_THREADS=ON .. &&
CXX=g++-12 cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE_THREADS=ON .. &&
cmake --build . --target document_stream_tests --target ondemand_document_stream_tests --target parse_many_test &&
ctest --output-on-failure -R parse_many_test &&
ctest --output-on-failure -R document_stream_tests
+16 -2
View File
@@ -9,11 +9,25 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake to build just the library
run: |
mkdir buildjustlib &&
cd buildjustlib &&
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DSIMDJSON_DEVELOPER_MODE=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp &&
c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson &&
cd ../tests/installation_tests/find &&
mkdir buildjustlib &&
cd buildjustlib &&
cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../buildjustlib/destination .. &&
cmake --build .
- name: Use cmake
run: |
mkdir builddebug &&
@@ -13,10 +13,10 @@ jobs:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
@@ -0,0 +1,22 @@
name: Ubuntu 24.04 CI (CXX 20, noexcept)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-24.04
strategy:
matrix:
cxx: [g++-13, clang++-16]
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- name: Prepare
run: cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_EXCEPTIONS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -B build
env:
CXX: ${{matrix.cxx}}
- name: Build
run: cmake --build build -j=2
- name: Test
run: ctest --output-on-failure --test-dir build
+22
View File
@@ -0,0 +1,22 @@
name: Ubuntu 24.04 CI (CXX 20)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-24.04
strategy:
matrix:
cxx: [g++-13, clang++-16]
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- name: Prepare
run: cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_DEVELOPER_MODE=ON -B build
env:
CXX: ${{matrix.cxx}}
- name: Build
run: cmake --build build -j=2
- name: Test
run: ctest --output-on-failure --test-dir build
@@ -7,10 +7,10 @@ jobs:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
@@ -31,4 +31,4 @@ jobs:
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
mkdir testfindpackage &&
cd testfindpackage &&
echo -e 'cmake_minimum_required(VERSION 3.1)\nproject(simdjsontester)\nset(CMAKE_CXX_STANDARD 17)\nfind_package(simdjson REQUIRED)'> CMakeLists.txt && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../destination .. && cmake --build .
echo -e 'cmake_minimum_required(VERSION 3.14)\nproject(simdjsontester)\nset(CMAKE_CXX_STANDARD 17)\nfind_package(simdjson REQUIRED)'> CMakeLists.txt && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../destination .. && cmake --build .
@@ -7,10 +7,10 @@ jobs:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
@@ -31,4 +31,4 @@ jobs:
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
mkdir testfindpackage &&
cd testfindpackage &&
echo -e 'cmake_minimum_required(VERSION 3.1)\nproject(simdjsontester)\nset(CMAKE_CXX_STANDARD 17)\nfind_package(simdjson REQUIRED)'> CMakeLists.txt && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../destination .. && cmake --build .
echo -e 'cmake_minimum_required(VERSION 3.14)\nproject(simdjsontester)\nset(CMAKE_CXX_STANDARD 17)\nfind_package(simdjson REQUIRED)'> CMakeLists.txt && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../destination .. && cmake --build .
@@ -3,14 +3,14 @@ name: Ubuntu 20.04 CI (GCC 9) With Memory Sanitizer
on: [push, pull_request]
jobs:
ubuntu-build:
ubuntu-build-address-sanitizier:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
@@ -21,6 +21,17 @@ jobs:
cmake -DSIMDJSON_SANITIZE=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j
ubuntu-build-undefined-sanitizer:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake with undefined sanitizer
run: |
mkdir builddebugundefsani &&
+25
View File
@@ -0,0 +1,25 @@
name: Ubuntu 24.04 CI
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-24.04
strategy:
matrix:
shared: [ON, OFF]
cxx: [g++-13, clang++-16]
sanitizer: [ON, OFF]
build_type: [RelWithDebInfo, Debug, Release]
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- name: Prepare
run: cmake -DCMAKE_BUILD_TYPE=${{matrix.build_type}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE=${{matrix.sanitizer}} -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
env:
CXX: ${{matrix.cxx}}
- name: Build
run: cmake --build build -j=2
- name: Test
run: ctest --output-on-failure --test-dir build
+4 -4
View File
@@ -10,12 +10,12 @@ jobs:
fail-fast: false
matrix:
include:
- {arch: ARM}
- {arch: ARM64}
- {arch: ARM64EC}
steps:
- name: checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Use cmake
run: |
cmake -A ${{ matrix.arch }} -DCMAKE_CROSSCOMPILING=1 -DSIMDJSON_DEVELOPER_MODE=ON -D SIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_EXCEPTIONS=OFF -B build &&
cmake --build build --verbose
cmake -A ${{ matrix.arch }} -DCMAKE_SYSTEM_VERSION="10.0.22621.0" -DCMAKE_CROSSCOMPILING=1 -DSIMDJSON_DEVELOPER_MODE=ON -D SIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_EXCEPTIONS=OFF -B build &&
cmake --build build --verbose
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF}
steps:
- name: checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Configure
run: |
cmake -DSIMDJSON_CXX_STANDARD=20 -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
+30
View File
@@ -0,0 +1,30 @@
name: VS17-CI-SANITIZE
on: [push, pull_request]
jobs:
ci:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
name: windows-vs17
runs-on: windows-latest
strategy:
fail-fast: false
matrix:
include:
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF, build_type: Debug}
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF, build_type: Release}
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF, build_type: RelWithDebInfo}
steps:
- name: checkout
uses: actions/checkout@v4
- name: Configure
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSANITIZE=ON -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
- name: Build
run: cmake --build build --config ${{matrix.build_type}} --verbose
- name: Run tests
run: |
cd build
ctest -C ${{matrix.build_type}} -LE explicitonly --output-on-failure
+12 -16
View File
@@ -13,32 +13,28 @@ jobs:
fail-fast: false
matrix:
include:
- {gen: Visual Studio 17 2022, arch: Win32, shared: ON}
- {gen: Visual Studio 17 2022, arch: Win32, shared: OFF}
- {gen: Visual Studio 17 2022, arch: x64, shared: ON}
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF}
- {gen: Visual Studio 17 2022, arch: Win32, shared: ON, build_type: Release}
- {gen: Visual Studio 17 2022, arch: Win32, shared: OFF, build_type: Release}
- {gen: Visual Studio 17 2022, arch: x64, shared: ON, build_type: Release}
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF, build_type: Debug}
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF, build_type: Release}
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF, build_type: RelWithDebInfo}
steps:
- name: checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Configure
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
- name: Build Debug
run: cmake --build build --config Debug --verbose
- name: Build Release
run: cmake --build build --config Release --verbose
- name: Run Release tests
run: cmake --build build --config ${{matrix.build_type}} --verbose
- name: Run tests
run: |
cd build
ctest -C Release -LE explicitonly --output-on-failure
- name: Run Debug tests
run: |
cd build
ctest -C Debug -LE explicitonly --output-on-failure
ctest -C ${{matrix.build_type}} -LE explicitonly --output-on-failure
- name: Install
run: |
cmake --install build --config Release
cmake --install build --config ${{matrix.build_type}}
- name: Test Installation
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -B build_install_test tests/installation_tests/find
cmake --build build_install_test --config Release
cmake --build build_install_test --config ${{matrix.build_type}}
+37
View File
@@ -0,0 +1,37 @@
name: VS17-CLANG-CI
on: [push, pull_request]
jobs:
ci:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
name: windows-vs17
runs-on: windows-latest
strategy:
fail-fast: false
matrix:
include:
- {gen: Visual Studio 17 2022, arch: x64, build_type: Debug}
- {gen: Visual Studio 17 2022, arch: x64, build_type: Release}
- {gen: Visual Studio 17 2022, arch: x64, build_type: RelWithDebInfo}
steps:
- name: checkout
uses: actions/checkout@v4
- name: Configure
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -T ClangCL -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
- name: Build
run: cmake --build build --config ${{matrix.build_type}} --verbose
- name: Run tests
run: |
cd build
ctest -C ${{matrix.build_type}} -LE explicitonly --output-on-failure
- name: Install
run: |
cmake --install build --config ${{matrix.build_type}}
- name: Test Installation
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -B build_install_test tests/installation_tests/find
cmake --build build_install_test --config ${{matrix.build_type}}
+10 -14
View File
@@ -13,29 +13,25 @@ jobs:
fail-fast: false
matrix:
include:
- {gen: Visual Studio 17 2022, arch: x64}
- {gen: Visual Studio 17 2022, arch: x64, build_type: Debug}
- {gen: Visual Studio 17 2022, arch: x64, build_type: Release}
- {gen: Visual Studio 17 2022, arch: x64, build_type: RelWithDebInfo}
steps:
- name: checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Configure
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -T ClangCL -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
- name: Build Debug
run: cmake --build build --config Debug --verbose
- name: Build Release
run: cmake --build build --config Release --verbose
- name: Run Release tests
- name: Build
run: cmake --build build --config ${{matrix.build_type}} --verbose
- name: Run tests
run: |
cd build
ctest -C Release -LE explicitonly --output-on-failure
- name: Run Debug tests
run: |
cd build
ctest -C Debug -LE explicitonly --output-on-failure
ctest -C ${{matrix.build_type}} -LE explicitonly --output-on-failure
- name: Install
run: |
cmake --install build --config Release
cmake --install build --config ${{matrix.build_type}}
- name: Test Installation
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -B build_install_test tests/installation_tests/find
cmake --build build_install_test --config Release
cmake --build build_install_test --config ${{matrix.build_type}}
+2 -2
View File
@@ -7,8 +7,8 @@ jobs:
name: windows-vs17
runs-on: windows-latest
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -38,7 +38,7 @@ cmake-build-release/
.history/
# Visual Studio artifacts
/VS/
/.vs/
# C/C++ build outputs
.build/
@@ -106,4 +106,4 @@ objs
!.vscode/extensions.json
# clangd
.cache
.cache
+32 -1
View File
@@ -3,6 +3,9 @@
{"column": 95 },
{"column": 120 }
],
"cmake.configureArgs": [
"-DSIMDJSON_DEVELOPER_MODE=ON"
],
"files.trimTrailingWhitespace": true,
"files.associations": {
".clangd": "yaml",
@@ -98,6 +101,34 @@
"queue": "cpp",
"shared_mutex": "cpp",
"ranges": "cpp",
"span": "cpp"
"span": "cpp",
"__verbose_abort": "cpp",
"charconv": "cpp",
"source_location": "cpp",
"strstream": "cpp",
"typeindex": "cpp",
"*.tcc": "cpp",
"memory_resource": "cpp",
"numbers": "cpp",
"semaphore": "cpp",
"stop_token": "cpp",
"cfenv": "cpp",
"format": "cpp",
"xlocmes": "cpp",
"xlocmon": "cpp",
"xlocnum": "cpp",
"xloctime": "cpp",
"xutility": "cpp",
"coroutine": "cpp",
"xfacet": "cpp",
"xhash": "cpp",
"xiosbase": "cpp",
"xlocale": "cpp",
"xlocbuf": "cpp",
"xlocinfo": "cpp",
"xmemory": "cpp",
"xstring": "cpp",
"xtr1common": "cpp",
"xtree": "cpp"
}
}
+128 -14
View File
@@ -1,9 +1,18 @@
cmake_minimum_required(VERSION 3.14)
# Build performance optimizations
set(CMAKE_EXPORT_COMPILE_COMMANDS ON CACHE BOOL "Export compile commands for faster IDE integration")
set_property(GLOBAL PROPERTY USE_FOLDERS ON)
# Enable parallel compilation on MSVC
if(MSVC)
add_compile_options(/MP)
endif()
project(
simdjson
# The version number is modified by tools/release.py
VERSION 3.2.1
VERSION 4.2.4
DESCRIPTION "Parsing gigabytes of JSON per second"
HOMEPAGE_URL "https://simdjson.org/"
LANGUAGES CXX C
@@ -20,8 +29,14 @@ string(
# ---- Options, variables ----
# These version numbers are modified by tools/release.py
set(SIMDJSON_LIB_VERSION "16.0.0" CACHE STRING "simdjson library version")
set(SIMDJSON_LIB_SOVERSION "16" CACHE STRING "simdjson library soversion")
set(SIMDJSON_LIB_VERSION "29.0.0" CACHE STRING "simdjson library version")
set(SIMDJSON_LIB_SOVERSION "29" CACHE STRING "simdjson library soversion")
option(SIMDJSON_BUILD_STATIC_LIB "Build simdjson_static library along with simdjson (only makes sense if BUILD_SHARED_LIBS=ON)" OFF)
if(SIMDJSON_BUILD_STATIC_LIB AND NOT BUILD_SHARED_LIBS)
message(WARNING "SIMDJSON_BUILD_STATIC_LIB only makes sense if BUILD_SHARED_LIBS is set to ON")
message(WARNING "You might be building and installing a two identical static libraries.")
endif()
option(SIMDJSON_ENABLE_THREADS "Link with thread support" ON)
@@ -37,6 +52,20 @@ if(SIMDJSON_DISABLE_DEPRECATED_API)
)
endif()
if(${CMAKE_VERSION} VERSION_GREATER_EQUAL "3.25.0")
option(SIMDJSON_STATIC_REFLECTION "Enables static reflection (experimental), requires C++26" OFF)
else()
set(SIMDJSON_STATIC_REFLECTION OFF CACHE BOOL "Enables static reflection (experimental)" FORCE)
message(WARNING "SIMDJSON_STATIC_REFLECTION is disabled because your CMake version is below 3.25")
endif()
if(SIMDJSON_STATIC_REFLECTION)
simdjson_add_props(
target_compile_definitions PUBLIC
SIMDJSON_STATIC_REFLECTION=1
)
endif()
option(SIMDJSON_DEVELOPMENT_CHECKS "Enable development-time aids, such as \
checks for incorrect API usage. Enabled by default in DEBUG." OFF)
if(SIMDJSON_DEVELOPMENT_CHECKS)
@@ -49,6 +78,7 @@ endif()
if(is_top_project)
option(SIMDJSON_DEVELOPER_MODE "Enable targets for developing simdjson" OFF)
option(BUILD_SHARED_LIBS "Build simdjson as a shared library" OFF)
option(SIMDJSON_SINGLEHEADER "Disable singleheader generation" ON)
endif()
include(cmake/handle-deprecations.cmake)
@@ -56,8 +86,43 @@ include(cmake/developer-options.cmake)
# ---- simdjson library ----
add_library(simdjson src/simdjson.cpp)
set(SIMDJSON_SOURCES src/simdjson.cpp)
add_library(simdjson ${SIMDJSON_SOURCES})
add_library(simdjson::simdjson ALIAS simdjson)
set(SIMDJSON_LIBRARIES simdjson)
# Enable precompiled headers for faster builds
if(CMAKE_VERSION VERSION_GREATER_EQUAL "3.16")
target_precompile_headers(simdjson PRIVATE
<algorithm>
<array>
<atomic>
<bit>
<cassert>
<cctype>
<cerrno>
<cstddef>
<cstdint>
<cstdlib>
<cstring>
<memory>
<string>
<utility>
<vector>
)
endif()
if(SIMDJSON_BUILD_STATIC_LIB)
add_library(simdjson_static STATIC ${SIMDJSON_SOURCES})
add_library(simdjson::simdjson_static ALIAS simdjson_static)
list(APPEND SIMDJSON_LIBRARIES simdjson_static)
# Reuse precompiled headers for static library
if(CMAKE_VERSION VERSION_GREATER_EQUAL "3.16")
target_precompile_headers(simdjson_static REUSE_FROM simdjson)
endif()
endif()
set_target_properties(
simdjson PROPERTIES
@@ -68,7 +133,7 @@ set_target_properties(
)
# FIXME: Use proper CMake integration for exports
if(MSVC AND BUILD_SHARED_LIBS)
if(WIN32 AND BUILD_SHARED_LIBS)
target_compile_definitions(
simdjson
PRIVATE SIMDJSON_BUILDING_WINDOWS_DYNAMIC_LIBRARY=1
@@ -82,7 +147,24 @@ simdjson_add_props(
PRIVATE "$<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}/src>"
)
simdjson_add_props(target_compile_features PUBLIC cxx_std_11)
# Optimize linker settings for faster builds
if(MSVC)
target_link_options(simdjson PRIVATE /INCREMENTAL)
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
target_link_options(simdjson PRIVATE /DEBUG:FASTLINK)
endif()
endif()
if(SIMDJSON_STATIC_REFLECTION)
# We would like to require C++26, but no compiler supports that!
# This is a hack:
simdjson_add_props(
target_compile_options PUBLIC
-freflection -fexpansion-statements -stdlib=libc++ -std=c++26
)
else()
simdjson_add_props(target_compile_features PUBLIC cxx_std_11)
endif()
# workaround for GNU GCC poor AVX load/store code generation
if(
@@ -95,6 +177,12 @@ if(
)
endif()
option(SIMDJSON_MINUS_ZERO_AS_FLOAT "Treat -0 as a floating-point value" OFF)
if(SIMDJSON_MINUS_ZERO_AS_FLOAT)
simdjson_add_props(target_compile_definitions PRIVATE SIMDJSON_MINUS_ZERO_AS_FLOAT=1)
endif(SIMDJSON_MINUS_ZERO_AS_FLOAT)
# GCC and Clang have horrendous Debug builds when using SIMD.
# A common fix is to use '-Og' instead.
# bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
@@ -108,7 +196,12 @@ if(
target_compile_options PRIVATE
$<$<CONFIG:DEBUG>:-Og>
)
endif()
# We still want to enable development checks in Debug mode
simdjson_add_props(
target_compile_definitions PUBLIC
SIMDJSON_DEVELOPMENT_CHECKS
)
endif()
if(SIMDJSON_ENABLE_THREADS)
find_package(Threads REQUIRED)
@@ -117,17 +210,22 @@ if(SIMDJSON_ENABLE_THREADS)
endif()
simdjson_apply_props(simdjson)
if(SIMDJSON_BUILD_STATIC_LIB)
simdjson_apply_props(simdjson_static)
endif()
# ---- Install rules ----
include(CMakePackageConfigHelpers)
include(GNUInstallDirs)
install(
FILES singleheader/simdjson.h
DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
COMPONENT simdjson_Development
)
if(SIMDJSON_SINGLEHEADER)
install(
FILES singleheader/simdjson.h
DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
COMPONENT simdjson_Development
)
endif()
install(
TARGETS simdjson
@@ -138,7 +236,6 @@ install(
ARCHIVE COMPONENT simdjson_Development
INCLUDES DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
)
configure_file(cmake/simdjson-config.cmake.in simdjson-config.cmake @ONLY)
write_basic_package_version_file(
@@ -167,6 +264,21 @@ install(
COMPONENT simdjson_Development
)
if(SIMDJSON_BUILD_STATIC_LIB)
install(
TARGETS simdjson_static
EXPORT simdjson_staticTargets
ARCHIVE COMPONENT simdjson_Development
INCLUDES DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
)
install(
EXPORT simdjson_staticTargets
NAMESPACE simdjson::
DESTINATION "${SIMDJSON_INSTALL_CMAKEDIR}"
COMPONENT simdjson_Development
)
endif()
# pkg-config
include(cmake/JoinPaths.cmake)
join_paths(PKGCONFIG_INCLUDEDIR "\${prefix}" "${CMAKE_INSTALL_INCLUDEDIR}")
@@ -234,6 +346,7 @@ enable_testing()
add_custom_target(all_tests)
add_subdirectory(windows)
include(cmake/CPM.cmake)
add_subdirectory(dependencies) ## This needs to be before tools because of cxxopts
add_subdirectory(tools) ## This needs to be before tests because of cxxopts
@@ -241,8 +354,9 @@ add_subdirectory(tools) ## This needs to be before tests because of cxxopts
# most of the data has been moved to https://github.com/simdjson/simdjson-data
add_subdirectory(jsonexamples)
if(SIMDJSON_SINGLEHEADER)
add_subdirectory(singleheader)
endif()
+3 -2
View File
@@ -52,10 +52,11 @@ General Guidelines
Contributors are encouraged to :
- Document their changes. Though we do not enforce a rule regarding code comments, we prefer that non-trivial algorithms and techniques be somewhat documented in the code.
- Follow as much as possible the existing code style. We do not enforce a specific code style, but we prefer consistency.
- Follow as much as possible the existing code style. We do not enforce a specific code style, but we prefer consistency. We avoid contractions (isn't, aren't) in the comments.
- Modify as few lines of code as possible when working on an issue. The more lines you modify, the harder it is for your fellow human beings to understand what is going on.
- Tools may report "problems" with the code, but we never delegate programming to tools: if there is a problem with the code, we need to understand it. Thus we will not "fix" code merely to please a static analyzer.
- Provide tests for any new feature. We will not merge a new feature without tests.
- Run before/after benchmarks so that we can appreciate the effect of the changes on the performance.
Pull Requests
--------------
@@ -91,7 +92,7 @@ We welcome contributions from women and less represented groups. If you need hel
Consider the following points when engaging with the project:
- We discourage arguments from authority: ideas are discusssed on their own merits and not based on who stated it.
- We discourage arguments from authority: ideas are discussed on their own merits and not based on who stated it.
- Be mindful that what you may view as an aggression is maybe merely a difference of opinion or a misunderstanding.
- Be mindful that a collection of small aggressions, even if mild in isolation, can become harmful.
+1 -1
View File
@@ -38,7 +38,7 @@ PROJECT_NAME = simdjson
# could be handy for archiving the generated documentation or if some version
# control system is used.
PROJECT_NUMBER = "3.2.1"
PROJECT_NUMBER = "4.2.4"
# Using the PROJECT_BRIEF tag one can provide an optional one line description
# for a project that appears at the top of each page and should give viewer a
+77 -18
View File
@@ -20,13 +20,54 @@ If you plan to contribute to simdjson, please read our [CONTRIBUTING](https://gi
Build Quickstart
------------------------------
For non-Windows system,
```bash
mkdir build
cd build
cmake -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build .
cmake -B -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build build
ctest --test-dir build
```
It is similar for Visual Studio users, please see the CMake or Visual Studio documentation.
By default the library is built in Release mode.
Assertions and development checks
------------------------------
We do not use conventional `assert` in simdjson. Instead we use the macro
`SIMDJSON_ASSUME`:
```cpp
SIMDJSON_ASSUME(something_that_is_true());
```
Sometimes, you need to do a bit more work that a simple check.
The `SIMDJSON_DEVELOPMENT_CHECKS` macro is true only in Debug mode unless manually set.
It is acceptable to add checks that you would not do in Release mode as long as
they are guarded:
```cpp
#if SIMDJSON_DEVELOPMENT_CHECKS
// do sanity checks here
```
Working with sanitizers
------------------------------
The simdjson library must be memory-safe. We cannot allow buffer overruns.
During development, if you system supports it, we recommend configuring
the project with `-D SIMDJSON_SANITIZE=ON`.
```bash
cmake -B -D SIMDJSON_SANITIZE=ON -D SIMDJSON_DEVELOPER_MODE=ON ..
cmake --build build
ctest --test-dir build
```
Design notes
------------------------------
@@ -69,6 +110,24 @@ workflows used by simdjson.
Directory Structure and Source
------------------------------
Before diving into the directory structure, here are key concepts used in the codebase:
- **Amalgamated File**: A file that is conditionally included in the amalgamation process. These are wrapped in `#ifndef SIMDJSON_CONDITIONAL_INCLUDE` blocks and are included based on the target implementation (e.g., ARM64, x86). They include implementation-specific files (e.g., `arm64.h`) and generic files (e.g., under `generic/`). Amalgamated files have associated dependency files (`dependencies.h`) to track includes.
- **Amalgamator File**: A file that orchestrates the inclusion of amalgamated files. Examples: `arm64.h`, `arm64/implementation.h`, `generic/amalgamated.h`. These are not themselves amalgamated but control conditional inclusions.
- **Free Dependency File**: A top-level header that is always included unconditionally. These do not have dependency files and represent the public API (e.g., main headers).
- **Implementation-Specific File**: A file tied to a specific CPU architecture or instruction set (e.g., `arm64/`, `haswell/`). These must be amalgamated.
- **Generic File**: A shared file (under `generic/` or `simdjson/generic/`) that contains common code included once per implementation.
- **Builtin File**: Special files under `simdjson/builtin/` that handle the builtin implementation, a fallback/default implementation used when no optimized implementation is available.
- **Conditional Include Block**: A section wrapped in `#ifndef SIMDJSON_CONDITIONAL_INCLUDE` for editor-only or implementation-specific content.
The script `singleheader/amalgation_helper.py` will generate an HTML report which you can use to visualize the status of each file.
simdjson's source structure, from the top level, looks like this:
* **CMakeLists.txt:** The main build system.
@@ -80,7 +139,7 @@ simdjson's source structure, from the top level, looks like this:
* simdjson/internal/*.h: the `simdjson::internal` namespace. Private classes and functions used by the rest of simdjson.
* simdjson/dom.h: the `simdjson::dom` namespace. Includes all public DOM classes.
* simdjson/dom/*.h: Declarations/definitions for individual DOM classes.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere.h: `simdjson::<implementation>` namesapce. Common implementation-specific tools like number and string parsing, as well as minification.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere.h: `simdjson::<implementation>` namespace. Common implementation-specific tools like number and string parsing, as well as minification.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/*.h: implementation-specific functions such as , etc.
* simdjson/generic/*.h: the bulk of the actual code, written generically and compiled for each implementation, using functions defined in the implementation's .h files.
* simdjson/generic/dependencies.h: dependencies on common, non-implementation-specific simdjson classes. This will be included before including amalgamated.h.
@@ -88,10 +147,16 @@ simdjson's source structure, from the top level, looks like this:
* simdjson/ondemand.h: the `simdjson::ondemand` namespace. Includes all public ondemand classes.
* simdjson/builtin.h: the `simdjson::builtin` namespace. Aliased to the most universal implementation available.
* simdjson/builtin/ondemand.h: the `simdjson::builtin::ondemand` namespace.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/ondemand.h: the `simdjson::<implementation>::ondemand` namespace. on demand compiled for the specific implementation.
* simdjson/generic/ondemand/*.h: individual on demand classes, generically written.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/ondemand.h: the `simdjson::<implementation>::ondemand` namespace. On-Demand compiled for the specific implementation.
* simdjson/generic/ondemand/*.h: individual On-Demand classes, generically written.
* simdjson/generic/ondemand/dependencies.h: dependencies on common, non-implementation-specific simdjson classes. This will be included before including amalgamated.h.
* simdjson/generic/ondemand/amalgamated.h: all generic ondemand classes for an implementation.
* simdjson/builder.h: the `simdjson::builder` namespace. Includes all public builder classes.
* simdjson/builtin/builder.h: the `simdjson::builtin::builder` namespace.
* simdjson/arm64|fallback|haswell|icelake|ppc64|westmere/builder.h: the `simdjson::<implementation>::builder` namespace. Builder compiled for the specific implementation.
* simdjson/generic/builder/*.h: individual Builder classes, generically written.
* simdjson/generic/builder/dependencies.h: dependencies on common, non-implementation-specific simdjson classes. This will be included before including amalgamated.h.
* simdjson/generic/builder/amalgamated.h: all generic builder classes for an implementation.
* **src:** The source files for non-inlined functionality (e.g. the architecture-specific parser
implementations).
* simdjson.cpp: A "main source" that includes all implementation files from src/. This is
@@ -99,16 +164,14 @@ simdjson's source structure, from the top level, looks like this:
* *.cpp: other misc. implementations, such as `simdjson::implementation` and the minifier.
* arm64|fallback|haswell|icelake|ppc64|westmere.cpp: Architecture-specific parser implementations.
* generic/*.h: `simdjson::<implementation>` namespace. Generic implementation of the parser, particularly the `dom_parser_implementation`.
* generic/stage1/*.h: `simdjson::<implementation>::stage1` namespace. Generic implementation of the simd-heavy tokenizer/indexer pass of the simdjson parser. Used for the On Demand interface
* generic/stage1/*.h: `simdjson::<implementation>::stage1` namespace. Generic implementation of the simd-heavy tokenizer/indexer pass of the simdjson parser. Used for the On-Demand interface
* generic/stage2/*.h: `simdjson::<implementation>::stage2` namespace. Generic implementation of the tape creator, which consumes the index from stage 1 and actually parses numbers and string and such. Used for the DOM interface.
Other important files and directories:
* **.drone.yml:** Definitions for Drone CI.
* **.appveyor.yml:** Definitions for Appveyor CI (Windows).
* **.circleci:** Definitions for Circle CI.
* **.github/workflows:** Definitions for GitHub Actions (CI).
* **singleheader:** Contains generated `simdjson.h` and `simdjson.cpp` that we release. The files `singleheader/simdjson.h` and `singleheader/simdjson.cpp` should never be edited by hand.
* **singleheader/amalgamate.py:** Generates `singleheader/simdjson.h` and `singleheader/simdjson.cpp` for release (python script).
* **singleheader/amalgamate.py:** Generates `singleheader/simdjson.h` and `singleheader/simdjson.cpp` for release (python script). If you add a new implementation (e.g., rvv), you need to edit this file (IMPLEMENTATIONS).
* **singleheader/amalgation_helper.py:** Generates and `amalgamation_report.html` that helps you understand the status of each file.
* **benchmark:** This is where we do benchmarking. Benchmarking is core to every change we make; the
cardinal rule is don't regress performance without knowing exactly why, and what you're trading
for it. Many of our benchmarks are microbenchmarks. We are effectively doing controlled scientific experiments for the purpose of understanding what affects our performance. So we simplify as much as possible. We try to avoid irrelevant factors such as page faults, interrupts, unnecessary system calls. We recommend checking the performance as follows:
@@ -283,8 +346,6 @@ If your compiler does not default on C++11 support or better you may get failing
Note that the name of directory (`build`) is arbitrary, you can name it as you want (e.g., `buildgcc`) and you can have as many different such directories as you would like (one per configuration).
## Usage (CMake on 64-bit Windows using Visual Studio 2019 or better)
Recent versions of Visual Studio support CMake natively, [please refer to the Visual Studio documentation](https://learn.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-170).
@@ -304,17 +365,15 @@ Though having Visual Studio installed is necessary, one can build simdjson using
- `mkdir build`
- `cd build`
- `cmake ..`
- `cmake --build . -config Release`
- `cmake --build . --config Release`
Furthermore, if you have installed LLVM clang on Windows, for example as a component of Visual Studio 2019, you can configure and build simdjson using LLVM clang on Windows using cmake:
- `mkdir build`
- `cd build`
- `cmake -T ClangCL ..`
- `cmake --build . -config Release`
- `cmake --build . --config Release`
## Various References
+1 -1
View File
@@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2018-2023 The simdjson authors
Copyright 2018-2025 The simdjson authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
+18
View File
@@ -0,0 +1,18 @@
Copyright 2018-2025 The simdjson authors
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+72 -18
View File
@@ -1,16 +1,13 @@
[![][license img]][license] [![][licensemit img]][licensemit]
[![Ubuntu 20.04 CI](https://github.com/simdjson/simdjson/workflows/Ubuntu%2020.04%20CI%20(GCC%209)/badge.svg)](https://simdjson.org/plots.html)
![VS16-CI](https://github.com/simdjson/simdjson/workflows/VS16-CI/badge.svg)
[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/simdjson.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:simdjson)
![MinGW64-CI](https://github.com/simdjson/simdjson/workflows/MinGW64-CI/badge.svg)
[![][license img]][license]
[![Doxygen Documentation](https://img.shields.io/badge/docs-doxygen-green.svg)](https://simdjson.github.io/simdjson/)
simdjson : Parsing gigabytes of JSON per second
===============================================
<img src="images/logo.png" width="10%" style="float: right">
<img src="images/official_logo/logo_noir/SVG/logo_simdjson_noir.svg" width="40%" style="float: right">
JSON is everywhere on the Internet. Servers spend a *lot* of time parsing it. We need a fresh
approach. The simdjson library uses commonly available SIMD instructions and microparallel algorithms
to parse JSON 4x faster than RapidJSON and 25x faster than JSON for Modern C++.
@@ -31,7 +28,9 @@ Table of Contents
* [Real-world usage](#real-world-usage)
* [Quick Start](#quick-start)
* [Documentation](#documentation)
* [Godbolt](#godbolt)
* [Performance results](#performance-results)
* [Packages](#packages)
* [Bindings and Ports of simdjson](#bindings-and-ports-of-simdjson)
* [About simdjson](#about-simdjson)
* [Funding](#funding)
@@ -42,9 +41,12 @@ Table of Contents
Real-world usage
----------------
- [Node.js](https://nodejs.org/)
- [ClickHouse](https://github.com/ClickHouse/ClickHouse)
- [Facebook/Meta Velox](https://velox-lib.io)
- [Meta Velox](https://velox-lib.io)
- [Google Pax](https://github.com/google/paxml)
- [milvus](https://github.com/milvus-io/milvus)
- [QuestDB](https://questdb.io/blog/questdb-release-8-0-3/)
- [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer)
- [Shopify HeapProfiler](https://github.com/Shopify/heap-profiler)
- [StarRocks](https://github.com/StarRocks/starrocks)
@@ -58,9 +60,18 @@ Real-world usage
- [vast](https://github.com/tenzir/vast)
- [ada-url](https://github.com/ada-url/ada)
- [fastgron](https://github.com/adamritter/fastgron)
- [WasmEdge](https://wasmedge.org)
- [RonDB](https://github.com/logicalclocks/rondb)
- [GreptimeDB](https://github.com/GreptimeTeam/greptimedb)
- [mamba](https://github.com/mamba-org/mamba)
- [Ladybird Browser](https://ladybird.org)
If you are planning to use simdjson in a product, please work from one of our releases.
Quick Start
-----------
@@ -68,16 +79,16 @@ The simdjson library is easily consumable with a single .h and .cpp file.
0. Prerequisites: `g++` (version 7 or better) or `clang++` (version 6 or better), and a 64-bit
system with a command-line shell (e.g., Linux, macOS, freeBSD). We also support programming
environments like Visual Studio and Xcode, but different steps are needed.
environments like Visual Studio and Xcode, but different steps are needed. Users of clang++ may need to specify the C++ version (e.g., `c++ -std=c++17`) since clang++ tends to default on C++98.
1. Pull [simdjson.h](singleheader/simdjson.h) and [simdjson.cpp](singleheader/simdjson.cpp) into a
directory, along with the sample file [twitter.json](jsonexamples/twitter.json).
directory, along with the sample file [twitter.json](jsonexamples/twitter.json). You can download them with the `wget` utility:
```
wget https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.h https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.cpp https://raw.githubusercontent.com/simdjson/simdjson/master/jsonexamples/twitter.json
```
2. Create `quickstart.cpp`:
```c++
```cpp
#include <iostream>
#include "simdjson.h"
using namespace simdjson;
@@ -95,22 +106,28 @@ int main(void) {
100 results.
```
Documentation
-------------
Usage documentation is available:
* [Basics](doc/basics.md) is an overview of how to use simdjson and its APIs.
* [Builder](doc/builder.md) is an overview of how to efficiently write JSON strings using simdjson.
* [Performance](doc/performance.md) shows some more advanced scenarios and how to tune for them.
* [Implementation Selection](doc/implementation-selection.md) describes runtime CPU detection and
how you can work with it.
* [API](https://simdjson.github.io/simdjson/) contains the automatically generated API documentation.
* [Compile-Time Parsing](doc/compile_time.md) presents our compile-time parsing function (C++26 only).
Godbolt
-------------
Some users may want to browse code along with the compiled assembly. You want to check out the following lists of examples:
* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/98Kx9Kqjn)
* [simdjson examples with errors without exceptions](https://godbolt.org/z/PKG7GdbPo)
* [C++26 reflection example](https://godbolt.org/z/K3Px64TqK)
* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/7G5qE4sr9)
* [simdjson examples with errors without exceptions](https://godbolt.org/z/e9dWb9E4v)
Performance results
-------------------
@@ -138,6 +155,9 @@ speed for [synthetic files over various sizes generated with a script](https://g
For NDJSON files, we can exceed 3 GB/s with [our multithreaded parsing functions](https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md).
Packages
------------------------------
[![Packaging status](https://repology.org/badge/vertical-allrepos/simdjson.svg)](https://repology.org/project/simdjson/versions)
Bindings and Ports of simdjson
@@ -162,9 +182,12 @@ We distinguish between "bindings" (which just wrap the C++ code) and a port to a
- [simdjsone](https://github.com/saleyn/simdjsone): erlang bindings.
- [lua-simdjson](https://github.com/FourierTransformer/lua-simdjson): lua bindings.
- [hermes-json](https://hackage.haskell.org/package/hermes-json): haskell bindings.
- [simdjzon](https://github.com/travisstaloch/simdjzon): zig port.
- [zimdjson](https://github.com/EzequielRamis/zimdjson): Zig port.
- [simdjzon](https://github.com/travisstaloch/simdjzon): Zig port.
- [JSON-Simd](https://github.com/rawleyfowler/JSON-simd): Raku bindings.
- [JSON::SIMD](https://metacpan.org/pod/JSON::SIMD): Perl bindings; fully-featured JSON module that uses simdjson for decoding.
- [gemmaJSON](https://github.com/sainttttt/gemmaJSON): Nim JSON parser based on simdjson bindings.
- [simdjson-java](https://github.com/simdjson/simdjson-java): Java port.
About simdjson
--------------
@@ -173,7 +196,11 @@ The simdjson library takes advantage of modern microarchitectures, parallelizing
instructions, reducing branch misprediction, and reducing data dependency to take advantage of each
CPU's multiple execution cores.
Some people [enjoy reading our paper](https://arxiv.org/abs/1902.08318): A description of the design
Our default front-end is called On-Demand, and we wrote a paper about it:
- John Keiser, Daniel Lemire, [On-Demand JSON: A Better Way to Parse Documents?](http://arxiv.org/abs/2312.17149), Software: Practice and Experience 54 (6), 2024.
Some people [enjoy reading the first (2019) simdjson paper](https://arxiv.org/abs/1902.08318): A description of the design
and implementation of simdjson is in our research article:
- Geoff Langdale, Daniel Lemire, [Parsing Gigabytes of JSON per Second](https://arxiv.org/abs/1902.08318), VLDB Journal 28 (6), 2019.
@@ -187,25 +214,52 @@ For the video inclined, <br />
[![simdjson at QCon San Francisco 2019](http://img.youtube.com/vi/wlvKAT7SZIQ/0.jpg)](http://www.youtube.com/watch?v=wlvKAT7SZIQ)<br />
(It was the best voted talk, we're kinda proud of it.)
Citing this work
-----------------
If you use simdjson in published research, please cite the software library. A suitable BibTeX entry is:
```bibtex
@misc{simdjson,
title={{The simdjson library: Parsing Gigabytes of JSON per Second}},
author={Daniel Lemire and Geoff Langdale and John Keiser and Paul Dreik and Francisco Thiesen and others},
year={2019},
howpublished={Software library},
note={https://github.com/simdjson/simdjson}
}
```
Funding
-------
The work is supported by the Natural Sciences and Engineering Research Council of Canada under grant
number RGPIN-2017-03910.
The work is supported by the Natural Sciences and Engineering Research Council of Canada under grants
RGPIN-2017-03910 and RGPIN-2024-03787.
[license]: LICENSE
[license img]: https://img.shields.io/badge/License-Apache%202-blue.svg
[licensemit]: LICENSE-MIT
[licensemit img]: https://img.shields.io/badge/License-MIT-blue.svg
Contributing to simdjson
------------------------
Head over to [CONTRIBUTING.md](CONTRIBUTING.md) for information on contributing to simdjson, and
[HACKING.md](HACKING.md) for information on source, building, and architecture/design.
Stars
------
[![Star History Chart](https://api.star-history.com/svg?repos=simdjson/simdjson&type=Date)](https://www.star-history.com/#simdjson/simdjson&Date)
License
-------
This code is made available under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html).
This code is made available under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) as well as under the MIT License. As a user, you can pick the license you prefer.
Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it is under the liberal (business-friendly) MIT license.
+7
View File
@@ -0,0 +1,7 @@
# Security Policy
## Reporting a Vulnerability
Please use the following contact information for reporting a vulnerability:
- [Daniel Lemire](https://github.com/lemire) - daniel@lemire.me
+19
View File
@@ -4,6 +4,9 @@ add_subdirectory(dom)
include_directories( . linux )
link_libraries(simdjson-windows-headers test-data)
link_libraries(simdjson)
if(SIMDJSON_STATIC_REFLECTION)
add_compile_definitions(SIMDJSON_STATIC_REFLECTION=1)
endif(SIMDJSON_STATIC_REFLECTION)
add_executable(benchfeatures benchfeatures.cpp)
add_executable(get_corpus_benchmark get_corpus_benchmark.cpp)
@@ -26,6 +29,22 @@ if (TARGET benchmark::benchmark)
if(TARGET nlohmann_json)
target_link_libraries(bench_ondemand PRIVATE nlohmann_json)
endif()
if(TARGET boostjson)
target_link_libraries(bench_ondemand PRIVATE boostjson)
endif()
endif()
endif()
include(CheckCXXCompilerFlag)
check_cxx_compiler_flag("-std=c++20" SIMDJSON_COMPILER_SUPPORTS_CXX20)
if(SIMDJSON_STATIC_REFLECTION)
add_subdirectory(static_reflect)
else()
if(SIMDJSON_EXCEPTIONS AND SIMDJSON_COMPILER_SUPPORTS_CXX20)
add_subdirectory(from)
add_subdirectory(car_builder)
endif()
endif(SIMDJSON_STATIC_REFLECTION)
+46
View File
@@ -0,0 +1,46 @@
# Accessor Performance Benchmarks (C++26)
These benchmarks compare the performance of runtime vs compile-time JSON accessors.
For the comparison to be meaningful, you must build simdjson with support for
C++26 reflexion. See the `p2996` repository in the main project directory.
## Files
- `accessor_benchmark.h` - Common benchmark framework and test data
- `runtime_accessors.h` - Runtime `at_path()` benchmarks
- `compile_time_accessors.h` - Compile-time `at_path_compiled()` benchmarks (requires C++26 reflection)
## Benchmarks
Each benchmark measures parsing + single field access:
1. **accessor_simple** - Simple field: `.name`
2. **accessor_nested** - Nested field: `.address.city`
3. **accessor_deep** - Deep nested field: `.address.coordinates.lat`
## Building (Linux/macOS)
```bash
cmake -B build -D SIMDJSON_STATIC_REFLECTION=ON -DSIMDJSON_DEVELOPER_MODE=ON
cmake --build build --target=bench_ondemand
```
The `SIMDJSON_STATIC_REFLECTION` will be made unnecessary once mainstream compilers
begin supporting C++26 sufficiently well.
## Running (Linux/macOS)
```bash
# Run all accessor benchmarks
./build/bench_ondemand --benchmark_filter="accessor"
```
## Results
We find that compile-time accessors show performance improvements that scale with path depth:
- Simple fields: ~1.2x faster
- Nested fields: ~1.5x faster
- Deep nested fields: ~1.8x faster
The speedup comes from eliminating runtime path parsing and conversion overhead.
@@ -0,0 +1,132 @@
#pragma once
#include "json_benchmark/file_runner.h"
#include <string>
namespace accessor_performance {
using namespace json_benchmark;
// Test JSON for accessor benchmarks
static const char* TEST_JSON = R"({
"name": "Alice",
"age": 30,
"email": "alice@example.com",
"address": {
"street": "123 Main St",
"city": "Boston",
"state": "MA",
"zip": 12345,
"coordinates": {
"lat": 42.3601,
"lon": -71.0589
}
},
"scores": [95, 87, 92, 88, 91],
"preferences": {
"theme": "dark",
"notifications": {
"email": true,
"push": false,
"sms": true
}
}
})";
// Struct definitions for compile-time validation
#if SIMDJSON_STATIC_REFLECTION
struct Coordinates {
double lat;
double lon;
};
struct Address {
std::string street;
std::string city;
std::string state;
int64_t zip;
Coordinates coordinates;
};
struct Notifications {
bool email;
bool push;
bool sms;
};
struct Preferences {
std::string theme;
Notifications notifications;
};
struct TestData {
std::string name;
int64_t age;
std::string email;
Address address;
std::vector<int64_t> scores;
Preferences preferences;
};
#endif // SIMDJSON_STATIC_REFLECTION
// Single-access benchmark runner: measures ONE field access per iteration
template<typename I>
struct single_access_runner : public file_runner<I> {
std::string result_string;
int64_t result_int{};
double result_double{};
bool result_bool{};
bool setup(benchmark::State &state) {
this->json = simdjson::padded_string(TEST_JSON, strlen(TEST_JSON));
state.SetBytesProcessed(int64_t(state.iterations()) * int64_t(this->json.size()));
return true;
}
bool before_run(benchmark::State &state) {
if (!file_runner<I>::before_run(state)) { return false; }
result_string.clear();
result_int = 0;
result_double = 0.0;
result_bool = false;
return true;
}
bool run(benchmark::State &) {
return this->implementation.run(this->json, result_string, result_int, result_double, result_bool);
}
template<typename R>
bool diff(benchmark::State &state, single_access_runner<R> &reference) {
if (result_string != reference.result_string ||
result_int != reference.result_int ||
result_double != reference.result_double ||
result_bool != reference.result_bool) {
std::cerr << "Accessor benchmark results differ!" << std::endl;
return false;
}
return true;
}
size_t items_per_iteration() {
return 1;
}
};
// Benchmark template definitions
struct runtime_at_path_simple;
template<typename I> simdjson_inline static void accessor_simple(benchmark::State &state) {
run_json_benchmark<single_access_runner<I>, single_access_runner<runtime_at_path_simple>>(state);
}
struct runtime_at_path_nested;
template<typename I> simdjson_inline static void accessor_nested(benchmark::State &state) {
run_json_benchmark<single_access_runner<I>, single_access_runner<runtime_at_path_nested>>(state);
}
struct runtime_at_path_deep;
template<typename I> simdjson_inline static void accessor_deep(benchmark::State &state) {
run_json_benchmark<single_access_runner<I>, single_access_runner<runtime_at_path_deep>>(state);
}
} // namespace accessor_performance
@@ -0,0 +1,56 @@
#pragma once
#if SIMDJSON_EXCEPTIONS && SIMDJSON_STATIC_REFLECTION
#include "accessor_benchmark.h"
namespace accessor_performance {
using namespace simdjson;
struct compile_time_at_path_simple {
ondemand::parser parser{};
bool run(simdjson::padded_string &json, std::string &result_str, int64_t&, double&, bool&) {
auto doc = parser.iterate(json);
std::string_view name;
auto r = ondemand::json_path::at_path_compiled<TestData, ".name">(doc);
if (r.get(name) != SUCCESS) return false;
result_str = name;
return true;
}
};
struct compile_time_at_path_nested {
ondemand::parser parser{};
bool run(simdjson::padded_string &json, std::string &result_str, int64_t&, double&, bool&) {
auto doc = parser.iterate(json);
std::string_view city;
auto r = ondemand::json_path::at_path_compiled<TestData, ".address.city">(doc);
if (r.get(city) != SUCCESS) return false;
result_str = city;
return true;
}
};
struct compile_time_at_path_deep {
ondemand::parser parser{};
bool run(simdjson::padded_string &json, std::string&, int64_t&, double &result_dbl, bool&) {
auto doc = parser.iterate(json);
double lat;
auto r = ondemand::json_path::at_path_compiled<TestData, ".address.coordinates.lat">(doc);
if (r.get(lat) != SUCCESS) return false;
result_dbl = lat;
return true;
}
};
BENCHMARK_TEMPLATE(accessor_simple, compile_time_at_path_simple)->UseManualTime();
BENCHMARK_TEMPLATE(accessor_nested, compile_time_at_path_nested)->UseManualTime();
BENCHMARK_TEMPLATE(accessor_deep, compile_time_at_path_deep)->UseManualTime();
} // namespace accessor_performance
#endif // SIMDJSON_EXCEPTIONS && SIMDJSON_STATIC_REFLECTION
@@ -0,0 +1,53 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "accessor_benchmark.h"
namespace accessor_performance {
using namespace simdjson;
struct runtime_at_path_simple {
ondemand::parser parser{};
bool run(simdjson::padded_string &json, std::string &result_str, int64_t&, double&, bool&) {
auto doc = parser.iterate(json);
std::string_view name;
if (doc.at_path(".name").get(name) != SUCCESS) return false;
result_str = name;
return true;
}
};
struct runtime_at_path_nested {
ondemand::parser parser{};
bool run(simdjson::padded_string &json, std::string &result_str, int64_t&, double&, bool&) {
auto doc = parser.iterate(json);
std::string_view city;
if (doc.at_path(".address.city").get(city) != SUCCESS) return false;
result_str = city;
return true;
}
};
struct runtime_at_path_deep {
ondemand::parser parser{};
bool run(simdjson::padded_string &json, std::string&, int64_t&, double &result_dbl, bool&) {
auto doc = parser.iterate(json);
double lat;
if (doc.at_path(".address.coordinates.lat").get(lat) != SUCCESS) return false;
result_dbl = lat;
return true;
}
};
BENCHMARK_TEMPLATE(accessor_simple, runtime_at_path_simple)->UseManualTime();
BENCHMARK_TEMPLATE(accessor_nested, runtime_at_path_nested)->UseManualTime();
BENCHMARK_TEMPLATE(accessor_deep, runtime_at_path_deep)->UseManualTime();
} // namespace accessor_performance
#endif // SIMDJSON_EXCEPTIONS
@@ -1,72 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "amazon_cellphones.h"
namespace amazon_cellphones {
using namespace simdjson;
template<bool threaded>
struct simdjson_singlestage {
using StringType = std::string;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::map<StringType, brand> &result) {
#ifdef SIMDJSON_THREADS_ENABLED
parser.threaded = threaded;
#endif
singlestage::document_stream stream = parser.iterate_many(json);
singlestage::document_stream::iterator i = stream.begin();
++i; // Skip first line
for (;i != stream.end(); ++i) {
auto doc = *i;
size_t index{0};
StringType copy;
double rating;
uint64_t reviews;
for ( auto value : doc ) {
switch (index)
{
case 1:
copy = StringType(std::string_view(value));
break;
case 5:
rating = double(value);
break;
case 7:
reviews = uint64_t(value);
break;
default:
break;
}
index++;
}
auto x = result.find(copy);
if (x == result.end()) { // If key not found, add new key
result.emplace(copy, amazon_cellphones::brand{
rating * reviews,
reviews
});
} else { // Otherwise, update key data
x->second.cumulative_rating += rating * reviews;
x->second.reviews_count += reviews;
}
}
return true;
}
};
BENCHMARK_TEMPLATE(amazon_cellphones, simdjson_singlestage<UNTHREADED>)->UseManualTime();
#ifdef SIMDJSON_THREADS_ENABLED
BENCHMARK_TEMPLATE(amazon_cellphones, simdjson_singlestage<THREADED>)->UseManualTime();
#endif
} // namespace amazon_cellphones
#endif // SIMDJSON_EXCEPTIONS
File diff suppressed because it is too large Load Diff
-158
View File
@@ -473,28 +473,6 @@ static void twitter_count(State& state) {
}
BENCHMARK(twitter_count);
#ifndef SIMDJSON_DISABLE_DEPRECATED_API
SIMDJSON_PUSH_DISABLE_WARNINGS
SIMDJSON_DISABLE_DEPRECATED_WARNING
static void iterator_twitter_count(State& state) {
// Prints the number of results in twitter.json
padded_string json = padded_string::load(TWITTER_JSON);
ParsedJson pj = build_parsed_json(json);
for (simdjson_unused auto _ : state) {
ParsedJson::Iterator iter(pj);
// uint64_t result_count = doc["search_metadata"]["count"];
if (!iter.move_to_key("search_metadata")) { return; }
if (!iter.move_to_key("count")) { return; }
if (!iter.is_integer()) { return; }
int64_t result_count = iter.get_integer();
if (result_count != 100) { return; }
}
}
BENCHMARK(iterator_twitter_count);
SIMDJSON_POP_DISABLE_WARNINGS
#endif // SIMDJSON_DISABLE_DEPRECATED_API
static void twitter_default_profile(State& state) {
// Count unique users with a default profile.
dom::parser parser;
@@ -578,54 +556,6 @@ static void error_code_twitter_default_profile(State& state) noexcept {
}
BENCHMARK(error_code_twitter_default_profile);
#ifndef SIMDJSON_DISABLE_DEPRECATED_API
SIMDJSON_PUSH_DISABLE_WARNINGS
SIMDJSON_DISABLE_DEPRECATED_WARNING
static void iterator_twitter_default_profile(State& state) {
// Count unique users with a default profile.
padded_string json;
auto error = padded_string::load(TWITTER_JSON).get(json);
if (error) { std::cerr << error << std::endl; return; }
ParsedJson pj = build_parsed_json(json);
for (simdjson_unused auto _ : state) {
set<string_view> default_users;
ParsedJson::Iterator iter(pj);
// for (dom::object tweet : doc["statuses"]) {
if (!(iter.move_to_key("statuses") && iter.is_array())) { return; }
if (iter.down()) { // first status
do {
// dom::object user = tweet["user"];
if (!(iter.move_to_key("user") && iter.is_object())) { return; }
// if (user["default_profile"]) {
if (iter.move_to_key("default_profile")) {
if (iter.is_true()) {
if (!iter.up()) { return; } // back to user
// default_users.insert(user["screen_name"]);
if (!(iter.move_to_key("screen_name") && iter.is_string())) { return; }
default_users.emplace(iter.get_string(), iter.get_string_length());
}
if (!iter.up()) { return; } // back to user
}
if (!iter.up()) { return; } // back to status
} while (iter.next()); // next status
}
if (default_users.size() != 86) { return; }
}
}
SIMDJSON_POP_DISABLE_WARNINGS
BENCHMARK(iterator_twitter_default_profile);
#endif // SIMDJSON_DISABLE_DEPRECATED_API
static void error_code_twitter_image_sizes(State& state) noexcept {
// Count unique image sizes
dom::parser parser;
@@ -680,92 +610,4 @@ static void parse_surrogate_pairs(State& state) {
}
BENCHMARK(parse_surrogate_pairs);
#ifndef SIMDJSON_DISABLE_DEPRECATED_API
SIMDJSON_PUSH_DISABLE_WARNINGS
SIMDJSON_DISABLE_DEPRECATED_WARNING
static void iterator_twitter_image_sizes(State& state) {
// Count unique image sizes
padded_string json;
auto error = padded_string::load(TWITTER_JSON).get(json);
if (error) { std::cerr << error << std::endl; return; }
ParsedJson pj = build_parsed_json(json);
for (simdjson_unused auto _ : state) {
set<tuple<uint64_t, uint64_t>> image_sizes;
ParsedJson::Iterator iter(pj);
// for (dom::object tweet : doc["statuses"]) {
if (!(iter.move_to_key("statuses") && iter.is_array())) { return; }
if (iter.down()) { // first status
do {
// dom::object media;
// not_found = tweet["entities"]["media"].get(media);
// if (!not_found) {
if (iter.move_to_key("entities")) {
if (!iter.is_object()) { return; }
if (iter.move_to_key("media")) {
if (!iter.is_array()) { return; }
// for (dom::object image : media) {
if (iter.down()) { // first media
do {
// for (auto [key, size] : dom::object(image["sizes"])) {
if (!(iter.move_to_key("sizes") && iter.is_object())) { return; }
if (iter.down()) { // first size
do {
iter.move_to_value();
// image_sizes.insert({ size["w"], size["h"] });
if (!(iter.move_to_key("w")) && !iter.is_integer()) { return; }
uint64_t width = iter.get_integer();
if (!iter.up()) { return; } // back to size
if (!(iter.move_to_key("h")) && !iter.is_integer()) { return; }
uint64_t height = iter.get_integer();
if (!iter.up()) { return; } // back to size
image_sizes.emplace(width, height);
} while (iter.next()); // next size
if (!iter.up()) { return; } // back to sizes
}
if (!iter.up()) { return; } // back to image
} while (iter.next()); // next image
if (!iter.up()) { return; } // back to media
}
if (!iter.up()) { return; } // back to entities
}
if (!iter.up()) { return; } // back to status
}
} while (iter.next()); // next status
}
if (image_sizes.size() != 15) { return; };
}
}
BENCHMARK(iterator_twitter_image_sizes);
#endif // SIMDJSON_DISABLE_DEPRECATED_API
#ifndef SIMDJSON_DISABLE_DEPRECATED_API
static void print_json(State& state) noexcept {
// Prints the number of results in twitter.json
dom::parser parser;
padded_string json;
auto error = padded_string::load(TWITTER_JSON).get(json);
if (error) { std::cerr << error << std::endl; return; }
int code = json_parse(json, parser);
if (code) { cerr << error_message(code) << endl; return; }
for (simdjson_unused auto _ : state) {
std::stringstream s;
if (!parser.print_json(s)) { cerr << "print_json failed" << endl; return; }
}
}
BENCHMARK(print_json);
#endif // SIMDJSON_DISABLE_DEPRECATED_API
SIMDJSON_POP_DISABLE_WARNINGS
BENCHMARK_MAIN();
+16 -13
View File
@@ -21,12 +21,15 @@ SIMDJSON_PUSH_DISABLE_ALL_WARNINGS
#include <nlohmann/json.hpp>
#endif
#ifdef SIMDJSON_COMPETITION_BOOSTJSON
#include <boost/json.hpp>
#endif
// This has to be last, for reasons I don't yet understand
#include <benchmark/benchmark.h>
SIMDJSON_POP_DISABLE_WARNINGS
#include "json2msgpack/simdjson_ondemand.h"
#include "json2msgpack/simdjson_singlestage.h"
#include "json2msgpack/simdjson_dom.h"
#include "json2msgpack/yyjson.h"
#include "json2msgpack/rapidjson.h"
@@ -34,9 +37,9 @@ SIMDJSON_POP_DISABLE_WARNINGS
#include "json2msgpack/sajson.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "json2msgpack/nlohmann_json.h"
#include "json2msgpack/boostjson.h"
#include "partial_tweets/simdjson_ondemand.h"
#include "partial_tweets/simdjson_singlestage.h"
#include "partial_tweets/simdjson_dom.h"
#include "partial_tweets/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -50,12 +53,11 @@ SIMDJSON_POP_DISABLE_WARNINGS
#if SIMDJSON_COMPETITION_SAX
#include "partial_tweets/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "partial_tweets/boostjson.h"
#include "distinct_user_id/simdjson_ondemand.h"
#include "distinct_user_id/simdjson_ondemand_json_pointer.h"
#include "distinct_user_id/simdjson_singlestage.h"
#include "distinct_user_id/simdjson_singlestage_json_pointer.h"
#include "distinct_user_id/simdjson_dom.h"
#include "distinct_user_id/simdjson_dom_json_pointer.h"
#include "distinct_user_id/yyjson.h"
@@ -70,9 +72,9 @@ SIMDJSON_POP_DISABLE_WARNINGS
#if SIMDJSON_COMPETITION_SAX
#include "distinct_user_id/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "distinct_user_id/boostjson.h"
#include "find_tweet/simdjson_ondemand.h"
#include "find_tweet/simdjson_singlestage.h"
#include "find_tweet/simdjson_dom.h"
#include "find_tweet/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -86,9 +88,9 @@ SIMDJSON_POP_DISABLE_WARNINGS
#if SIMDJSON_COMPETITION_SAX
#include "find_tweet/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "find_tweet/boostjson.h"
#include "top_tweet/simdjson_ondemand.h"
#include "top_tweet/simdjson_singlestage.h"
#include "top_tweet/simdjson_dom.h"
#include "top_tweet/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -102,10 +104,10 @@ SIMDJSON_POP_DISABLE_WARNINGS
#if SIMDJSON_COMPETITION_SAX
#include "top_tweet/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "top_tweet/boostjson.h"
#include "kostya/simdjson_ondemand.h"
#include "kostya/simdjson_singlestage.h"
#include "kostya/simdjson_dom.h"
#include "kostya/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -119,15 +121,12 @@ SIMDJSON_POP_DISABLE_WARNINGS
#if SIMDJSON_COMPETITION_SAX
#include "kostya/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "kostya/boostjson.h"
#include "large_random/simdjson_ondemand.h"
#if SIMDJSON_COMPETITION_ONDEMAND_UNORDERED
#include "large_random/simdjson_ondemand_unordered.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_UNORDERED
#include "large_random/simdjson_singlestage.h"
#if SIMDJSON_COMPETITION_SINGLESTAGE_UNORDERED
#include "large_random/simdjson_singlestage_unordered.h"
#endif // SIMDJSON_COMPETITION_SINGLESTAGE_UNORDERED
#include "large_random/simdjson_dom.h"
#include "large_random/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -141,13 +140,17 @@ SIMDJSON_POP_DISABLE_WARNINGS
#if SIMDJSON_COMPETITION_SAX
#include "large_random/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "large_random/boostjson.h"
#include "amazon_cellphones/simdjson_dom.h"
#include "amazon_cellphones/simdjson_ondemand.h"
#include "amazon_cellphones/simdjson_singlestage.h"
#include "large_amazon_cellphones/simdjson_dom.h"
#include "large_amazon_cellphones/simdjson_ondemand.h"
#include "large_amazon_cellphones/simdjson_singlestage.h"
#include "accessor_performance/runtime_accessors.h"
#if SIMDJSON_STATIC_REFLECTION
#include "accessor_performance/compile_time_accessors.h"
#endif
BENCHMARK_MAIN();
-30
View File
@@ -169,22 +169,6 @@ BENCHMARK(parse_gsoc)->Repetitions(10)->ComputeStatistics("max", [](const std::v
})->DisplayAggregatesOnly(true);
#ifndef SIMDJSON_DISABLE_DEPRECATED_API
SIMDJSON_PUSH_DISABLE_WARNINGS
SIMDJSON_DISABLE_DEPRECATED_WARNING
static void json_parse(State& state) {
ParsedJson pj;
if (!pj.allocate_capacity(EMPTY_ARRAY.length())) { return; }
for (simdjson_unused auto _ : state) {
auto error = json_parse(EMPTY_ARRAY, pj);
if (error) { return; }
}
}
SIMDJSON_POP_DISABLE_WARNINGS
BENCHMARK(json_parse);
#endif // SIMDJSON_DISABLE_DEPRECATED_API
static void parser_parse_error_code(State& state) {
dom::parser parser;
if (parser.allocate(EMPTY_ARRAY.length())) { return; }
@@ -213,20 +197,6 @@ BENCHMARK(parser_parse_exception);
#endif // SIMDJSON_EXCEPTIONS
#ifndef SIMDJSON_DISABLE_DEPRECATED_API
SIMDJSON_PUSH_DISABLE_WARNINGS
SIMDJSON_DISABLE_DEPRECATED_WARNING
static void build_parsed_json(State& state) {
for (simdjson_unused auto _ : state) {
dom::parser parser = simdjson::build_parsed_json(EMPTY_ARRAY);
if (!parser.valid) { return; }
}
}
SIMDJSON_POP_DISABLE_WARNINGS
BENCHMARK(build_parsed_json);
#endif
static void document_parse_error_code(State& state) {
for (simdjson_unused auto _ : state) {
dom::parser parser;
+618
View File
@@ -0,0 +1,618 @@
#!/bin/bash
# JSON Parsing Compilation Benchmark: Reflection Usage vs Manual Parsing
# Compares compilation times when ACTUALLY USING reflection for parsing vs manual parsing
# This measures the compile-time cost of reflection-based automatic deserialization
set -e
echo "=== simdjson Reflection Usage Compilation Benchmark ==="
echo "Measuring compilation impact of ACTUALLY USING reflection for parsing"
echo "Starting at: $(date)"
echo
echo "╔════════════════════════════════════════════════════════════════════════════╗"
echo "║ METHODOLOGY ║"
echo "╠════════════════════════════════════════════════════════════════════════════╣"
echo "║ ║"
echo "║ FAIR COMPARISON STRATEGY: ║"
echo "║ This benchmark compares two DIFFERENT approaches to parsing the same JSON: ║"
echo "║ ║"
echo "║ • MANUAL PARSING: Traditional simdjson with explicit .get() calls ║"
echo "║ - Uses doc[\"field\"].get(variable) for each field ║"
echo "║ - No reflection involved ║"
echo "║ ║"
echo "║ • REFLECTION PARSING: Automatic deserialization with reflection ║"
echo "║ - Uses doc.get<MyStruct>() for automatic field mapping ║"
echo "║ - Relies on compile-time reflection to generate parsing code ║"
echo "║ ║"
echo "║ WHAT WE'RE MEASURING: ║"
echo "║ • Compile-time cost of reflection-based automatic deserialization ║"
echo "║ • Template instantiation overhead for reflection parsing ║"
echo "║ • Code generation complexity from using reflection features ║"
echo "║ ║"
echo "║ TEST SCENARIOS: ║"
echo "║ ║"
echo "║ 1. SIMPLE STRUCT: Basic fields (string, int, bool) ║"
echo "║ - Measures baseline reflection overhead ║"
echo "║ ║"
echo "║ 2. NESTED STRUCT: Multiple levels of nested objects ║"
echo "║ - Measures reflection complexity scaling ║"
echo "║ ║"
echo "║ 3. COMPLEX STRUCT: Arrays, optional fields, mixed types ║"
echo "║ - Measures real-world reflection usage impact ║"
echo "║ ║"
echo "║ WHY THIS IS MEANINGFUL: ║"
echo "║ • Shows actual cost of using reflection features ║"
echo "║ • Measures compile-time code generation overhead ║"
echo "║ • Helps developers understand reflection's compilation impact ║"
echo "║ • Compares equivalent functionality implemented two different ways ║"
echo "║ ║"
echo "╚════════════════════════════════════════════════════════════════════════════╝"
echo
# Configuration
ITERATIONS=10
JOBS=4
# ───────────────────────────── BOX-PRINT HELPER ────────────────────────────
BOX_WIDTH=74 # characters between the pipes
print_box_line() { # usage: print_box_line "text"
printf "║ %-*s ║\n" "${BOX_WIDTH}" "$1"
}
# Function to test if a compiler supports reflection with debug output
test_reflection_support() {
local compiler="$1"
echo " → Testing compiler: $compiler"
if [ ! -x "$compiler" ] && ! command -v "$compiler" >/dev/null 2>&1; then
echo " → Compiler not found or not executable"
return 1
fi
# Check compiler version first
echo " → Compiler version: $("$compiler" --version 2>/dev/null | head -n1 || echo "version check failed")"
# Simple test: check if compiler accepts reflection flags
local test_file=$(mktemp /tmp/reflection_test_XXXXXX.cpp)
cat > "$test_file" << 'EOF'
int main() {
return 0;
}
EOF
echo " → Testing basic reflection flags..."
local test_exe=$(mktemp /tmp/reflection_test_XXXXXX)
local basic_result=$("$compiler" -freflection -fexpansion-statements -std=c++26 "$test_file" -o "$test_exe" 2>&1)
local basic_exit_code=$?
if [ $basic_exit_code -ne 0 ]; then
echo " → Basic flags FAILED with exit code $basic_exit_code"
echo " → Error output: $basic_result"
rm -f "$test_file" "$test_exe"
return 1
fi
echo " → Basic flags: OK"
rm -f "$test_exe"
# Test reflection syntax
echo " → Testing reflection syntax..."
cat > "$test_file" << 'EOF'
struct Test {
int x;
};
int main() {
auto refl = ^^Test;
return 0;
}
EOF
local syntax_result=$("$compiler" -freflection -fexpansion-statements -std=c++26 "$test_file" -o "$test_exe" 2>&1)
local syntax_exit_code=$?
if [ $syntax_exit_code -eq 0 ]; then
echo " → Reflection syntax: OK"
echo " → ✓ REFLECTION SUPPORT CONFIRMED"
rm -f "$test_file" "$test_exe"
return 0
else
echo " → Reflection syntax FAILED with exit code $syntax_exit_code"
echo " → Error output: $syntax_result"
rm -f "$test_file" "$test_exe"
return 1
fi
}
# Find a compiler with reflection support
echo "Searching for clang++ with reflection support..."
REFLECTION_CXX=""
REFLECTION_CC=""
# List of potential clang++ locations to check
POTENTIAL_COMPILERS=(
"/usr/local/bin/clang++"
"/opt/clang/bin/clang++"
"/usr/bin/clang++"
"clang++"
)
# If CXX is already set, test it first
if [ -n "$CXX" ]; then
echo "Testing user-specified compiler: $CXX"
if test_reflection_support "$CXX"; then
REFLECTION_CXX="$CXX"
echo "✓ User-specified compiler supports reflection: $CXX"
else
echo "✗ User-specified compiler does not support reflection: $CXX"
echo "Will search for alternative..."
fi
fi
# If we don't have a working compiler yet, search for one
if [ -z "$REFLECTION_CXX" ]; then
for compiler in "${POTENTIAL_COMPILERS[@]}"; do
echo "Testing: $compiler"
if test_reflection_support "$compiler"; then
REFLECTION_CXX="$compiler"
echo "✓ Found reflection-enabled compiler: $compiler"
break
else
echo "✗ No reflection support: $compiler"
fi
done
fi
# Check if we found a working compiler
if [ -z "$REFLECTION_CXX" ]; then
echo
echo "╔════════════════════════════════════════════════════════════════════════════╗"
echo "║ ERROR ║"
echo "╠════════════════════════════════════════════════════════════════════════════╣"
echo "║ ║"
echo "║ No clang++ compiler with reflection support found! ║"
echo "║ ║"
echo "║ This benchmark requires a compiler that supports C++26 reflection. ║"
echo "║ ║"
echo "║ Options: ║"
echo "║ 1. Use the Docker container: ./p2996/run_docker.sh ║"
echo "║ 2. Build clang with reflection from: https://github.com/bloomberg/clang-p2996 ║"
echo "║ 3. Set CXX environment variable to point to reflection-enabled clang++ ║"
echo "║ ║"
echo "║ Example: CXX=/path/to/reflection-clang++ ./benchmark_script.sh ║"
echo "║ ║"
echo "╚════════════════════════════════════════════════════════════════════════════╝"
exit 1
fi
# Set the compilers
export CXX="$REFLECTION_CXX"
# Find corresponding C compiler
if [ -n "$CC" ]; then
REFLECTION_CC="$CC"
elif [ "$REFLECTION_CXX" = "/usr/local/bin/clang++" ]; then
REFLECTION_CC="/usr/local/bin/clang"
elif [ "$REFLECTION_CXX" = "/opt/clang/bin/clang++" ]; then
REFLECTION_CC="/opt/clang/bin/clang"
elif [ "$REFLECTION_CXX" = "/usr/bin/clang++" ]; then
REFLECTION_CC="/usr/bin/clang"
else
REFLECTION_CC="clang"
fi
export CC="$REFLECTION_CC"
echo
echo "Using reflection-enabled compiler: $($CXX --version | head -n1)"
echo "Using C compiler: $($CC --version | head -n1)"
echo
# Function to create manual parsing test
create_manual_parsing_test() {
local test_name="$1"
local struct_complexity="$2"
cat > "${test_name}_manual.cpp" << 'EOF'
#include <simdjson.h>
#include <iostream>
#include <string>
#include <vector>
#include <optional>
// Test structures
struct Person {
std::string name;
int age;
bool active;
};
struct Address {
std::string street;
std::string city;
int zipcode;
};
struct Employee {
Person person;
Address address;
std::vector<std::string> skills;
std::optional<std::string> department;
double salary;
};
// Manual parsing functions
bool parse_person_manual(simdjson::ondemand::value& val, Person& person) {
auto obj = val.get_object();
if (obj.error()) return false;
for (auto field : obj) {
std::string_view key = field.unescaped_key();
if (key == "name") {
std::string_view name_val;
if (field.value().get(name_val)) return false;
person.name = name_val;
} else if (key == "age") {
if (field.value().get(person.age)) return false;
} else if (key == "active") {
if (field.value().get(person.active)) return false;
}
}
return true;
}
bool parse_address_manual(simdjson::ondemand::value& val, Address& address) {
auto obj = val.get_object();
if (obj.error()) return false;
for (auto field : obj) {
std::string_view key = field.unescaped_key();
if (key == "street") {
std::string_view street_val;
if (field.value().get(street_val)) return false;
address.street = street_val;
} else if (key == "city") {
std::string_view city_val;
if (field.value().get(city_val)) return false;
address.city = city_val;
} else if (key == "zipcode") {
if (field.value().get(address.zipcode)) return false;
}
}
return true;
}
bool parse_employee_manual(simdjson::ondemand::document& doc, Employee& employee) {
auto obj = doc.get_object();
if (obj.error()) return false;
for (auto field : obj) {
std::string_view key = field.unescaped_key();
if (key == "person") {
auto person_val = field.value();
if (!parse_person_manual(person_val, employee.person)) return false;
} else if (key == "address") {
auto addr_val = field.value();
if (!parse_address_manual(addr_val, employee.address)) return false;
} else if (key == "skills") {
auto skills_array = field.value().get_array();
if (skills_array.error()) return false;
for (auto skill : skills_array) {
std::string_view skill_val;
if (skill.get(skill_val)) return false;
employee.skills.emplace_back(skill_val);
}
} else if (key == "department") {
std::string_view dept_val;
if (!field.value().get(dept_val)) {
employee.department = dept_val;
}
} else if (key == "salary") {
if (field.value().get(employee.salary)) return false;
}
}
return true;
}
int main() {
simdjson::ondemand::parser parser;
std::string json_str = R"({
"person": {
"name": "John Doe",
"age": 30,
"active": true
},
"address": {
"street": "123 Main St",
"city": "Anytown",
"zipcode": 12345
},
"skills": ["C++", "JSON", "Programming"],
"department": "Engineering",
"salary": 85000.50
})";
simdjson::ondemand::document doc;
auto error = parser.iterate(simdjson::pad(json_str)).get(doc);
if (error) {
std::cerr << "Parse error" << std::endl;
return 1;
}
Employee employee;
if (!parse_employee_manual(doc, employee)) {
std::cerr << "Manual parsing failed" << std::endl;
return 1;
}
std::cout << "Manual parsing successful: " << employee.person.name
<< ", age " << employee.person.age << std::endl;
return 0;
}
EOF
}
# Function to create reflection parsing test
create_reflection_parsing_test() {
local test_name="$1"
local struct_complexity="$2"
cat > "${test_name}_reflection.cpp" << 'EOF'
#include <simdjson.h>
#include <iostream>
#include <string>
#include <vector>
#include <optional>
// Test structures (same as manual version)
struct Person {
std::string name;
int age;
bool active;
};
struct Address {
std::string street;
std::string city;
int zipcode;
};
struct Employee {
Person person;
Address address;
std::vector<std::string> skills;
std::optional<std::string> department;
double salary;
};
int main() {
simdjson::ondemand::parser parser;
std::string json_str = R"({
"person": {
"name": "John Doe",
"age": 30,
"active": true
},
"address": {
"street": "123 Main St",
"city": "Anytown",
"zipcode": 12345
},
"skills": ["C++", "JSON", "Programming"],
"department": "Engineering",
"salary": 85000.50
})";
simdjson::ondemand::document doc;
auto error = parser.iterate(simdjson::pad(json_str)).get(doc);
if (error) {
std::cerr << "Parse error" << std::endl;
return 1;
}
// Use reflection-based automatic deserialization
Employee employee;
auto result = doc.get<Employee>();
if (result.error()) {
std::cerr << "Reflection parsing failed" << std::endl;
return 1;
}
employee = result.value();
std::cout << "Reflection parsing successful: " << employee.person.name
<< ", age " << employee.person.age << std::endl;
return 0;
}
EOF
}
# Function to time compilation of parsing approach
time_parsing_compilation() {
local description="$1"
local test_file="$2"
local use_reflection="$3"
local iteration="$4"
echo "[$iteration] $description"
# Clean build
rm -rf build_parsing_test
mkdir build_parsing_test
cd build_parsing_test
# Copy test file
cp "../$test_file" .
echo " Configuring..."
if [ "$use_reflection" = "true" ]; then
cmake -DCMAKE_CXX_COMPILER="$CXX" \
-DSIMDJSON_DEVELOPER_MODE=ON \
-DSIMDJSON_STATIC_REFLECTION=ON \
-DBUILD_SHARED_LIBS=OFF \
../.. >/dev/null 2>&1
else
cmake -DCMAKE_CXX_COMPILER="$CXX" \
-DSIMDJSON_DEVELOPER_MODE=ON \
-DSIMDJSON_STATIC_REFLECTION=OFF \
-DBUILD_SHARED_LIBS=OFF \
../.. >/dev/null 2>&1
fi
echo " Building simdjson..."
cmake --build . --target simdjson >/dev/null 2>&1
echo " Compiling parsing test..."
# Time just the test compilation
start_time=$(date +%s.%N)
"$CXX" -std=c++17 -I../../include "$test_file" -L. -lsimdjson -o parsing_test >/dev/null 2>&1
end_time=$(date +%s.%N)
# Calculate time duration
time_taken=$(echo "$end_time $start_time" | awk '{printf "%.3f", $1 - $2}')
echo " Completed in: ${time_taken}s"
cd ..
rm -rf build_parsing_test
echo "$time_taken"
}
# Create test files
echo "Creating test files..."
create_manual_parsing_test "complex" "complex"
create_reflection_parsing_test "complex" "complex"
# Arrays to store times
times_manual=""
times_reflection=""
echo
echo "=== MANUAL PARSING COMPILATION ==="
echo "Testing traditional simdjson parsing with explicit .get() calls"
echo
for i in $(seq 1 $ITERATIONS); do
time_result=$(time_parsing_compilation "Compiling manual parsing test" "complex_manual.cpp" "false" "$i" | tail -n1)
times_manual="$times_manual $time_result"
done
echo
echo "=== REFLECTION PARSING COMPILATION ==="
echo "Testing automatic deserialization with doc.get<Struct>()"
echo
for i in $(seq 1 $ITERATIONS); do
time_result=$(time_parsing_compilation "Compiling reflection parsing test" "complex_reflection.cpp" "true" "$i" | tail -n1)
times_reflection="$times_reflection $time_result"
done
echo
echo "╔════════════════════════════════════════════════════════════════════════════╗"
echo "║ REFLECTION USAGE COMPILATION RESULTS ║"
echo "╠════════════════════════════════════════════════════════════════════════════╣"
echo "║ ║"
echo "║ MANUAL PARSING (explicit .get() calls): ║"
count=1
for t in $times_manual; do
if [[ "$t" =~ ^[0-9]+\.?[0-9]*$ ]]; then
line=$(printf "Run %2d: %7.3f seconds" "$count" "$t")
print_box_line "$line"
count=$((count + 1))
fi
done
echo "║ ║"
echo "║ REFLECTION PARSING (automatic doc.get<Struct>()): ║"
count=1
for t in $times_reflection; do
if [[ "$t" =~ ^[0-9]+\.?[0-9]*$ ]]; then
line=$(printf "Run %2d: %7.3f seconds" "$count" "$t")
print_box_line "$line"
count=$((count + 1))
fi
done
echo "╚════════════════════════════════════════════════════════════════════════════╝"
echo
echo "╔════════════════════════════════════════════════════════════════════════════╗"
echo "║ ANALYSIS SUMMARY ║"
echo "╠════════════════════════════════════════════════════════════════════════════╣"
echo "║ ║"
# Calculate averages and percentages - filter to only numeric values first
manual_numbers=""
reflection_numbers=""
for t in $times_manual; do
if [[ "$t" =~ ^[0-9]+\.?[0-9]*$ ]]; then
manual_numbers="$manual_numbers $t"
fi
done
for t in $times_reflection; do
if [[ "$t" =~ ^[0-9]+\.?[0-9]*$ ]]; then
reflection_numbers="$reflection_numbers $t"
fi
done
if [ -n "$manual_numbers" ] && [ -n "$reflection_numbers" ]; then
manual_avg=$(echo "$manual_numbers" | awk '{sum=0; for(i=1;i<=NF;i++) sum+=$i; print sum/NF}')
reflection_avg=$(echo "$reflection_numbers" | awk '{sum=0; for(i=1;i<=NF;i++) sum+=$i; print sum/NF}')
overhead=$(echo "$reflection_avg $manual_avg" | awk '{printf "%.3f", $1 - $2}')
if [ $(echo "$manual_avg > 0" | awk '{print ($1 > 0)}') -eq 1 ]; then
percent=$(echo "$reflection_avg $manual_avg" | awk '{printf "%.1f", ($1 - $2) / $2 * 100}')
else
percent="0"
fi
print_box_line "MANUAL PARSING RESULTS:"
print_box_line "$(printf "Average compilation time: %.3fs" "$manual_avg")"
print_box_line ""
print_box_line "REFLECTION PARSING RESULTS:"
print_box_line "$(printf "Average compilation time: %.3fs" "$reflection_avg")"
print_box_line ""
print_box_line "REFLECTION OVERHEAD:"
print_box_line "$(printf "Additional time: %.3fs (%+.1f%%)" "$overhead" "$percent")"
print_box_line ""
else
echo "║ ERROR: Could not extract valid timing data ║"
echo "║ Manual times: $times_manual"
echo "║ Reflection times: $times_reflection"
echo "║ ║"
fi
echo "╠════════════════════════════════════════════════════════════════════════════╣"
echo "║ INTERPRETATION ║"
echo "╠════════════════════════════════════════════════════════════════════════════╣"
echo "║ ║"
echo "║ WHAT THESE RESULTS SHOW: ║"
echo "║ ║"
echo "║ • COMPILE-TIME COST: How much longer reflection parsing takes to compile ║"
echo "║ - Higher % = more expensive template instantiation and codegen ║"
echo "║ ║"
echo "║ • CODE GENERATION OVERHEAD: Reflection creates parsing code at compile ║"
echo "║ time, which requires more template processing than manual parsing ║"
echo "║ ║"
echo "║ • DEVELOPER TRADE-OFF: Reflection provides automatic deserialization ║"
echo "║ but at the cost of increased compilation time ║"
echo "║ ║"
echo "║ EVALUATION: ║"
echo "║ • Low overhead (0-20%): Reflection is compile-time efficient ║"
echo "║ • Medium overhead (20-50%): Noticeable but potentially acceptable ║"
echo "║ • High overhead (50%+): Significant compilation cost for reflection ║"
echo "║ ║"
echo "║ REAL-WORLD IMPACT: ║"
echo "║ • Small projects: Absolute time matters more than percentage ║"
echo "║ • Large projects: Percentage overhead compounds across many files ║"
echo "║ • CI/CD pipelines: Longer builds affect development velocity ║"
echo "║ ║"
echo "╚════════════════════════════════════════════════════════════════════════════╝"
# Clean up test files
rm -f complex_manual.cpp complex_reflection.cpp
echo
echo "Completed at: $(date)"
+13 -9
View File
@@ -445,7 +445,7 @@ struct benchmarker {
return 100.0 * a / b;
}
void print(bool tabbed_output) const {
void print(bool tabbed_output, bool stage1_only) const {
if (tabbed_output) {
char* filename_copy = reinterpret_cast<char*>(malloc(strlen(filename)+1));
SIMDJSON_PUSH_DISABLE_WARNINGS
@@ -503,17 +503,21 @@ struct benchmarker {
stats->blocks_with_16_structurals_flipped, percent(stats->blocks_with_16_structurals_flipped, stats->blocks));
}
printf("\n");
printf("All Stages (excluding allocation)\n");
print_aggregate("| " , all_stages_without_allocation.best);
// frequently, allocation is a tiny fraction of the running time so we omit it
if(allocate_stage.best.elapsed_sec() > 0.01 * all_stages_without_allocation.best.elapsed_sec()) {
printf("|- Allocation\n");
print_aggregate("| ", allocate_stage.best);
if(!stage1_only) {
printf("All Stages (excluding allocation)\n");
print_aggregate("| " , all_stages_without_allocation.best);
// frequently, allocation is a tiny fraction of the running time so we omit it
if(allocate_stage.best.elapsed_sec() > 0.01 * all_stages_without_allocation.best.elapsed_sec()) {
printf("|- Allocation\n");
print_aggregate("| ", allocate_stage.best);
}
}
printf("|- Stage 1\n");
print_aggregate("| ", stage1.best);
printf("|- Stage 2\n");
print_aggregate("| ", stage2.best);
if(!stage1_only) {
printf("|- Stage 2\n");
print_aggregate("| ", stage2.best);
}
if (collector.has_events()) {
double freq1 = (stage1.best.cycles() / stage1.best.elapsed_sec()) / 1000000000.0;
double freq2 = (stage2.best.cycles() / stage2.best.elapsed_sec()) / 1000000000.0;
+14
View File
@@ -0,0 +1,14 @@
# Executable
add_executable(benchmark_car_builder benchmark_car_builder.cpp)
# Compile for C++20.
target_compile_features(benchmark_car_builder PRIVATE cxx_std_20)
# Check if -march=native is supported
include(CheckCXXCompilerFlag)
check_cxx_compiler_flag("-march=native" SIMDJSON_SUPPORTS_MARCH_NATIVE)
if(SIMDJSON_SUPPORTS_MARCH_NATIVE)
target_compile_options(benchmark_car_builder PRIVATE -march=native)
endif()
target_include_directories(benchmark_car_builder PRIVATE ${CMAKE_CURRENT_LIST_DIR}/..)
@@ -0,0 +1,127 @@
#include "event_counter.h"
#include <random>
#include <vector>
#include <simdjson.h>
event_collector collector;
struct Car {
std::string make;
std::string model;
int64_t year; // We deliberately do not include the tire pressure.
};
std::vector<Car> generate_random_cars(size_t count) {
static const std::vector<std::string> makes = {"Toyota", "Honda", "Ford",
"BMW", "Mazda"};
static const std::vector<std::string> models = {"Camry", "Civic", "Focus",
"320i", "3"};
static thread_local std::mt19937 rng{std::random_device{}()};
std::uniform_int_distribution<int> make_dist(0, makes.size() - 1);
std::uniform_int_distribution<int> model_dist(0, models.size() - 1);
std::uniform_int_distribution<int64_t> year_dist(2000, 2025);
std::uniform_real_distribution<double> pressure_dist(30.0, 45.0);
std::vector<Car> cars;
cars.reserve(count);
for (size_t i = 0; i < count; ++i) {
Car car;
car.make = makes[make_dist(rng)];
car.model = models[model_dist(rng)];
car.year = year_dist(rng);
cars.push_back(std::move(car));
}
return cars;
}
std::string_view serialize(simdjson::builder::string_builder &sb,
const std::vector<Car> &cars) {
sb.clear();
sb.start_array();
for (const auto &car : cars) {
sb.start_object();
sb.append_key_value("make", car.make);
sb.append_comma();
sb.append_key_value("model", car.model);
sb.append_comma();
sb.append_key_value("year", car.year);
sb.end_object();
}
sb.end_array();
std::string_view result;
if (sb.view().get(result)) {
return ""; // unexpected (error)
}
return result;
}
double pretty_print(const std::string &name, size_t num_chars,
std::pair<event_aggregate, size_t> result) {
const auto &agg = result.first;
size_t N = result.second;
num_chars *= N;
printf("%-40s : %8.2f ns %8.2f GB/s", name.c_str(),
agg.elapsed_ns() / num_chars, num_chars / agg.elapsed_ns());
if (collector.has_events()) {
printf(" %8.2f GHz %8.2f cycles/char %8.2f ins./char %8.2f i/c",
agg.cycles() / agg.elapsed_ns(), agg.cycles() / num_chars,
agg.instructions() / num_chars, agg.instructions() / agg.cycles());
}
printf("\n");
return num_chars / agg.elapsed_ns();
}
template <class function_type>
std::pair<event_aggregate, size_t>
bench(const function_type &&function, size_t min_repeat = 100,
size_t min_time_ns = 40'000'000, size_t max_repeat = 10000000) {
size_t N = min_repeat;
if (N == 0) {
N = 1;
}
event_aggregate warm_aggregate{};
for (size_t i = 0; i < N; i++) {
std::atomic_thread_fence(std::memory_order_acquire);
collector.start();
function();
std::atomic_thread_fence(std::memory_order_release);
event_count allocate_count = collector.end();
warm_aggregate << allocate_count;
if ((i + 1 == N) && (warm_aggregate.total_elapsed_ns() < min_time_ns) &&
(N < max_repeat)) {
N *= 10;
}
}
event_aggregate aggregate{};
for (size_t i = 0; i < 10; i++) {
std::atomic_thread_fence(std::memory_order_acquire);
collector.start();
for (size_t i = 0; i < N; i++) {
function();
}
std::atomic_thread_fence(std::memory_order_release);
event_count allocate_count = collector.end();
aggregate << allocate_count;
}
return {aggregate, N};
}
void run_benchmarks() {
std::vector<Car> source = generate_random_cars(100000);
simdjson::builder::string_builder sb;
size_t volume = serialize(sb, source).size();
pretty_print("string_builder", volume, bench([&source, &sb]() -> size_t {
return serialize(sb, source).size();
}));
}
int main() {
for (size_t trial = 0; trial < 3; trial++) {
printf("Trial %zu:\n", trial + 1);
run_benchmarks();
printf("\n");
}
return EXIT_SUCCESS;
}
+29
View File
@@ -0,0 +1,29 @@
#pragma once
#ifdef SIMDJSON_COMPETITION_BOOSTJSON
#include "distinct_user_id.h"
namespace distinct_user_id {
struct boostjson {
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
auto root = boost::json::parse(json);
for (const auto &tweet : root.at("statuses").as_array()) {
result.push_back(tweet.at("user").at("id").to_number<uint64_t>());
if (tweet.as_object().if_contains("retweeted_status")) {
result.push_back(tweet.at("retweeted_status").at("user").at("id").to_number<uint64_t>());
}
}
return true;
}
};
BENCHMARK_TEMPLATE(distinct_user_id, boostjson)->UseManualTime();
} // namespace distinct_user_id
#endif // SIMDJSON_COMPETITION_BOOSTJSON
@@ -1,37 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "distinct_user_id.h"
namespace distinct_user_id {
using namespace simdjson;
struct simdjson_singlestage {
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
// Walk the document, parsing as we go
auto doc = parser.iterate(json);
for (singlestage::object tweet : doc.find_field("statuses")) {
// We believe that all statuses have a matching
// user, and we are willing to throw when they do not.
result.push_back(tweet.find_field("user").find_field("id"));
// Not all tweets have a "retweeted_status", but when they do
// we want to go and find the user within.
auto retweet = tweet.find_field("retweeted_status");
if (!retweet.error()) {
result.push_back(retweet.find_field("user").find_field("id"));
}
}
return true;
}
};
BENCHMARK_TEMPLATE(distinct_user_id, simdjson_singlestage)->UseManualTime();
} // namespace distinct_user_id
#endif // SIMDJSON_EXCEPTIONS
@@ -1,37 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "distinct_user_id.h"
namespace distinct_user_id {
using namespace simdjson;
struct simdjson_singlestage_json_pointer {
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
// Walk the document, parsing as we go
auto doc = parser.iterate(json);
for (singlestage::object tweet : doc.find_field("statuses")) {
// We believe that all statuses have a matching
// user, and we are willing to throw when they do not.
result.push_back(tweet.at_pointer("/user/id"));
// Not all tweets have a "retweeted_status", but when they do
// we want to go and find the user within.
auto retweet_id = tweet.at_pointer("/retweeted_status/user/id");
if (retweet_id.error() != NO_SUCH_FIELD) {
result.push_back(retweet_id);
}
}
return true;
}
};
BENCHMARK_TEMPLATE(distinct_user_id, simdjson_singlestage_json_pointer)->UseManualTime();
} // namespace distinct_user_id
#endif // SIMDJSON_EXCEPTIONS
+9 -2
View File
@@ -44,7 +44,10 @@ struct yyjson_base {
struct yyjson : yyjson_base {
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
return yyjson_base::run(yyjson_read(json.data(), json.size(), 0), result);
yyjson_doc *doc = yyjson_read(json.data(), json.size(), 0);
bool b = yyjson_base::run(doc, result);
yyjson_doc_free(doc);
return b;
}
};
BENCHMARK_TEMPLATE(distinct_user_id, yyjson)->UseManualTime();
@@ -52,11 +55,15 @@ BENCHMARK_TEMPLATE(distinct_user_id, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson_base {
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result);
yyjson_doc *doc = yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0);
bool b = yyjson_base::run(doc, result);
yyjson_doc_free(doc);
return b;
}
};
BENCHMARK_TEMPLATE(distinct_user_id, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace distinct_user_id
#endif // SIMDJSON_COMPETITION_YYJSON
+1 -1
View File
@@ -218,7 +218,7 @@ int main(int argc, char *argv[]) {
if (!options.verbose) { progress.erase(); }
for (size_t i=0; i<options.files.size(); i++) {
benchmarkers[i]->print(options.tabbed_output);
benchmarkers[i]->print(options.tabbed_output, options.stage1_only);
delete benchmarkers[i];
}
+2 -3
View File
@@ -25,12 +25,11 @@ int main(int argc, char *argv[]) {
exit(1);
}
const char *filename = argv[1];
auto v = simdjson::padded_string::load(filename);
if (v.error()) {
simdjson::padded_string p;
if (simdjson::padded_string::load(filename).get(p)) {
std::cerr << "Could not load the file " << filename << std::endl;
return EXIT_FAILURE;
}
const simdjson::padded_string& p = v.value_unsafe();
if (test_baseline) {
std::wclog << "Baseline: Getline + normal parse... " << std::endl;
std::cout << "Gigabytes/second\t"
+29 -3
View File
@@ -34,11 +34,15 @@
#include <string>
#include <vector>
#include "linux-perf-events.h"
#ifdef __linux__
#include "linux-perf-events.h"
#include <libgen.h>
#endif
#if __APPLE__ && __aarch64__
#include "apple/apple_arm_events.h"
#endif
#include "simdjson.h"
using std::string;
@@ -118,6 +122,7 @@ struct event_aggregate {
}
double elapsed_sec() const { return total.elapsed_sec() / iterations; }
double total_elapsed_ns() const { return total.elapsed_ns(); }
double elapsed_ns() const { return total.elapsed_ns() / iterations; }
double cycles() const { return total.cycles() / iterations; }
double instructions() const { return total.instructions() / iterations; }
@@ -134,7 +139,7 @@ struct event_collector {
#if defined(__linux__)
LinuxEvents<PERF_TYPE_HARDWARE> linux_events;
event_collector(simdjson_unused bool quiet = false) : linux_events(vector<int>{
event_collector() : linux_events(vector<int>{
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
PERF_COUNT_HW_CPU_CYCLES,
PERF_COUNT_HW_INSTRUCTIONS,
@@ -149,8 +154,17 @@ struct event_collector {
bool has_events() {
return linux_events.is_working();
}
#elif __APPLE__ && __aarch64__
AppleEvents apple_events;
performance_counters diff;
event_collector() : diff(0) {
apple_events.setup_performance_counters();
}
bool has_events() {
return apple_events.setup_performance_counters();
}
#else
event_collector(simdjson_unused bool _quiet = false) {}
event_collector() {}
bool has_events() {
return false;
}
@@ -159,6 +173,8 @@ struct event_collector {
simdjson_inline void start() {
#if defined(__linux)
linux_events.start();
#elif __APPLE__ && __aarch64__
if(has_events()) { diff = apple_events.get_counters(); }
#endif
start_clock = steady_clock::now();
}
@@ -166,6 +182,16 @@ struct event_collector {
time_point<steady_clock> end_clock = steady_clock::now();
#if defined(__linux)
linux_events.end(count.event_counts);
#elif __APPLE__ && __aarch64__
if(has_events()) {
performance_counters end = apple_events.get_counters();
diff = end - diff;
}
count.event_counts[0] = diff.cycles;
count.event_counts[1] = diff.instructions;
count.event_counts[2] = diff.missed_branches;
count.event_counts[3] = 0;
count.event_counts[4] = 0;
#endif
count.elapsed = end_clock - start_clock;
return count;
+30
View File
@@ -0,0 +1,30 @@
#pragma once
#ifdef SIMDJSON_COMPETITION_BOOSTJSON
#include "find_tweet.h"
namespace find_tweet {
struct boostjson {
using StringType=std::string;
bool run(simdjson::padded_string &json, uint64_t find_id, std::string &result) {
auto root = boost::json::parse(json);
for (const auto &tweet : root.at("statuses").as_array()) {
if (tweet.at("id") == find_id) {
result = tweet.at("text").as_string();
return true;
}
}
return false;
}
};
BENCHMARK_TEMPLATE(find_tweet, boostjson)->UseManualTime();
} // namespace find_tweet
#endif // SIMDJSON_COMPETITION_BOOSTJSON
+1 -1
View File
@@ -13,7 +13,7 @@ struct nlohmann_json {
auto root = nlohmann::json::parse(json.data(), json.data() + json.size());
for (auto tweet : root["statuses"]) {
if (tweet["id"] == find_id) {
result = tweet["text"];
result = to_string(tweet["text"]);
return true;
}
}
@@ -1,33 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "find_tweet.h"
namespace find_tweet {
using namespace simdjson;
struct simdjson_singlestage {
using StringType=std::string_view;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, uint64_t find_id, std::string_view &result) {
// Walk the document, parsing as we go
auto doc = parser.iterate(json);
for (auto tweet : doc.find_field("statuses")) {
if (uint64_t(tweet.find_field("id")) == find_id) {
result = tweet.find_field("text");
return true;
}
}
return false;
}
};
BENCHMARK_TEMPLATE(find_tweet, simdjson_singlestage)->UseManualTime();
} // namespace find_tweet
#endif // SIMDJSON_EXCEPTIONS
+10 -2
View File
@@ -36,18 +36,26 @@ struct yyjson_base {
struct yyjson : yyjson_base {
bool run(simdjson::padded_string &json, uint64_t find_id, std::string_view &result) {
return yyjson_base::run(yyjson_read(json.data(), json.size(), 0), find_id, result);
yyjson_doc *doc = yyjson_read(json.data(), json.size(), 0);
bool b = yyjson_base::run(doc, find_id, result);
yyjson_doc_free(doc);
return b;
}
};
BENCHMARK_TEMPLATE(find_tweet, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson_base {
bool run(simdjson::padded_string &json, uint64_t find_id, std::string_view &result) {
return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), find_id, result);
yyjson_doc *doc = yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0);
bool b = yyjson_base::run(doc, find_id, result);
yyjson_doc_free(doc);
return b;
}
};
BENCHMARK_TEMPLATE(find_tweet, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace find_tweet
#endif // SIMDJSON_COMPETITION_YYJSON
+15
View File
@@ -0,0 +1,15 @@
# Executable
add_executable(from_benchmark from_benchmark.cpp)
# Compile for C++20.
if(CMAKE_CXX_STANDARD LESS 20)
target_compile_features(from_benchmark PRIVATE cxx_std_20)
endif()
# Check if -march=native is supported
include(CheckCXXCompilerFlag)
check_cxx_compiler_flag("-march=native" SIMDJSON_SUPPORTS_MARCH_NATIVE)
if(SIMDJSON_SUPPORTS_MARCH_NATIVE)
target_compile_options(from_benchmark PRIVATE -march=native)
endif()
target_include_directories(from_benchmark PRIVATE ${CMAKE_CURRENT_LIST_DIR}/..)
+77
View File
@@ -0,0 +1,77 @@
#ifndef BENCHMARK_HELPERS_H
#define BENCHMARK_HELPERS_H
#include "event_counter.h"
#include <atomic>
event_collector collector;
template <class function_type>
std::pair<event_aggregate, size_t>
bench(const function_type &&function, size_t min_repeat = 10,
size_t min_time_ns = 40'000'000, size_t max_repeat = 10000000) {
size_t N = min_repeat;
if (N == 0) {
N = 1;
}
event_aggregate warm_aggregate{};
for (size_t i = 0; i < N; i++) {
std::atomic_thread_fence(std::memory_order_acquire);
collector.start();
function();
std::atomic_thread_fence(std::memory_order_release);
event_count allocate_count = collector.end();
warm_aggregate << allocate_count;
if ((i + 1 == N) && (warm_aggregate.total_elapsed_ns() < min_time_ns) &&
(N < max_repeat)) {
N *= 10;
}
}
event_aggregate aggregate{};
for (size_t i = 0; i < 10; i++) {
std::atomic_thread_fence(std::memory_order_acquire);
collector.start();
for (size_t i = 0; i < N; i++) {
function();
}
std::atomic_thread_fence(std::memory_order_release);
event_count allocate_count = collector.end();
aggregate << allocate_count;
}
return {aggregate, N};
}
double pretty_print(const std::string &name, size_t num_chars,
std::pair<event_aggregate, size_t> result) {
const auto &agg = result.first;
size_t N = result.second;
num_chars *= N;
printf("%-40s : %8.2f ns %8.2f GB/s", name.c_str(),
agg.elapsed_ns() / num_chars, num_chars / agg.elapsed_ns());
if (collector.has_events()) {
printf(" %8.2f GHz %8.2f cycles/char %8.2f ins./char %8.2f i/c",
agg.cycles() / agg.elapsed_ns(), agg.cycles() / num_chars,
agg.instructions() / num_chars, agg.instructions() / agg.cycles());
}
printf("\n");
return num_chars / agg.elapsed_ns();
}
double pretty_print_array(const std::string &name, size_t num_chars, size_t num_elements,
std::pair<event_aggregate, size_t> result) {
const auto &agg = result.first;
size_t N = result.second;
num_chars *= N;
printf("%-40s : %8.2f ns/char %8.2f ns/value %8.2f GB/s", name.c_str(),
agg.elapsed_ns() / num_chars, agg.elapsed_ns() / num_elements, num_chars / agg.elapsed_ns());
if (collector.has_events()) {
printf(" %8.2f GHz %8.2f cycles/char %8.2f ins./char %8.2f ins./value %8.2f i/c",
agg.cycles() / agg.elapsed_ns(), agg.cycles() / num_chars,
agg.instructions() / num_chars, agg.instructions() / num_elements, agg.instructions() / agg.cycles());
}
printf("\n");
return num_chars / agg.elapsed_ns();
}
#endif
+146
View File
@@ -0,0 +1,146 @@
#ifndef CAR_HELPERS_H
#define CAR_HELPERS_H
#include <iomanip>
#include <random>
#include <sstream>
#include <vector>
#include <simdjson.h>
struct Car {
std::string make;
std::string model;
int64_t year; // We deliberately do not include the tire pressure.
};
using namespace simdjson;
std::string random_car_json() {
static const std::vector<std::string> makes = {"Toyota", "Honda", "Ford",
"BMW", "Mazda"};
static const std::vector<std::string> models = {"Camry", "Civic", "Focus",
"320i", "3"};
static thread_local std::mt19937 rng{std::random_device{}()};
std::uniform_int_distribution<int> make_dist(0, makes.size() - 1);
std::uniform_int_distribution<int> model_dist(0, models.size() - 1);
std::uniform_int_distribution<int> year_dist(2000, 2025);
std::uniform_real_distribution<double> pressure_dist(30.0, 45.0);
std::ostringstream oss;
oss << R"({ "make": ")" << makes[make_dist(rng)] << R"(", "model": ")"
<< models[model_dist(rng)] << R"(", "year": )" << year_dist(rng)
<< R"(, "tire_pressure": [ )" << std::fixed << std::setprecision(1)
<< pressure_dist(rng) << ", " << pressure_dist(rng) << R"( ] })";
return oss.str();
}
std::string generate_car_json_stream(size_t N) {
std::ostringstream oss;
for (size_t i = 0; i < N; ++i) {
oss << random_car_json();
oss << "\n";
}
return oss.str();
}
std::string generate_car_json_array(size_t N) {
std::ostringstream oss;
oss << "[\n";
for (size_t i = 0; i < N; ++i) {
oss << random_car_json();
if (i < N - 1) {
oss << ",";
}
oss << "\n";
}
oss << "]";
return oss.str();
}
// We want to maximize C++ portability, but this is not needed with static
// reflection (C++26).
template <>
simdjson_inline simdjson_result<Car>
simdjson::ondemand::document::get() & noexcept {
ondemand::object obj;
auto error = get_object().get(obj);
if (error) {
return error;
}
Car car;
if ((error = obj["make"].get_string(car.make))) {
return error;
}
if ((error = obj["model"].get_string(car.model))) {
return error;
}
if ((error = obj["year"].get_int64().get(car.year))) {
return error;
}
return car;
}
template <>
simdjson_inline simdjson_result<Car> simdjson::ondemand::value::get() noexcept {
ondemand::object obj;
auto error = get_object().get(obj);
if (error) {
return error;
}
Car car;
if ((error = obj["make"].get_string(car.make))) {
return error;
}
if ((error = obj["model"].get_string(car.model))) {
return error;
}
if ((error = obj["year"].get_int64().get(car.year))) {
return error;
}
return car;
}
template <>
simdjson_inline simdjson_result<Car>
simdjson::ondemand::document_reference::get() & noexcept {
ondemand::object obj;
auto error = get_object().get(obj);
if (error) {
return error;
}
Car car;
if ((error = obj["make"].get_string(car.make))) {
return error;
}
if ((error = obj["model"].get_string(car.model))) {
return error;
}
if ((error = obj["year"].get_int64().get(car.year))) {
return error;
}
return car;
}
template <>
simdjson_inline simdjson_result<Car>
simdjson::ondemand::document_reference::get() && noexcept {
ondemand::object obj;
auto error = get_object().get(obj);
if (error) {
return error;
}
Car car;
if ((error = obj["make"].get_string(car.make))) {
return error;
}
if ((error = obj["model"].get_string(car.model))) {
return error;
}
if ((error = obj["year"].get_int64().get(car.year))) {
return error;
}
return car;
}
#endif
+132
View File
@@ -0,0 +1,132 @@
// We are going to assume C++20.
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <simdjson.h>
#include <string>
#include "benchmark_helpers.h"
#include "car_helpers.h"
using namespace simdjson;
void run_benchmarks() {
printf("Running benchmarks on tiny input...\n");
simdjson::padded_string json =
R"({ "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9 ] })"_padded;
volatile size_t dummy = 0;
ondemand::parser parser;
// Classic On-Demand
pretty_print("simdjson classic", json.size(),
bench([&json, &dummy, &parser]() {
ondemand::document doc = parser.iterate(json);
Car car = doc.get<Car>();
dummy = dummy + car.year;
}));
// Benchmark simdjson::from() without parser
pretty_print("simdjson::from<Car>() (no parser)", json.size(),
bench([&json, &dummy]() {
Car car = simdjson::from(json);
dummy = dummy + car.year;
}));
// Benchmark simdjson::from() with parser
pretty_print("simdjson::from<Car>() (with parser)", json.size(),
bench([&json, &dummy, &parser]() {
Car car = simdjson::from(parser, json);
dummy = dummy + car.year;
}));
}
void run_array_benchmarks() {
printf("Running benchmarks on large array...\n");
size_t N = 1'000'000;
simdjson::padded_string json = generate_car_json_array(N);
volatile size_t dummy = 0;
ondemand::parser parser;
// Classic On-Demand
pretty_print_array("simdjson classic", json.size(), N,
bench([&json, &dummy, &parser]() {
dummy = 0;
ondemand::document doc = parser.iterate(json);
ondemand::array array = doc.get_array();
for (auto value : array) {
Car car = value.get<Car>();
dummy = dummy + car.year;
}
}));
// from with array
pretty_print_array("simdjson::from(json).array()", json.size(), N,
bench([&json, &dummy, &parser]() {
dummy = 0;
for (auto value : simdjson::from(json).array()) {
Car car = value.get<Car>();
dummy = dummy + car.year;
}
}));
#if SIMDJSON_SUPPORTS_RANGES_FROM
// Benchmark simdjson::from() without parser (EXPERIMENTAL)
pretty_print_array("simdjson::from<Car>() (experimental, no parser)",
json.size(), N, bench([&json, &dummy]() {
dummy = 0;
for (Car car :
simdjson::from(json) | simdjson::as<Car>()) {
dummy = dummy + car.year;
}
}));
// Benchmark simdjson::from() with parser (EXPERIMENTAL)
pretty_print_array("simdjson::from<Car>() (experimental, with parser)",
json.size(), N, bench([&json, &dummy, &parser]() {
dummy = 0;
for (Car car : simdjson::from(parser, json) |
simdjson::as<Car>()) {
dummy = dummy + car.year;
}
}));
#endif // SIMDJSON_SUPPORTS_RANGES_FROM
}
void run_stream_benchmarks() {
printf("Running stream benchmarks...\n");
simdjson::padded_string json = generate_car_json_stream(1'000'000);
volatile size_t dummy = 0;
ondemand::parser parser;
// Classic On-Demand
pretty_print("simdjson classic", json.size(),
bench([&json, &dummy, &parser]() {
ondemand::document_stream stream = parser.iterate_many(json);
for (auto doc : stream) {
Car car = doc.get<Car>();
dummy = dummy + car.year;
}
}));
// from_many
pretty_print("simdjson with thread local", json.size(),
bench([&json, &dummy, &parser]() {
ondemand::document_stream stream =
ondemand::parser::get_parser().iterate_many(json);
for (auto doc : stream) {
Car car = doc.get<Car>();
dummy = dummy + car.year;
}
}));
}
int main() {
for (size_t trial = 0; trial < 3; trial++) {
printf("Trial %zu:\n", trial + 1);
run_array_benchmarks();
run_stream_benchmarks();
run_benchmarks();
printf("\n");
}
return EXIT_SUCCESS;
}
+5 -1
View File
@@ -9,7 +9,11 @@ simdjson_never_inline
double bench(std::string filename, simdjson::padded_string& p) {
std::chrono::time_point<std::chrono::steady_clock> start_clock =
std::chrono::steady_clock::now();
simdjson::padded_string::load(filename).value_unsafe().swap(p);
auto error = simdjson::padded_string::load(filename).get(p);
if(error) {
std::cerr << simdjson::error_message(error) << std::endl;
std::abort();
}
std::chrono::time_point<std::chrono::steady_clock> end_clock =
std::chrono::steady_clock::now();
std::chrono::duration<double> elapsed = end_clock - start_clock;
+104
View File
@@ -0,0 +1,104 @@
#pragma once
#ifdef SIMDJSON_COMPETITION_BOOSTJSON
#include "json2msgpack.h"
namespace json2msgpack {
struct boostjson2msgpack {
inline std::string_view to_msgpack(const boost::json::value &root, uint8_t *buf) {
buff = buf;
recursive_processor(root);
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
private:
uint8_t *buff{};
inline void write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
inline void write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
inline void write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
inline void write_string(const std::string & str) {
write_byte(0xdb);
write_uint32(uint32_t(str.size()));
::memcpy(buff, str.data(), str.size());
buff += str.size();
}
inline void recursive_processor(const boost::json::value &element) {
switch(element.kind()) {
case boost::json::kind::array: {
write_byte(0xdd);
const auto &array = element.as_array();
write_uint32(static_cast<uint32_t>(array.size()));
for (const auto &child : array) {
recursive_processor(child);
}
} break;
case boost::json::kind::object: {
write_byte(0xdf);
const auto &object = element.as_object();
write_uint32(static_cast<uint32_t>(object.size()));
for (const auto &child : object) {
write_string(child.key_c_str());
recursive_processor(child.value());
}
} break;
case boost::json::kind::int64:
case boost::json::kind::uint64:
case boost::json::kind::double_:
write_double(element.to_number<double>());
break;
case boost::json::kind::string:
write_string(element.as_string().c_str());
break;
case boost::json::kind::bool_:
write_byte(0xc2 + element.as_bool());
break;
case boost::json::kind::null:
write_byte(0xc0);
break;
default:
printf("unexpected\n");
break;
}
}
};
struct boostjson {
using StringType=std::string;
boostjson2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer, std::string_view &result) {
auto root = boost::json::parse(json);
result = parser.to_msgpack(root, reinterpret_cast<uint8_t *>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, boostjson)->UseManualTime();
} // namespace json2msgpack
#endif // SIMDJSON_COMPETITION_BOOSTJSON
+14 -14
View File
@@ -8,7 +8,7 @@ namespace json2msgpack {
using namespace simdjson;
/**
* @brief The simdjsonondemand2msgpack struct is used to quickly convert
* @brief The simdjson2msgpack struct is used to quickly convert
* JSON strings to msgpack views. You must provide a pointer to
* a large memory region where the msgpack gets written. The
* buffer should be large enough to store the msgpack output (which
@@ -17,7 +17,7 @@ using namespace simdjson;
*
* Recommended usage:
*
* simdjsonondemand2msgpack parser{};
* simdjson2msgpack parser{};
* simdjson::padded_string json = "[1,2]"_padded; // some JSON
* uint8_t * buffer = new uint8_t[3*json.size() + simdjson::SIMDJSON_PADDING]; // large buffer
*
@@ -26,10 +26,10 @@ using namespace simdjson;
* The result (msgpack) is a string view to a msgpack serialization of the input JSON,
* it points inside the buffer you provided.
*
* You may reuse the simdjsonondemand2msgpack instance though you should use
* You may reuse the simdjson2msgpack instance though you should use
* one per thread.
*/
struct simdjsonondemand2msgpack {
struct simdjson2msgpack {
/**
* @brief Converts the provided JSON into msgpack.
*
@@ -58,7 +58,7 @@ private:
};
std::string_view
simdjsonondemand2msgpack::to_msgpack(const simdjson::padded_string &json,
simdjson2msgpack::to_msgpack(const simdjson::padded_string &json,
uint8_t *buf) {
buff = buf;
ondemand::document doc = parser.iterate(json);
@@ -106,33 +106,33 @@ simdjsonondemand2msgpack::to_msgpack(const simdjson::padded_string &json,
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void simdjsonondemand2msgpack::write_double(const double d) noexcept {
void simdjson2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void simdjsonondemand2msgpack::write_byte(const uint8_t b) noexcept {
void simdjson2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void simdjsonondemand2msgpack::write_uint32(const uint32_t w) noexcept {
void simdjson2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
uint8_t *simdjsonondemand2msgpack::skip_uint32() noexcept {
uint8_t *simdjson2msgpack::skip_uint32() noexcept {
uint8_t *ret = buff;
buff += sizeof(uint32_t);
return ret;
}
void simdjsonondemand2msgpack::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
void simdjson2msgpack::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
::memcpy(p, &w, sizeof(w));
}
void simdjsonondemand2msgpack::write_raw_string(
void simdjson2msgpack::write_raw_string(
simdjson::ondemand::raw_json_string in) {
write_byte(0xdb);
uint8_t *location = skip_uint32();
@@ -140,7 +140,7 @@ void simdjsonondemand2msgpack::write_raw_string(
write_uint32_at(uint32_t(v.size()), location);
}
void simdjsonondemand2msgpack::recursive_processor(simdjson::ondemand::value element) {
void simdjson2msgpack::recursive_processor(simdjson::ondemand::value element) {
switch (element.type()) {
case simdjson::ondemand::json_type::array: {
uint32_t counter = 0;
@@ -185,7 +185,7 @@ void simdjsonondemand2msgpack::recursive_processor(simdjson::ondemand::value ele
}
void simdjsonondemand2msgpack::recursive_processor_ref(simdjson::ondemand::value& element) {
void simdjson2msgpack::recursive_processor_ref(simdjson::ondemand::value& element) {
switch (element.type()) {
case simdjson::ondemand::json_type::array: {
uint32_t counter = 0;
@@ -234,7 +234,7 @@ void simdjsonondemand2msgpack::recursive_processor_ref(simdjson::ondemand::value
struct simdjson_ondemand {
using StringType = std::string_view;
simdjsonondemand2msgpack parser{};
simdjson2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
@@ -1,250 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "json2msgpack.h"
namespace json2msgpack {
using namespace simdjson;
/**
* @brief The simdjsonsinglestage2msgpack struct is used to quickly convert
* JSON strings to msgpack views. You must provide a pointer to
* a large memory region where the msgpack gets written. The
* buffer should be large enough to store the msgpack output (which
* can never be 3x larger than the input JSON) with an additional
* simdjson::SIMDJSON_PADDING bytes.
*
* Recommended usage:
*
* simdjsonsinglestage2msgpack parser{};
* simdjson::padded_string json = "[1,2]"_padded; // some JSON
* uint8_t * buffer = new uint8_t[3*json.size() + simdjson::SIMDJSON_PADDING]; // large buffer
*
* std::string_view msgpack = parser.to_msgpack(json, buffer);
*
* The result (msgpack) is a string view to a msgpack serialization of the input JSON,
* it points inside the buffer you provided.
*
* You may reuse the simdjsonsinglestage2msgpack instance though you should use
* one per thread.
*/
struct simdjsonsinglestage2msgpack {
/**
* @brief Converts the provided JSON into msgpack.
*
* @param json JSON input
* @param buf temporary buffer (must be large enough, with simdjson::SIMDJSON_PADDING bytes
* of padding)
* @return std::string_view msgpack output, writing to the temporary buffer
*/
inline std::string_view to_msgpack(const simdjson::padded_string &json,
uint8_t *buf);
private:
simdjson_inline void write_double(const double d) noexcept;
simdjson_inline void write_byte(const uint8_t b) noexcept;
simdjson_inline void write_uint32(const uint32_t w) noexcept;
simdjson_inline uint8_t *skip_uint32() noexcept;
simdjson_inline void write_uint32_at(const uint32_t w,
uint8_t *p) noexcept;
simdjson_inline void
write_raw_string(simdjson::singlestage::raw_json_string rjs);
inline void recursive_processor(simdjson::singlestage::value element);
inline void recursive_processor_ref(simdjson::singlestage::value& element);
simdjson::singlestage::parser parser;
uint8_t *buff{};
};
std::string_view
simdjsonsinglestage2msgpack::to_msgpack(const simdjson::padded_string &json,
uint8_t *buf) {
buff = buf;
singlestage::document doc = parser.iterate(json);
if (doc.is_scalar()) {
// we have a special case where the JSON document is a single document...
switch (doc.type()) {
case simdjson::singlestage::json_type::number:
write_double(doc.get_double());
break;
case simdjson::singlestage::json_type::string:
write_raw_string(doc.get_raw_json_string());
break;
case simdjson::singlestage::json_type::boolean:
write_byte(0xc2 + doc.get_bool());
break;
case simdjson::singlestage::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(doc.is_null()) {
write_byte(0xc0);
}
break;
case simdjson::singlestage::json_type::array:
case simdjson::singlestage::json_type::object:
default:
// impossible
SIMDJSON_UNREACHABLE();
}
} else {
simdjson::singlestage::value val = doc;
#define SIMDJSON_GCC_COMPILER ((__GNUC__) && !(__clang__) && !(__INTEL_COMPILER))
#if SIMDJSON_GCC_COMPILER
// the GCC compiler does well with by-value passing.
// GCC has superior recursive inlining:
// https://stackoverflow.com/questions/29186186/why-does-gcc-generate-a-faster-program-than-clang-in-this-recursive-fibonacci-co
// https://godbolt.org/z/TeK4doE51
recursive_processor(val);
#else
recursive_processor_ref(val);
#endif
}
if (!doc.at_end()) {
throw "There are unexpectedly tokens after the end of the json in the json2msgpack sample data";
}
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void simdjsonsinglestage2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void simdjsonsinglestage2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void simdjsonsinglestage2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
uint8_t *simdjsonsinglestage2msgpack::skip_uint32() noexcept {
uint8_t *ret = buff;
buff += sizeof(uint32_t);
return ret;
}
void simdjsonsinglestage2msgpack::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
::memcpy(p, &w, sizeof(w));
}
void simdjsonsinglestage2msgpack::write_raw_string(
simdjson::singlestage::raw_json_string in) {
write_byte(0xdb);
uint8_t *location = skip_uint32();
std::string_view v = parser.unescape(in, buff);
write_uint32_at(uint32_t(v.size()), location);
}
void simdjsonsinglestage2msgpack::recursive_processor(simdjson::singlestage::value element) {
switch (element.type()) {
case simdjson::singlestage::json_type::array: {
uint32_t counter = 0;
write_byte(0xdd);
uint8_t *location = skip_uint32();
for (auto child : element.get_array()) {
counter++;
recursive_processor(child.value());
}
write_uint32_at(counter, location);
} break;
case simdjson::singlestage::json_type::object: {
uint32_t counter = 0;
write_byte(0xdf);
uint8_t *location = skip_uint32();
for (auto field : element.get_object()) {
counter++;
write_raw_string(field.key());
recursive_processor(field.value());
}
write_uint32_at(counter, location);
} break;
case simdjson::singlestage::json_type::number:
write_double(element.get_double());
break;
case simdjson::singlestage::json_type::string:
write_raw_string(element.get_raw_json_string());
break;
case simdjson::singlestage::json_type::boolean:
write_byte(0xc2 + element.get_bool());
break;
case simdjson::singlestage::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
write_byte(0xc0);
}
break;
default:
SIMDJSON_UNREACHABLE();
}
}
void simdjsonsinglestage2msgpack::recursive_processor_ref(simdjson::singlestage::value& element) {
switch (element.type()) {
case simdjson::singlestage::json_type::array: {
uint32_t counter = 0;
write_byte(0xdd);
uint8_t *location = skip_uint32();
for (auto child : element.get_array()) {
counter++;
simdjson::singlestage::value v = child.value();
recursive_processor_ref(v);
}
write_uint32_at(counter, location);
} break;
case simdjson::singlestage::json_type::object: {
uint32_t counter = 0;
write_byte(0xdf);
uint8_t *location = skip_uint32();
for (auto field : element.get_object()) {
counter++;
write_raw_string(field.key());
simdjson::singlestage::value v = field.value();
recursive_processor_ref(v);
}
write_uint32_at(counter, location);
} break;
case simdjson::singlestage::json_type::number:
write_double(element.get_double());
break;
case simdjson::singlestage::json_type::string:
write_raw_string(element.get_raw_json_string());
break;
case simdjson::singlestage::json_type::boolean:
write_byte(0xc2 + element.get_bool());
break;
case simdjson::singlestage::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
write_byte(0xc0);
}
break;
default:
SIMDJSON_UNREACHABLE();
}
}
struct simdjson_singlestage {
using StringType = std::string_view;
simdjsonsinglestage2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
result = parser.to_msgpack(json, reinterpret_cast<uint8_t *>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, simdjson_singlestage)->UseManualTime();
} // namespace json2msgpack
#endif // SIMDJSON_EXCEPTIONS
+3 -1
View File
@@ -100,6 +100,7 @@ struct yyjson : yyjson2msgpack {
std::string_view &result) {
yyjson_doc *doc = yyjson_read(json.data(), json.size(), 0);
result = to_msgpack(doc, reinterpret_cast<uint8_t*>(buffer));
yyjson_doc_free(doc);
return true;
}
};
@@ -113,6 +114,7 @@ struct yyjson_insitu : yyjson2msgpack {
yyjson_doc *doc =
yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0);
result = to_msgpack(doc, reinterpret_cast<uint8_t*>(buffer));
yyjson_doc_free(doc);
return true;
}
};
@@ -120,4 +122,4 @@ BENCHMARK_TEMPLATE(json2msgpack, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace json2msgpack
#endif // SIMDJSON_COMPETITION_YYJSON
#endif // SIMDJSON_COMPETITION_YYJSON
+1 -1
View File
@@ -24,7 +24,7 @@ struct file_runner : public runner_base<I> {
simdjson_warn_unused bool before_run(benchmark::State &state) {
if (!runner_base<I>::after_run(state)) { return false; };
// Copy the original json in case we did *in situ* last time
// Copy the original JSON in case we did *in situ* last time
std::memcpy(json.data(), original_json.data(), original_json.size());
return true;
}
@@ -13,15 +13,13 @@ void maybe_display_implementation() {
std::cout << "simdjson::dom implementation: " << simdjson::get_active_implementation()->name() << std::endl;
std::cout << "simdjson::ondemand implementation (stage 1): " << simdjson::get_active_implementation()->name() << std::endl;
std::cout << "simdjson::ondemand implementation (stage 2): " << simdjson::builtin_implementation()->name() << std::endl;
std::cout << "simdjson::singlestage implementation (stage 1): " << simdjson::get_active_implementation()->name() << std::endl;
std::cout << "simdjson::singlestage implementation (stage 2): " << simdjson::builtin_implementation()->name() << std::endl;
}
}
template<typename B, typename R> static void run_json_benchmark(benchmark::State &state) {
maybe_display_implementation();
event_collector collector(true);
event_collector collector;
event_aggregate events;
// Warmup and equality check (make sure the data is right!)
+1 -1
View File
@@ -13,7 +13,7 @@ struct string_runner : public runner_base<I> {
simdjson_warn_unused bool before_run(benchmark::State &state) {
if (!runner_base<I>::after_run(state)) { return false; };
// Copy the original json in case we did *in situ*
// Copy the original JSON in case we did *in situ*
std::memcpy(json.data(), original_json.data(), original_json.size());
return true;
}
+29
View File
@@ -0,0 +1,29 @@
#pragma once
#ifdef SIMDJSON_COMPETITION_BOOSTJSON
#include "kostya.h"
namespace kostya {
struct boostjson {
static constexpr diff_flags DiffFlags = diff_flags::IMPRECISE_FLOATS;
bool run(simdjson::padded_string &json, std::vector<point> &result) {
auto root = boost::json::parse(json);
for (const auto &point : root.at("coordinates").as_array()) {
result.emplace_back(json_benchmark::point{
point.at("x").to_number<double>(),
point.at("y").to_number<double>(),
point.at("z").to_number<double>()
});
}
return true;
}
};
BENCHMARK_TEMPLATE(kostya, boostjson)->UseManualTime();
} // namespace kostya
#endif // SIMDJSON_COMPETITION_BOOSTJSON
-29
View File
@@ -1,29 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "kostya.h"
namespace kostya {
using namespace simdjson;
struct simdjson_singlestage {
static constexpr diff_flags DiffFlags = diff_flags::NONE;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::vector<point> &result) {
auto doc = parser.iterate(json);
for (singlestage::object point : doc.find_field("coordinates")) {
result.emplace_back(json_benchmark::point{point.find_field("x"), point.find_field("y"), point.find_field("z")});
}
return true;
}
};
BENCHMARK_TEMPLATE(kostya, simdjson_singlestage)->UseManualTime();
} // namespace kostya
#endif // SIMDJSON_EXCEPTIONS

Some files were not shown because too many files have changed in this diff Show More