Compare commits

..

49 Commits

Author SHA1 Message Date
John Keiser d1de135ebd Updates 2023-09-13 11:57:43 -04:00
John Keiser 0edf1008c9 Updates 2023-09-13 11:21:58 -04:00
John Keiser 6a0d7bcb55 Right-justify int columns in markdown tables 2023-09-13 11:10:01 -04:00
John Keiser 6f196d0af0 Add results 2023-09-12 15:02:17 -04:00
John Keiser c3414a1234 Show more transitions 2023-09-12 14:58:33 -04:00
John Keiser 5d2107b470 Add jsonexamples generator and miss reduction script 2023-09-12 14:36:11 -04:00
John Keiser 90aa198430 Moar jsonexamples 2023-09-12 14:36:11 -04:00
John Keiser 02b488b88d All branch tests for twitter.json 2023-09-12 14:36:10 -04:00
John Keiser 47e3e95867 Branch tests with twitter.json. 2023-09-12 14:36:10 -04:00
Daniel Lemire 588c0677f5 Redesigning visit_primitive so that it is optimized for strings and
numbers.
2023-09-05 11:05:15 -04:00
Daniel Lemire 68ba9a1b2a Adding a few spaces (minor formatting) (#2059) 2023-08-30 20:23:58 -04:00
Daniel Lemire 6fed6bd29b Update README.md 2023-08-30 19:26:34 -04:00
Piotr Rżysko cc24bb4114 Removed number parsing fallback (#2056) 2023-08-29 18:28:21 -04:00
John Keiser 14ec0ca8f4 Merge pull request #2052 from simdjson/jkeiser/structural_indexer_step
Write out 4 structural indexes at a time instead of 8
2023-08-29 15:24:54 -07:00
John Keiser 8dabd02c3a Default to step = 4 2023-08-29 15:48:52 -04:00
Daniel Lemire c9692005fd This allows us to pass SIMDJSON_STRUCTURAL_INDEXER_STEP as a command-line option. (#2055) 2023-08-25 13:47:20 -04:00
Daniel Lemire 2e0a9397c9 Merge branch 'master' into jkeiser/structural_indexer_step 2023-08-25 11:34:13 -04:00
Daniel Lemire f7922075c3 Adding support for Apple event counters (#2054) 2023-08-25 11:33:24 -04:00
Daniel Lemire b2e20e04c9 Standard compatibility fixes (#2053)
* Standard compatibility fixes

* missing commit

* Should work.

* Fix.

* Fix.

* Should work now.

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-08-25 10:18:02 -04:00
John Keiser adc9d18efd Change step for structural_indexes to 2 2023-08-23 19:02:05 -07:00
Daniel Lemire bf849e3619 Preparing release 2023-08-22 14:05:14 -04:00
Daniel Lemire c09e8dc225 Icelake processes 64 bytes at a time when processing strings. (#2050)
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-08-18 14:37:10 -04:00
Daniel Lemire 5bf585ef02 Fixing failing thread sanitizer (#2051) 2023-08-18 14:35:12 -04:00
Daniel Lemire caff09cafc Update iterate_many.md 2023-08-14 12:18:09 -04:00
Daniel Lemire f043db4dca Preparing release. 2023-08-02 16:01:39 -04:00
Daniel Lemire e3bd0828d1 Fix for issue 2045. (#2046)
* Fix for issue 2045.

* Retauring single-header code.
2023-08-02 16:00:45 -04:00
John Keiser 9408298e0e Merge pull request #2042 from simdjson/jkeiser/escape3
New escape algorithm with significantly less data dependency
2023-08-02 10:38:09 -07:00
John Keiser 573bbac2d2 Reamalgamate again 2023-08-02 08:28:35 -07:00
John Keiser ac78c625df simdjson_inline -> simdjson_really_inline 2023-08-02 08:28:05 -07:00
John Keiser 390a66c6e8 Make next_structurals inline 2023-08-02 08:28:05 -07:00
John Keiser 210e0a56a1 Reamalgamate 2023-08-02 08:28:05 -07:00
John Keiser 084f662f32 Don't use C++20 designated initializers 2023-08-02 08:28:05 -07:00
John Keiser c878ff2500 Fix compiler warnings 2023-08-02 08:28:05 -07:00
John Keiser 06afe9ecb3 New escape algorithm with significantly less data dependency 2023-08-02 08:28:05 -07:00
Daniel Lemire e658502adb We are having issues with llvm.sh (#2044) 2023-08-02 11:24:28 -04:00
Daniel Lemire 6e7d415040 These lines should not be necessary. (#2043) 2023-08-01 19:33:27 -04:00
John Keiser 2ccd5b9c97 Merge pull request #2039 from simdjson/jkeiser/more-headers
Disallow including headers from implementation files
2023-07-21 12:09:57 -07:00
Daniel Lemire 1cdcbf79b3 We do actually support legacy 32-bit systems, although not with fast kernels. (#2041) 2023-07-21 13:53:10 -04:00
Calum McConnell 7013ed98cc Peek(0) is an error on 32-bit systems, where pointers are indistinguishable from int32_t (#2040) 2023-07-21 13:52:53 -04:00
John Keiser b383c717d7 Have amalgamate.py verify it's actually amalgamating all the files 2023-07-20 15:11:52 -07:00
John Keiser b8743bced0 Fix amalgamate.py under python 3.7 2023-07-20 14:14:44 -07:00
John Keiser ae9ba2bed2 Rename SIMDJSON_AMALGAMATED to SIMDJSON_CONDITIONAL_INCLUDE 2023-07-20 11:43:19 -07:00
Daniel Lemire 939844d79f Adding tests for issue 2037. (#2038)
* Adding tests for issue 2037.

* Fixed typo

* Simplifying.
2023-07-20 14:11:33 -04:00
John Keiser 4f4e81668e Change all include paths to <> instead of "" to avoid relative path includes 2023-07-20 10:48:25 -07:00
John Keiser bb54946b78 Disallow including headers from implementation files 2023-07-20 10:01:22 -07:00
John Keiser 4dec6ed5a7 Merge pull request #2036 from simdjson/jkeiser/more-header-fixes
A few more header fixes
2023-07-18 10:26:40 -07:00
John Keiser 1dfd48195d Merge pull request #2035 from simdjson/jkeiser/sanitizer-ci
Run sanitizers in parallel
2023-07-17 23:37:34 -07:00
John Keiser 23afc6b695 Merge branch 'master' into jkeiser/sanitizer-ci 2023-07-17 16:09:20 -07:00
John Keiser 80bc9e5051 Run sanitizers in parallel 2023-07-17 12:57:50 -07:00
332 changed files with 1082442 additions and 52712 deletions
+22 -1
View File
@@ -1,5 +1,8 @@
CompileFlags:
CompilationDatabase: build
Add:
- -Werror -Wall -Wextra -Weffc++ -Wsign-compare -Wshadow -Wwrite-strings -Wpointer-arith -Winit-self -Wconversion -Wno-sign-conversion
- -Wundefined-inline
Diagnostics:
Suppress:
- misc-unused-alias-decls
@@ -16,10 +19,28 @@ If:
CompileFlags:
Add:
- -Wno-unneeded-internal-declaration
- -Wno-undefined-inline # TODO fix and remove these violations
- -Wno-undefined-internal # TODO fix and remove these violations
- -Wno-unused-function
- -Wno-unused-const-variable
Diagnostics:
Suppress:
- pp_including_mainfile_in_preamble
---
# Amalgamated files that require or partly define an implementation
If:
PathMatch:
- .*/(arm64|fallback|haswell|icelake|ppc64|westmere)/begin.h
- .*/generic/.*
Diagnostics:
Suppress:
- pragma_attribute_no_pop_eof
---
# clang has a bad time detecting the push/pop together in src/ for some reason
If:
PathMatch:
- include/simdjson/.*/end.h
- src/(arm64|fallback|haswell|icelake|ppc64|westmere).cpp
Diagnostics:
Suppress:
- pragma_attribute_no_pop_eof
- pragma_attribute_stack_mismatch
+1 -1
View File
@@ -49,7 +49,7 @@ It is fine to report bugs against our main branch, but if that is what you are d
- Version [e.g. 22]
- Optimization setting (e.g., -O3)
We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux systems. Please ensure that your configuration is supported before labelling the issue as a bug. In particular, we do not support legacy 32-bit systems.
We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux systems. Please ensure that your configuration is supported before labelling the issue as a bug.
**Indicate whether you are willing or able to provide a bug fix as a pull request**
+2 -1
View File
@@ -20,7 +20,7 @@ jobs:
# fuzzers that change behaviour with SIMDJSON_FORCE_IMPLEMENTATION
defaultimplfuzzers: atpointer dump dump_raw_tape element minify parser print_json
# fuzzers that loop over the implementations themselves, or don't need to switch.
implfuzzers: implementations minifyimpl ndjson ondemand padded utf8 # TODO add singlestage
implfuzzers: implementations minifyimpl ndjson ondemand padded utf8
implementations: haswell westmere fallback
UBSAN_OPTIONS: halt_on_error=1
MAXLEN: -max_len=4000
@@ -34,6 +34,7 @@ jobs:
sudo apt update
sudo apt-get install --quiet ninja-build valgrind zip unzip lsb-release wget software-properties-common gnupg
wget https://apt.llvm.org/llvm.sh
sudo apt-get purge --auto-remove llvm python3-lldb-14 llvm-14
chmod +x llvm.sh
sudo ./llvm.sh $CLANGVERSION
+13 -1
View File
@@ -3,7 +3,7 @@ name: Ubuntu 20.04 CI (GCC 9) With Memory Sanitizer
on: [push, pull_request]
jobs:
ubuntu-build:
ubuntu-build-address-sanitizier:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
@@ -21,6 +21,18 @@ jobs:
cmake -DSIMDJSON_SANITIZE=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j
ubuntu-build-undefined-sanitizer:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
ctest --output-on-failure -LE explicitonly -j
- name: Use cmake with undefined sanitizer
run: |
mkdir builddebugundefsani &&
+14
View File
@@ -14,6 +14,20 @@ jobs:
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake to build just the library
run: |
mkdir buildjustlib &&
cd buildjustlib &&
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DSIMDJSON_DEVELOPER_MODE=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp &&
c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson &&
cd ../tests/installation_tests/find &&
mkdir buildjustlib &&
cd buildjustlib &&
cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../buildjustlib/destination .. &&
cmake --build .
- name: Use cmake
run: |
mkdir builddebug &&
-2
View File
@@ -14,8 +14,6 @@ jobs:
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install clang++-13
run: sudo apt-get install -y clang++-13
- name: Use cmake
run: |
mkdir build &&
-2
View File
@@ -14,8 +14,6 @@ jobs:
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install clang++-14
run: sudo apt-get install -y clang++-14
- name: Use cmake
run: |
mkdir build &&
@@ -1,4 +1,4 @@
name: Ubuntu 20.04 CI (GCC 9) with Thread Sanitizer
name: Ubuntu 22.04 CI (GCC 11) with Thread Sanitizer
on: [push, pull_request]
@@ -7,7 +7,7 @@ jobs:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
+14
View File
@@ -14,6 +14,20 @@ jobs:
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake to build just the library
run: |
mkdir buildjustlib &&
cd buildjustlib &&
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DSIMDJSON_DEVELOPER_MODE=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp &&
c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson &&
cd ../tests/installation_tests/find &&
mkdir buildjustlib &&
cd buildjustlib &&
cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../buildjustlib/destination .. &&
cmake --build .
- name: Use cmake
run: |
mkdir builddebug &&
+3 -2
View File
@@ -3,7 +3,7 @@
{"column": 95 },
{"column": 120 }
],
"files.trimTrailingWhitespace": true,
"files.trimTrailingWhitespace": false,
"files.associations": {
".clangd": "yaml",
"array": "cpp",
@@ -99,5 +99,6 @@
"shared_mutex": "cpp",
"ranges": "cpp",
"span": "cpp"
}
},
"editor.trimAutoWhitespace": true
}
+1 -1
View File
@@ -3,7 +3,7 @@ cmake_minimum_required(VERSION 3.14)
project(
simdjson
# The version number is modified by tools/release.py
VERSION 3.2.1
VERSION 3.2.3
DESCRIPTION "Parsing gigabytes of JSON per second"
HOMEPAGE_URL "https://simdjson.org/"
LANGUAGES CXX C
+1 -1
View File
@@ -38,7 +38,7 @@ PROJECT_NAME = simdjson
# could be handy for archiving the generated documentation or if some version
# control system is used.
PROJECT_NUMBER = "3.2.1"
PROJECT_NUMBER = "3.2.3"
# Using the PROJECT_BRIEF tag one can provide an optional one line description
# for a project that appears at the top of each page and should give viewer a
-2
View File
@@ -1,8 +1,6 @@
[![Ubuntu 20.04 CI](https://github.com/simdjson/simdjson/workflows/Ubuntu%2020.04%20CI%20(GCC%209)/badge.svg)](https://simdjson.org/plots.html)
![VS16-CI](https://github.com/simdjson/simdjson/workflows/VS16-CI/badge.svg)
[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/simdjson.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:simdjson)
![MinGW64-CI](https://github.com/simdjson/simdjson/workflows/MinGW64-CI/badge.svg)
[![][license img]][license]
[![Doxygen Documentation](https://img.shields.io/badge/docs-doxygen-green.svg)](https://simdjson.github.io/simdjson/)
@@ -1,72 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "amazon_cellphones.h"
namespace amazon_cellphones {
using namespace simdjson;
template<bool threaded>
struct simdjson_singlestage {
using StringType = std::string;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::map<StringType, brand> &result) {
#ifdef SIMDJSON_THREADS_ENABLED
parser.threaded = threaded;
#endif
singlestage::document_stream stream = parser.iterate_many(json);
singlestage::document_stream::iterator i = stream.begin();
++i; // Skip first line
for (;i != stream.end(); ++i) {
auto doc = *i;
size_t index{0};
StringType copy;
double rating;
uint64_t reviews;
for ( auto value : doc ) {
switch (index)
{
case 1:
copy = StringType(std::string_view(value));
break;
case 5:
rating = double(value);
break;
case 7:
reviews = uint64_t(value);
break;
default:
break;
}
index++;
}
auto x = result.find(copy);
if (x == result.end()) { // If key not found, add new key
result.emplace(copy, amazon_cellphones::brand{
rating * reviews,
reviews
});
} else { // Otherwise, update key data
x->second.cumulative_rating += rating * reviews;
x->second.reviews_count += reviews;
}
}
return true;
}
};
BENCHMARK_TEMPLATE(amazon_cellphones, simdjson_singlestage<UNTHREADED>)->UseManualTime();
#ifdef SIMDJSON_THREADS_ENABLED
BENCHMARK_TEMPLATE(amazon_cellphones, simdjson_singlestage<THREADED>)->UseManualTime();
#endif
} // namespace amazon_cellphones
#endif // SIMDJSON_EXCEPTIONS
File diff suppressed because it is too large Load Diff
-13
View File
@@ -26,7 +26,6 @@ SIMDJSON_PUSH_DISABLE_ALL_WARNINGS
SIMDJSON_POP_DISABLE_WARNINGS
#include "json2msgpack/simdjson_ondemand.h"
#include "json2msgpack/simdjson_singlestage.h"
#include "json2msgpack/simdjson_dom.h"
#include "json2msgpack/yyjson.h"
#include "json2msgpack/rapidjson.h"
@@ -36,7 +35,6 @@ SIMDJSON_POP_DISABLE_WARNINGS
#include "json2msgpack/nlohmann_json.h"
#include "partial_tweets/simdjson_ondemand.h"
#include "partial_tweets/simdjson_singlestage.h"
#include "partial_tweets/simdjson_dom.h"
#include "partial_tweets/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -54,8 +52,6 @@ SIMDJSON_POP_DISABLE_WARNINGS
#include "distinct_user_id/simdjson_ondemand.h"
#include "distinct_user_id/simdjson_ondemand_json_pointer.h"
#include "distinct_user_id/simdjson_singlestage.h"
#include "distinct_user_id/simdjson_singlestage_json_pointer.h"
#include "distinct_user_id/simdjson_dom.h"
#include "distinct_user_id/simdjson_dom_json_pointer.h"
#include "distinct_user_id/yyjson.h"
@@ -72,7 +68,6 @@ SIMDJSON_POP_DISABLE_WARNINGS
#endif // SIMDJSON_COMPETITION_SAX
#include "find_tweet/simdjson_ondemand.h"
#include "find_tweet/simdjson_singlestage.h"
#include "find_tweet/simdjson_dom.h"
#include "find_tweet/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -88,7 +83,6 @@ SIMDJSON_POP_DISABLE_WARNINGS
#endif // SIMDJSON_COMPETITION_SAX
#include "top_tweet/simdjson_ondemand.h"
#include "top_tweet/simdjson_singlestage.h"
#include "top_tweet/simdjson_dom.h"
#include "top_tweet/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -105,7 +99,6 @@ SIMDJSON_POP_DISABLE_WARNINGS
#include "kostya/simdjson_ondemand.h"
#include "kostya/simdjson_singlestage.h"
#include "kostya/simdjson_dom.h"
#include "kostya/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -124,10 +117,6 @@ SIMDJSON_POP_DISABLE_WARNINGS
#if SIMDJSON_COMPETITION_ONDEMAND_UNORDERED
#include "large_random/simdjson_ondemand_unordered.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_UNORDERED
#include "large_random/simdjson_singlestage.h"
#if SIMDJSON_COMPETITION_SINGLESTAGE_UNORDERED
#include "large_random/simdjson_singlestage_unordered.h"
#endif // SIMDJSON_COMPETITION_SINGLESTAGE_UNORDERED
#include "large_random/simdjson_dom.h"
#include "large_random/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
@@ -144,10 +133,8 @@ SIMDJSON_POP_DISABLE_WARNINGS
#include "amazon_cellphones/simdjson_dom.h"
#include "amazon_cellphones/simdjson_ondemand.h"
#include "amazon_cellphones/simdjson_singlestage.h"
#include "large_amazon_cellphones/simdjson_dom.h"
#include "large_amazon_cellphones/simdjson_ondemand.h"
#include "large_amazon_cellphones/simdjson_singlestage.h"
BENCHMARK_MAIN();
+13 -9
View File
@@ -445,7 +445,7 @@ struct benchmarker {
return 100.0 * a / b;
}
void print(bool tabbed_output) const {
void print(bool tabbed_output, bool stage1_only) const {
if (tabbed_output) {
char* filename_copy = reinterpret_cast<char*>(malloc(strlen(filename)+1));
SIMDJSON_PUSH_DISABLE_WARNINGS
@@ -503,17 +503,21 @@ struct benchmarker {
stats->blocks_with_16_structurals_flipped, percent(stats->blocks_with_16_structurals_flipped, stats->blocks));
}
printf("\n");
printf("All Stages (excluding allocation)\n");
print_aggregate("| " , all_stages_without_allocation.best);
// frequently, allocation is a tiny fraction of the running time so we omit it
if(allocate_stage.best.elapsed_sec() > 0.01 * all_stages_without_allocation.best.elapsed_sec()) {
printf("|- Allocation\n");
print_aggregate("| ", allocate_stage.best);
if(!stage1_only) {
printf("All Stages (excluding allocation)\n");
print_aggregate("| " , all_stages_without_allocation.best);
// frequently, allocation is a tiny fraction of the running time so we omit it
if(allocate_stage.best.elapsed_sec() > 0.01 * all_stages_without_allocation.best.elapsed_sec()) {
printf("|- Allocation\n");
print_aggregate("| ", allocate_stage.best);
}
}
printf("|- Stage 1\n");
print_aggregate("| ", stage1.best);
printf("|- Stage 2\n");
print_aggregate("| ", stage2.best);
if(!stage1_only) {
printf("|- Stage 2\n");
print_aggregate("| ", stage2.best);
}
if (collector.has_events()) {
double freq1 = (stage1.best.cycles() / stage1.best.elapsed_sec()) / 1000000000.0;
double freq2 = (stage2.best.cycles() / stage2.best.elapsed_sec()) / 1000000000.0;
@@ -1,37 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "distinct_user_id.h"
namespace distinct_user_id {
using namespace simdjson;
struct simdjson_singlestage {
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
// Walk the document, parsing as we go
auto doc = parser.iterate(json);
for (singlestage::object tweet : doc.find_field("statuses")) {
// We believe that all statuses have a matching
// user, and we are willing to throw when they do not.
result.push_back(tweet.find_field("user").find_field("id"));
// Not all tweets have a "retweeted_status", but when they do
// we want to go and find the user within.
auto retweet = tweet.find_field("retweeted_status");
if (!retweet.error()) {
result.push_back(retweet.find_field("user").find_field("id"));
}
}
return true;
}
};
BENCHMARK_TEMPLATE(distinct_user_id, simdjson_singlestage)->UseManualTime();
} // namespace distinct_user_id
#endif // SIMDJSON_EXCEPTIONS
@@ -1,37 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "distinct_user_id.h"
namespace distinct_user_id {
using namespace simdjson;
struct simdjson_singlestage_json_pointer {
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
// Walk the document, parsing as we go
auto doc = parser.iterate(json);
for (singlestage::object tweet : doc.find_field("statuses")) {
// We believe that all statuses have a matching
// user, and we are willing to throw when they do not.
result.push_back(tweet.at_pointer("/user/id"));
// Not all tweets have a "retweeted_status", but when they do
// we want to go and find the user within.
auto retweet_id = tweet.at_pointer("/retweeted_status/user/id");
if (retweet_id.error() != NO_SUCH_FIELD) {
result.push_back(retweet_id);
}
}
return true;
}
};
BENCHMARK_TEMPLATE(distinct_user_id, simdjson_singlestage_json_pointer)->UseManualTime();
} // namespace distinct_user_id
#endif // SIMDJSON_EXCEPTIONS
+1 -1
View File
@@ -218,7 +218,7 @@ int main(int argc, char *argv[]) {
if (!options.verbose) { progress.erase(); }
for (size_t i=0; i<options.files.size(); i++) {
benchmarkers[i]->print(options.tabbed_output);
benchmarkers[i]->print(options.tabbed_output, options.stage1_only);
delete benchmarkers[i];
}
+28 -3
View File
@@ -34,11 +34,15 @@
#include <string>
#include <vector>
#include "linux-perf-events.h"
#ifdef __linux__
#include "linux-perf-events.h"
#include <libgen.h>
#endif
#if __APPLE__ && __aarch64__
#include "apple/apple_arm_events.h"
#endif
#include "simdjson.h"
using std::string;
@@ -134,7 +138,7 @@ struct event_collector {
#if defined(__linux__)
LinuxEvents<PERF_TYPE_HARDWARE> linux_events;
event_collector(simdjson_unused bool quiet = false) : linux_events(vector<int>{
event_collector() : linux_events(vector<int>{
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
PERF_COUNT_HW_CPU_CYCLES,
PERF_COUNT_HW_INSTRUCTIONS,
@@ -149,8 +153,17 @@ struct event_collector {
bool has_events() {
return linux_events.is_working();
}
#elif __APPLE__ && __aarch64__
AppleEvents apple_events;
performance_counters diff;
event_collector() : diff(0) {
apple_events.setup_performance_counters();
}
bool has_events() {
return apple_events.setup_performance_counters();
}
#else
event_collector(simdjson_unused bool _quiet = false) {}
event_collector() {}
bool has_events() {
return false;
}
@@ -159,6 +172,8 @@ struct event_collector {
simdjson_inline void start() {
#if defined(__linux)
linux_events.start();
#elif __APPLE__ && __aarch64__
if(has_events()) { diff = apple_events.get_counters(); }
#endif
start_clock = steady_clock::now();
}
@@ -166,6 +181,16 @@ struct event_collector {
time_point<steady_clock> end_clock = steady_clock::now();
#if defined(__linux)
linux_events.end(count.event_counts);
#elif __APPLE__ && __aarch64__
if(has_events()) {
performance_counters end = apple_events.get_counters();
diff = end - diff;
}
count.event_counts[0] = diff.cycles;
count.event_counts[1] = diff.instructions;
count.event_counts[2] = diff.missed_branches;
count.event_counts[3] = 0;
count.event_counts[4] = 0;
#endif
count.elapsed = end_clock - start_clock;
return count;
@@ -1,33 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "find_tweet.h"
namespace find_tweet {
using namespace simdjson;
struct simdjson_singlestage {
using StringType=std::string_view;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, uint64_t find_id, std::string_view &result) {
// Walk the document, parsing as we go
auto doc = parser.iterate(json);
for (auto tweet : doc.find_field("statuses")) {
if (uint64_t(tweet.find_field("id")) == find_id) {
result = tweet.find_field("text");
return true;
}
}
return false;
}
};
BENCHMARK_TEMPLATE(find_tweet, simdjson_singlestage)->UseManualTime();
} // namespace find_tweet
#endif // SIMDJSON_EXCEPTIONS
+14 -14
View File
@@ -8,7 +8,7 @@ namespace json2msgpack {
using namespace simdjson;
/**
* @brief The simdjsonondemand2msgpack struct is used to quickly convert
* @brief The simdjson2msgpack struct is used to quickly convert
* JSON strings to msgpack views. You must provide a pointer to
* a large memory region where the msgpack gets written. The
* buffer should be large enough to store the msgpack output (which
@@ -17,7 +17,7 @@ using namespace simdjson;
*
* Recommended usage:
*
* simdjsonondemand2msgpack parser{};
* simdjson2msgpack parser{};
* simdjson::padded_string json = "[1,2]"_padded; // some JSON
* uint8_t * buffer = new uint8_t[3*json.size() + simdjson::SIMDJSON_PADDING]; // large buffer
*
@@ -26,10 +26,10 @@ using namespace simdjson;
* The result (msgpack) is a string view to a msgpack serialization of the input JSON,
* it points inside the buffer you provided.
*
* You may reuse the simdjsonondemand2msgpack instance though you should use
* You may reuse the simdjson2msgpack instance though you should use
* one per thread.
*/
struct simdjsonondemand2msgpack {
struct simdjson2msgpack {
/**
* @brief Converts the provided JSON into msgpack.
*
@@ -58,7 +58,7 @@ private:
};
std::string_view
simdjsonondemand2msgpack::to_msgpack(const simdjson::padded_string &json,
simdjson2msgpack::to_msgpack(const simdjson::padded_string &json,
uint8_t *buf) {
buff = buf;
ondemand::document doc = parser.iterate(json);
@@ -106,33 +106,33 @@ simdjsonondemand2msgpack::to_msgpack(const simdjson::padded_string &json,
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void simdjsonondemand2msgpack::write_double(const double d) noexcept {
void simdjson2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void simdjsonondemand2msgpack::write_byte(const uint8_t b) noexcept {
void simdjson2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void simdjsonondemand2msgpack::write_uint32(const uint32_t w) noexcept {
void simdjson2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
uint8_t *simdjsonondemand2msgpack::skip_uint32() noexcept {
uint8_t *simdjson2msgpack::skip_uint32() noexcept {
uint8_t *ret = buff;
buff += sizeof(uint32_t);
return ret;
}
void simdjsonondemand2msgpack::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
void simdjson2msgpack::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
::memcpy(p, &w, sizeof(w));
}
void simdjsonondemand2msgpack::write_raw_string(
void simdjson2msgpack::write_raw_string(
simdjson::ondemand::raw_json_string in) {
write_byte(0xdb);
uint8_t *location = skip_uint32();
@@ -140,7 +140,7 @@ void simdjsonondemand2msgpack::write_raw_string(
write_uint32_at(uint32_t(v.size()), location);
}
void simdjsonondemand2msgpack::recursive_processor(simdjson::ondemand::value element) {
void simdjson2msgpack::recursive_processor(simdjson::ondemand::value element) {
switch (element.type()) {
case simdjson::ondemand::json_type::array: {
uint32_t counter = 0;
@@ -185,7 +185,7 @@ void simdjsonondemand2msgpack::recursive_processor(simdjson::ondemand::value ele
}
void simdjsonondemand2msgpack::recursive_processor_ref(simdjson::ondemand::value& element) {
void simdjson2msgpack::recursive_processor_ref(simdjson::ondemand::value& element) {
switch (element.type()) {
case simdjson::ondemand::json_type::array: {
uint32_t counter = 0;
@@ -234,7 +234,7 @@ void simdjsonondemand2msgpack::recursive_processor_ref(simdjson::ondemand::value
struct simdjson_ondemand {
using StringType = std::string_view;
simdjsonondemand2msgpack parser{};
simdjson2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
@@ -1,250 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "json2msgpack.h"
namespace json2msgpack {
using namespace simdjson;
/**
* @brief The simdjsonsinglestage2msgpack struct is used to quickly convert
* JSON strings to msgpack views. You must provide a pointer to
* a large memory region where the msgpack gets written. The
* buffer should be large enough to store the msgpack output (which
* can never be 3x larger than the input JSON) with an additional
* simdjson::SIMDJSON_PADDING bytes.
*
* Recommended usage:
*
* simdjsonsinglestage2msgpack parser{};
* simdjson::padded_string json = "[1,2]"_padded; // some JSON
* uint8_t * buffer = new uint8_t[3*json.size() + simdjson::SIMDJSON_PADDING]; // large buffer
*
* std::string_view msgpack = parser.to_msgpack(json, buffer);
*
* The result (msgpack) is a string view to a msgpack serialization of the input JSON,
* it points inside the buffer you provided.
*
* You may reuse the simdjsonsinglestage2msgpack instance though you should use
* one per thread.
*/
struct simdjsonsinglestage2msgpack {
/**
* @brief Converts the provided JSON into msgpack.
*
* @param json JSON input
* @param buf temporary buffer (must be large enough, with simdjson::SIMDJSON_PADDING bytes
* of padding)
* @return std::string_view msgpack output, writing to the temporary buffer
*/
inline std::string_view to_msgpack(const simdjson::padded_string &json,
uint8_t *buf);
private:
simdjson_inline void write_double(const double d) noexcept;
simdjson_inline void write_byte(const uint8_t b) noexcept;
simdjson_inline void write_uint32(const uint32_t w) noexcept;
simdjson_inline uint8_t *skip_uint32() noexcept;
simdjson_inline void write_uint32_at(const uint32_t w,
uint8_t *p) noexcept;
simdjson_inline void
write_raw_string(simdjson::singlestage::raw_json_string rjs);
inline void recursive_processor(simdjson::singlestage::value element);
inline void recursive_processor_ref(simdjson::singlestage::value& element);
simdjson::singlestage::parser parser;
uint8_t *buff{};
};
std::string_view
simdjsonsinglestage2msgpack::to_msgpack(const simdjson::padded_string &json,
uint8_t *buf) {
buff = buf;
singlestage::document doc = parser.iterate(json);
if (doc.is_scalar()) {
// we have a special case where the JSON document is a single document...
switch (doc.type()) {
case simdjson::singlestage::json_type::number:
write_double(doc.get_double());
break;
case simdjson::singlestage::json_type::string:
write_raw_string(doc.get_raw_json_string());
break;
case simdjson::singlestage::json_type::boolean:
write_byte(0xc2 + doc.get_bool());
break;
case simdjson::singlestage::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(doc.is_null()) {
write_byte(0xc0);
}
break;
case simdjson::singlestage::json_type::array:
case simdjson::singlestage::json_type::object:
default:
// impossible
SIMDJSON_UNREACHABLE();
}
} else {
simdjson::singlestage::value val = doc;
#define SIMDJSON_GCC_COMPILER ((__GNUC__) && !(__clang__) && !(__INTEL_COMPILER))
#if SIMDJSON_GCC_COMPILER
// the GCC compiler does well with by-value passing.
// GCC has superior recursive inlining:
// https://stackoverflow.com/questions/29186186/why-does-gcc-generate-a-faster-program-than-clang-in-this-recursive-fibonacci-co
// https://godbolt.org/z/TeK4doE51
recursive_processor(val);
#else
recursive_processor_ref(val);
#endif
}
if (!doc.at_end()) {
throw "There are unexpectedly tokens after the end of the json in the json2msgpack sample data";
}
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void simdjsonsinglestage2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void simdjsonsinglestage2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void simdjsonsinglestage2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
uint8_t *simdjsonsinglestage2msgpack::skip_uint32() noexcept {
uint8_t *ret = buff;
buff += sizeof(uint32_t);
return ret;
}
void simdjsonsinglestage2msgpack::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
::memcpy(p, &w, sizeof(w));
}
void simdjsonsinglestage2msgpack::write_raw_string(
simdjson::singlestage::raw_json_string in) {
write_byte(0xdb);
uint8_t *location = skip_uint32();
std::string_view v = parser.unescape(in, buff);
write_uint32_at(uint32_t(v.size()), location);
}
void simdjsonsinglestage2msgpack::recursive_processor(simdjson::singlestage::value element) {
switch (element.type()) {
case simdjson::singlestage::json_type::array: {
uint32_t counter = 0;
write_byte(0xdd);
uint8_t *location = skip_uint32();
for (auto child : element.get_array()) {
counter++;
recursive_processor(child.value());
}
write_uint32_at(counter, location);
} break;
case simdjson::singlestage::json_type::object: {
uint32_t counter = 0;
write_byte(0xdf);
uint8_t *location = skip_uint32();
for (auto field : element.get_object()) {
counter++;
write_raw_string(field.key());
recursive_processor(field.value());
}
write_uint32_at(counter, location);
} break;
case simdjson::singlestage::json_type::number:
write_double(element.get_double());
break;
case simdjson::singlestage::json_type::string:
write_raw_string(element.get_raw_json_string());
break;
case simdjson::singlestage::json_type::boolean:
write_byte(0xc2 + element.get_bool());
break;
case simdjson::singlestage::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
write_byte(0xc0);
}
break;
default:
SIMDJSON_UNREACHABLE();
}
}
void simdjsonsinglestage2msgpack::recursive_processor_ref(simdjson::singlestage::value& element) {
switch (element.type()) {
case simdjson::singlestage::json_type::array: {
uint32_t counter = 0;
write_byte(0xdd);
uint8_t *location = skip_uint32();
for (auto child : element.get_array()) {
counter++;
simdjson::singlestage::value v = child.value();
recursive_processor_ref(v);
}
write_uint32_at(counter, location);
} break;
case simdjson::singlestage::json_type::object: {
uint32_t counter = 0;
write_byte(0xdf);
uint8_t *location = skip_uint32();
for (auto field : element.get_object()) {
counter++;
write_raw_string(field.key());
simdjson::singlestage::value v = field.value();
recursive_processor_ref(v);
}
write_uint32_at(counter, location);
} break;
case simdjson::singlestage::json_type::number:
write_double(element.get_double());
break;
case simdjson::singlestage::json_type::string:
write_raw_string(element.get_raw_json_string());
break;
case simdjson::singlestage::json_type::boolean:
write_byte(0xc2 + element.get_bool());
break;
case simdjson::singlestage::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
write_byte(0xc0);
}
break;
default:
SIMDJSON_UNREACHABLE();
}
}
struct simdjson_singlestage {
using StringType = std::string_view;
simdjsonsinglestage2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
result = parser.to_msgpack(json, reinterpret_cast<uint8_t *>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, simdjson_singlestage)->UseManualTime();
} // namespace json2msgpack
#endif // SIMDJSON_EXCEPTIONS
@@ -13,15 +13,13 @@ void maybe_display_implementation() {
std::cout << "simdjson::dom implementation: " << simdjson::get_active_implementation()->name() << std::endl;
std::cout << "simdjson::ondemand implementation (stage 1): " << simdjson::get_active_implementation()->name() << std::endl;
std::cout << "simdjson::ondemand implementation (stage 2): " << simdjson::builtin_implementation()->name() << std::endl;
std::cout << "simdjson::singlestage implementation (stage 1): " << simdjson::get_active_implementation()->name() << std::endl;
std::cout << "simdjson::singlestage implementation (stage 2): " << simdjson::builtin_implementation()->name() << std::endl;
}
}
template<typename B, typename R> static void run_json_benchmark(benchmark::State &state) {
maybe_display_implementation();
event_collector collector(true);
event_collector collector;
event_aggregate events;
// Warmup and equality check (make sure the data is right!)
-29
View File
@@ -1,29 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "kostya.h"
namespace kostya {
using namespace simdjson;
struct simdjson_singlestage {
static constexpr diff_flags DiffFlags = diff_flags::NONE;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::vector<point> &result) {
auto doc = parser.iterate(json);
for (singlestage::object point : doc.find_field("coordinates")) {
result.emplace_back(json_benchmark::point{point.find_field("x"), point.find_field("y"), point.find_field("z")});
}
return true;
}
};
BENCHMARK_TEMPLATE(kostya, simdjson_singlestage)->UseManualTime();
} // namespace kostya
#endif // SIMDJSON_EXCEPTIONS
@@ -1,72 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "large_amazon_cellphones.h"
namespace large_amazon_cellphones {
using namespace simdjson;
template<bool threaded>
struct simdjson_singlestage {
using StringType = std::string;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::map<StringType, brand> &result) {
#ifdef SIMDJSON_THREADS_ENABLED
parser.threaded = threaded;
#endif
singlestage::document_stream stream = parser.iterate_many(json);
singlestage::document_stream::iterator i = stream.begin();
++i; // Skip first line
for (;i != stream.end(); ++i) {
auto doc = *i;
size_t index{0};
StringType copy;
double rating;
uint64_t reviews;
for ( auto value : doc ) {
switch (index)
{
case 1:
copy = StringType(std::string_view(value));
break;
case 5:
rating = double(value);
break;
case 7:
reviews = uint64_t(value);
break;
default:
break;
}
index++;
}
auto x = result.find(copy);
if (x == result.end()) { // If key not found, add new key
result.emplace(copy, large_amazon_cellphones::brand{
rating * reviews,
reviews
});
} else { // Otherwise, update key data
x->second.cumulative_rating += rating * reviews;
x->second.reviews_count += reviews;
}
}
return true;
}
};
BENCHMARK_TEMPLATE(large_amazon_cellphones, simdjson_singlestage<UNTHREADED>)->UseManualTime();
#ifdef SIMDJSON_THREADS_ENABLED
BENCHMARK_TEMPLATE(large_amazon_cellphones, simdjson_singlestage<THREADED>)->UseManualTime();
#endif
} // namespace amazon_cellphones
#endif // SIMDJSON_EXCEPTIONS
@@ -1,29 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "large_random.h"
namespace large_random {
using namespace simdjson;
struct simdjson_singlestage {
static constexpr diff_flags DiffFlags = diff_flags::NONE;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::vector<point> &result) {
auto doc = parser.iterate(json);
for (singlestage::object coord : doc) {
result.emplace_back(json_benchmark::point{coord.find_field("x"), coord.find_field("y"), coord.find_field("z")});
}
return true;
}
};
BENCHMARK_TEMPLATE(large_random, simdjson_singlestage)->UseManualTime();
} // namespace large_random
#endif // SIMDJSON_EXCEPTIONS
@@ -1,29 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "large_random.h"
namespace large_random {
using namespace simdjson;
struct simdjson_singlestage_unordered {
static constexpr diff_flags DiffFlags = diff_flags::NONE;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, std::vector<point> &result) {
auto doc = parser.iterate(json);
for (singlestage::object coord : doc) {
result.emplace_back(json_benchmark::point{coord["x"], coord["y"], coord["z"]});
}
return true;
}
};
BENCHMARK_TEMPLATE(large_random, simdjson_singlestage_unordered)->UseManualTime();
} // namespace large_random
#endif // SIMDJSON_EXCEPTIONS
-37
View File
@@ -1,37 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "largerandom.h"
namespace largerandom {
using namespace simdjson;
class OnDemand {
public:
simdjson_inline bool Run(const padded_string &json);
simdjson_inline const std::vector<my_point> &Result() { return container; }
simdjson_inline size_t ItemCount() { return container.size(); }
private:
singlestage::parser parser{};
std::vector<my_point> container{};
};
simdjson_inline bool OnDemand::Run(const padded_string &json) {
container.clear();
auto doc = parser.iterate(json);
for (singlestage::object coord : doc) {
container.emplace_back(my_point{coord.find_field("x"), coord.find_field("y"), coord.find_field("z")});
}
return true;
}
BENCHMARK_TEMPLATE(LargeRandom, OnDemand);
} // namespace largerandom
#endif // SIMDJSON_EXCEPTIONS
@@ -1,48 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "partial_tweets.h"
namespace partial_tweets {
using namespace simdjson;
struct simdjson_singlestage {
using StringType=std::string_view;
singlestage::parser parser{};
simdjson_inline uint64_t nullable_int(singlestage::value value) {
if (value.is_null()) { return 0; }
return value;
}
simdjson_inline twitter_user<std::string_view> read_user(singlestage::object user) {
return { user.find_field("id"), user.find_field("screen_name") };
}
bool run(simdjson::padded_string &json, std::vector<tweet<std::string_view>> &result) {
// Walk the document, parsing the tweets as we go
auto doc = parser.iterate(json);
for (singlestage::object tweet : doc.find_field("statuses")) {
result.emplace_back(partial_tweets::tweet<std::string_view>{
tweet.find_field("created_at"),
tweet.find_field("id"),
tweet.find_field("text"),
nullable_int(tweet.find_field("in_reply_to_status_id")),
read_user(tweet.find_field("user")),
tweet.find_field("retweet_count"),
tweet.find_field("favorite_count")
});
}
return true;
}
};
BENCHMARK_TEMPLATE(partial_tweets, simdjson_singlestage)->UseManualTime();
} // namespace partial_tweets
#endif // SIMDJSON_EXCEPTIONS
-63
View File
@@ -1,63 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "partial_tweets.h"
namespace partial_tweets {
using namespace simdjson;
class SingleStage {
public:
SingleStage() {
if(!displayed_implementation) {
std::cout << "On Demand implementation: " << builtin_implementation()->name() << std::endl;
displayed_implementation = true;
}
}
simdjson_inline bool Run(const padded_string &json);
simdjson_inline const std::vector<tweet> &Result() { return tweets; }
simdjson_inline size_t ItemCount() { return tweets.size(); }
private:
singlestage::parser parser{};
std::vector<tweet> tweets{};
simdjson_inline uint64_t nullable_int(singlestage::value value) {
if (value.is_null()) { return 0; }
return value;
}
simdjson_inline twitter_user read_user(singlestage::object user) {
return { user.find_field("id"), user.find_field("screen_name") };
}
static inline bool displayed_implementation = false;
};
simdjson_inline bool SingleStage::Run(const padded_string &json) {
tweets.clear();
// Walk the document, parsing the tweets as we go
auto doc = parser.iterate(json);
for (singlestage::object tweet : doc.find_field("statuses")) {
tweets.emplace_back(partial_tweets::tweet{
tweet.find_field("created_at"),
tweet.find_field("id"),
tweet.find_field("text"),
nullable_int(tweet.find_field("in_reply_to_status_id")),
read_user(tweet.find_field("user")),
tweet.find_field("retweet_count"),
tweet.find_field("favorite_count")
});
}
return true;
}
BENCHMARK_TEMPLATE(PartialTweets, SingleStage);
} // namespace partial_tweets
#endif // SIMDJSON_EXCEPTIONS
@@ -1,80 +0,0 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "top_tweet.h"
namespace top_tweet {
using namespace simdjson;
struct simdjson_singlestage {
using StringType=std::string_view;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, int64_t max_retweet_count, top_tweet_result<StringType> &result) {
result.retweet_count = -1;
// We save these DOM values for later so we don't have to parse them
// into string_views until we're sure which ones we want to parse
// NOTE: simdjson does not presently support reuse of objects or arrays--just scalars. This is
// why we have to grab the text and screen_name fields instead of just saving the tweet object.
singlestage::value screen_name, text;
auto doc = parser.iterate(json);
for (auto tweet : doc["statuses"]) {
// Since text, user.screen_name, and retweet_count generally appear in order, it's nearly free
// for us to retrieve them here (and will cost a bit more if we do it in the if
// statement).
auto tweet_text = tweet["text"];
auto tweet_screen_name = tweet["user"]["screen_name"];
int64_t retweet_count = tweet["retweet_count"];
if (retweet_count <= max_retweet_count && retweet_count >= result.retweet_count) {
result.retweet_count = retweet_count;
// TODO std::move should not be necessary
text = std::move(tweet_text);
screen_name = std::move(tweet_screen_name);
}
}
// Now that we know which was the most retweeted, parse the values in it
result.screen_name = screen_name;
result.text = text;
return result.retweet_count != -1;
}
};
BENCHMARK_TEMPLATE(top_tweet, simdjson_singlestage)->UseManualTime();
struct simdjson_singlestage_forward_only {
using StringType=std::string_view;
singlestage::parser parser{};
bool run(simdjson::padded_string &json, int64_t max_retweet_count, top_tweet_result<StringType> &result) {
result.retweet_count = -1;
auto doc = parser.iterate(json);
for (auto tweet : doc["statuses"]) {
// Since text, user.screen_name, and retweet_count generally appear in order, it's nearly free
// for us to retrieve them here (and will cost a bit more if we do it in the if
// statement).
auto tweet_text = tweet["text"];
auto tweet_screen_name = tweet["user"]["screen_name"];
int64_t retweet_count = tweet["retweet_count"];
if (retweet_count <= max_retweet_count && retweet_count >= result.retweet_count) {
result.retweet_count = retweet_count;
result.text = tweet_text;
result.screen_name = tweet_screen_name;
}
}
return result.retweet_count != -1;
}
};
BENCHMARK_TEMPLATE(top_tweet, simdjson_singlestage_forward_only)->UseManualTime();
} // namespace top_tweet
#endif // SIMDJSON_EXCEPTIONS
+5
View File
@@ -117,7 +117,12 @@ set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_MACOSX_RPATH OFF)
set(CMAKE_THREAD_PREFER_PTHREAD ON)
set(THREADS_PREFER_PTHREAD_FLAG ON)
set(SIMDJSON_STRUCTURAL_INDEXER_STEP CACHE STRING "the SIMDJSON_STRUCTURAL_INDEXER_STEP variable")
if(SIMDJSON_STRUCTURAL_INDEXER_STEP)
message(STATUS "Setting SIMDJSON_STRUCTURAL_INDEXER_STEP to ${SIMDJSON_STRUCTURAL_INDEXER_STEP}.")
add_compile_definitions(SIMDJSON_STRUCTURAL_INDEXER_STEP=${SIMDJSON_STRUCTURAL_INDEXER_STEP})
endif()
# LTO seems to create all sorts of fun problems. Let us
# disable temporarily.
#include(CheckIPOSupported)
+31 -31
View File
@@ -355,7 +355,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
* **Field Access:** To get the value of the "foo" field in an object, use `object["foo"]`. This will
scan through the object looking for the field with the matching string, doing a character-by-character
comparison. It may generate the error `simdjson::NO_SUCH_FIELD` if there is no such key in the object, it may throw an exception (see [Error Handling](#error-handling)). For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do
not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. Keep in mind that On Demand does not buffer or save the result of the parsing: if you repeatedly access `object["foo"]`, then it must repeatedly seek the key and parse the content. The library does not provide a distinct function to check if a key is present, instead we recommend you attempt to access the key: e.g., by doing `ondemand::value val{}; if(!object["foo"].get(val)) {...}`, you have that `val` contains the requested value inside the if clause. It is your responsibility as a user to temporarily keep a reference to the value (`auto v = object["foo"]`), or to consume the content and store it in your own data structures. If you consume an
not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. Keep in mind that On Demand does not buffer or save the result of the parsing: if you repeatedly access `object["foo"]`, then it must repeatedly seek the key and parse the content. The library does not provide a distinct function to check if a key is present, instead we recommend you attempt to access the key: e.g., by doing `ondemand::value val{}; if (!object["foo"].get(val)) {...}`, you have that `val` contains the requested value inside the if clause. It is your responsibility as a user to temporarily keep a reference to the value (`auto v = object["foo"]`), or to consume the content and store it in your own data structures. If you consume an
object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]` then your code
is in error. Furthermore, you can only consume one field at a time, on the same object. The
value instance you get from `content["bids"]` becomes invalid when you call `content["asks"]`.
@@ -381,7 +381,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
> // parses and writes out the key, after unescaping it,
> // to a string buffer. It causes a performance penalty.
> std::string_view keyv = field.unescaped_key();
> if(keyv == "key") { std::cout << uint64_t(field.value()); }
> if (keyv == "key") { std::cout << uint64_t(field.value()); }
> }
> ```
>
@@ -445,7 +445,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
> {
> ondemand::parser parser;
> for (ondemand::object car : parser.iterate(cars_json)) {
> if(uint64_t(car["year"]) > 2000) {
> if (uint64_t(car["year"]) > 2000) {
> arrays.push_back(simdjson::to_json_string(car["tire_pressure"]));
> }
> }
@@ -454,7 +454,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
> std::ostringstream oss;
> oss << "[";
> for(size_t i = 0; i < arrays.size(); i++) {
> if(i>0) { oss << ","; }
> if (i>0) { oss << ","; }
> oss << arrays[i];
> }
> oss << "]";
@@ -597,7 +597,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
case ondemand::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
if (element.is_null()) {
cout << "null";
}
break;
@@ -910,11 +910,11 @@ bool simple_error_example() {
ondemand::parser parser;
auto json = R"({"bad number":3.14.1 })"_padded;
ondemand::document doc;
if( parser.iterate(json).get(doc) != SUCCESS ) { return false; }
if (parser.iterate(json).get(doc) != SUCCESS) { return false; }
double x;
auto error = doc["bad number"].get_double().get(x);
// returns "simdjson::NUMBER_ERROR"
if(error != SUCCESS) {
if (error != SUCCESS) {
std::cout << error << std::endl;
return false;
}
@@ -976,10 +976,10 @@ it selects the key `"count"` within that object.
int main(void) {
simdjson::ondemand::parser parser;
auto error = padded_string::load("twitter.json").get(json);
if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
simdjson::ondemand::document tweets;
error = parser.iterate(json).get(tweets);
if( error ) { std::cerr << error << std::endl; return EXIT_FAILURE; }
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
simdjson::ondemand::value res;
error = tweets["search_metadata"]["count"].get(res);
if (error != SUCCESS) {
@@ -1010,12 +1010,12 @@ int main(void) {
simdjson::ondemand::document tweets;
padded_string json;
auto error = padded_string::load("twitter.json").get(json);
if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
error = parser.iterate(json).get(tweets);
if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
uint64_t identifier;
error = tweets["statuses"].at(0)["id"].get(identifier);
if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
std::cout << identifier << std::endl;
}
```
@@ -1039,40 +1039,40 @@ bool parse() {
// Iterating through an array of objects
auto error = parser.iterate(cars_json).get(doc);
if(error) { std::cerr << error << std::endl; return false; }
if (error) { std::cerr << error << std::endl; return false; }
ondemand::array cars; // invalid until the get() succeeds
error = doc.get_array().get(cars);
for (auto car_value : cars) {
ondemand::object car; // invalid until the get() succeeds
error = car_value.get_object().get(car);
if(error) { std::cerr << error << std::endl; return false; }
if (error) { std::cerr << error << std::endl; return false; }
// Accessing a field by name
std::string_view make;
std::string_view model;
error = car["make"].get(make);
if(error) { std::cerr << error << std::endl; return false; }
if (error) { std::cerr << error << std::endl; return false; }
error = car["model"].get(model);
if(error) { std::cerr << error << std::endl; return false; }
if (error) { std::cerr << error << std::endl; return false; }
cout << "Make/Model: " << make << "/" << model << endl;
// Casting a JSON element to an integer
uint64_t year{};
error = car["year"].get(year);
if(error) { std::cerr << error << std::endl; return false; }
if (error) { std::cerr << error << std::endl; return false; }
cout << "- This car is " << 2020 - year << " years old." << endl;
// Iterating through an array of floats
double total_tire_pressure = 0;
ondemand::array pressures;
error = car["tire_pressure"].get_array().get(pressures);
if(error) { std::cerr << error << std::endl; return false; }
if (error) { std::cerr << error << std::endl; return false; }
for (auto tire_pressure_value : pressures) {
double tire_pressure;
error = tire_pressure_value.get_double().get(tire_pressure);
if(error) { std::cerr << error << std::endl; return false; }
if (error) { std::cerr << error << std::endl; return false; }
total_tire_pressure += tire_pressure;
}
cout << "- Average tire pressure: " << (total_tire_pressure / 4) << endl;
@@ -1088,7 +1088,7 @@ after you have initialized them and checked that there is no error:
ondemand::object car; // invalid until the get() succeeds
// the `car` instance should not use used before it is initialized
error = car_value.get_object().get(car);
if(error) {
if (error) {
// the `car` instance should not use used
} else {
// the `car` instance can be safely used
@@ -1102,20 +1102,20 @@ having to handle exceptions.
ondemand::parser parser;
ondemand::document doc;
auto error = parser.iterate(json).get(doc);
if(error) { return false; }
if (error) { return false; }
ondemand::object object; // invalid until the get() succeeds
error = doc.get_object().get(object);
if(error) { return false; }
if (error) { return false; }
for(auto field : object) {
// We could replace 'field.key() with field.unescaped_key(),
// and ondemand::raw_json_string by std::string_view.
ondemand::raw_json_string keyv;
error = field.key().get(keyv);
if(error) { return false; }
if(keyv == "key") {
if (error) { return false; }
if (keyv == "key") {
uint64_t intvalue;
error = field.value().get(intvalue);
if(error) { return false; }
if (error) { return false; }
std::cout << intvalue;
}
}
@@ -1294,7 +1294,7 @@ content.
for (uint64_t values : array) {
std::cout << values << std::endl;
}
if(!doc.at_end()) {
if (!doc.at_end()) {
// In this instance, we will be left pointing at 'foo' since we have consumed the array [1,2].
std::cerr << "trailing content at byte index " << doc.current_location() - json.data() << std::endl;
}
@@ -1324,7 +1324,7 @@ before printout the data.
auto doc = parser.iterate(cars_json);
for (simdjson_unused ondemand::object car : doc) {
if(car["make"] == "Toyota") { count++; }
if (car["make"] == "Toyota") { count++; }
}
std::cout << "We have " << count << " Toyota cars.\n";
doc.rewind(); // requires simdjson 1.0 or better
@@ -1392,18 +1392,18 @@ ondemand::parser parser;
ondemand::document_stream stream;
size_t counter{0};
auto error = parser.iterate_many(json, 50).get(stream);
if( error ) { /* handle the error */ }
if (error) { /* handle the error */ }
for (auto doc: stream) {
if(counter < 6) {
if (counter < 6) {
int64_t val;
error = doc.at_pointer("/4").get(val);
if( error ) { /* handle the error */ }
if (error) { /* handle the error */ }
std::cout << "5 = " << val << std::endl;
} else {
ondemand::value val;
error = doc.at_pointer("/4").get(val);
// error == simdjson::CAPACITY
if(error) {
if (error) {
std::cerr << error << std::endl;
// We left 293 bytes unprocessed at the tail end of the input.
std::cout << " unprocessed bytes at the end: " << stream.truncated_bytes() << std::endl;
+2 -2
View File
@@ -329,10 +329,10 @@ int main(void) {
simdjson::dom::parser parser;
simdjson::dom::element tweets; // invalid until the get() succeeds
auto error = parser.load("twitter.json").get(tweets);
if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
uint64_t identifier;
error = tweets["statuses"].at(0)["id"].get(identifier);
if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
std::cout << identifier << std::endl;
return EXIT_SUCCESS;
}
+3 -3
View File
@@ -101,8 +101,8 @@ by comparing it with the null pointer.
```c++
auto my_implementation = simdjson::get_available_implementations()["haswell"];
if(! my_implementation) { exit(1); }
if(! my_implementation->supported_by_runtime_system()) { exit(1); }
if (! my_implementation) { exit(1); }
if (! my_implementation->supported_by_runtime_system()) { exit(1); }
simdjson::get_active_implementation() = my_implementation;
```
@@ -113,7 +113,7 @@ You should call `supported_by_runtime_system()` to compare the processor's featu
```c++
for (auto implementation : simdjson::get_available_implementations()) {
if(implementation->supported_by_runtime_system()) {
if (implementation->supported_by_runtime_system()) {
cout << implementation->name() << ": " << implementation->description() << endl;
}
}
+7 -4
View File
@@ -17,6 +17,9 @@ Contents
- [Motivations](#motivations)
- [How it works](#how-it-works)
- [Context](#context)
- [Design](#design)
- [Threads](#threads)
- [Support](#support)
- [API](#api)
- [Use cases](#use-cases)
@@ -193,12 +196,12 @@ Let us illustrate the idea with code:
simdjson::ondemand::parser parser;
simdjson::ondemand::document_stream stream;
auto error = parser.iterate_many(json).get(stream);
if( error ) { /* do something */ }
if (error) { /* do something */ }
auto i = stream.begin();
size_t count{0};
for(; i != stream.end(); ++i) {
auto doc = *i;
if(!i.error()) {
if (!i.error()) {
std::cout << "got full document at " << i.current_index() << std::endl;
std::cout << i.source() << std::endl;
count++;
@@ -234,7 +237,7 @@ Consider the following example where a truncated document (`{"key":"intentionall
simdjson::ondemand::parser parser;
simdjson::ondemand::document_stream stream;
auto error = parser.iterate_many(json,json.size()).get(stream);
if(error) { std::cerr << error << std::endl; return; }
if (error) { std::cerr << error << std::endl; return; }
for(auto i = stream.begin(); i != stream.end(); ++i) {
std::cout << i.source() << std::endl;
}
@@ -266,7 +269,7 @@ Example:
// we pass 'true' to the allow_comma parameter, the batch size will be set to at least
// the document size.
auto error = parser.iterate_many(json, 32, true).get(doc_stream);
if(error) { std::cerr << error << std::endl; return; }
if (error) { std::cerr << error << std::endl; return; }
for (auto doc : doc_stream) {
std::cout << doc.type() << std::endl;
}
+4 -4
View File
@@ -679,11 +679,11 @@ in production systems:
ondemand::object c1 = parent["child1"];
// c1 owns the focus
//
if(std::string_view(c1["name"]) != "John") { ... }
if (std::string_view(c1["name"]) != "John") { ... }
// c2 attempts to grab the focus from parent but fails
ondemand::object c2 = parent["child2"];
// c2 is now in an unsafe state and the following line would be unsafe
// if(std::string_view(c2["name"]) != "Daniel") { return false; }
// if (std::string_view(c2["name"]) != "Daniel") { return false; }
```
A correct usage is given by the following example:
@@ -697,7 +697,7 @@ in production systems:
{
ondemand::object c1 = parent["child1"];
// c1 grabbed the focus from parent
if(std::string_view(c1["name"]) != "John") { return false; }
if (std::string_view(c1["name"]) != "John") { return false; }
}
// c1 went out of scope, so its destructor was called and the focus
// was handed back to parent.
@@ -705,7 +705,7 @@ in production systems:
ondemand::object c2 = parent["child2"];
// c2 grabbed the focus from parent
// the following is safe:
if(std::string_view(c2["name"]) != "Daniel") { return false; }
if (std::string_view(c2["name"]) != "Daniel") { return false; }
}
```
+3 -3
View File
@@ -184,12 +184,12 @@ Let us illustrate the idea with code:
simdjson::dom::parser parser;
simdjson::dom::document_stream stream;
auto error = parser.parse_many(json).get(stream);
if( error ) { /* do something */ }
if (error) { /* do something */ }
auto i = stream.begin();
size_t count{0};
for(; i != stream.end(); ++i) {
auto doc = *i;
if(!doc.error()) {
if (!doc.error()) {
std::cout << "got full document at " << i.current_index() << std::endl;
std::cout << i.source() << std::endl;
count++;
@@ -225,7 +225,7 @@ Consider the following example where a truncated document (`{"key":"intentionall
simdjson::dom::parser parser;
simdjson::dom::document_stream stream;
auto error = parser.parse_many(json,json.size()).get(stream);
if(error) { std::cerr << error << std::endl; return; }
if (error) { std::cerr << error << std::endl; return; }
for(auto doc : stream) {
std::cout << doc << std::endl;
}
-10
View File
@@ -35,14 +35,4 @@ IF(${CMAKE_SYSTEM_NAME} MATCHES "Linux")
add_quickstart_test(quickstart_ondemand_noexceptions quickstart_ondemand_noexceptions.cpp NO_EXCEPTIONS LABELS quickstart_ondemand acceptance)
add_quickstart_test(quickstart_ondemand_noexceptions11 quickstart_ondemand_noexceptions.cpp NO_EXCEPTIONS CXX_STANDARD c++11 LABELS quickstart_ondemand)
# SingleStage Quick Start
if (SIMDJSON_EXCEPTIONS)
add_quickstart_test(quickstart_singlestage quickstart_singlestage.cpp LABELS quickstart_singlestage acceptance)
add_quickstart_test(quickstart_singlestage11 quickstart_singlestage.cpp CXX_STANDARD c++11 LABELS quickstart_singlestage acceptance)
add_quickstart_test(quickstart_singlestage14 quickstart_singlestage.cpp CXX_STANDARD c++14 LABELS quickstart_singlestage)
endif()
add_quickstart_test(quickstart_singlestage_noexceptions quickstart_singlestage_noexceptions.cpp NO_EXCEPTIONS LABELS quickstart_singlestage acceptance)
add_quickstart_test(quickstart_singlestage_noexceptions11 quickstart_singlestage_noexceptions.cpp NO_EXCEPTIONS CXX_STANDARD c++11 LABELS quickstart_singlestage)
endif()
@@ -1,9 +0,0 @@
#include <iostream>
#include "simdjson.h"
using namespace simdjson;
int main(void) {
ondemand::parser parser;
padded_string json = padded_string::load("twitter.json");
ondemand::document tweets = parser.iterate(json);
std::cout << uint64_t(tweets["search_metadata"]["count"]) << " results." << std::endl;
}
@@ -1,21 +0,0 @@
#include <iostream>
#include "simdjson.h"
using namespace simdjson;
int main(void) {
padded_string json;
auto error = padded_string::load("twitter.json").get(json);
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
ondemand::parser parser;
ondemand::document tweets;
error = parser.iterate(json).get(tweets);
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
uint64_t count;
error = tweets["search_metadata"]["count"].get(count);
if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
std::cout << count << " results." << std::endl;
return EXIT_SUCCESS;
}
+277
View File
@@ -0,0 +1,277 @@
number of iterations 3000
| Original (jsonexamples/twitter-padded-numbers.json) 773
|- 18-digit numbers (jsonexamples/twitter18.json) 757
|- 8-digit numbers (jsonexamples/twitter8.json) 766
|- 1-digit numbers (jsonexamples/twitter1.json) 544
| |- strings -> no backslashes (jsonexamples/twitter1-nobackslash.json) 499
| |- strings -> no utf8 (jsonexamples/twitter1-nobackslash-noutf8.json) 489
| |- strings -> empty strings (jsonexamples/twitter1-emptystring.json) 351
| |- no number/bool/null -> all strings (jsonexamples/twitter-allstrings.json) 216
| |- no objects -> all arrays (jsonexamples/twitter-allstrings-arrays.json) 427
| |- arrays -> no nested arrays (jsonexamples/twitter-allstrings-flatarray.json) 3
jsonexamples/twitter-padded-numbers.json
========================================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 2289 ( 22.2 %) - escape 604 ( 5.9 %) - 0 structurals 1270 ( 12.3 %) - 1+ structurals 9037 ( 87.7 %) - 8+ structurals 2576 ( 25.0 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 1096 ( 10.6 %) - escape 646 ( 6.3 %) - 0 structurals 932 ( 9.0 %) - 1+ structurals 932 ( 9.0 %) - 8+ structurals 2843 ( 27.6 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 20.7212 ns per block ( 97.66%) - 0.3238 ns per byte - 3.8647 ns per structural - 3.0885 GB/s
| Cycles : 64.1524 per block ( 97.86%) - 1.0024 per byte - 11.9652 per structural - 3.096 GHz est. frequency
| Instructions : 200.2565 per block (100.00%) - 3.1291 per byte - 37.3502 per structural - 3.122 per cycle
| Misses : 873 branch misses ( 93.08%) - 0 cache misses ( 0.00%) - 28032.00 cache references
|- Stage 1
| Speed : 6.0073 ns per block ( 28.31%) - 0.0939 ns per byte - 1.1204 ns per structural - 10.6534 GB/s
| Cycles : 18.6130 per block ( 28.39%) - 0.2908 per byte - 3.4715 per structural - 3.098 GHz est. frequency
| Instructions : 61.8107 per block ( 30.87%) - 0.9658 per byte - 11.5284 per structural - 3.321 per cycle
| Misses : 93 branch misses ( 9.92%) - 0 cache misses ( 0.00%) - 12229.00 cache references
|- Stage 2
| Speed : 14.6555 ns per block ( 69.07%) - 0.2290 ns per byte - 2.7334 ns per structural - 4.3668 GB/s
| Cycles : 45.3587 per block ( 69.19%) - 0.7088 per byte - 8.4599 per structural - 3.095 GHz est. frequency
| Instructions : 138.4458 per block ( 69.13%) - 2.1633 per byte - 25.8217 per structural - 3.052 per cycle
| Misses : 773 branch misses ( 82.42%) - 0 cache misses ( 0.00%) - 15925.00 cache references
4682.2 documents parsed per second (best)
jsonexamples/twitter18.json
===========================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 2289 ( 22.2 %) - escape 604 ( 5.9 %) - 0 structurals 1270 ( 12.3 %) - 1+ structurals 9037 ( 87.7 %) - 8+ structurals 2585 ( 25.1 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 1096 ( 10.6 %) - escape 646 ( 6.3 %) - 0 structurals 932 ( 9.0 %) - 1+ structurals 932 ( 9.0 %) - 8+ structurals 2847 ( 27.6 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 21.1739 ns per block ( 93.63%) - 0.3309 ns per byte - 3.9492 ns per structural - 3.0225 GB/s
| Cycles : 65.5570 per block ( 97.40%) - 1.0244 per byte - 12.2271 per structural - 3.096 GHz est. frequency
| Instructions : 205.6160 per block (100.00%) - 3.2129 per byte - 38.3498 per structural - 3.136 per cycle
| Misses : 856 branch misses ( 97.20%) - 0 cache misses ( 0.00%) - 28297.00 cache references
|- Stage 1
| Speed : 5.9968 ns per block ( 26.52%) - 0.0937 ns per byte - 1.1185 ns per structural - 10.6720 GB/s
| Cycles : 18.5808 per block ( 27.61%) - 0.2903 per byte - 3.4655 per structural - 3.098 GHz est. frequency
| Instructions : 61.8107 per block ( 30.06%) - 0.9658 per byte - 11.5284 per structural - 3.327 per cycle
| Misses : 103 branch misses ( 11.70%) - 0 cache misses ( 0.00%) - 12271.00 cache references
|- Stage 2
| Speed : 15.0964 ns per block ( 66.75%) - 0.2359 ns per byte - 2.8157 ns per structural - 4.2393 GB/s
| Cycles : 46.7271 per block ( 69.43%) - 0.7301 per byte - 8.7151 per structural - 3.095 GHz est. frequency
| Instructions : 143.8053 per block ( 69.94%) - 2.2470 per byte - 26.8213 per structural - 3.078 per cycle
| Misses : 757 branch misses ( 85.96%) - 0 cache misses ( 0.00%) - 16107.00 cache references
4582.1 documents parsed per second (best)
jsonexamples/twitter8.json
==========================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 2289 ( 22.2 %) - escape 604 ( 5.9 %) - 0 structurals 1270 ( 12.3 %) - 1+ structurals 9037 ( 87.7 %) - 8+ structurals 2576 ( 25.0 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 1096 ( 10.6 %) - escape 646 ( 6.3 %) - 0 structurals 932 ( 9.0 %) - 1+ structurals 932 ( 9.0 %) - 8+ structurals 2843 ( 27.6 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 20.6976 ns per block ( 97.48%) - 0.3234 ns per byte - 3.8603 ns per structural - 3.0920 GB/s
| Cycles : 64.0733 per block ( 97.72%) - 1.0012 per byte - 11.9504 per structural - 3.096 GHz est. frequency
| Instructions : 200.2565 per block (100.00%) - 3.1291 per byte - 37.3502 per structural - 3.125 per cycle
| Misses : 860 branch misses ( 91.48%) - 0 cache misses ( 0.00%) - 28700.00 cache references
|- Stage 1
| Speed : 6.0103 ns per block ( 28.31%) - 0.0939 ns per byte - 1.1210 ns per structural - 10.6481 GB/s
| Cycles : 18.6231 per block ( 28.40%) - 0.2910 per byte - 3.4734 per structural - 3.099 GHz est. frequency
| Instructions : 61.8107 per block ( 30.87%) - 0.9658 per byte - 11.5284 per structural - 3.319 per cycle
| Misses : 102 branch misses ( 10.85%) - 0 cache misses ( 0.00%) - 11933.00 cache references
|- Stage 2
| Speed : 14.6407 ns per block ( 68.95%) - 0.2288 ns per byte - 2.7307 ns per structural - 4.3712 GB/s
| Cycles : 45.3059 per block ( 69.10%) - 0.7079 per byte - 8.4501 per structural - 3.095 GHz est. frequency
| Instructions : 138.4458 per block ( 69.13%) - 2.1633 per byte - 25.8217 per structural - 3.056 per cycle
| Misses : 766 branch misses ( 81.48%) - 0 cache misses ( 0.00%) - 16566.00 cache references
4687.6 documents parsed per second (best)
jsonexamples/twitter1.json
==========================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 2289 ( 22.2 %) - escape 604 ( 5.9 %) - 0 structurals 1270 ( 12.3 %) - 1+ structurals 9037 ( 87.7 %) - 8+ structurals 2584 ( 25.1 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 1096 ( 10.6 %) - escape 646 ( 6.3 %) - 0 structurals 932 ( 9.0 %) - 1+ structurals 932 ( 9.0 %) - 8+ structurals 2859 ( 27.7 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 20.1200 ns per block ( 93.76%) - 0.3144 ns per byte - 3.7526 ns per structural - 3.1808 GB/s
| Cycles : 62.2865 per block ( 97.52%) - 0.9733 per byte - 11.6172 per structural - 3.096 GHz est. frequency
| Instructions : 194.5666 per block (100.00%) - 3.0402 per byte - 36.2889 per structural - 3.124 per cycle
| Misses : 673 branch misses ( 97.88%) - 0 cache misses ( 0.00%) - 28320.00 cache references
|- Stage 1
| Speed : 6.0046 ns per block ( 27.98%) - 0.0938 ns per byte - 1.1199 ns per structural - 10.6582 GB/s
| Cycles : 18.6140 per block ( 29.14%) - 0.2909 per byte - 3.4717 per structural - 3.100 GHz est. frequency
| Instructions : 61.8107 per block ( 31.77%) - 0.9658 per byte - 11.5284 per structural - 3.321 per cycle
| Misses : 113 branch misses ( 16.43%) - 0 cache misses ( 0.00%) - 12175.00 cache references
|- Stage 2
| Speed : 14.0716 ns per block ( 65.57%) - 0.2199 ns per byte - 2.6245 ns per structural - 4.5480 GB/s
| Cycles : 43.5464 per block ( 68.18%) - 0.6804 per byte - 8.1219 per structural - 3.095 GHz est. frequency
| Instructions : 132.7559 per block ( 68.23%) - 2.0744 per byte - 24.7605 per structural - 3.049 per cycle
| Misses : 544 branch misses ( 79.11%) - 0 cache misses ( 0.00%) - 16212.00 cache references
4822.1 documents parsed per second (best)
jsonexamples/twitter1-nobackslash.json
======================================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 2289 ( 22.2 %) - escape 0 ( 0.0 %) - 0 structurals 1270 ( 12.3 %) - 1+ structurals 9037 ( 87.7 %) - 8+ structurals 2584 ( 25.1 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 1096 ( 10.6 %) - escape 0 ( 0.0 %) - 0 structurals 932 ( 9.0 %) - 1+ structurals 932 ( 9.0 %) - 8+ structurals 2859 ( 27.7 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 19.3473 ns per block ( 94.75%) - 0.3023 ns per byte - 3.6085 ns per structural - 3.3078 GB/s
| Cycles : 59.8930 per block ( 97.21%) - 0.9359 per byte - 11.1707 per structural - 3.096 GHz est. frequency
| Instructions : 191.2882 per block (100.00%) - 2.9890 per byte - 35.6774 per structural - 3.194 per cycle
| Misses : 624 branch misses ( 99.02%) - 2 cache misses ( 30.96%) - 28632.00 cache references
|- Stage 1
| Speed : 5.9519 ns per block ( 29.15%) - 0.0930 ns per byte - 1.1101 ns per structural - 10.7526 GB/s
| Cycles : 18.4460 per block ( 29.94%) - 0.2882 per byte - 3.4404 per structural - 3.099 GHz est. frequency
| Instructions : 61.1661 per block ( 31.98%) - 0.9558 per byte - 11.4082 per structural - 3.316 per cycle
| Misses : 92 branch misses ( 14.60%) - 1 cache misses ( 15.48%) - 12277.00 cache references
|- Stage 2
| Speed : 13.3283 ns per block ( 65.28%) - 0.2083 ns per byte - 2.4859 ns per structural - 4.8016 GB/s
| Cycles : 41.2453 per block ( 66.95%) - 0.6445 per byte - 7.6927 per structural - 3.095 GHz est. frequency
| Instructions : 130.1221 per block ( 68.02%) - 2.0332 per byte - 24.2693 per structural - 3.155 per cycle
| Misses : 499 branch misses ( 79.19%) - 1 cache misses ( 15.48%) - 16358.00 cache references
5014.7 documents parsed per second (best)
jsonexamples/twitter1-nobackslash-noutf8.json
=============================================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 1270 ( 12.3 %) - 1+ structurals 9037 ( 87.7 %) - 8+ structurals 2584 ( 25.1 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 932 ( 9.0 %) - 1+ structurals 932 ( 9.0 %) - 8+ structurals 2859 ( 27.7 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 18.9506 ns per block ( 97.78%) - 0.2961 ns per byte - 3.5345 ns per structural - 3.3771 GB/s
| Cycles : 58.6756 per block ( 97.97%) - 0.9168 per byte - 10.9437 per structural - 3.096 GHz est. frequency
| Instructions : 186.6244 per block (100.00%) - 2.9161 per byte - 34.8076 per structural - 3.181 per cycle
| Misses : 634 branch misses ( 99.08%) - 0 cache misses ( 0.00%) - 28596.00 cache references
|- Stage 1
| Speed : 5.4613 ns per block ( 28.18%) - 0.0853 ns per byte - 1.0186 ns per structural - 11.7184 GB/s
| Cycles : 16.9236 per block ( 28.26%) - 0.2644 per byte - 3.1565 per structural - 3.099 GHz est. frequency
| Instructions : 56.5024 per block ( 30.28%) - 0.8829 per byte - 10.5383 per structural - 3.339 per cycle
| Misses : 130 branch misses ( 20.32%) - 0 cache misses ( 0.00%) - 12365.00 cache references
|- Stage 2
| Speed : 13.4108 ns per block ( 69.19%) - 0.2096 ns per byte - 2.5013 ns per structural - 4.7721 GB/s
| Cycles : 41.5073 per block ( 69.31%) - 0.6486 per byte - 7.7416 per structural - 3.095 GHz est. frequency
| Instructions : 130.1221 per block ( 69.72%) - 2.0332 per byte - 24.2693 per structural - 3.135 per cycle
| Misses : 489 branch misses ( 76.42%) - 0 cache misses ( 0.00%) - 16279.00 cache references
5119.7 documents parsed per second (best)
jsonexamples/twitter1-emptystring.json
======================================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 1269 ( 12.3 %) - 1+ structurals 9038 ( 87.7 %) - 8+ structurals 2856 ( 27.7 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 928 ( 9.0 %) - 1+ structurals 928 ( 9.0 %) - 8+ structurals 3123 ( 30.3 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 18.0587 ns per block ( 93.58%) - 0.2822 ns per byte - 3.3682 ns per structural - 3.5439 GB/s
| Cycles : 55.9116 per block ( 97.35%) - 0.8736 per byte - 10.4282 per structural - 3.096 GHz est. frequency
| Instructions : 183.3181 per block (100.00%) - 2.8644 per byte - 34.1909 per structural - 3.279 per cycle
| Misses : 473 branch misses (101.19%) - 1 cache misses ( 15.11%) - 18833.00 cache references
|- Stage 1
| Speed : 5.4026 ns per block ( 28.00%) - 0.0844 ns per byte - 1.0077 ns per structural - 11.8457 GB/s
| Cycles : 16.7461 per block ( 29.16%) - 0.2617 per byte - 3.1233 per structural - 3.100 GHz est. frequency
| Instructions : 56.5028 per block ( 30.82%) - 0.8829 per byte - 10.5384 per structural - 3.374 per cycle
| Misses : 112 branch misses ( 23.96%) - 0 cache misses ( 0.00%) - 10717.00 cache references
|- Stage 2
| Speed : 12.6022 ns per block ( 65.30%) - 0.1969 ns per byte - 2.3505 ns per structural - 5.0783 GB/s
| Cycles : 39.0031 per block ( 67.91%) - 0.6094 per byte - 7.2745 per structural - 3.095 GHz est. frequency
| Instructions : 126.8154 per block ( 69.18%) - 1.9816 per byte - 23.6525 per structural - 3.251 per cycle
| Misses : 351 branch misses ( 75.09%) - 1 cache misses ( 15.11%) - 8120.00 cache references
5372.6 documents parsed per second (best)
jsonexamples/twitter-allstrings.json
====================================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 1269 ( 12.3 %) - 1+ structurals 9038 ( 87.7 %) - 8+ structurals 2853 ( 27.7 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 928 ( 9.0 %) - 1+ structurals 928 ( 9.0 %) - 8+ structurals 3107 ( 30.1 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 19.6482 ns per block ( 97.20%) - 0.3070 ns per byte - 3.6646 ns per structural - 3.2572 GB/s
| Cycles : 60.8255 per block ( 97.62%) - 0.9504 per byte - 11.3446 per structural - 3.096 GHz est. frequency
| Instructions : 183.8463 per block (100.00%) - 2.8727 per byte - 34.2895 per structural - 3.023 per cycle
| Misses : 300 branch misses ( 96.97%) - 0 cache misses ( 0.00%) - 18361.00 cache references
|- Stage 1
| Speed : 5.4053 ns per block ( 26.74%) - 0.0845 ns per byte - 1.0081 ns per structural - 11.8399 GB/s
| Cycles : 16.7545 per block ( 26.89%) - 0.2618 per byte - 3.1249 per structural - 3.100 GHz est. frequency
| Instructions : 56.5028 per block ( 30.73%) - 0.8829 per byte - 10.5384 per structural - 3.372 per cycle
| Misses : 98 branch misses ( 31.68%) - 1 cache misses ( 18.41%) - 10786.00 cache references
|- Stage 2
| Speed : 14.0367 ns per block ( 69.44%) - 0.2193 ns per byte - 2.6180 ns per structural - 4.5593 GB/s
| Cycles : 43.4371 per block ( 69.71%) - 0.6787 per byte - 8.1015 per structural - 3.095 GHz est. frequency
| Instructions : 127.3447 per block ( 69.27%) - 1.9898 per byte - 23.7513 per structural - 2.932 per cycle
| Misses : 216 branch misses ( 69.82%) - 0 cache misses ( 0.00%) - 7835.00 cache references
4937.9 documents parsed per second (best)
jsonexamples/twitter-allstrings-arrays.json
===========================================
10306 blocks - 659626 bytes - 55262 structurals ( 8.4 %)
special blocks with: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 1269 ( 12.3 %) - 1+ structurals 9038 ( 87.7 %) - 8+ structurals 2853 ( 27.7 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 928 ( 9.0 %) - 1+ structurals 928 ( 9.0 %) - 8+ structurals 3107 ( 30.1 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 23.6768 ns per block ( 96.29%) - 0.3700 ns per byte - 4.4160 ns per structural - 2.7030 GB/s
| Cycles : 72.7530 per block ( 95.79%) - 1.1368 per byte - 13.5693 per structural - 3.073 GHz est. frequency
| Instructions : 225.0789 per block (100.00%) - 3.5170 per byte - 41.9798 per structural - 3.094 per cycle
| Misses : 547 branch misses (104.38%) - 0 cache misses ( 0.00%) - 17259.00 cache references
|- Stage 1
| Speed : 5.4063 ns per block ( 21.99%) - 0.0845 ns per byte - 1.0083 ns per structural - 11.8376 GB/s
| Cycles : 16.7598 per block ( 22.07%) - 0.2619 per byte - 3.1259 per structural - 3.100 GHz est. frequency
| Instructions : 56.5028 per block ( 25.10%) - 0.8829 per byte - 10.5384 per structural - 3.371 per cycle
| Misses : 101 branch misses ( 19.27%) - 0 cache misses ( 0.00%) - 10570.00 cache references
|- Stage 2
| Speed : 18.2062 ns per block ( 74.04%) - 0.2845 ns per byte - 3.3957 ns per structural - 3.5152 GB/s
| Cycles : 55.7937 per block ( 73.46%) - 0.8718 per byte - 10.4062 per structural - 3.065 GHz est. frequency
| Instructions : 168.5761 per block ( 74.90%) - 2.6341 per byte - 31.4414 per structural - 3.021 per cycle
| Misses : 427 branch misses ( 81.48%) - 0 cache misses ( 0.00%) - 6797.00 cache references
4097.7 documents parsed per second (best)
jsonexamples/twitter-allstrings-flatarray.json
==============================================
10306 blocks - 659626 bytes - 49890 structurals ( 7.6 %)
special blocks with: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 1279 ( 12.4 %) - 1+ structurals 9028 ( 87.6 %) - 8+ structurals 2100 ( 20.4 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 946 ( 9.2 %) - 1+ structurals 946 ( 9.2 %) - 8+ structurals 2668 ( 25.9 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 20.5314 ns per block ( 98.77%) - 0.3208 ns per byte - 4.2417 ns per structural - 3.1171 GB/s
| Cycles : 63.5535 per block ( 98.95%) - 0.9931 per byte - 13.1298 per structural - 3.095 GHz est. frequency
| Instructions : 208.9967 per block (100.00%) - 3.2657 per byte - 43.1776 per structural - 3.289 per cycle
| Misses : 94 branch misses ( 99.52%) - 0 cache misses ( 0.00%) - 15420.00 cache references
|- Stage 1
| Speed : 5.3985 ns per block ( 25.97%) - 0.0844 ns per byte - 1.1153 ns per structural - 11.8548 GB/s
| Cycles : 16.7318 per block ( 26.05%) - 0.2614 per byte - 3.4567 per structural - 3.099 GHz est. frequency
| Instructions : 56.4917 per block ( 27.03%) - 0.8827 per byte - 11.6709 per structural - 3.376 per cycle
| Misses : 87 branch misses ( 92.11%) - 0 cache misses ( 0.00%) - 9889.00 cache references
|- Stage 2
| Speed : 15.0691 ns per block ( 72.49%) - 0.2355 ns per byte - 3.1132 ns per structural - 4.2470 GB/s
| Cycles : 46.6318 per block ( 72.61%) - 0.7286 per byte - 9.6339 per structural - 3.095 GHz est. frequency
| Instructions : 152.5050 per block ( 72.97%) - 2.3830 per byte - 31.5067 per structural - 3.270 per cycle
| Misses : 3 branch misses ( 3.18%) - 0 cache misses ( 0.00%) - 5610.00 cache references
4725.5 documents parsed per second (best)
jsonexamples/twitter-allstrings-flatobjects.json
================================================
10306 blocks - 659626 bytes - 56091 structurals ( 8.5 %)
special blocks with: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 1269 ( 12.3 %) - 1+ structurals 9038 ( 87.7 %) - 8+ structurals 2973 ( 28.8 %) - 16+ structurals 0 ( 0.0 %)
special block flips: utf8 0 ( 0.0 %) - escape 0 ( 0.0 %) - 0 structurals 928 ( 9.0 %) - 1+ structurals 928 ( 9.0 %) - 8+ structurals 3093 ( 30.0 %) - 16+ structurals 0 ( 0.0 %)
All Stages (excluding allocation)
| Speed : 19.4292 ns per block ( 97.71%) - 0.3036 ns per byte - 3.5702 ns per structural - 3.2939 GB/s
| Cycles : 60.1577 per block ( 97.90%) - 0.9400 per byte - 11.0543 per structural - 3.096 GHz est. frequency
| Instructions : 181.6747 per block (100.00%) - 2.8388 per byte - 33.3836 per structural - 3.020 per cycle
| Misses : 508 branch misses (100.57%) - 0 cache misses ( 0.00%) - 17421.00 cache references
|- Stage 1
| Speed : 5.4138 ns per block ( 27.22%) - 0.0846 ns per byte - 0.9948 ns per structural - 11.8213 GB/s
| Cycles : 16.7833 per block ( 27.31%) - 0.2622 per byte - 3.0840 per structural - 3.100 GHz est. frequency
| Instructions : 56.5028 per block ( 31.10%) - 0.8829 per byte - 10.3827 per structural - 3.367 per cycle
| Misses : 99 branch misses ( 19.60%) - 0 cache misses ( 0.00%) - 10517.00 cache references
|- Stage 2
| Speed : 13.9445 ns per block ( 70.12%) - 0.2179 ns per byte - 2.5624 ns per structural - 4.5895 GB/s
| Cycles : 43.1562 per block ( 70.23%) - 0.6743 per byte - 7.9302 per structural - 3.095 GHz est. frequency
| Instructions : 125.1719 per block ( 68.90%) - 1.9559 per byte - 23.0010 per structural - 2.900 per cycle
| Misses : 402 branch misses ( 79.59%) - 0 cache misses ( 0.00%) - 6973.00 cache references
4993.6 documents parsed per second (best)
-1
View File
@@ -63,7 +63,6 @@ if(SIMDJSON_ENABLE_FUZZING)
implement_fuzzer(fuzz_minifyimpl) # minify *without* parsing, plus compare implementations
implement_fuzzer(fuzz_ndjson) # the ndjson api
implement_fuzzer(fuzz_ondemand)
# implement_fuzzer(fuzz_singlestage) # TODO why no worky
implement_fuzzer(fuzz_padded)
implement_fuzzer(fuzz_parser)
implement_fuzzer(fuzz_print_json)
-74
View File
@@ -1,74 +0,0 @@
#include "FuzzUtils.h"
#include "simdjson.h"
#include <cstddef>
#include <cstdint>
#include <string>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
FuzzData fd(Data, Size);
const int action = fd.getInt<0, 12>();
// split the remainder of the document into strings
auto strings = fd.splitIntoStrings();
while (strings.size() < 1) {
strings.emplace_back();
}
#if SIMDJSON_EXCEPTIONS
try {
#endif
simdjson::singlestage::parser parser;
simdjson::padded_string padded(strings[0]);
auto doc = parser.iterate(padded);
if (doc.error()) {
return 0;
}
for (auto item : doc) {
switch (action) {
case 0: {
simdjson_unused auto x = item.get_string();
} break;
case 1: {
simdjson_unused auto x = item.get_bool();
} break;
case 2: {
simdjson_unused auto x = item.get_array();
} break;
case 3: {
simdjson_unused auto x = item.get_int64();
} break;
case 4: {
simdjson_unused auto x = item.get_double();
} break;
case 5: {
simdjson_unused auto x = item.get_object();
} break;
case 6: {
simdjson_unused auto x = item.get_uint64();
} break;
case 7: {
simdjson_unused auto x = item.get_raw_json_string();
} break;
case 8: {
simdjson_unused auto x = item.is_null();
} break;
case 9: {
simdjson_unused auto x = item.begin();
} break;
case 10: {
simdjson_unused auto x = item.end();
} break;
case 11: {
for (auto e : item) {
simdjson_unused auto x = e.is_null();
}
} break;
default:;
}
}
#if SIMDJSON_EXCEPTIONS
} catch (...) {
}
#endif
return 0;
}
-1
View File
@@ -53,6 +53,5 @@
#include "simdjson/dom.h"
#include "simdjson/ondemand.h"
#include "simdjson/singlestage.h"
#endif // SIMDJSON_H
+2 -9
View File
@@ -1,9 +1,9 @@
#ifndef SIMDJSON_ARM64_BASE_H
#define SIMDJSON_ARM64_BASE_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/base.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
/**
@@ -23,11 +23,4 @@ template <typename T> struct simd8x64;
} // namespace arm64
} // namespace simdjson
#ifndef SIMDJSON_AMALGAMATED
// If we're editing one of the files in this directory, begin the implementation!
#ifndef SIMDJSON_IMPLEMENTATION
#include "simdjson/arm64/begin.h"
#endif
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_ARM64_BASE_H
+2
View File
@@ -6,3 +6,5 @@
#include "simdjson/arm64/numberparsing_defs.h"
#include "simdjson/arm64/simd.h"
#include "simdjson/arm64/stringparsing_defs.h"
#define SIMDJSON_SKIP_BACKSLASH_SHORT_CIRCUIT 1
+4 -2
View File
@@ -1,11 +1,13 @@
#ifndef SIMDJSON_ARM64_BITMANIPULATION_H
#define SIMDJSON_ARM64_BITMANIPULATION_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/arm64/base.h"
#include "simdjson/arm64/intrinsics.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace arm64 {
namespace {
// We sometimes call trailing_zero on inputs that are zero,
@@ -98,7 +100,7 @@ simdjson_inline bool add_overflow(uint64_t value1, uint64_t value2, uint64_t *re
}
} // unnamed namespace
} // namespace SIMDJSON_IMPLEMENTATION
} // namespace arm64
} // namespace simdjson
#endif // SIMDJSON_ARM64_BITMANIPULATION_H
+3 -1
View File
@@ -1,10 +1,12 @@
#ifndef SIMDJSON_ARM64_BITMASK_H
#define SIMDJSON_ARM64_BITMASK_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/arm64/base.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace arm64 {
namespace {
//
+5
View File
@@ -1 +1,6 @@
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/arm64/base.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#undef SIMDJSON_SKIP_BACKSLASH_SHORT_CIRCUIT
#undef SIMDJSON_IMPLEMENTATION
+3 -2
View File
@@ -1,10 +1,11 @@
#ifndef SIMDJSON_ARM64_IMPLEMENTATION_H
#define SIMDJSON_ARM64_IMPLEMENTATION_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/base.h"
#include "simdjson/implementation.h"
#include "simdjson/internal/instruction_set.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace arm64 {
+4 -2
View File
@@ -1,12 +1,14 @@
#ifndef SIMDJSON_ARM64_INTRINSICS_H
#define SIMDJSON_ARM64_INTRINSICS_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/arm64/base.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
// This should be the correct header whether
// you use visual studio or other compilers.
#include <arm_neon.h>
#include "simdjson/arm64/base.h"
static_assert(sizeof(uint8x16_t) <= simdjson::SIMDJSON_PADDING, "insufficient padding for arm64");
#endif // SIMDJSON_ARM64_INTRINSICS_H
+7 -8
View File
@@ -1,22 +1,21 @@
#ifndef SIMDJSON_ARM64_NUMBERPARSING_DEFS_H
#define SIMDJSON_ARM64_NUMBERPARSING_DEFS_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/arm64/base.h"
#include "simdjson/arm64/intrinsics.h"
#include "simdjson/internal/numberparsing_tables.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#include <cstring>
#if _M_ARM64
// __umulh requires intrin.h
#include <intrin.h>
#endif // _M_ARM64
#include <cstring>
#ifndef SIMDJSON_AMALGAMATED
#include "simdjson/internal/numberparsing_tables.h"
#endif // SIMDJSON_AMALGAMATED
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace arm64 {
namespace numberparsing {
// we don't have SSE, so let us use a scalar function
@@ -49,7 +48,7 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
}
} // namespace numberparsing
} // namespace SIMDJSON_IMPLEMENTATION
} // namespace arm64
} // namespace simdjson
#define SIMDJSON_SWAR_NUMBER_PARSING 1
+4 -5
View File
@@ -1,15 +1,14 @@
#ifndef SIMDJSON_ARM64_SIMD_H
#define SIMDJSON_ARM64_SIMD_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/arm64/base.h"
#include "simdjson/arm64/bitmanipulation.h"
#ifndef SIMDJSON_AMALGAMATED
#include "simdjson/internal/simdprune_tables.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace arm64 {
namespace {
namespace simd {
@@ -528,7 +527,7 @@ simdjson_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x3, int
} // namespace simd
} // unnamed namespace
} // namespace SIMDJSON_IMPLEMENTATION
} // namespace arm64
} // namespace simdjson
#endif // SIMDJSON_ARM64_SIMD_H
-8
View File
@@ -1,8 +0,0 @@
#ifndef SIMDJSON_ARM64_SINGLESTAGE_H
#define SIMDJSON_ARM64_SINGLESTAGE_H
#include "simdjson/arm64/begin.h"
#include "simdjson/generic/singlestage/amalgamated.h"
#include "simdjson/arm64/end.h"
#endif // SIMDJSON_ARM64_SINGLESTAGE_H
+4 -2
View File
@@ -1,12 +1,14 @@
#ifndef SIMDJSON_ARM64_STRINGPARSING_DEFS_H
#define SIMDJSON_ARM64_STRINGPARSING_DEFS_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/arm64/base.h"
#include "simdjson/arm64/simd.h"
#include "simdjson/arm64/bitmanipulation.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace arm64 {
namespace {
using namespace simd;
@@ -45,7 +47,7 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
}
} // unnamed namespace
} // namespace SIMDJSON_IMPLEMENTATION
} // namespace arm64
} // namespace simdjson
#endif // SIMDJSON_ARM64_STRINGPARSING_DEFS_H
+2 -2
View File
@@ -6,7 +6,7 @@
#include "simdjson/generic/dependencies.h"
#define SIMDJSON_AMALGAMATED
#define SIMDJSON_CONDITIONAL_INCLUDE
#if SIMDJSON_BUILTIN_IMPLEMENTATION_IS(arm64)
#include "simdjson/arm64.h"
@@ -24,6 +24,6 @@
#error Unknown SIMDJSON_BUILTIN_IMPLEMENTATION
#endif
#undef SIMDJSON_AMALGAMATED
#undef SIMDJSON_CONDITIONAL_INCLUDE
#endif // SIMDJSON_BUILTIN_H
+2 -2
View File
@@ -5,7 +5,7 @@
#include "simdjson/generic/dependencies.h"
#define SIMDJSON_AMALGAMATED
#define SIMDJSON_CONDITIONAL_INCLUDE
#if SIMDJSON_BUILTIN_IMPLEMENTATION_IS(arm64)
#include "simdjson/arm64/implementation.h"
@@ -23,7 +23,7 @@
#error Unknown SIMDJSON_BUILTIN_IMPLEMENTATION
#endif
#undef SIMDJSON_AMALGAMATED
#undef SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
/**
+2 -2
View File
@@ -6,7 +6,7 @@
#include "simdjson/generic/ondemand/dependencies.h"
#define SIMDJSON_AMALGAMATED
#define SIMDJSON_CONDITIONAL_INCLUDE
#if SIMDJSON_BUILTIN_IMPLEMENTATION_IS(arm64)
#include "simdjson/arm64/ondemand.h"
@@ -24,7 +24,7 @@
#error Unknown SIMDJSON_BUILTIN_IMPLEMENTATION
#endif
#undef SIMDJSON_AMALGAMATED
#undef SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
/**
-36
View File
@@ -1,36 +0,0 @@
#ifndef SIMDJSON_BUILTIN_SINGLESTAGE_H
#define SIMDJSON_BUILTIN_SINGLESTAGE_H
#include "simdjson/builtin.h"
#include "simdjson/builtin/base.h"
#include "simdjson/generic/singlestage/dependencies.h"
#define SIMDJSON_AMALGAMATED
#if SIMDJSON_BUILTIN_IMPLEMENTATION_IS(arm64)
#include "simdjson/arm64/singlestage.h"
#elif SIMDJSON_BUILTIN_IMPLEMENTATION_IS(fallback)
#include "simdjson/fallback/singlestage.h"
#elif SIMDJSON_BUILTIN_IMPLEMENTATION_IS(haswell)
#include "simdjson/haswell/singlestage.h"
#elif SIMDJSON_BUILTIN_IMPLEMENTATION_IS(icelake)
#include "simdjson/icelake/singlestage.h"
#elif SIMDJSON_BUILTIN_IMPLEMENTATION_IS(ppc64)
#include "simdjson/ppc64/singlestage.h"
#elif SIMDJSON_BUILTIN_IMPLEMENTATION_IS(westmere)
#include "simdjson/westmere/singlestage.h"
#else
#error Unknown SIMDJSON_BUILTIN_IMPLEMENTATION
#endif
#undef SIMDJSON_AMALGAMATED
namespace simdjson {
/**
* @copydoc simdjson::SIMDJSON_BUILTIN_IMPLEMENTATION::singlestage
*/
namespace singlestage = SIMDJSON_BUILTIN_IMPLEMENTATION::singlestage;
} // namespace simdjson
#endif // SIMDJSON_BUILTIN_SINGLESTAGE_H
+8
View File
@@ -32,4 +32,12 @@
#error simdjson requires a compiler compliant with the C++11 standard
#endif
#ifndef SIMDJSON_IF_CONSTEXPR
#if SIMDJSON_CPLUSPLUS17
#define SIMDJSON_IF_CONSTEXPR if constexpr
#else
#define SIMDJSON_IF_CONSTEXPR if
#endif
#endif
#endif // SIMDJSON_COMPILER_CHECK_H
+3 -1
View File
@@ -5,10 +5,12 @@
#include "simdjson/dom/base.h"
#include "simdjson/dom/document.h"
#include "simdjson/dom/element.h"
#include "simdjson/dom/element-inl.h"
#include "simdjson/internal/tape_ref-inl.h"
#include "simdjson/internal/jsonformatutils.h"
#include <cstring>
namespace simdjson {
namespace dom {
+4
View File
@@ -7,8 +7,12 @@
#include "simdjson/dom/object.h"
#include "simdjson/internal/tape_type.h"
#include "simdjson/dom/object-inl.h"
#include "simdjson/error-inl.h"
#include <ostream>
#include <limits>
namespace simdjson {
//
+2
View File
@@ -7,6 +7,8 @@
#include "simdjson/dom/parser.h"
#include "simdjson/dom/element.h"
#include "simdjson/dom/parser-inl.h"
namespace simdjson {
//
+5
View File
@@ -5,6 +5,11 @@
#include "simdjson/dom/object.h"
#include "simdjson/dom/document.h"
#include "simdjson/dom/element-inl.h"
#include "simdjson/error-inl.h"
#include <cstring>
namespace simdjson {
//
@@ -3,10 +3,15 @@
#include "simdjson/dom/base.h"
#include "simdjson/dom/parsedjson_iterator.h"
#include "simdjson/internal/tape_ref-inl.h"
#include "simdjson/internal/jsonformatutils.h"
#include <ostream>
#include "simdjson/dom/parser-inl.h"
#include "simdjson/internal/tape_ref-inl.h"
#include <cstring>
#include <iterator>
#include <limits>
#include <ostream>
#ifndef SIMDJSON_DISABLE_DEPRECATED_API
+1
View File
@@ -8,6 +8,7 @@
#include "simdjson/error-inl.h"
#include "simdjson/padded_string-inl.h"
#include "simdjson/dom/document_stream-inl.h"
#include "simdjson/dom/element-inl.h"
#include <climits>
+42 -2
View File
@@ -7,6 +7,12 @@
#include "simdjson/dom/parser.h"
#include "simdjson/internal/tape_type.h"
#include "simdjson/dom/array-inl.h"
#include "simdjson/dom/object-inl.h"
#include "simdjson/internal/tape_ref-inl.h"
#include <cstring>
namespace simdjson {
namespace dom {
inline bool parser::print_json(std::ostream &os) const noexcept {
@@ -17,12 +23,46 @@ inline bool parser::print_json(std::ostream &os) const noexcept {
os << answer;
return true;
}
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::element value) {
simdjson::internal::string_builder<> sb;
sb.append(value);
return (out << sb.str());
}
#if SIMDJSON_EXCEPTIONS
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::element> x) {
if (x.error()) { throw simdjson::simdjson_error(x.error()); }
return (out << x.value());
}
#endif
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::array value) {
simdjson::internal::string_builder<> sb;
sb.append(value);
return (out << sb.str());
}
#if SIMDJSON_EXCEPTIONS
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::array> x) {
if (x.error()) { throw simdjson::simdjson_error(x.error()); }
return (out << x.value());
}
#endif
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::object value) {
simdjson::internal::string_builder<> sb;
sb.append(value);
return (out << sb.str());
}
#if SIMDJSON_EXCEPTIONS
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::object> x) {
if (x.error()) { throw simdjson::simdjson_error(x.error()); }
return (out << x.value());
}
#endif
} // namespace dom
/***
* Number utility functions
**/
namespace {
/**@private
* Escape sequence like \b or \u0001
+6 -27
View File
@@ -146,16 +146,9 @@ namespace dom {
* @param value The element.
* @throw if there is an error with the underlying output stream. simdjson itself will not throw.
*/
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::element value) {
simdjson::internal::string_builder<> sb;
sb.append(value);
return (out << sb.str());
}
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::element value);
#if SIMDJSON_EXCEPTIONS
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::element> x) {
if (x.error()) { throw simdjson::simdjson_error(x.error()); }
return (out << x.value());
}
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::element> x);
#endif
/**
* Print JSON to an output stream.
@@ -164,16 +157,9 @@ inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<sim
* @param value The array.
* @throw if there is an error with the underlying output stream. simdjson itself will not throw.
*/
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::array value) {
simdjson::internal::string_builder<> sb;
sb.append(value);
return (out << sb.str());
}
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::array value);
#if SIMDJSON_EXCEPTIONS
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::array> x) {
if (x.error()) { throw simdjson::simdjson_error(x.error()); }
return (out << x.value());
}
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::array> x);
#endif
/**
* Print JSON to an output stream.
@@ -182,16 +168,9 @@ inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<sim
* @param value The object.
* @throw if there is an error with the underlying output stream. simdjson itself will not throw.
*/
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::object value) {
simdjson::internal::string_builder<> sb;
sb.append(value);
return (out << sb.str());
}
inline std::ostream& operator<<(std::ostream& out, simdjson::dom::object value);
#if SIMDJSON_EXCEPTIONS
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::object> x) {
if (x.error()) { throw simdjson::simdjson_error(x.error()); }
return (out << x.value());
}
inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::dom::object> x);
#endif
} // namespace dom
+2 -2
View File
@@ -1,10 +1,10 @@
#ifndef SIMDJSON_ERROR_INL_H
#define SIMDJSON_ERROR_INL_H
#include <iostream>
#include "simdjson/error.h"
#include <iostream>
namespace simdjson {
namespace internal {
// We store the error code so we can validate the error message is associated with the right code
+2
View File
@@ -2,7 +2,9 @@
#define SIMDJSON_ERROR_H
#include "simdjson/base.h"
#include <string>
#include <ostream>
namespace simdjson {
+2 -9
View File
@@ -1,9 +1,9 @@
#ifndef SIMDJSON_FALLBACK_BASE_H
#define SIMDJSON_FALLBACK_BASE_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/base.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
/**
@@ -16,11 +16,4 @@ class implementation;
} // namespace fallback
} // namespace simdjson
#ifndef SIMDJSON_AMALGAMATED
// If we're editing one of the files in this directory, begin the implementation!
#ifndef SIMDJSON_IMPLEMENTATION
#include "simdjson/fallback/begin.h"
#endif
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_FALLBACK_BASE_H
+4 -2
View File
@@ -1,10 +1,12 @@
#ifndef SIMDJSON_FALLBACK_BITMANIPULATION_H
#define SIMDJSON_FALLBACK_BITMANIPULATION_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/fallback/base.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace fallback {
namespace {
#if defined(_MSC_VER) && !defined(_M_ARM64) && !defined(_M_X64)
@@ -40,7 +42,7 @@ simdjson_inline int leading_zeroes(uint64_t input_num) {
}
} // unnamed namespace
} // namespace SIMDJSON_IMPLEMENTATION
} // namespace fallback
} // namespace simdjson
#endif // SIMDJSON_FALLBACK_BITMANIPULATION_H
+4
View File
@@ -1 +1,5 @@
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/fallback/base.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#undef SIMDJSON_IMPLEMENTATION
+3 -2
View File
@@ -1,9 +1,10 @@
#ifndef SIMDJSON_FALLBACK_IMPLEMENTATION_H
#define SIMDJSON_FALLBACK_IMPLEMENTATION_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/fallback/base.h"
#include "simdjson/implementation.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace fallback {
@@ -1,11 +1,12 @@
#ifndef SIMDJSON_FALLBACK_NUMBERPARSING_DEFS_H
#define SIMDJSON_FALLBACK_NUMBERPARSING_DEFS_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/fallback/base.h"
#ifndef SIMDJSON_AMALGAMATED
#include "simdjson/internal/numberparsing_tables.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#include <cstring>
#ifdef JSON_TEST_NUMBERS // for unit testing
void found_invalid_number(const uint8_t *buf);
@@ -15,7 +16,7 @@ void found_float(double result, const uint8_t *buf);
#endif
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace fallback {
namespace numberparsing {
// credit: https://johnnylee-sde.github.io/Fast-numeric-string-to-int/
@@ -71,7 +72,7 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
}
} // namespace numberparsing
} // namespace SIMDJSON_IMPLEMENTATION
} // namespace fallback
} // namespace simdjson
#define SIMDJSON_SWAR_NUMBER_PARSING 1
-8
View File
@@ -1,8 +0,0 @@
#ifndef SIMDJSON_FALLBACK_SINGLESTAGE_H
#define SIMDJSON_FALLBACK_SINGLESTAGE_H
#include "simdjson/fallback/begin.h"
#include "simdjson/generic/singlestage/amalgamated.h"
#include "simdjson/fallback/end.h"
#endif // SIMDJSON_FALLBACK_SINGLESTAGE_H
@@ -1,10 +1,12 @@
#ifndef SIMDJSON_FALLBACK_STRINGPARSING_DEFS_H
#define SIMDJSON_FALLBACK_STRINGPARSING_DEFS_H
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#include "simdjson/fallback/base.h"
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
namespace fallback {
namespace {
// Holds backslashes and quotes locations.
@@ -28,7 +30,7 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
}
} // unnamed namespace
} // namespace SIMDJSON_IMPLEMENTATION
} // namespace fallback
} // namespace simdjson
#endif // SIMDJSON_FALLBACK_STRINGPARSING_DEFS_H
+1 -1
View File
@@ -1,4 +1,4 @@
#if defined(SIMDJSON_AMALGAMATED) && !defined(SIMDJSON_GENERIC_DEPENDENCIES_H)
#if defined(SIMDJSON_CONDITIONAL_INCLUDE) && !defined(SIMDJSON_GENERIC_DEPENDENCIES_H)
#error simdjson/generic/dependencies.h must be included before simdjson/generic/amalgamated.h!
#endif
+4 -2
View File
@@ -1,10 +1,12 @@
#ifndef SIMDJSON_GENERIC_ATOMPARSING_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ATOMPARSING_H
#include "simdjson/generic/base.h"
#include "simdjson/generic/jsoncharutils.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#include <cstring>
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
+2 -2
View File
@@ -1,6 +1,6 @@
#ifndef SIMDJSON_GENERIC_BASE_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_BASE_H
#include "simdjson/base.h"
// If we haven't got an implementation yet, we're in the editor, editing a generic file! Just
@@ -23,7 +23,7 @@
#error "All possible implementations (including fallback) have been disabled! simdjson will not run."
#endif
#endif // SIMDJSON_IMPLEMENTATION
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
+3 -2
View File
@@ -1,5 +1,5 @@
#ifdef SIMDJSON_AMALGAMATED
#error simdjson/generic/dependencies.h must be included before defining SIMDJSON_AMALGAMATED!
#ifdef SIMDJSON_CONDITIONAL_INCLUDE
#error simdjson/generic/dependencies.h must be included before defining SIMDJSON_CONDITIONAL_INCLUDE!
#endif
#ifndef SIMDJSON_GENERIC_DEPENDENCIES_H
@@ -10,6 +10,7 @@
// Otherwise, amalgamation will fail.
#include "simdjson/base.h"
#include "simdjson/implementation.h"
#include "simdjson/implementation_detection.h"
#include "simdjson/internal/instruction_set.h"
#include "simdjson/internal/dom_parser_implementation.h"
#include "simdjson/internal/jsoncharutils_tables.h"
@@ -1,10 +1,10 @@
#ifndef SIMDJSON_GENERIC_DOM_PARSER_IMPLEMENTATION_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_DOM_PARSER_IMPLEMENTATION_H
#include "simdjson/generic/base.h"
#include "simdjson/internal/dom_parser_implementation.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -1,10 +1,10 @@
#ifndef SIMDJSON_GENERIC_IMPLEMENTATION_SIMDJSON_RESULT_BASE_INL_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_IMPLEMENTATION_SIMDJSON_RESULT_BASE_INL_H
#include "simdjson/generic/base.h"
#include "simdjson/generic/implementation_simdjson_result_base.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -1,9 +1,9 @@
#ifndef SIMDJSON_GENERIC_IMPLEMENTATION_SIMDJSON_RESULT_BASE_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_IMPLEMENTATION_SIMDJSON_RESULT_BASE_H
#include "simdjson/generic/base.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
+2 -12
View File
@@ -1,11 +1,11 @@
#ifndef SIMDJSON_GENERIC_JSONCHARUTILS_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_JSONCHARUTILS_H
#include "simdjson/generic/base.h"
#include "simdjson/internal/jsoncharutils_tables.h"
#include "simdjson/internal/numberparsing_tables.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -96,16 +96,6 @@ static simdjson_inline uint64_t _umul128(uint64_t ab, uint64_t cd, uint64_t *hi)
}
#endif
inline std::string_view trim(const std::string_view str) noexcept {
// We can almost surely do better by rolling our own find_first_not_of function.
size_t first = str.find_first_not_of(" \t\n\r");
// If we have the empty string (just white space), then no trimming is possible, and
// we return the empty string_view.
if (std::string_view::npos == first) { return std::string_view(); }
size_t last = str.find_last_not_of(" \t\n\r");
return str.substr(first, (last - first + 1));
}
} // namespace jsoncharutils
} // unnamed namespace
} // namespace SIMDJSON_IMPLEMENTATION
+7 -10
View File
@@ -1,14 +1,15 @@
#ifndef SIMDJSON_GENERIC_NUMBERPARSING_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_NUMBERPARSING_H
#include "simdjson/generic/base.h"
#include "simdjson/generic/jsoncharutils.h"
#include "simdjson/internal/numberparsing_tables.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#include <limits>
#include <ostream>
#include <cstring>
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -185,8 +186,7 @@ simdjson_inline bool compute_float_64(int64_t power, uint64_t i, bool negative,
// power_of_five_128[index]. Usually, that's good enough to approximate i * 5^q
// to the desired approximation using one multiplication. Sometimes it does not suffice.
// Then we store the next most significant 64 bits in power_of_five_128[index + 1], and
// then we get a better approximation to i * 5^q. In very rare cases, even that
// will not suffice, though it is seemingly very hard to find such a scenario.
// then we get a better approximation to i * 5^q.
//
// That's for when q>=0. The logic for q<0 is somewhat similar but it is somewhat
// more complicated.
@@ -201,12 +201,9 @@ simdjson_inline bool compute_float_64(int64_t power, uint64_t i, bool negative,
simdjson::internal::value128 secondproduct = full_multiplication(i, simdjson::internal::power_of_five_128[index + 1]);
firstproduct.low += secondproduct.high;
if(secondproduct.high > firstproduct.low) { firstproduct.high++; }
// At this point, we might need to add at most one to firstproduct, but this
// can only change the value of firstproduct.high if firstproduct.low is maximal.
if(simdjson_unlikely(firstproduct.low == 0xFFFFFFFFFFFFFFFF)) {
// This is very unlikely, but if so, we need to do much more work!
return false;
}
// As it has been proven by Noble Mushtak and Daniel Lemire in "Fast Number Parsing Without
// Fallback" (https://arxiv.org/abs/2212.06644), at this point we are sure that the product
// is sufficiently accurate, and more computation is not needed.
}
uint64_t lower = firstproduct.low;
uint64_t upper = firstproduct.high;
@@ -1,4 +1,4 @@
#if defined(SIMDJSON_AMALGAMATED) && !defined(SIMDJSON_GENERIC_ONDEMAND_DEPENDENCIES_H)
#if defined(SIMDJSON_CONDITIONAL_INCLUDE) && !defined(SIMDJSON_GENERIC_ONDEMAND_DEPENDENCIES_H)
#error simdjson/generic/ondemand/dependencies.h must be included before simdjson/generic/ondemand/amalgamated.h!
#endif
@@ -1,6 +1,6 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_ARRAY_INL_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_ARRAY_INL_H
#include "simdjson/generic/ondemand/base.h"
#include "simdjson/generic/ondemand/array.h"
@@ -8,7 +8,7 @@
#include "simdjson/generic/ondemand/json_iterator.h"
#include "simdjson/generic/ondemand/value.h"
#include "simdjson/generic/ondemand/value_iterator-inl.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
+2 -2
View File
@@ -1,11 +1,11 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_ARRAY_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_ARRAY_H
#include "simdjson/generic/ondemand/base.h"
#include "simdjson/generic/implementation_simdjson_result_base.h"
#include "simdjson/generic/ondemand/value_iterator.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -1,12 +1,12 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_ARRAY_ITERATOR_INL_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_ARRAY_ITERATOR_INL_H
#include "simdjson/generic/ondemand/base.h"
#include "simdjson/generic/ondemand/array_iterator.h"
#include "simdjson/generic/ondemand/value-inl.h"
#include "simdjson/generic/ondemand/value_iterator-inl.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -1,11 +1,11 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_ARRAY_ITERATOR_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_ARRAY_ITERATOR_H
#include "simdjson/generic/implementation_simdjson_result_base.h"
#include "simdjson/generic/ondemand/base.h"
#include "simdjson/generic/ondemand/value_iterator.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
+2 -2
View File
@@ -1,9 +1,9 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_BASE_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_BASE_H
#include "simdjson/generic/base.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -1,5 +1,5 @@
#ifdef SIMDJSON_AMALGAMATED
#error simdjson/generic/ondemand/dependencies.h must be included before defining SIMDJSON_AMALGAMATED!
#ifdef SIMDJSON_CONDITIONAL_INCLUDE
#error simdjson/generic/ondemand/dependencies.h must be included before defining SIMDJSON_CONDITIONAL_INCLUDE!
#endif
#ifndef SIMDJSON_GENERIC_ONDEMAND_DEPENDENCIES_H
@@ -1,6 +1,6 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_INL_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_INL_H
#include "simdjson/generic/ondemand/base.h"
#include "simdjson/generic/ondemand/array-inl.h"
@@ -12,7 +12,7 @@
#include "simdjson/generic/ondemand/raw_json_string.h"
#include "simdjson/generic/ondemand/value.h"
#include "simdjson/generic/ondemand/value_iterator-inl.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
+2 -2
View File
@@ -1,10 +1,10 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_H
#include "simdjson/generic/ondemand/base.h"
#include "simdjson/generic/ondemand/json_iterator.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -1,12 +1,12 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_STREAM_INL_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_STREAM_INL_H
#include "simdjson/generic/ondemand/base.h"
#include "simdjson/generic/ondemand/document_stream.h"
#include "simdjson/generic/ondemand/document-inl.h"
#include "simdjson/generic/implementation_simdjson_result_base-inl.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#include <algorithm>
#include <stdexcept>
@@ -1,12 +1,12 @@
#ifndef SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_STREAM_H
#ifndef SIMDJSON_AMALGAMATED
#ifndef SIMDJSON_CONDITIONAL_INCLUDE
#define SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_STREAM_H
#include "simdjson/generic/ondemand/base.h"
#include "simdjson/generic/implementation_simdjson_result_base.h"
#include "simdjson/generic/ondemand/document.h"
#include "simdjson/generic/ondemand/parser.h"
#endif // SIMDJSON_AMALGAMATED
#endif // SIMDJSON_CONDITIONAL_INCLUDE
#ifdef SIMDJSON_THREADS_ENABLED
#include <thread>

Some files were not shown because too many files have changed in this diff Show More