Compare commits

...

31 Commits

Author SHA1 Message Date
Daniel Lemire 933c2ebeac Preparing release 2022-07-28 21:46:45 -04:00
Daniel Lemire db3e813aa6 Verifying and fixing issue 1876 (#1877)
* Verifying and fixing issue 1876

* Typo
2022-07-28 21:45:54 -04:00
Daniel Lemire 9c95a48fe6 cleaning on-demand benchmarks (#1875)
* Setting RapidJSON and yyjson to their latest version.

* Let us stop dumping all of the benchmarks (it is confusing) on screen.
2022-07-28 20:28:29 -04:00
Daniel Lemire cb20f7e7df Update CONTRIBUTING.md 2022-07-20 09:46:52 -04:00
Dirk Stolle 18b9168eec remove empty if block (#1873)
I guess it will be thrown away by the compiler's optimizer
anyway, but there is no need to keep this in the code.
2022-07-19 20:36:51 -04:00
Daniel Lemire 241ce7304c New version. 2022-07-19 16:47:06 -04:00
Daniel Lemire 5dbe96da96 We change slightly on development checks are enabled. (#1869)
* We change slightly on development checks are enabled.

* Removing garbagy code.
2022-07-19 16:40:02 -04:00
Daniel Lemire 40b397a3d4 Fixing issue 1870 (#1871) 2022-07-19 15:17:12 -04:00
strager 5510089d45 Improve build times for debug builds (#1859)
* Rename simdjson_really_inline -> simdjson_inline

I want to change the simdjson_really_inline macro to sometimes not force
inlining. After that upcoming change, the name simdjson_really_inline
will no longer makes sense.

Rename simdjson_really_inline to simdjson_inline. This patch should not
change semantics; simdjson_inline still forces inlining as before.

Some functions still need to be really inlined for ABI reasons.
(GCC's -Wpsabi complains otherwise.) Leave those functions marked as
simdjson_really_inline.

* Improve build times for debug builds

simdjson_inline is used for most simdjson functions. It forces inlining.
In unoptimized/debug builds, this can lead to a lot of machine code
being generated (especially with Address Sanitizer), causing slow
compilation.

Change simdjson_inline to force inlining only for optimized builds.

Sometimes, the programmer might want a slightly-optimized build and want
fast compilation (e.g. GCC's -Og mode). Allow simdjson users to define
the simdjson_inline macro themselves (e.g. on the command line:
-Dsimdjson_inline=inline) in cases where the default behavior is
undesired.

This patch reduced build times by over 75% for ondemand_object_tests.cpp
with GCC 9.4.0 and CMAKE_BUILD_TYPE=Debug on my AMD 5950X:

Before: 6.885 6.683 6.971 6.957 6.949 seconds (5 samples)
After:  1.492 1.551 1.494 1.490 1.531 seconds (5 samples)
2022-07-19 15:14:33 -04:00
Daniel Lemire 62a57907a7 Adding DOM benchmark to msgpack (#1866)
* We need simdjson dom for the json2msgpack benchmark

* Minor tweaking.
2022-07-14 13:47:10 -04:00
Daniel Lemire becbe99e81 Update README.md 2022-07-14 09:45:04 -04:00
Daniel Lemire bd3d67e889 Documenting a specific use case where you need a value if and only if it is another key is not present (#1865)
* Documenting a specific use case.

* Adding more comments and documentation.
2022-07-09 10:54:28 -04:00
Dirk Stolle a89d57d0d4 update dependency nlohmann/json for benchmarks to current version 3.10.5 (#1862) 2022-07-08 12:32:05 -04:00
Dirk Stolle e04d400c64 update jsoncpp to version 1.9.5 (#1863) 2022-07-08 12:31:43 -04:00
Daniel Lemire 1075e8609c Updating single header, prior to release. 2022-07-04 21:55:09 -04:00
Daniel Lemire 3fde8a4eac New release candidate (#1856)
* Patch for possible AVX-512 overflow.

* Updating the test for new padding.

* Preparing new version.

* replace binary integer literals with hex literals for C++11 compatibility (#1855)

Binary integer literals are a C++14 feature, so those are not supported
in C++11 and should be replaced by hexadecimal literals instead.

Fixes #1854.

Co-authored-by: Dirk Stolle <striezel-dev@web.de>
2022-07-04 21:54:19 -04:00
Daniel Lemire 62bdb9a2f7 [noci] update users 2022-07-02 21:44:48 -04:00
Daniel Lemire 2fbacb0058 New version 2022-06-30 11:49:26 -04:00
Daniel Lemire 509066f06a adding msgpack benchmarks (#1853) 2022-06-30 10:29:50 -04:00
Daniel Lemire 070f0b26a3 Removing dead code. (#1852) 2022-06-25 00:11:31 -04:00
Daniel Lemire 1a195623a5 Improve string performance in ondemand by making the string processing runtime dispatched. (#1849)
* This should improve string performance in ondemand by making the string processing runtime dispatched.
2022-06-24 09:57:16 -04:00
Dirk Stolle 5f7a56e7f1 add SIMDJSON_IMPLEMENTATION_ICELAKE to implementation-selection.md (#1848) 2022-06-22 13:09:19 -04:00
Daniel Lemire 5e60f0482d Update implementation-selection.md 2022-06-21 21:29:12 -04:00
Daniel Lemire e6c90b8efb Preparing release. 2022-06-15 15:23:03 -04:00
Daniel Lemire 7c450fbb70 Adding clang 13 tests. (#1844) 2022-06-15 15:21:33 -04:00
Daniel Lemire 4e1e002cb0 This verifies and fixes issue 1834. (#1843) 2022-06-15 13:42:04 -04:00
Benson Muite de196dd7a3 [skip ci] Grammar and typo fixes (#1842) 2022-06-14 16:16:39 -04:00
Daniel Lemire 08cb8dd81c Simpler counters. (#1841) 2022-06-07 15:19:51 -04:00
Dirk Stolle 57d54792ba update actions/cache + actions/checkout in GitHub Actions to v3 (#1839) 2022-06-06 08:51:50 -04:00
Dirk Stolle 79879802f9 run tests with GCC 8, too (#1837) 2022-06-03 20:52:07 -04:00
Daniel Lemire fbe955e9a4 gcc12 without warnings. (#1836) 2022-06-02 22:53:39 -04:00
209 changed files with 10380 additions and 8661 deletions
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
@@ -6,7 +6,7 @@ jobs:
whitespace: whitespace:
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- name: Remove whitespace and check the diff - name: Remove whitespace and check the diff
run: | run: |
set -eu set -eu
+3 -3
View File
@@ -37,14 +37,14 @@ jobs:
chmod +x llvm.sh chmod +x llvm.sh
sudo ./llvm.sh $CLANGVERSION sudo ./llvm.sh $CLANGVERSION
- uses: actions/checkout@v1 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- uses: actions/cache@v2 - uses: actions/cache@v3
id: cache-corpus id: cache-corpus
with: with:
path: out/ path: out/
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: macos-latest runs-on: macos-latest
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: macos-latest runs-on: macos-latest
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -24,8 +24,8 @@ jobs:
CMAKE_GENERATOR: Ninja CMAKE_GENERATOR: Ninja
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -26,8 +26,8 @@ jobs:
CMAKE_GENERATOR: Ninja CMAKE_GENERATOR: Ninja
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -15,8 +15,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-18.04 runs-on: ubuntu-18.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -13,8 +13,8 @@ jobs:
CC: clang-7 CC: clang-7
CXX: clang++-7 CXX: clang++-7
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -10,8 +10,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-18.04 runs-on: ubuntu-18.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-18.04 runs-on: ubuntu-18.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -15,8 +15,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+38
View File
@@ -0,0 +1,38 @@
name: Ubuntu 20.04 CI (GCC 8)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04
env:
CXX: g++-8
CC: gcc-8
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install GCC 8
run: sudo apt-get install -y g++-8
- name: Use cmake
run: |
mkdir builddebug &&
cd builddebug &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . &&
ctest -j --output-on-failure -LE explicitonly &&
cd .. &&
mkdir build &&
cd build &&
cmake -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
ctest -j --output-on-failure -LE explicitonly &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
cd ../tests/installation_tests/find &&
mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../build/destination .. && cmake --build .
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+25
View File
@@ -0,0 +1,25 @@
name: Ubuntu 22.04 CI (CLANG 13)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install clang++-13
run: sudo apt-get install -y clang++-13
- name: Use cmake
run: |
mkdir build &&
cd build &&
CXX=clang++-13 cmake -DSIMDJSON_DEVELOPER_MODE=ON .. &&
cmake --build . &&
ctest -j --output-on-failure -LE explicitonly
+25
View File
@@ -0,0 +1,25 @@
name: Ubuntu 22.04 CI (GCC 12)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Install gcc12
run: sudo apt-get install -y g++-12
- name: Use cmake
run: |
mkdir build &&
cd build &&
CXX=g++-12 cmake -DSIMDJSON_DEVELOPER_MODE=ON .. &&
cmake --build . &&
ctest -j --output-on-failure -LE explicitonly
+2 -2
View File
@@ -9,8 +9,8 @@ jobs:
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04 runs-on: ubuntu-22.04
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+1 -1
View File
@@ -14,7 +14,7 @@ jobs:
- {arch: ARM64} - {arch: ARM64}
steps: steps:
- name: checkout - name: checkout
uses: actions/checkout@v2 uses: actions/checkout@v3
- name: Use cmake - name: Use cmake
run: | run: |
cmake -A ${{ matrix.arch }} -DCMAKE_CROSSCOMPILING=1 -DSIMDJSON_DEVELOPER_MODE=ON -D SIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_EXCEPTIONS=OFF -B build && cmake -A ${{ matrix.arch }} -DCMAKE_CROSSCOMPILING=1 -DSIMDJSON_DEVELOPER_MODE=ON -D SIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_EXCEPTIONS=OFF -B build &&
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF} - {gen: Visual Studio 17 2022, arch: x64, shared: OFF}
steps: steps:
- name: checkout - name: checkout
uses: actions/checkout@v2 uses: actions/checkout@v3
- name: Configure - name: Configure
run: | run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
+1 -1
View File
@@ -16,7 +16,7 @@ jobs:
- {gen: Visual Studio 17 2022, arch: x64} - {gen: Visual Studio 17 2022, arch: x64}
steps: steps:
- name: checkout - name: checkout
uses: actions/checkout@v2 uses: actions/checkout@v3
- name: Configure - name: Configure
run: | run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -T ClangCL -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -T ClangCL -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
+2 -2
View File
@@ -7,8 +7,8 @@ jobs:
name: windows-vs17 name: windows-vs17
runs-on: windows-latest runs-on: windows-latest
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v3
- uses: actions/cache@v2 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+11 -1
View File
@@ -86,6 +86,16 @@
"vector": "cpp", "vector": "cpp",
"*.ipp": "cpp", "*.ipp": "cpp",
"__functional_base_03": "cpp", "__functional_base_03": "cpp",
"filesystem": "cpp" "filesystem": "cpp",
"*.inc": "cpp",
"compare": "cpp",
"concepts": "cpp",
"variant": "cpp",
"__bits": "cpp",
"csignal": "cpp",
"future": "cpp",
"queue": "cpp",
"shared_mutex": "cpp",
"ranges": "cpp"
} }
} }
+3 -3
View File
@@ -3,7 +3,7 @@ cmake_minimum_required(VERSION 3.14)
project( project(
simdjson simdjson
# The version number is modified by tools/release.py # The version number is modified by tools/release.py
VERSION 2.0.3 VERSION 2.2.2
DESCRIPTION "Parsing gigabytes of JSON per second" DESCRIPTION "Parsing gigabytes of JSON per second"
HOMEPAGE_URL "https://simdjson.org/" HOMEPAGE_URL "https://simdjson.org/"
LANGUAGES CXX C LANGUAGES CXX C
@@ -20,8 +20,8 @@ string(
# ---- Options, variables ---- # ---- Options, variables ----
# These version numbers are modified by tools/release.py # These version numbers are modified by tools/release.py
set(SIMDJSON_LIB_VERSION "11.0.0" CACHE STRING "simdjson library version") set(SIMDJSON_LIB_VERSION "13.0.0" CACHE STRING "simdjson library version")
set(SIMDJSON_LIB_SOVERSION "11" CACHE STRING "simdjson library soversion") set(SIMDJSON_LIB_SOVERSION "13" CACHE STRING "simdjson library soversion")
option(SIMDJSON_ENABLE_THREADS "Link with thread support" ON) option(SIMDJSON_ENABLE_THREADS "Link with thread support" ON)
+5 -5
View File
@@ -54,7 +54,7 @@ Contributors are encouraged to :
- Document their changes. Though we do not enforce a rule regarding code comments, we prefer that non-trivial algorithms and techniques be somewhat documented in the code. - Document their changes. Though we do not enforce a rule regarding code comments, we prefer that non-trivial algorithms and techniques be somewhat documented in the code.
- Follow as much as possible the existing code style. We do not enforce a specific code style, but we prefer consistency. - Follow as much as possible the existing code style. We do not enforce a specific code style, but we prefer consistency.
- Modify as few lines of code as possible when working on an issue. The more lines you modify, the harder it is for your fellow human beings to understand what is going on. - Modify as few lines of code as possible when working on an issue. The more lines you modify, the harder it is for your fellow human beings to understand what is going on.
- Tools may report "problems" with the code, but we never delegate programming to tools: if there is a problem with the code, we need to understand it. Thus we will not "fix" code merely to please a static analyzer if we do not understand. - Tools may report "problems" with the code, but we never delegate programming to tools: if there is a problem with the code, we need to understand it. Thus we will not "fix" code merely to please a static analyzer.
- Provide tests for any new feature. We will not merge a new feature without tests. - Provide tests for any new feature. We will not merge a new feature without tests.
Pull Requests Pull Requests
@@ -62,13 +62,13 @@ Pull Requests
Pull requests are always invited. However, we ask that you follow these guidelines: Pull requests are always invited. However, we ask that you follow these guidelines:
- It is wiser to discuss your ideas first as part of an issue before you start coding. If you omit this step and code first, be prepare to have your code receive scrutiny and be dropped. - It is wise to discuss your ideas first as part of an issue before you start coding. If you omit this step and code first, be prepared to have your code receive scrutiny and be dropped.
- Users should provide a rationale for their changes. Does it improve performance? Does it add a feature? Does it improve maintainability? Does fix a bug? This must be explicitly stated as part of the pull request. Do not propose changes based on taste or intuition. We do not delegate programming to tools: that some tool suggested a code change is not reason enough to change the code. - Users should provide a rationale for their changes. Does it improve performance? Does it add a feature? Does it improve maintainability? Does it fix a bug? This must be explicitly stated as part of the pull request. Do not propose changes based on taste or intuition. We do not delegate programming to tools: that some tool suggested a code change is not reason enough to change the code.
1. When your code improves performance, please document the gains with a benchmark using hard numbers. 1. When your code improves performance, please document the gains with a benchmark using hard numbers.
2. If your code fixes a bug, please be either fix a failing test, or propose a new test. 2. If your code fixes a bug, please either fix a failing test, or propose a new test.
3. Other types of changes must be clearly motivated. We openly discourage changes with no identifiable benefits. 3. Other types of changes must be clearly motivated. We openly discourage changes with no identifiable benefits.
- Changes should be focused and minimal. You should change as few lines of code as possible. Please do not reformat or touch files needlessly. - Changes should be focused and minimal. You should change as few lines of code as possible. Please do not reformat or touch files needlessly.
- New features must be accompanied of new tests, in general. - New features must be accompanied by new tests, in general.
- Your code should pass our continuous-integration tests. It is your responsibility to ensure that your proposal pass the tests. We do not merge pull requests that would break our build. - Your code should pass our continuous-integration tests. It is your responsibility to ensure that your proposal pass the tests. We do not merge pull requests that would break our build.
- An exception to this would be changes to non-code files, such as documentation and assets, or trivial changes to code, such as comments, where it is encouraged to explicitly ask for skipping a CI run using the `[skip ci]` prefix in your Pull Request title **and** in the first line of the most recent commit in a push. Example for such a commit: `[skip ci] Fixed typo in power_of_ten's docs` - An exception to this would be changes to non-code files, such as documentation and assets, or trivial changes to code, such as comments, where it is encouraged to explicitly ask for skipping a CI run using the `[skip ci]` prefix in your Pull Request title **and** in the first line of the most recent commit in a push. Example for such a commit: `[skip ci] Fixed typo in power_of_ten's docs`
This benefits the project in such a way that the CI pipeline is not burdened by running jobs on changes that don't change any behavior in the code, which reduces wait times for other Pull Requests that do change behavior and require testing. This benefits the project in such a way that the CI pipeline is not burdened by running jobs on changes that don't change any behavior in the code, which reduces wait times for other Pull Requests that do change behavior and require testing.
+1 -1
View File
@@ -38,7 +38,7 @@ PROJECT_NAME = simdjson
# could be handy for archiving the generated documentation or if some version # could be handy for archiving the generated documentation or if some version
# control system is used. # control system is used.
PROJECT_NUMBER = "2.0.3" PROJECT_NUMBER = "2.2.2"
# Using the PROJECT_BRIEF tag one can provide an optional one line description # Using the PROJECT_BRIEF tag one can provide an optional one line description
# for a project that appears at the top of each page and should give viewer a # for a project that appears at the top of each page and should give viewer a
+5 -3
View File
@@ -111,10 +111,12 @@ For NDJSON files, we can exceed 3 GB/s with [our multithreaded parsing function
Real-world usage Real-world usage
---------------- ----------------
- [Microsoft FishStore](https://github.com/microsoft/FishStore) - [ClickHouse](https://github.com/ClickHouse/ClickHouse)
- [Yandex ClickHouse](https://github.com/yandex/ClickHouse)
- [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer) - [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer)
- [Shopify HeapProfiler](https://github.com/Shopify/heap-profiler) - [Shopify HeapProfiler](https://github.com/Shopify/heap-profiler)
- [StarRocks](https://github.com/StarRocks/starrocks)
- [Microsoft FishStore](https://github.com/microsoft/FishStore)
If you are planning to use simdjson in a product, please work from one of our releases. If you are planning to use simdjson in a product, please work from one of our releases.
@@ -139,7 +141,7 @@ We distinguish between "bindings" (which just wrap the C++ code) and a port to a
- [simdjson_erlang](https://github.com/ChomperT/simdjson_erlang): erlang bindings. - [simdjson_erlang](https://github.com/ChomperT/simdjson_erlang): erlang bindings.
- [lua-simdjson](https://github.com/FourierTransformer/lua-simdjson): lua bindings. - [lua-simdjson](https://github.com/FourierTransformer/lua-simdjson): lua bindings.
- [hermes-json](https://hackage.haskell.org/package/hermes-json): haskell bindings. - [hermes-json](https://hackage.haskell.org/package/hermes-json): haskell bindings.
- [simdjzon](https://github.com/travisstaloch/simdjzon): zig port.
About simdjson About simdjson
-------------- --------------
@@ -15,11 +15,11 @@ using namespace json_benchmark;
struct brand { struct brand {
double cumulative_rating; double cumulative_rating;
uint64_t reviews_count; uint64_t reviews_count;
simdjson_really_inline bool operator==(const brand &other) const { simdjson_inline bool operator==(const brand &other) const {
return cumulative_rating == other.cumulative_rating && return cumulative_rating == other.cumulative_rating &&
reviews_count == other.reviews_count; reviews_count == other.reviews_count;
} }
simdjson_really_inline bool operator!=(const brand &other) const { return !(*this == other); } simdjson_inline bool operator!=(const brand &other) const { return !(*this == other); }
}; };
simdjson_unused static std::ostream &operator<<(std::ostream &o, const brand &b) { simdjson_unused static std::ostream &operator<<(std::ostream &o, const brand &b) {
@@ -66,8 +66,8 @@ struct runner : public file_runner<I> {
template<bool threaded> template<bool threaded>
struct simdjson_dom; struct simdjson_dom;
template<typename I> simdjson_really_inline static void amazon_cellphones(benchmark::State &state) { template<typename I> simdjson_inline static void amazon_cellphones(benchmark::State &state) {
run_json_benchmark<runner<I>, runner<simdjson_dom<UNTHREADED>>>(state); run_json_benchmark<runner<I>, runner<simdjson_dom<UNTHREADED>>>(state);
} }
} // namespace amazon_cellphones } // namespace amazon_cellphones
+105 -57
View File
@@ -25,6 +25,111 @@ SIMDJSON_PUSH_DISABLE_ALL_WARNINGS
#include <benchmark/benchmark.h> #include <benchmark/benchmark.h>
SIMDJSON_POP_DISABLE_WARNINGS SIMDJSON_POP_DISABLE_WARNINGS
#include "json2msgpack/simdjson_ondemand.h"
#include "json2msgpack/simdjson_dom.h"
#include "json2msgpack/yyjson.h"
#include "json2msgpack/rapidjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "json2msgpack/sajson.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "json2msgpack/nlohmann_json.h"
#include "partial_tweets/simdjson_ondemand.h"
#include "partial_tweets/simdjson_dom.h"
#include "partial_tweets/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "partial_tweets/sajson.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "partial_tweets/rapidjson.h"
#if SIMDJSON_COMPETITION_SAX
#include "partial_tweets/rapidjson_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "partial_tweets/nlohmann_json.h"
#if SIMDJSON_COMPETITION_SAX
#include "partial_tweets/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "distinct_user_id/simdjson_ondemand.h"
#include "distinct_user_id/simdjson_ondemand_json_pointer.h"
#include "distinct_user_id/simdjson_dom.h"
#include "distinct_user_id/simdjson_dom_json_pointer.h"
#include "distinct_user_id/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "distinct_user_id/sajson.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "distinct_user_id/rapidjson.h"
#if SIMDJSON_COMPETITION_SAX
#include "distinct_user_id/rapidjson_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "distinct_user_id/nlohmann_json.h"
#if SIMDJSON_COMPETITION_SAX
#include "distinct_user_id/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "find_tweet/simdjson_ondemand.h"
#include "find_tweet/simdjson_dom.h"
#include "find_tweet/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "find_tweet/sajson.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "find_tweet/rapidjson.h"
#if SIMDJSON_COMPETITION_SAX
#include "find_tweet/rapidjson_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "find_tweet/nlohmann_json.h"
#if SIMDJSON_COMPETITION_SAX
#include "find_tweet/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "top_tweet/simdjson_ondemand.h"
#include "top_tweet/simdjson_dom.h"
#include "top_tweet/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "top_tweet/sajson.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "top_tweet/rapidjson.h"
#if SIMDJSON_COMPETITION_SAX
#include "top_tweet/rapidjson_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "top_tweet/nlohmann_json.h"
#if SIMDJSON_COMPETITION_SAX
#include "top_tweet/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "kostya/simdjson_ondemand.h"
#include "kostya/simdjson_dom.h"
#include "kostya/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "kostya/sajson.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "kostya/rapidjson.h"
#if SIMDJSON_COMPETITION_SAX
#include "kostya/rapidjson_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "kostya/nlohmann_json.h"
#if SIMDJSON_COMPETITION_SAX
#include "kostya/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "large_random/simdjson_ondemand.h"
#if SIMDJSON_COMPETITION_ONDEMAND_UNORDERED
#include "large_random/simdjson_ondemand_unordered.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_UNORDERED
#include "large_random/simdjson_dom.h"
#include "large_random/yyjson.h"
#if SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "large_random/sajson.h"
#endif // SIMDJSON_COMPETITION_ONDEMAND_SAJSON
#include "large_random/rapidjson.h"
#if SIMDJSON_COMPETITION_SAX
#include "large_random/rapidjson_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "large_random/nlohmann_json.h"
#if SIMDJSON_COMPETITION_SAX
#include "large_random/nlohmann_json_sax.h"
#endif // SIMDJSON_COMPETITION_SAX
#include "amazon_cellphones/simdjson_dom.h" #include "amazon_cellphones/simdjson_dom.h"
#include "amazon_cellphones/simdjson_ondemand.h" #include "amazon_cellphones/simdjson_ondemand.h"
@@ -32,61 +137,4 @@ SIMDJSON_POP_DISABLE_WARNINGS
#include "large_amazon_cellphones/simdjson_dom.h" #include "large_amazon_cellphones/simdjson_dom.h"
#include "large_amazon_cellphones/simdjson_ondemand.h" #include "large_amazon_cellphones/simdjson_ondemand.h"
#include "partial_tweets/simdjson_dom.h"
#include "partial_tweets/simdjson_ondemand.h"
#include "partial_tweets/yyjson.h"
#include "partial_tweets/sajson.h"
#include "partial_tweets/rapidjson.h"
#include "partial_tweets/rapidjson_sax.h"
#include "partial_tweets/nlohmann_json.h"
#include "partial_tweets/nlohmann_json_sax.h"
#include "large_random/simdjson_dom.h"
#include "large_random/simdjson_ondemand.h"
#include "large_random/simdjson_ondemand_unordered.h"
#include "large_random/yyjson.h"
#include "large_random/sajson.h"
#include "large_random/rapidjson.h"
#include "large_random/rapidjson_sax.h"
#include "large_random/nlohmann_json.h"
#include "large_random/nlohmann_json_sax.h"
#include "kostya/simdjson_dom.h"
#include "kostya/simdjson_ondemand.h"
#include "kostya/yyjson.h"
#include "kostya/sajson.h"
#include "kostya/rapidjson.h"
#include "kostya/rapidjson_sax.h"
#include "kostya/nlohmann_json.h"
#include "kostya/nlohmann_json_sax.h"
#include "distinct_user_id/simdjson_dom.h"
#include "distinct_user_id/simdjson_dom_json_pointer.h"
#include "distinct_user_id/simdjson_ondemand.h"
#include "distinct_user_id/simdjson_ondemand_json_pointer.h"
#include "distinct_user_id/yyjson.h"
#include "distinct_user_id/sajson.h"
#include "distinct_user_id/rapidjson.h"
#include "distinct_user_id/rapidjson_sax.h"
#include "distinct_user_id/nlohmann_json.h"
#include "distinct_user_id/nlohmann_json_sax.h"
#include "find_tweet/simdjson_dom.h"
#include "find_tweet/simdjson_ondemand.h"
#include "find_tweet/yyjson.h"
#include "find_tweet/sajson.h"
#include "find_tweet/rapidjson.h"
#include "find_tweet/rapidjson_sax.h"
#include "find_tweet/nlohmann_json.h"
#include "find_tweet/nlohmann_json_sax.h"
#include "top_tweet/simdjson_dom.h"
#include "top_tweet/simdjson_ondemand.h"
#include "top_tweet/yyjson.h"
#include "top_tweet/sajson.h"
#include "top_tweet/rapidjson.h"
#include "top_tweet/rapidjson_sax.h"
#include "top_tweet/nlohmann_json.h"
#include "top_tweet/nlohmann_json_sax.h"
BENCHMARK_MAIN(); BENCHMARK_MAIN();
+40 -4
View File
@@ -164,7 +164,7 @@ struct feature_benchmarker {
} }
simdjson_really_inline void run_iterations(size_t iterations, bool stage1_only=false) { simdjson_inline void run_iterations(size_t iterations, bool stage1_only=false) {
struct7.run_iterations(iterations, stage1_only); struct7.run_iterations(iterations, stage1_only);
struct7_miss.run_iterations(iterations, stage1_only); struct7_miss.run_iterations(iterations, stage1_only);
struct7_full.run_iterations(iterations, stage1_only); struct7_full.run_iterations(iterations, stage1_only);
@@ -204,10 +204,13 @@ struct feature_benchmarker {
} }
// Rate of 1-7-structural misses per 8-structural flip // Rate of 1-7-structural misses per 8-structural flip
double struct1_7_miss_rate(BenchmarkStage stage) const { double struct1_7_miss_rate(BenchmarkStage stage) const {
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
return 1;
#else
if (!has_events()) { return 1; } if (!has_events()) { return 1; }
return struct7_miss[stage].best.branch_misses() - struct7[stage].best.branch_misses() / double(struct7_miss.stats->blocks_with_1_structural_flipped); return struct7_miss[stage].best.branch_misses() - struct7[stage].best.branch_misses() / double(struct7_miss.stats->blocks_with_1_structural_flipped);
#endif
} }
// Extra cost of an 8-15 structural block over a 1-7 structural block // Extra cost of an 8-15 structural block over a 1-7 structural block
double struct8_15_cost(BenchmarkStage stage) const { double struct8_15_cost(BenchmarkStage stage) const {
return cost_per_block(stage, struct15, struct15.stats->blocks_with_8_structurals, struct7); return cost_per_block(stage, struct15, struct15.stats->blocks_with_8_structurals, struct7);
@@ -218,8 +221,12 @@ struct feature_benchmarker {
} }
// Rate of 8-15-structural misses per 8-structural flip // Rate of 8-15-structural misses per 8-structural flip
double struct8_15_miss_rate(BenchmarkStage stage) const { double struct8_15_miss_rate(BenchmarkStage stage) const {
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
return 1;
#else
if (!has_events()) { return 1; } if (!has_events()) { return 1; }
return double(struct15_miss[stage].best.branch_misses() - struct15[stage].best.branch_misses()) / double(struct15_miss.stats->blocks_with_8_structurals_flipped); return double(struct15_miss[stage].best.branch_misses() - struct15[stage].best.branch_misses()) / double(struct15_miss.stats->blocks_with_8_structurals_flipped);
#endif
} }
// Extra cost of a 16+-structural block over an 8-15 structural block (actual varies based on # of structurals!) // Extra cost of a 16+-structural block over an 8-15 structural block (actual varies based on # of structurals!)
@@ -232,10 +239,15 @@ struct feature_benchmarker {
} }
// Rate of 16-structural misses per 16-structural flip // Rate of 16-structural misses per 16-structural flip
double struct16_miss_rate(BenchmarkStage stage) const { double struct16_miss_rate(BenchmarkStage stage) const {
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
return 1;
#else
if (!has_events()) { return 1; } if (!has_events()) { return 1; }
return double(struct23_miss[stage].best.branch_misses() - struct23[stage].best.branch_misses()) / double(struct23_miss.stats->blocks_with_16_structurals_flipped); return double(struct23_miss[stage].best.branch_misses() - struct23[stage].best.branch_misses()) / double(struct23_miss.stats->blocks_with_16_structurals_flipped);
#endif
} }
// Extra cost of having UTF-8 in a block // Extra cost of having UTF-8 in a block
double utf8_cost(BenchmarkStage stage) const { double utf8_cost(BenchmarkStage stage) const {
return cost_per_block(stage, utf8, utf8.stats->blocks_with_utf8, struct7_full); return cost_per_block(stage, utf8, utf8.stats->blocks_with_utf8, struct7_full);
@@ -246,10 +258,13 @@ struct feature_benchmarker {
} }
// Rate of UTF-8 misses per UTF-8 flip // Rate of UTF-8 misses per UTF-8 flip
double utf8_miss_rate(BenchmarkStage stage) const { double utf8_miss_rate(BenchmarkStage stage) const {
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
return 1;
#else
if (!has_events()) { return 1; } if (!has_events()) { return 1; }
return double(utf8_miss[stage].best.branch_misses() - utf8[stage].best.branch_misses()) / double(utf8_miss.stats->blocks_with_utf8_flipped); return double(utf8_miss[stage].best.branch_misses() - utf8[stage].best.branch_misses()) / double(utf8_miss.stats->blocks_with_utf8_flipped);
#endif
} }
// Extra cost of having escapes in a block // Extra cost of having escapes in a block
double escape_cost(BenchmarkStage stage) const { double escape_cost(BenchmarkStage stage) const {
return cost_per_block(stage, escape, escape.stats->blocks_with_escapes, struct7_full); return cost_per_block(stage, escape, escape.stats->blocks_with_escapes, struct7_full);
@@ -260,10 +275,15 @@ struct feature_benchmarker {
} }
// Rate of escape misses per escape flip // Rate of escape misses per escape flip
double escape_miss_rate(BenchmarkStage stage) const { double escape_miss_rate(BenchmarkStage stage) const {
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
return 1;
#else
if (!has_events()) { return 1; } if (!has_events()) { return 1; }
return double(escape_miss[stage].best.branch_misses() - escape[stage].best.branch_misses()) / double(escape_miss.stats->blocks_with_escapes_flipped); return double(escape_miss[stage].best.branch_misses() - escape[stage].best.branch_misses()) / double(escape_miss.stats->blocks_with_escapes_flipped);
#endif
} }
double calc_expected_feature_cost(BenchmarkStage stage, const benchmarker& file) const { double calc_expected_feature_cost(BenchmarkStage stage, const benchmarker& file) const {
// Expected base ns/block (empty) // Expected base ns/block (empty)
json_stats& stats = *file.stats; json_stats& stats = *file.stats;
@@ -300,7 +320,6 @@ struct feature_benchmarker {
double calc_expected(BenchmarkStage stage, const benchmarker& file) const { double calc_expected(BenchmarkStage stage, const benchmarker& file) const {
return calc_expected_feature_cost(stage, file) + calc_expected_miss_cost(stage, file); return calc_expected_feature_cost(stage, file) + calc_expected_miss_cost(stage, file);
} }
void print(const option_struct& options) const { void print(const option_struct& options) const {
printf("\n"); printf("\n");
printf("Features in ns/block (64 bytes):\n"); printf("Features in ns/block (64 bytes):\n");
@@ -359,6 +378,22 @@ struct feature_benchmarker {
} }
}; };
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
void print_file_effectiveness(BenchmarkStage stage, const char* filename, const benchmarker& results, const feature_benchmarker& features) {
double actual = results[stage].best.elapsed_ns() / double(results.stats->blocks);
double calc = features.calc_expected(stage, results);
double calc_misses = features.calc_expected_misses(stage, results);
double calc_miss_cost = features.calc_expected_miss_cost(stage, results);
printf(" | %-8s ", benchmark_stage_name(stage));
printf("| %-15s ", filename);
printf("| %8.3g ", features.calc_expected_feature_cost(stage, results));
printf("| %8.3g ", calc_miss_cost);
printf("| %8.3g ", calc);
printf("| %8.3g ", actual);
printf("| %+8.3g ", actual - calc);
printf("| %13llu ", (long long unsigned)(calc_misses));
}
#else
void print_file_effectiveness(BenchmarkStage stage, const char* filename, const benchmarker& results, const feature_benchmarker& features) { void print_file_effectiveness(BenchmarkStage stage, const char* filename, const benchmarker& results, const feature_benchmarker& features) {
double actual = results[stage].best.elapsed_ns() / double(results.stats->blocks); double actual = results[stage].best.elapsed_ns() / double(results.stats->blocks);
double calc = features.calc_expected(stage, results); double calc = features.calc_expected(stage, results);
@@ -382,6 +417,7 @@ void print_file_effectiveness(BenchmarkStage stage, const char* filename, const
} }
printf("|\n"); printf("|\n");
} }
#endif
int main(int argc, char *argv[]) { int main(int argc, char *argv[]) {
// Read options // Read options
+4 -3
View File
@@ -308,7 +308,7 @@ struct benchmarker {
return all_stages_without_allocation.iterations; return all_stages_without_allocation.iterations;
} }
simdjson_really_inline void run_iteration(bool stage1_only, bool hotbuffers=false) { simdjson_inline void run_iteration(bool stage1_only, bool hotbuffers=false) {
// Allocate dom::parser // Allocate dom::parser
collector.start(); collector.start();
dom::parser parser; dom::parser parser;
@@ -384,7 +384,7 @@ struct benchmarker {
loop << all_loop_count; loop << all_loop_count;
} }
simdjson_really_inline void run_iterations(size_t iterations, bool stage1_only, bool hotbuffers=false) { simdjson_inline void run_iterations(size_t iterations, bool stage1_only, bool hotbuffers=false) {
for (size_t i = 0; i<iterations; i++) { for (size_t i = 0; i<iterations; i++) {
run_iteration(stage1_only, hotbuffers); run_iteration(stage1_only, hotbuffers);
} }
@@ -423,7 +423,7 @@ struct benchmarker {
stage.instructions() / static_cast<double>(stats->structurals), stage.instructions() / static_cast<double>(stats->structurals),
stage.instructions() / static_cast<double>(stage.cycles()) stage.instructions() / static_cast<double>(stage.cycles())
); );
#if !SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
// NOTE: removed cycles/miss because it is a somewhat misleading stat // NOTE: removed cycles/miss because it is a somewhat misleading stat
printf("%s%-13s: %7.0f branch misses (%6.2f%%) - %.0f cache misses (%6.2f%%) - %.2f cache references\n", printf("%s%-13s: %7.0f branch misses (%6.2f%%) - %.0f cache misses (%6.2f%%) - %.2f cache references\n",
prefix, prefix,
@@ -434,6 +434,7 @@ struct benchmarker {
percent(stage.cache_misses(), all_stages_without_allocation.cache_misses()), percent(stage.cache_misses(), all_stages_without_allocation.cache_misses()),
stage.cache_references() stage.cache_references()
); );
#endif
} }
} }
@@ -46,7 +46,7 @@ struct runner : public file_runner<I> {
struct simdjson_dom; struct simdjson_dom;
template<typename I> simdjson_really_inline static void distinct_user_id(benchmark::State &state) { template<typename I> simdjson_inline static void distinct_user_id(benchmark::State &state) {
run_json_benchmark<runner<I>, runner<simdjson_dom>>(state); run_json_benchmark<runner<I>, runner<simdjson_dom>>(state);
} }
+3 -2
View File
@@ -46,13 +46,14 @@ struct rapidjson : rapidjson_base {
}; };
BENCHMARK_TEMPLATE(distinct_user_id, rapidjson)->UseManualTime(); BENCHMARK_TEMPLATE(distinct_user_id, rapidjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct rapidjson_insitu : rapidjson_base { struct rapidjson_insitu : rapidjson_base {
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) { bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag>(json.data()), result); return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag|kParseInsituFlag>(json.data()), result);
} }
}; };
BENCHMARK_TEMPLATE(distinct_user_id, rapidjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(distinct_user_id, rapidjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace partial_tweets } // namespace partial_tweets
#endif // SIMDJSON_COMPETITION_RAPIDJSON #endif // SIMDJSON_COMPETITION_RAPIDJSON
+4 -2
View File
@@ -9,12 +9,14 @@ namespace distinct_user_id {
struct sajson { struct sajson {
size_t ast_buffer_size{0}; size_t ast_buffer_size{0};
size_t *ast_buffer{nullptr}; size_t *ast_buffer{nullptr};
simdjson_really_inline std::string_view get_string_view(const ::sajson::value &obj, std::string_view key) { ~sajson() { free(ast_buffer); }
simdjson_inline std::string_view get_string_view(const ::sajson::value &obj, std::string_view key) {
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
if (val.get_type() != ::sajson::TYPE_STRING) { throw "field is not a string"; } if (val.get_type() != ::sajson::TYPE_STRING) { throw "field is not a string"; }
return { val.as_cstring(), val.get_string_length() }; return { val.as_cstring(), val.get_string_length() };
} }
simdjson_really_inline uint64_t get_str_uint64(const ::sajson::value &obj, std::string_view key) { simdjson_inline uint64_t get_str_uint64(const ::sajson::value &obj, std::string_view key) {
// Since sajson only supports 53-bit numbers, and IDs in twitter.json can be > 53 bits, we read the corresponding id_str and parse that. // Since sajson only supports 53-bit numbers, and IDs in twitter.json can be > 53 bits, we read the corresponding id_str and parse that.
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
if (val.get_type() != ::sajson::TYPE_STRING) { throw "field not a string"; } if (val.get_type() != ::sajson::TYPE_STRING) { throw "field not a string"; }
+2 -1
View File
@@ -49,13 +49,14 @@ struct yyjson : yyjson_base {
}; };
BENCHMARK_TEMPLATE(distinct_user_id, yyjson)->UseManualTime(); BENCHMARK_TEMPLATE(distinct_user_id, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson_base { struct yyjson_insitu : yyjson_base {
bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) { bool run(simdjson::padded_string &json, std::vector<uint64_t> &result) {
return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result); return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result);
} }
}; };
BENCHMARK_TEMPLATE(distinct_user_id, yyjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(distinct_user_id, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace distinct_user_id } // namespace distinct_user_id
#endif // SIMDJSON_COMPETITION_YYJSON #endif // SIMDJSON_COMPETITION_YYJSON
+1 -1
View File
@@ -41,7 +41,7 @@ using stat_t = struct stat_s;
simdjson_really_inline void simdjson_process_atom(stat_t &s, simdjson_inline void simdjson_process_atom(stat_t &s,
simdjson::dom::element element) { simdjson::dom::element element) {
if (element.is<int64_t>()) { if (element.is<int64_t>()) {
s.integer_count++; s.integer_count++;
+27 -4
View File
@@ -1,6 +1,15 @@
#ifndef __EVENT_COUNTER_H #ifndef __EVENT_COUNTER_H
#define __EVENT_COUNTER_H #define __EVENT_COUNTER_H
#ifndef SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
#ifdef __aarch64__
// on ARM, we use just cycles and instructions
#define SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS 1
#else
// elsewhere, we try to use four counters.
#define SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS 0
#endif
#endif
#include <cassert> #include <cassert>
#include <cctype> #include <cctype>
#ifndef _MSC_VER #ifndef _MSC_VER
@@ -46,6 +55,12 @@ struct event_count {
event_count(const event_count& other): elapsed(other.elapsed), event_counts(other.event_counts) { } event_count(const event_count& other): elapsed(other.elapsed), event_counts(other.event_counts) { }
// The types of counters (so we can read the getter more easily) // The types of counters (so we can read the getter more easily)
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
enum event_counter_types {
CPU_CYCLES,
INSTRUCTIONS
};
#else
enum event_counter_types { enum event_counter_types {
CPU_CYCLES, CPU_CYCLES,
INSTRUCTIONS, INSTRUCTIONS,
@@ -53,15 +68,16 @@ struct event_count {
CACHE_REFERENCES, CACHE_REFERENCES,
CACHE_MISSES CACHE_MISSES
}; };
#endif
double elapsed_sec() const { return duration<double>(elapsed).count(); } double elapsed_sec() const { return duration<double>(elapsed).count(); }
double elapsed_ns() const { return duration<double, std::nano>(elapsed).count(); } double elapsed_ns() const { return duration<double, std::nano>(elapsed).count(); }
double cycles() const { return static_cast<double>(event_counts[CPU_CYCLES]); } double cycles() const { return static_cast<double>(event_counts[CPU_CYCLES]); }
double instructions() const { return static_cast<double>(event_counts[INSTRUCTIONS]); } double instructions() const { return static_cast<double>(event_counts[INSTRUCTIONS]); }
#if !SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
double branch_misses() const { return static_cast<double>(event_counts[BRANCH_MISSES]); } double branch_misses() const { return static_cast<double>(event_counts[BRANCH_MISSES]); }
double cache_references() const { return static_cast<double>(event_counts[CACHE_REFERENCES]); } double cache_references() const { return static_cast<double>(event_counts[CACHE_REFERENCES]); }
double cache_misses() const { return static_cast<double>(event_counts[CACHE_MISSES]); } double cache_misses() const { return static_cast<double>(event_counts[CACHE_MISSES]); }
#endif
event_count& operator=(const event_count& other) { event_count& operator=(const event_count& other) {
this->elapsed = other.elapsed; this->elapsed = other.elapsed;
this->event_counts = other.event_counts; this->event_counts = other.event_counts;
@@ -105,9 +121,11 @@ struct event_aggregate {
double elapsed_ns() const { return total.elapsed_ns() / iterations; } double elapsed_ns() const { return total.elapsed_ns() / iterations; }
double cycles() const { return total.cycles() / iterations; } double cycles() const { return total.cycles() / iterations; }
double instructions() const { return total.instructions() / iterations; } double instructions() const { return total.instructions() / iterations; }
#if !SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
double branch_misses() const { return total.branch_misses() / iterations; } double branch_misses() const { return total.branch_misses() / iterations; }
double cache_references() const { return total.cache_references() / iterations; } double cache_references() const { return total.cache_references() / iterations; }
double cache_misses() const { return total.cache_misses() / iterations; } double cache_misses() const { return total.cache_misses() / iterations; }
#endif
}; };
struct event_collector { struct event_collector {
@@ -117,11 +135,16 @@ struct event_collector {
#if defined(__linux__) #if defined(__linux__)
LinuxEvents<PERF_TYPE_HARDWARE> linux_events; LinuxEvents<PERF_TYPE_HARDWARE> linux_events;
event_collector(bool quiet = false) : linux_events(vector<int>{ event_collector(bool quiet = false) : linux_events(vector<int>{
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
PERF_COUNT_HW_CPU_CYCLES,
PERF_COUNT_HW_INSTRUCTIONS,
#else
PERF_COUNT_HW_CPU_CYCLES, PERF_COUNT_HW_CPU_CYCLES,
PERF_COUNT_HW_INSTRUCTIONS, PERF_COUNT_HW_INSTRUCTIONS,
PERF_COUNT_HW_BRANCH_MISSES, PERF_COUNT_HW_BRANCH_MISSES,
PERF_COUNT_HW_CACHE_REFERENCES, PERF_COUNT_HW_CACHE_REFERENCES,
PERF_COUNT_HW_CACHE_MISSES PERF_COUNT_HW_CACHE_MISSES
#endif
}, quiet) {} }, quiet) {}
bool has_events() { bool has_events() {
return linux_events.is_working(); return linux_events.is_working();
@@ -133,13 +156,13 @@ struct event_collector {
} }
#endif #endif
simdjson_really_inline void start() { simdjson_inline void start() {
#if defined(__linux) #if defined(__linux)
linux_events.start(); linux_events.start();
#endif #endif
start_clock = steady_clock::now(); start_clock = steady_clock::now();
} }
simdjson_really_inline event_count& end() { simdjson_inline event_count& end() {
time_point<steady_clock> end_clock = steady_clock::now(); time_point<steady_clock> end_clock = steady_clock::now();
#if defined(__linux) #if defined(__linux)
linux_events.end(count.event_counts); linux_events.end(count.event_counts);
+1 -1
View File
@@ -33,7 +33,7 @@ struct runner : public file_runner<I> {
struct simdjson_dom; struct simdjson_dom;
template<typename I> simdjson_really_inline static void find_tweet(benchmark::State &state) { template<typename I> simdjson_inline static void find_tweet(benchmark::State &state) {
run_json_benchmark<runner<I>, runner<simdjson_dom>>(state); run_json_benchmark<runner<I>, runner<simdjson_dom>>(state);
} }
+3 -2
View File
@@ -40,13 +40,14 @@ struct rapidjson : rapidjson_base {
}; };
BENCHMARK_TEMPLATE(find_tweet, rapidjson)->UseManualTime(); BENCHMARK_TEMPLATE(find_tweet, rapidjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct rapidjson_insitu : rapidjson_base { struct rapidjson_insitu : rapidjson_base {
bool run(simdjson::padded_string &json, uint64_t find_id, std::string_view &result) { bool run(simdjson::padded_string &json, uint64_t find_id, std::string_view &result) {
return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag>(json.data()), find_id, result); return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag|kParseInsituFlag>(json.data()), find_id, result);
} }
}; };
BENCHMARK_TEMPLATE(find_tweet, rapidjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(find_tweet, rapidjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace find_tweet } // namespace find_tweet
#endif // SIMDJSON_COMPETITION_RAPIDJSON #endif // SIMDJSON_COMPETITION_RAPIDJSON
+4 -2
View File
@@ -11,12 +11,14 @@ struct sajson {
size_t ast_buffer_size{0}; size_t ast_buffer_size{0};
size_t *ast_buffer{nullptr}; size_t *ast_buffer{nullptr};
simdjson_really_inline std::string_view get_string_view(const ::sajson::value &obj, std::string_view key) { ~sajson() { free(ast_buffer); }
simdjson_inline std::string_view get_string_view(const ::sajson::value &obj, std::string_view key) {
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
if (val.get_type() != ::sajson::TYPE_STRING) { throw "field is not a string"; } if (val.get_type() != ::sajson::TYPE_STRING) { throw "field is not a string"; }
return { val.as_cstring(), val.get_string_length() }; return { val.as_cstring(), val.get_string_length() };
} }
simdjson_really_inline uint64_t get_str_uint64(const ::sajson::value &obj, std::string_view key) { simdjson_inline uint64_t get_str_uint64(const ::sajson::value &obj, std::string_view key) {
// Since sajson only supports 53-bit numbers, and IDs in twitter.json can be > 53 bits, we read the corresponding id_str and parse that. // Since sajson only supports 53-bit numbers, and IDs in twitter.json can be > 53 bits, we read the corresponding id_str and parse that.
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
if (val.get_type() != ::sajson::TYPE_STRING) { throw "field not a string"; } if (val.get_type() != ::sajson::TYPE_STRING) { throw "field not a string"; }
+2 -2
View File
@@ -40,14 +40,14 @@ struct yyjson : yyjson_base {
} }
}; };
BENCHMARK_TEMPLATE(find_tweet, yyjson)->UseManualTime(); BENCHMARK_TEMPLATE(find_tweet, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson_base { struct yyjson_insitu : yyjson_base {
bool run(simdjson::padded_string &json, uint64_t find_id, std::string_view &result) { bool run(simdjson::padded_string &json, uint64_t find_id, std::string_view &result) {
return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), find_id, result); return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), find_id, result);
} }
}; };
BENCHMARK_TEMPLATE(find_tweet, yyjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(find_tweet, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace find_tweet } // namespace find_tweet
#endif // SIMDJSON_COMPETITION_YYJSON #endif // SIMDJSON_COMPETITION_YYJSON
+48
View File
@@ -0,0 +1,48 @@
#pragma once
#include "json_benchmark/file_runner.h"
namespace json2msgpack {
using namespace json_benchmark;
template <typename I> struct runner : public file_runner<I> {
std::string_view result;
std::unique_ptr<char[]> buffer;
bool setup(benchmark::State &state) {
bool isok = this->load_json(state, TWITTER_JSON);
if (isok) {
// Let us allocate a sizeable buffer.
buffer = std::unique_ptr<char[]>(new char[this->json.size() * 4 + 1024]);
}
return isok;
}
bool before_run(benchmark::State &state) {
if (!file_runner<I>::before_run(state)) {
return false;
}
// Clear the buffer.
::memset(buffer.get(), 0, this->json.size() * 4 + 1024);
return true;
}
bool run(benchmark::State &) {
return this->implementation.run(this->json, buffer.get(), result);
}
template <typename R>
bool diff(benchmark::State &state, runner<R> &reference) {
return diff_results(state, result.size(), reference.result.size(), diff_flags::NONE);
}
};
struct simdjson_ondemand;
template <typename I>
simdjson_inline static void json2msgpack(benchmark::State &state) {
run_json_benchmark<runner<I>, runner<simdjson_ondemand>>(state);
}
} // namespace json2msgpack
+117
View File
@@ -0,0 +1,117 @@
#pragma once
#ifdef SIMDJSON_COMPETITION_NLOHMANN_JSON
#include "json2msgpack.h"
namespace json2msgpack {
using namespace nlohmann;
struct nlohmann_json2msgpack {
inline std::string_view to_msgpack(const simdjson::padded_string &json,
uint8_t *buf);
private:
inline void write_double(const double d) noexcept;
inline void write_byte(const uint8_t b) noexcept;
inline void write_uint32(const uint32_t w) noexcept;
inline void write_string(const std::string& str);
inline void recursive_processor(basic_json<> element);
uint8_t *buff{};
};
std::string_view nlohmann_json2msgpack::to_msgpack(const simdjson::padded_string &json,
uint8_t *buf) {
buff = buf;
auto val = nlohmann::json::parse(json.data(), json.data() + json.size());
recursive_processor(val);
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void nlohmann_json2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void nlohmann_json2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void nlohmann_json2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
void nlohmann_json2msgpack::write_string(const std::string & str) {
write_byte(0xdb);
write_uint32(uint32_t(str.size()));
::memcpy(buff, str.data(), str.size());
buff += str.size();
}
void nlohmann_json2msgpack::recursive_processor(json element) {
switch (element.type()) {
case nlohmann::detail::value_t::array: {
uint32_t counter = 0;
write_byte(0xdd);
std::vector<json> array = element.get<std::vector<json>>();
write_uint32(uint32_t(array.size()));
for (auto child : array) {
recursive_processor(child);
}
} break;
case nlohmann::detail::value_t::object: {
write_byte(0xdf);
std::map<std::string,json> object = element.get<std::map<std::string,json>>();
write_uint32(uint32_t(object.size()));
for (auto field : object) {
write_string(field.first);
recursive_processor(field.second);
}
} break;
case nlohmann::detail::value_t::number_integer:
case nlohmann::detail::value_t::number_unsigned:
case nlohmann::detail::value_t::number_float:
write_double(double(element));
break;
case nlohmann::detail::value_t::string:
write_string(std::string(element));
break;
case nlohmann::detail::value_t::boolean:
write_byte(0xc2 + bool(element));
break;
case nlohmann::detail::value_t::null:
write_byte(0xc0);
break;
case nlohmann::detail::value_t::discarded:
case nlohmann::detail::value_t::binary:
default:
printf("unexpected\n");
break;
}
}
struct nlohmann_json {
using StringType = std::string_view;
nlohmann_json2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
result = parser.to_msgpack(json, reinterpret_cast<uint8_t *>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, nlohmann_json)->UseManualTime();
} // namespace json2msgpack
#endif // SIMDJSON_COMPETITION_NLOHMANN_JSON
+142
View File
@@ -0,0 +1,142 @@
#pragma once
#ifdef SIMDJSON_COMPETITION_RAPIDJSON
#include "json2msgpack.h"
namespace json2msgpack {
using namespace rapidjson;
template <int parseflag>
struct rapidjson2msgpack {
inline std::string_view to_msgpack(char *json, uint8_t *buf);
private:
inline void write_double(const double d) noexcept;
inline void write_byte(const uint8_t b) noexcept;
inline void write_uint32(const uint32_t w) noexcept;
inline void write_uint32_at(const uint32_t w, uint8_t *p) noexcept;
void write_string(const char * s, size_t length) noexcept;
inline void recursive_processor(Value &v);
uint8_t *buff{};
};
template <int parseflag>
std::string_view rapidjson2msgpack<parseflag>::to_msgpack(char *json, uint8_t *buf) {
buff = buf;
Document doc{};
if(parseflag & kParseInsituFlag) {
doc.ParseInsitu<parseflag>(json);
} else {
doc.Parse<parseflag>(json);
}
recursive_processor(doc);
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
template <int parseflag>
void rapidjson2msgpack<parseflag>::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
template <int parseflag>
void rapidjson2msgpack<parseflag>::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
template <int parseflag>
void rapidjson2msgpack<parseflag>::write_string(const char * c, size_t len) noexcept {
write_byte(0xdb);
write_uint32(uint32_t(len));
::memcpy(buff, c, len);
buff += len;
}
template <int parseflag>
void rapidjson2msgpack<parseflag>::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
template <int parseflag>
void rapidjson2msgpack<parseflag>::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
::memcpy(p, &w, sizeof(w));
}
template <int parseflag>
void rapidjson2msgpack<parseflag>::recursive_processor(Value &v) {
switch (v.GetType()) {
case kArrayType:
write_byte(0xdd);
write_uint32(v.Size());
for (Value::ValueIterator i = v.Begin(); i != v.End(); ++i) {
recursive_processor(*i);
}
break;
case kObjectType:
write_byte(0xdf);
write_uint32(uint32_t(v.MemberEnd()-v.MemberBegin()));
for (Value::MemberIterator m = v.MemberBegin(); m != v.MemberEnd();
++m) {
write_string(m->name.GetString(), m->name.GetStringLength());
recursive_processor(m->value);
}
break;
case kStringType:
write_string(v.GetString(), v.GetStringLength());
break;
case kNumberType:
write_double(v.GetDouble());
break;
case kFalseType:
write_byte(0xc2);
break;
case kTrueType:
write_byte(0xc3);
break;
case kNullType:
write_byte(0xc0);
break;
}
}
template <int parseflag>
struct rapidjson_base {
using StringType = std::string_view;
rapidjson2msgpack<parseflag> parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
result =
parser.to_msgpack(json.data(), reinterpret_cast<uint8_t *>(buffer));
return true;
}
};
using rapidjson = rapidjson_base<kParseValidateEncodingFlag|kParseFullPrecisionFlag>;
BENCHMARK_TEMPLATE(json2msgpack, rapidjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_APPROX
using rapidjson_approx = rapidjson_base<kParseValidateEncodingFlag>;
BENCHMARK_TEMPLATE(json2msgpack, rapidjson_approx)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_APPROX
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
using rapidjson_insitu = rapidjson_base<kParseValidateEncodingFlag|kParseInsituFlag>;
BENCHMARK_TEMPLATE(json2msgpack, rapidjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace json2msgpack
#endif // SIMDJSON_COMPETITION_RAPIDJSON
+131
View File
@@ -0,0 +1,131 @@
#pragma once
#ifdef SIMDJSON_COMPETITION_SAJSON
#include "json2msgpack.h"
namespace json2msgpack {
using namespace sajson;
struct sajson2msgpack {
inline std::string_view to_msgpack(char *json, size_t size, uint8_t *buf);
virtual ~sajson2msgpack() { free(ast_buffer); }
private:
inline void write_double(const double d) noexcept;
inline void write_byte(const uint8_t b) noexcept;
inline void write_uint32(const uint32_t w) noexcept;
inline void write_string(const char * s, size_t length) noexcept;
inline void recursive_processor(const sajson::value &v);
uint8_t *buff{};
size_t ast_buffer_size{0};
size_t *ast_buffer{nullptr};
};
std::string_view sajson2msgpack::to_msgpack(char *json, size_t size, uint8_t *buf) {
buff = buf;
if (!ast_buffer) {
ast_buffer_size = size;
ast_buffer = (size_t *)std::malloc(ast_buffer_size * sizeof(size_t));
}
auto doc = parse(
bounded_allocation(ast_buffer, ast_buffer_size),
mutable_string_view(size, json)
);
auto root = doc.get_root();
recursive_processor(root);
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void sajson2msgpack::write_string(const char * c, size_t len) noexcept {
write_byte(0xdb);
write_uint32(uint32_t(len));
::memcpy(buff, c, len);
buff += len;
}
void sajson2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void sajson2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void sajson2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
void sajson2msgpack::recursive_processor(const sajson::value &node) {
using namespace sajson;
switch (node.get_type()) {
case TYPE_NULL:
write_byte(0xc0);
break;
case TYPE_FALSE:
write_byte(0xc2);
break;
case TYPE_TRUE:
write_byte(0xc3);
break;
case TYPE_ARRAY: {
auto length = node.get_length();
write_byte(0xdf);
write_uint32(uint32_t(length));
for (size_t i = 0; i < length; ++i) {
recursive_processor(node.get_array_element(i));
}
break;
}
case TYPE_OBJECT: {
auto length = node.get_length();
write_byte(0xdd);
write_uint32(uint32_t(length));
for (auto i = 0u; i < length; ++i) {
auto s = node.get_object_key(i);
write_string(s.data(), s.length());
recursive_processor(node.get_object_value(i));
}
break;
}
case TYPE_STRING:
write_string(node.as_cstring(), node.get_string_length());
break;
case TYPE_DOUBLE:
case TYPE_INTEGER:
write_double(node.get_number_value());
break;
default:
assert(false && "unknown node type");
}
}
struct sajson {
using StringType = std::string_view;
sajson2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
result =
parser.to_msgpack(json.data(), json.size(), reinterpret_cast<uint8_t *>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, sajson)->UseManualTime();
} // namespace json2msgpack
#endif // SIMDJSON_COMPETITION_SAJSON
+138
View File
@@ -0,0 +1,138 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "json2msgpack.h"
namespace json2msgpack {
using namespace simdjson;
struct simdjsondom2msgpack {
/**
* @brief Converts the provided JSON into msgpack.
*
* @param json JSON input
* @param buf temporary buffer (must be large enough, with simdjson::SIMDJSON_PADDING bytes
* of padding)
* @return std::string_view msgpack output, writting to the temporary buffer
*/
inline std::string_view to_msgpack(const simdjson::padded_string &json,
uint8_t *buf);
private:
simdjson_really_inline void write_double(const double d) noexcept;
simdjson_really_inline void write_string(const std::string_view v) noexcept;
simdjson_really_inline void write_byte(const uint8_t b) noexcept;
simdjson_really_inline void write_uint32(const uint32_t w) noexcept;
simdjson_really_inline uint8_t *skip_uint32() noexcept;
simdjson_really_inline void write_uint32_at(const uint32_t w,
uint8_t *p) noexcept;
inline void recursive_processor(simdjson::dom::element element);
dom::parser parser;
uint8_t *buff{};
};
std::string_view
simdjsondom2msgpack::to_msgpack(const simdjson::padded_string &json,
uint8_t *buf) {
buff = buf;
recursive_processor(parser.parse(json));
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void simdjsondom2msgpack::write_string(const std::string_view v) noexcept {
write_byte(0xdb);
write_uint32(uint32_t(v.size()));
::memcpy(buff, v.data(), v.size());
buff += v.size();
}
void simdjsondom2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void simdjsondom2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void simdjsondom2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
uint8_t *simdjsondom2msgpack::skip_uint32() noexcept {
uint8_t *ret = buff;
buff += sizeof(uint32_t);
return ret;
}
void simdjsondom2msgpack::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
::memcpy(p, &w, sizeof(w));
}
void simdjsondom2msgpack::recursive_processor(simdjson::dom::element element) {
switch (element.type()) {
case dom::element_type::ARRAY: {
uint32_t counter = 0;
write_byte(0xdd);
uint8_t *location = skip_uint32();
for (auto child : dom::array(element)) {
counter++;
recursive_processor(child);
}
write_uint32_at(counter, location);}
break;
case dom::element_type::OBJECT:{
uint32_t counter = 0;
write_byte(0xdf);
uint8_t *location = skip_uint32();
for (dom::key_value_pair field : dom::object(element)) {
counter++;
write_string(field.key);
recursive_processor(field.value);
}
write_uint32_at(counter, location);
}
break;
case dom::element_type::INT64:
case dom::element_type::UINT64:
case dom::element_type::DOUBLE:
write_double( double(element));
break;
case dom::element_type::STRING:
write_string(std::string_view(element));
break;
case dom::element_type::BOOL:
write_byte(0xc2 + bool(element));
break;
case dom::element_type::NULL_VALUE:
write_byte(0xc0);
break;
default:
break;
}
}
struct simdjson_dom {
using StringType = std::string_view;
simdjsondom2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
result = parser.to_msgpack(json, reinterpret_cast<uint8_t *>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, simdjson_dom)->UseManualTime();
} // namespace json2msgpack
#endif // SIMDJSON_EXCEPTIONS
+182
View File
@@ -0,0 +1,182 @@
#pragma once
#if SIMDJSON_EXCEPTIONS
#include "json2msgpack.h"
namespace json2msgpack {
using namespace simdjson;
/**
* @brief The simdjson2msgpack struct is used to quickly convert
* JSON strings to msgpack views. You must provide a pointer to
* a large memory region where the msgpack gets written. The
* buffer should be large enough to store the msgpack output (which
* can never be 3x larger than the input JSON) with an additional
* simdjson::SIMDJSON_PADDING bytes.
*
* Recommended usage:
*
* simdjson2msgpack parser{};
* simdjson::padded_string json = "[1,2]"_padded; // some JSON
* uint8_t * buffer = new uint8_t[3*json.size() + simdjson::SIMDJSON_PADDING]; // large buffer
*
* std::string_view msgpack = parser.to_msgpack(json, buffer);
*
* The result (msgpack) is a string view to a msgpack serialization of the input JSON,
* it points inside the buffer you provided.
*
* You may reuse the simdjson2msgpack instance though you should use
* one per thread.
*/
struct simdjson2msgpack {
/**
* @brief Converts the provided JSON into msgpack.
*
* @param json JSON input
* @param buf temporary buffer (must be large enough, with simdjson::SIMDJSON_PADDING bytes
* of padding)
* @return std::string_view msgpack output, writting to the temporary buffer
*/
inline std::string_view to_msgpack(const simdjson::padded_string &json,
uint8_t *buf);
private:
simdjson_inline void write_double(const double d) noexcept;
simdjson_inline void write_byte(const uint8_t b) noexcept;
simdjson_inline void write_uint32(const uint32_t w) noexcept;
simdjson_inline uint8_t *skip_uint32() noexcept;
simdjson_inline void write_uint32_at(const uint32_t w,
uint8_t *p) noexcept;
simdjson_inline void
write_raw_string(simdjson::ondemand::raw_json_string rjs);
inline void recursive_processor(simdjson::ondemand::value element);
simdjson::ondemand::parser parser;
uint8_t *buff{};
};
std::string_view
simdjson2msgpack::to_msgpack(const simdjson::padded_string &json,
uint8_t *buf) {
buff = buf;
ondemand::document doc = parser.iterate(json);
if (doc.is_scalar()) {
// we have a special case where the JSON document is a single document...
switch (doc.type()) {
case simdjson::ondemand::json_type::number:
write_double(doc.get_double());
break;
case simdjson::ondemand::json_type::string:
write_raw_string(doc.get_raw_json_string());
break;
case simdjson::ondemand::json_type::boolean:
write_byte(0xc2 + doc.get_bool());
break;
case simdjson::ondemand::json_type::null:
write_byte(0xc0);
break;
case simdjson::ondemand::json_type::array:
case simdjson::ondemand::json_type::object:
default:
// impossible
break;
}
} else {
simdjson::ondemand::value val = doc;
recursive_processor(val);
}
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void simdjson2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void simdjson2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void simdjson2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
uint8_t *simdjson2msgpack::skip_uint32() noexcept {
uint8_t *ret = buff;
buff += sizeof(uint32_t);
return ret;
}
void simdjson2msgpack::write_uint32_at(const uint32_t w, uint8_t *p) noexcept {
::memcpy(p, &w, sizeof(w));
}
void simdjson2msgpack::write_raw_string(
simdjson::ondemand::raw_json_string in) {
write_byte(0xdb);
uint8_t *location = skip_uint32();
std::string_view v = parser.unescape(in, buff);
write_uint32_at(uint32_t(v.size()), location);
}
void simdjson2msgpack::recursive_processor(simdjson::ondemand::value element) {
switch (element.type()) {
case simdjson::ondemand::json_type::array: {
uint32_t counter = 0;
write_byte(0xdd);
uint8_t *location = skip_uint32();
for (auto child : element.get_array()) {
counter++;
recursive_processor(child.value());
}
write_uint32_at(counter, location);
} break;
case simdjson::ondemand::json_type::object: {
uint32_t counter = 0;
write_byte(0xdf);
uint8_t *location = skip_uint32();
for (auto field : element.get_object()) {
counter++;
write_raw_string(field.key());
recursive_processor(field.value());
}
write_uint32_at(counter, location);
} break;
case simdjson::ondemand::json_type::number:
write_double(element.get_double());
break;
case simdjson::ondemand::json_type::string:
write_raw_string(element.get_raw_json_string());
break;
case simdjson::ondemand::json_type::boolean:
write_byte(0xc2 + element.get_bool());
break;
case simdjson::ondemand::json_type::null:
write_byte(0xc0);
break;
default:
SIMDJSON_UNREACHABLE();
}
}
struct simdjson_ondemand {
using StringType = std::string_view;
simdjson2msgpack parser{};
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
result = parser.to_msgpack(json, reinterpret_cast<uint8_t *>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, simdjson_ondemand)->UseManualTime();
} // namespace json2msgpack
#endif // SIMDJSON_EXCEPTIONS
+123
View File
@@ -0,0 +1,123 @@
#pragma once
#ifdef SIMDJSON_COMPETITION_YYJSON
#include "json2msgpack.h"
namespace json2msgpack {
struct yyjson2msgpack {
inline std::string_view to_msgpack(yyjson_doc *doc, uint8_t *buf);
private:
inline void write_double(const double d) noexcept;
inline void write_byte(const uint8_t b) noexcept;
inline void write_uint32(const uint32_t w) noexcept;
inline void write_string(const char *s, size_t length) noexcept;
inline void recursive_processor(yyjson_val *obj);
uint8_t *buff{};
};
std::string_view yyjson2msgpack::to_msgpack(yyjson_doc *doc, uint8_t *buf) {
buff = buf;
yyjson_val *root = yyjson_doc_get_root(doc);
recursive_processor(root);
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
}
void yyjson2msgpack::write_string(const char *c, size_t len) noexcept {
write_byte(0xdb);
write_uint32(uint32_t(len));
::memcpy(buff, c, len);
buff += len;
}
void yyjson2msgpack::write_double(const double d) noexcept {
*buff++ = 0xcb;
::memcpy(buff, &d, sizeof(d));
buff += sizeof(d);
}
void yyjson2msgpack::write_byte(const uint8_t b) noexcept {
*buff = b;
buff++;
}
void yyjson2msgpack::write_uint32(const uint32_t w) noexcept {
::memcpy(buff, &w, sizeof(w));
buff += sizeof(w);
}
void yyjson2msgpack::recursive_processor(yyjson_val *obj) {
size_t idx, max;
yyjson_val *val;
yyjson_val *key;
switch (yyjson_get_type(obj)) {
case YYJSON_TYPE_STR:
write_string(yyjson_get_str(obj), yyjson_get_len(obj));
break;
case YYJSON_TYPE_ARR:
write_byte(0xdf);
write_uint32(uint32_t(yyjson_arr_size(obj)));
yyjson_arr_foreach(obj, idx, max, val) { recursive_processor(val); }
break;
case YYJSON_TYPE_OBJ:
write_byte(0xdd);
write_uint32(uint32_t(yyjson_obj_size(obj)));
yyjson_obj_foreach(obj, idx, max, key, val) {
write_string(yyjson_get_str(key), yyjson_get_len(key));
recursive_processor(val);
}
break;
case YYJSON_TYPE_BOOL:
write_byte(0xc2 + yyjson_get_bool(obj));
break;
case YYJSON_TYPE_NULL:
write_byte(0xc0);
break;
case YYJSON_TYPE_NUM:
switch (yyjson_get_subtype(obj)) {
case YYJSON_SUBTYPE_UINT:
write_double(double(yyjson_get_uint(obj)));
break;
case YYJSON_SUBTYPE_SINT:
write_double(double(yyjson_get_sint(obj)));
break;
case YYJSON_SUBTYPE_REAL:
write_double(yyjson_get_real(obj));
break;
default:
SIMDJSON_UNREACHABLE();
}
break;
default:
SIMDJSON_UNREACHABLE();
}
}
struct yyjson : yyjson2msgpack {
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
yyjson_doc *doc = yyjson_read(json.data(), json.size(), 0);
result = to_msgpack(doc, reinterpret_cast<uint8_t*>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson2msgpack {
bool run(simdjson::padded_string &json, char *buffer,
std::string_view &result) {
yyjson_doc *doc =
yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0);
result = to_msgpack(doc, reinterpret_cast<uint8_t*>(buffer));
return true;
}
};
BENCHMARK_TEMPLATE(json2msgpack, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace json2msgpack
#endif // SIMDJSON_COMPETITION_YYJSON
@@ -58,10 +58,11 @@ template<typename B, typename R> static void run_json_benchmark(benchmark::State
if (collector.has_events()) { if (collector.has_events()) {
state.counters["instructions"] = events.instructions(); state.counters["instructions"] = events.instructions();
state.counters["cycles"] = events.cycles(); state.counters["cycles"] = events.cycles();
#if !SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
state.counters["branch_miss"] = events.branch_misses(); state.counters["branch_miss"] = events.branch_misses();
state.counters["cache_miss"] = events.cache_misses(); state.counters["cache_miss"] = events.cache_misses();
state.counters["cache_ref"] = events.cache_references(); state.counters["cache_ref"] = events.cache_references();
#endif
state.counters["instructions_per_byte"] = events.instructions() / double(bench.bytes_per_iteration()); state.counters["instructions_per_byte"] = events.instructions() / double(bench.bytes_per_iteration());
state.counters["instructions_per_cycle"] = events.instructions() / events.cycles(); state.counters["instructions_per_cycle"] = events.instructions() / events.cycles();
state.counters["cycles_per_byte"] = events.cycles() / double(bench.bytes_per_iteration()); state.counters["cycles_per_byte"] = events.cycles() / double(bench.bytes_per_iteration());
@@ -69,9 +70,11 @@ template<typename B, typename R> static void run_json_benchmark(benchmark::State
state.counters["best_instructions"] = events.best.instructions(); state.counters["best_instructions"] = events.best.instructions();
state.counters["best_cycles"] = events.best.cycles(); state.counters["best_cycles"] = events.best.cycles();
#if !SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
state.counters["best_branch_miss"] = events.best.branch_misses(); state.counters["best_branch_miss"] = events.best.branch_misses();
state.counters["best_cache_miss"] = events.best.cache_misses(); state.counters["best_cache_miss"] = events.best.cache_misses();
state.counters["best_cache_ref"] = events.best.cache_references(); state.counters["best_cache_ref"] = events.best.cache_references();
#endif
state.counters["best_instructions_per_byte"] = events.best.instructions() / double(bench.bytes_per_iteration()); state.counters["best_instructions_per_byte"] = events.best.instructions() / double(bench.bytes_per_iteration());
state.counters["best_instructions_per_cycle"] = events.best.instructions() / events.best.cycles(); state.counters["best_instructions_per_cycle"] = events.best.instructions() / events.best.cycles();
@@ -92,9 +95,11 @@ template<typename B, typename R> static void run_json_benchmark(benchmark::State
if (collector.has_events()) { if (collector.has_events()) {
label << " instructions=" << setw(12) << uint64_t(events.best.instructions()) << setw(0); label << " instructions=" << setw(12) << uint64_t(events.best.instructions()) << setw(0);
label << " cycles=" << setw(12) << uint64_t(events.best.cycles()) << setw(0); label << " cycles=" << setw(12) << uint64_t(events.best.cycles()) << setw(0);
#if !SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
label << " branch_miss=" << setw(8) << uint64_t(events.best.branch_misses()) << setw(0); label << " branch_miss=" << setw(8) << uint64_t(events.best.branch_misses()) << setw(0);
label << " cache_miss=" << setw(8) << uint64_t(events.best.cache_misses()) << setw(0); label << " cache_miss=" << setw(8) << uint64_t(events.best.cache_misses()) << setw(0);
label << " cache_ref=" << setw(10) << uint64_t(events.best.cache_references()) << setw(0); label << " cache_ref=" << setw(10) << uint64_t(events.best.cache_references()) << setw(0);
#endif
} }
label << " items=" << setw(10) << bench.items_per_iteration() << setw(0); label << " items=" << setw(10) << bench.items_per_iteration() << setw(0);
+1 -1
View File
@@ -79,7 +79,7 @@ static const simdjson::padded_string &get_built_json_array() {
struct simdjson_dom; struct simdjson_dom;
template<typename I> simdjson_really_inline static void kostya(benchmark::State &state) { template<typename I> simdjson_inline static void kostya(benchmark::State &state) {
run_json_benchmark<runner<I>, runner<simdjson_dom>>(state); run_json_benchmark<runner<I>, runner<simdjson_dom>>(state);
} }
+1 -1
View File
@@ -42,7 +42,7 @@ struct nlohmann_json_sax {
return true; return true;
} }
bool number_unsigned(number_unsigned_t val) override { // Need this event because coordinate value can be equal to 1 bool number_unsigned(number_unsigned_t val) override { // Need this event because coordinate value can be equal to 1
buffer[k] = val; buffer[k] = double(val);
if (k == 2) { if (k == 2) {
result.emplace_back(json_benchmark::point{buffer[0],buffer[1],buffer[2]}); result.emplace_back(json_benchmark::point{buffer[0],buffer[1],buffer[2]});
k = 0; k = 0;
+10 -9
View File
@@ -13,7 +13,7 @@ struct rapidjson_base {
Document doc; Document doc;
simdjson_really_inline double get_double(Value &object, std::string_view key) { simdjson_inline double get_double(Value &object, std::string_view key) {
auto field = object.FindMember(key.data()); auto field = object.FindMember(key.data());
if (field == object.MemberEnd()) { throw "Missing double field"; } if (field == object.MemberEnd()) { throw "Missing double field"; }
if (!field->value.IsNumber()) { throw "Field is not double"; } if (!field->value.IsNumber()) { throw "Field is not double"; }
@@ -34,28 +34,29 @@ struct rapidjson_base {
return true; return true;
} }
}; };
#if SIMDJSON_COMPETITION_ONDEMAND_APPROX
struct rapidjson : rapidjson_base { struct rapidjson_approx : rapidjson_base {
bool run(simdjson::padded_string &json, std::vector<point> &result) { bool run(simdjson::padded_string &json, std::vector<point> &result) {
return rapidjson_base::run(doc.Parse<kParseValidateEncodingFlag>(json.data()), result); return rapidjson_base::run(doc.Parse<kParseValidateEncodingFlag>(json.data()), result);
} }
}; };
BENCHMARK_TEMPLATE(kostya, rapidjson)->UseManualTime(); BENCHMARK_TEMPLATE(kostya, rapidjson_approx)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_APPROX
struct rapidjson_lossless : rapidjson_base { struct rapidjson : rapidjson_base {
bool run(simdjson::padded_string &json, std::vector<point> &result) { bool run(simdjson::padded_string &json, std::vector<point> &result) {
return rapidjson_base::run(doc.Parse<kParseValidateEncodingFlag | kParseFullPrecisionFlag>(json.data()), result); return rapidjson_base::run(doc.Parse<kParseValidateEncodingFlag | kParseFullPrecisionFlag>(json.data()), result);
} }
}; };
BENCHMARK_TEMPLATE(kostya, rapidjson_lossless)->UseManualTime(); BENCHMARK_TEMPLATE(kostya, rapidjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct rapidjson_insitu : rapidjson_base { struct rapidjson_insitu : rapidjson_base {
bool run(simdjson::padded_string &json, std::vector<point> &result) { bool run(simdjson::padded_string &json, std::vector<point> &result) {
return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag>(json.data()), result); return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag|kParseInsituFlag>(json.data()), result);
} }
}; };
BENCHMARK_TEMPLATE(kostya, rapidjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(kostya, rapidjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace kostya } // namespace kostya
#endif // SIMDJSON_COMPETITION_RAPIDJSON #endif // SIMDJSON_COMPETITION_RAPIDJSON
+2 -1
View File
@@ -11,8 +11,9 @@ struct sajson {
size_t ast_buffer_size{0}; size_t ast_buffer_size{0};
size_t *ast_buffer{nullptr}; size_t *ast_buffer{nullptr};
~sajson() { free(ast_buffer); }
simdjson_really_inline double get_double(const ::sajson::value &obj, std::string_view key) { simdjson_inline double get_double(const ::sajson::value &obj, std::string_view key) {
using namespace sajson; using namespace sajson;
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
+3 -3
View File
@@ -9,7 +9,7 @@ namespace kostya {
struct yyjson_base { struct yyjson_base {
static constexpr diff_flags DiffFlags = diff_flags::NONE; static constexpr diff_flags DiffFlags = diff_flags::NONE;
simdjson_really_inline double get_double(yyjson_val *obj, std::string_view key) { simdjson_inline double get_double(yyjson_val *obj, std::string_view key) {
yyjson_val *val = yyjson_obj_getn(obj, key.data(), key.length()); yyjson_val *val = yyjson_obj_getn(obj, key.data(), key.length());
if (!val) { throw "missing point field!"; } if (!val) { throw "missing point field!"; }
if (yyjson_get_type(val) != YYJSON_TYPE_NUM) { throw "Number is not a type!"; } if (yyjson_get_type(val) != YYJSON_TYPE_NUM) { throw "Number is not a type!"; }
@@ -53,14 +53,14 @@ struct yyjson : yyjson_base {
} }
}; };
BENCHMARK_TEMPLATE(kostya, yyjson)->UseManualTime(); BENCHMARK_TEMPLATE(kostya, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson_base { struct yyjson_insitu : yyjson_base {
bool run(simdjson::padded_string &json, std::vector<point> &result) { bool run(simdjson::padded_string &json, std::vector<point> &result) {
return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result); return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result);
} }
}; };
BENCHMARK_TEMPLATE(kostya, yyjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(kostya, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace kostya } // namespace kostya
#endif // SIMDJSON_COMPETITION_YYJSON #endif // SIMDJSON_COMPETITION_YYJSON
@@ -16,11 +16,11 @@ using namespace json_benchmark;
struct brand { struct brand {
double cumulative_rating; double cumulative_rating;
uint64_t reviews_count; uint64_t reviews_count;
simdjson_really_inline bool operator==(const brand &other) const { simdjson_inline bool operator==(const brand &other) const {
return cumulative_rating == other.cumulative_rating && return cumulative_rating == other.cumulative_rating &&
reviews_count == other.reviews_count; reviews_count == other.reviews_count;
} }
simdjson_really_inline bool operator!=(const brand &other) const { return !(*this == other); } simdjson_inline bool operator!=(const brand &other) const { return !(*this == other); }
}; };
simdjson_unused static std::ostream &operator<<(std::ostream &o, const brand &b) { simdjson_unused static std::ostream &operator<<(std::ostream &o, const brand &b) {
@@ -87,8 +87,8 @@ static const simdjson::padded_string &get_built_json() {
template<bool threaded> template<bool threaded>
struct simdjson_dom; struct simdjson_dom;
template<typename I> simdjson_really_inline static void large_amazon_cellphones(benchmark::State &state) { template<typename I> simdjson_inline static void large_amazon_cellphones(benchmark::State &state) {
run_json_benchmark<runner<I>, runner<simdjson_dom<UNTHREADED>>>(state); run_json_benchmark<runner<I>, runner<simdjson_dom<UNTHREADED>>>(state);
} }
} // namespace large_amazon_cellphones } // namespace large_amazon_cellphones
+1 -1
View File
@@ -34,7 +34,7 @@ struct nlohmann_json_sax {
return true; return true;
} }
bool number_unsigned(number_unsigned_t val) override { bool number_unsigned(number_unsigned_t val) override {
buffer[k] = val; buffer[k] = double(val);
if (k == 2) { if (k == 2) {
result.emplace_back(json_benchmark::point{buffer[0],buffer[1],buffer[2]}); result.emplace_back(json_benchmark::point{buffer[0],buffer[1],buffer[2]});
k = 0; k = 0;
+10 -8
View File
@@ -13,7 +13,7 @@ struct rapidjson_base {
Document doc; Document doc;
simdjson_really_inline double get_double(Value &object, std::string_view key) { simdjson_inline double get_double(Value &object, std::string_view key) {
auto field = object.FindMember(key.data()); auto field = object.FindMember(key.data());
if (field == object.MemberEnd()) { throw "Missing double field"; } if (field == object.MemberEnd()) { throw "Missing double field"; }
if (!field->value.IsNumber()) { throw "Field is not double"; } if (!field->value.IsNumber()) { throw "Field is not double"; }
@@ -31,28 +31,30 @@ struct rapidjson_base {
return true; return true;
} }
}; };
#if SIMDJSON_COMPETITION_ONDEMAND_APPROX
struct rapidjson : rapidjson_base { struct rapidjson_approx : rapidjson_base {
bool run(simdjson::padded_string &json, std::vector<point> &result) { bool run(simdjson::padded_string &json, std::vector<point> &result) {
return rapidjson_base::run(doc.Parse<kParseValidateEncodingFlag>(json.data()), result); return rapidjson_base::run(doc.Parse<kParseValidateEncodingFlag>(json.data()), result);
} }
}; };
BENCHMARK_TEMPLATE(large_random, rapidjson)->UseManualTime(); BENCHMARK_TEMPLATE(large_random, rapidjson_approx)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_APPROX
struct rapidjson_lossless : rapidjson_base { struct rapidjson : rapidjson_base {
bool run(simdjson::padded_string &json, std::vector<point> &result) { bool run(simdjson::padded_string &json, std::vector<point> &result) {
return rapidjson_base::run(doc.Parse<kParseValidateEncodingFlag | kParseFullPrecisionFlag>(json.data()), result); return rapidjson_base::run(doc.Parse<kParseValidateEncodingFlag | kParseFullPrecisionFlag>(json.data()), result);
} }
}; };
BENCHMARK_TEMPLATE(large_random, rapidjson_lossless)->UseManualTime(); BENCHMARK_TEMPLATE(large_random, rapidjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct rapidjson_insitu : rapidjson_base { struct rapidjson_insitu : rapidjson_base {
bool run(simdjson::padded_string &json, std::vector<point> &result) { bool run(simdjson::padded_string &json, std::vector<point> &result) {
return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag>(json.data()), result); return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag|kParseInsituFlag>(json.data()), result);
} }
}; };
BENCHMARK_TEMPLATE(large_random, rapidjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(large_random, rapidjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace large_random } // namespace large_random
+2 -1
View File
@@ -11,8 +11,9 @@ struct sajson {
size_t ast_buffer_size{0}; size_t ast_buffer_size{0};
size_t *ast_buffer{nullptr}; size_t *ast_buffer{nullptr};
~sajson() { free(ast_buffer); }
simdjson_really_inline double get_double(const ::sajson::value &obj, std::string_view key) { simdjson_inline double get_double(const ::sajson::value &obj, std::string_view key) {
using namespace sajson; using namespace sajson;
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
+3 -3
View File
@@ -9,7 +9,7 @@ namespace large_random {
struct yyjson_base { struct yyjson_base {
static constexpr diff_flags DiffFlags = diff_flags::NONE; static constexpr diff_flags DiffFlags = diff_flags::NONE;
simdjson_really_inline double get_double(yyjson_val *obj, std::string_view key) { simdjson_inline double get_double(yyjson_val *obj, std::string_view key) {
yyjson_val *val = yyjson_obj_getn(obj, key.data(), key.length()); yyjson_val *val = yyjson_obj_getn(obj, key.data(), key.length());
if (!val) { throw "missing point field!"; } if (!val) { throw "missing point field!"; }
if (yyjson_get_type(val) != YYJSON_TYPE_NUM) { throw "Number is not a type!"; } if (yyjson_get_type(val) != YYJSON_TYPE_NUM) { throw "Number is not a type!"; }
@@ -51,14 +51,14 @@ struct yyjson : yyjson_base {
} }
}; };
BENCHMARK_TEMPLATE(large_random, yyjson)->UseManualTime(); BENCHMARK_TEMPLATE(large_random, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson_base { struct yyjson_insitu : yyjson_base {
bool run(simdjson::padded_string &json, std::vector<point> &result) { bool run(simdjson::padded_string &json, std::vector<point> &result) {
return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result); return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result);
} }
}; };
BENCHMARK_TEMPLATE(large_random, yyjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(large_random, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace large_random } // namespace large_random
#endif // SIMDJSON_COMPETITION_YYJSON #endif // SIMDJSON_COMPETITION_YYJSON
+6 -6
View File
@@ -10,28 +10,28 @@ using namespace simdjson;
class Iter { class Iter {
public: public:
simdjson_really_inline bool Run(const padded_string &json); simdjson_inline bool Run(const padded_string &json);
simdjson_really_inline const std::vector<my_point> &Result() { return container; } simdjson_inline const std::vector<my_point> &Result() { return container; }
simdjson_really_inline size_t ItemCount() { return container.size(); } simdjson_inline size_t ItemCount() { return container.size(); }
private: private:
ondemand::parser parser{}; ondemand::parser parser{};
std::vector<my_point> container{}; std::vector<my_point> container{};
simdjson_really_inline double first_double(ondemand::json_iterator &iter) { simdjson_inline double first_double(ondemand::json_iterator &iter) {
if (iter.start_object().error() || iter.field_key().error() || iter.field_value()) { throw "Invalid field"; } if (iter.start_object().error() || iter.field_key().error() || iter.field_value()) { throw "Invalid field"; }
return iter.consume_double(); return iter.consume_double();
} }
simdjson_really_inline double next_double(ondemand::json_iterator &iter) { simdjson_inline double next_double(ondemand::json_iterator &iter) {
if (!iter.has_next_field() || iter.field_key().error() || iter.field_value()) { throw "Invalid field"; } if (!iter.has_next_field() || iter.field_key().error() || iter.field_value()) { throw "Invalid field"; }
return iter.consume_double(); return iter.consume_double();
} }
}; };
simdjson_really_inline bool Iter::Run(const padded_string &json) { simdjson_inline bool Iter::Run(const padded_string &json) {
container.clear(); container.clear();
auto iter = parser.iterate_raw(json).value(); auto iter = parser.iterate_raw(json).value();
+4 -4
View File
@@ -10,16 +10,16 @@ using namespace simdjson;
class OnDemand { class OnDemand {
public: public:
simdjson_really_inline bool Run(const padded_string &json); simdjson_inline bool Run(const padded_string &json);
simdjson_really_inline const std::vector<my_point> &Result() { return container; } simdjson_inline const std::vector<my_point> &Result() { return container; }
simdjson_really_inline size_t ItemCount() { return container.size(); } simdjson_inline size_t ItemCount() { return container.size(); }
private: private:
ondemand::parser parser{}; ondemand::parser parser{};
std::vector<my_point> container{}; std::vector<my_point> container{};
}; };
simdjson_really_inline bool OnDemand::Run(const padded_string &json) { simdjson_inline bool OnDemand::Run(const padded_string &json) {
container.clear(); container.clear();
auto doc = parser.iterate(json); auto doc = parser.iterate(json);
+16 -16
View File
@@ -11,13 +11,13 @@ using namespace simdjson::builtin::stage2;
class Sax { class Sax {
public: public:
simdjson_really_inline bool Run(const padded_string &json) noexcept; simdjson_inline bool Run(const padded_string &json) noexcept;
simdjson_really_inline const std::vector<my_point> &Result() { return container; } simdjson_inline const std::vector<my_point> &Result() { return container; }
simdjson_really_inline size_t ItemCount() { return container.size(); } simdjson_inline size_t ItemCount() { return container.size(); }
private: private:
simdjson_really_inline error_code RunNoExcept(const padded_string &json) noexcept; simdjson_inline error_code RunNoExcept(const padded_string &json) noexcept;
error_code Allocate(size_t new_capacity); error_code Allocate(size_t new_capacity);
std::unique_ptr<uint8_t[]> string_buf{}; std::unique_ptr<uint8_t[]> string_buf{};
size_t capacity{}; size_t capacity{};
@@ -34,21 +34,21 @@ public:
explicit sax_point_reader_visitor(std::vector<my_point> &_points) : points(_points) {} explicit sax_point_reader_visitor(std::vector<my_point> &_points) : points(_points) {}
simdjson_really_inline error_code visit_object_start(json_iterator &) { simdjson_inline error_code visit_object_start(json_iterator &) {
idx = 0; idx = 0;
return SUCCESS; return SUCCESS;
} }
simdjson_really_inline error_code visit_primitive(json_iterator &, const uint8_t *value) { simdjson_inline error_code visit_primitive(json_iterator &, const uint8_t *value) {
if(idx == GOT_SOMETHING_ELSE) { return simdjson::SUCCESS; } if(idx == GOT_SOMETHING_ELSE) { return simdjson::SUCCESS; }
return numberparsing::parse_double(value).get(buffer[idx]); return numberparsing::parse_double(value).get(buffer[idx]);
} }
simdjson_really_inline error_code visit_object_end(json_iterator &) { simdjson_inline error_code visit_object_end(json_iterator &) {
points.emplace_back(my_point{buffer[0], buffer[1], buffer[2]}); points.emplace_back(my_point{buffer[0], buffer[1], buffer[2]});
return SUCCESS; return SUCCESS;
} }
simdjson_really_inline error_code visit_document_start(json_iterator &) { return SUCCESS; } simdjson_inline error_code visit_document_start(json_iterator &) { return SUCCESS; }
simdjson_really_inline error_code visit_key(json_iterator &, const uint8_t * key) { simdjson_inline error_code visit_key(json_iterator &, const uint8_t * key) {
switch(key[1]) { switch(key[1]) {
// Technically, we should check the other characters // Technically, we should check the other characters
// in the key, but we are cheating to go as fast // in the key, but we are cheating to go as fast
@@ -67,13 +67,13 @@ public:
} }
return SUCCESS; return SUCCESS;
} }
simdjson_really_inline error_code visit_array_start(json_iterator &) { return SUCCESS; } simdjson_inline error_code visit_array_start(json_iterator &) { return SUCCESS; }
simdjson_really_inline error_code visit_array_end(json_iterator &) { return SUCCESS; } simdjson_inline error_code visit_array_end(json_iterator &) { return SUCCESS; }
simdjson_really_inline error_code visit_document_end(json_iterator &) { return SUCCESS; } simdjson_inline error_code visit_document_end(json_iterator &) { return SUCCESS; }
simdjson_really_inline error_code visit_empty_array(json_iterator &) { return SUCCESS; } simdjson_inline error_code visit_empty_array(json_iterator &) { return SUCCESS; }
simdjson_really_inline error_code visit_empty_object(json_iterator &) { return SUCCESS; } simdjson_inline error_code visit_empty_object(json_iterator &) { return SUCCESS; }
simdjson_really_inline error_code visit_root_primitive(json_iterator &, const uint8_t *) { return SUCCESS; } simdjson_inline error_code visit_root_primitive(json_iterator &, const uint8_t *) { return SUCCESS; }
simdjson_really_inline error_code increment_count(json_iterator &) { return SUCCESS; } simdjson_inline error_code increment_count(json_iterator &) { return SUCCESS; }
}; };
// NOTE: this assumes the dom_parser is already allocated // NOTE: this assumes the dom_parser is already allocated
+2 -2
View File
@@ -9,7 +9,7 @@ namespace partial_tweets {
struct nlohmann_json { struct nlohmann_json {
using StringType=std::string; using StringType=std::string;
simdjson_really_inline uint64_t nullable_int(nlohmann::json value) { simdjson_inline uint64_t nullable_int(nlohmann::json value) {
if (value.is_null()) { return 0; } if (value.is_null()) { return 0; }
return value; return value;
} }
@@ -37,4 +37,4 @@ BENCHMARK_TEMPLATE(partial_tweets, nlohmann_json)->UseManualTime();
} // namespace partial_tweets } // namespace partial_tweets
#endif // SIMDJSON_COMPETITION_NLOHMANN_JSON #endif // SIMDJSON_COMPETITION_NLOHMANN_JSON
+6 -6
View File
@@ -17,27 +17,27 @@ public:
displayed_implementation = true; displayed_implementation = true;
} }
} }
simdjson_really_inline bool Run(const padded_string &json); simdjson_inline bool Run(const padded_string &json);
simdjson_really_inline const std::vector<tweet> &Result() { return tweets; } simdjson_inline const std::vector<tweet> &Result() { return tweets; }
simdjson_really_inline size_t ItemCount() { return tweets.size(); } simdjson_inline size_t ItemCount() { return tweets.size(); }
private: private:
ondemand::parser parser{}; ondemand::parser parser{};
std::vector<tweet> tweets{}; std::vector<tweet> tweets{};
simdjson_really_inline uint64_t nullable_int(ondemand::value value) { simdjson_inline uint64_t nullable_int(ondemand::value value) {
if (value.is_null()) { return 0; } if (value.is_null()) { return 0; }
return value; return value;
} }
simdjson_really_inline twitter_user read_user(ondemand::object user) { simdjson_inline twitter_user read_user(ondemand::object user) {
return { user.find_field("id"), user.find_field("screen_name") }; return { user.find_field("id"), user.find_field("screen_name") };
} }
static inline bool displayed_implementation = false; static inline bool displayed_implementation = false;
}; };
simdjson_really_inline bool OnDemand::Run(const padded_string &json) { simdjson_inline bool OnDemand::Run(const padded_string &json) {
tweets.clear(); tweets.clear();
// Walk the document, parsing the tweets as we go // Walk the document, parsing the tweets as we go
+1 -1
View File
@@ -39,7 +39,7 @@ struct runner : public file_runner<I> {
struct simdjson_dom; struct simdjson_dom;
template<typename I> simdjson_really_inline static void partial_tweets(benchmark::State &state) { template<typename I> simdjson_inline static void partial_tweets(benchmark::State &state) {
run_json_benchmark<runner<I>, runner<simdjson_dom>>(state); run_json_benchmark<runner<I>, runner<simdjson_dom>>(state);
} }
+7 -7
View File
@@ -13,27 +13,27 @@ struct rapidjson_base {
Document doc{}; Document doc{};
simdjson_really_inline std::string_view get_string_view(Value &object, std::string_view key) { simdjson_inline std::string_view get_string_view(Value &object, std::string_view key) {
// TODO use version that supports passing string length? // TODO use version that supports passing string length?
auto field = object.FindMember(key.data()); auto field = object.FindMember(key.data());
if (field == object.MemberEnd()) { throw "Missing object field"; } if (field == object.MemberEnd()) { throw "Missing object field"; }
if (!field->value.IsString()) { throw "Field is not a string"; } if (!field->value.IsString()) { throw "Field is not a string"; }
return { field->value.GetString(), field->value.GetStringLength() }; return { field->value.GetString(), field->value.GetStringLength() };
} }
simdjson_really_inline uint64_t get_uint64(Value &object, std::string_view key) { simdjson_inline uint64_t get_uint64(Value &object, std::string_view key) {
auto field = object.FindMember(key.data()); auto field = object.FindMember(key.data());
if (field == object.MemberEnd()) { throw "Missing object field"; } if (field == object.MemberEnd()) { throw "Missing object field"; }
if (!field->value.IsUint64()) { throw "Field is not uint64"; } if (!field->value.IsUint64()) { throw "Field is not uint64"; }
return field->value.GetUint64(); return field->value.GetUint64();
} }
simdjson_really_inline uint64_t get_nullable_uint64(Value &object, std::string_view key) { simdjson_inline uint64_t get_nullable_uint64(Value &object, std::string_view key) {
auto field = object.FindMember(key.data()); auto field = object.FindMember(key.data());
if (field == object.MemberEnd()) { throw "Missing nullable uint64 field"; } if (field == object.MemberEnd()) { throw "Missing nullable uint64 field"; }
if (field->value.IsNull()) { return 0; } if (field->value.IsNull()) { return 0; }
if (!field->value.IsUint64()) { throw "Field is not nullable uint64"; } if (!field->value.IsUint64()) { throw "Field is not nullable uint64"; }
return field->value.GetUint64(); return field->value.GetUint64();
} }
simdjson_really_inline partial_tweets::twitter_user<std::string_view> get_user(Value &object, std::string_view key) { simdjson_inline partial_tweets::twitter_user<std::string_view> get_user(Value &object, std::string_view key) {
auto field = object.FindMember(key.data()); auto field = object.FindMember(key.data());
if (field == object.MemberEnd()) { throw "Missing user field"; } if (field == object.MemberEnd()) { throw "Missing user field"; }
if (!field->value.IsObject()) { throw "User field is not an object"; } if (!field->value.IsObject()) { throw "User field is not an object"; }
@@ -67,14 +67,14 @@ struct rapidjson : rapidjson_base {
} }
}; };
BENCHMARK_TEMPLATE(partial_tweets, rapidjson)->UseManualTime(); BENCHMARK_TEMPLATE(partial_tweets, rapidjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct rapidjson_insitu : rapidjson_base { struct rapidjson_insitu : rapidjson_base {
bool run(simdjson::padded_string &json, std::vector<tweet<std::string_view>> &result) { bool run(simdjson::padded_string &json, std::vector<tweet<std::string_view>> &result) {
return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag>(json.data()), result); return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag|kParseInsituFlag>(json.data()), result);
} }
}; };
BENCHMARK_TEMPLATE(partial_tweets, rapidjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(partial_tweets, rapidjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace partial_tweets } // namespace partial_tweets
#endif // SIMDJSON_COMPETITION_RAPIDJSON #endif // SIMDJSON_COMPETITION_RAPIDJSON
+7 -5
View File
@@ -11,12 +11,14 @@ struct sajson {
size_t ast_buffer_size{0}; size_t ast_buffer_size{0};
size_t *ast_buffer{nullptr}; size_t *ast_buffer{nullptr};
simdjson_really_inline std::string_view get_string_view(const ::sajson::value &obj, std::string_view key) { ~sajson() { free(ast_buffer); }
simdjson_inline std::string_view get_string_view(const ::sajson::value &obj, std::string_view key) {
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
if (val.get_type() != ::sajson::TYPE_STRING) { throw "field is not a string"; } if (val.get_type() != ::sajson::TYPE_STRING) { throw "field is not a string"; }
return { val.as_cstring(), val.get_string_length() }; return { val.as_cstring(), val.get_string_length() };
} }
simdjson_really_inline uint64_t get_uint52(const ::sajson::value &obj, std::string_view key) { simdjson_inline uint64_t get_uint52(const ::sajson::value &obj, std::string_view key) {
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
switch (val.get_type()) { switch (val.get_type()) {
case ::sajson::TYPE_INTEGER: { case ::sajson::TYPE_INTEGER: {
@@ -28,7 +30,7 @@ struct sajson {
throw "field not integer"; throw "field not integer";
} }
} }
simdjson_really_inline uint64_t get_str_uint64(const ::sajson::value &obj, std::string_view key) { simdjson_inline uint64_t get_str_uint64(const ::sajson::value &obj, std::string_view key) {
// Since sajson only supports 53-bit numbers, and IDs in twitter.json can be > 53 bits, we read the corresponding id_str and parse that. // Since sajson only supports 53-bit numbers, and IDs in twitter.json can be > 53 bits, we read the corresponding id_str and parse that.
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
if (val.get_type() != ::sajson::TYPE_STRING) { throw "field not a string"; } if (val.get_type() != ::sajson::TYPE_STRING) { throw "field not a string"; }
@@ -38,7 +40,7 @@ struct sajson {
if (endptr != &str[val.get_string_length()]) { throw "field is a string, but not an integer string"; } if (endptr != &str[val.get_string_length()]) { throw "field is a string, but not an integer string"; }
return result; return result;
} }
simdjson_really_inline uint64_t get_nullable_str_uint64(const ::sajson::value &obj, std::string_view key) { simdjson_inline uint64_t get_nullable_str_uint64(const ::sajson::value &obj, std::string_view key) {
auto val = obj.get_value_of_key({key.data(), key.length()}); auto val = obj.get_value_of_key({key.data(), key.length()});
if (val.get_type() == ::sajson::TYPE_NULL) { return 0; } if (val.get_type() == ::sajson::TYPE_NULL) { return 0; }
if (val.get_type() != ::sajson::TYPE_STRING) { throw "field not a string"; } if (val.get_type() != ::sajson::TYPE_STRING) { throw "field not a string"; }
@@ -48,7 +50,7 @@ struct sajson {
if (endptr != &str[val.get_string_length()]) { throw "field is a string, but not an integer string"; } if (endptr != &str[val.get_string_length()]) { throw "field is a string, but not an integer string"; }
return result; return result;
} }
simdjson_really_inline partial_tweets::twitter_user<std::string_view> get_user(const ::sajson::value &obj, std::string_view key) { simdjson_inline partial_tweets::twitter_user<std::string_view> get_user(const ::sajson::value &obj, std::string_view key) {
auto user = obj.get_value_of_key({key.data(), key.length()}); auto user = obj.get_value_of_key({key.data(), key.length()});
if (user.get_type() != ::sajson::TYPE_OBJECT) { throw "user is not an object"; } if (user.get_type() != ::sajson::TYPE_OBJECT) { throw "user is not an object"; }
return { get_str_uint64(user, "id_str"), get_string_view(user, "screen_name") }; return { get_str_uint64(user, "id_str"), get_string_view(user, "screen_name") };
+2 -2
View File
@@ -13,7 +13,7 @@ struct simdjson_dom {
dom::parser parser{}; dom::parser parser{};
simdjson_really_inline uint64_t nullable_int(dom::element element) { simdjson_inline uint64_t nullable_int(dom::element element) {
if (element.is_null()) { return 0; } if (element.is_null()) { return 0; }
return element; return element;
} }
@@ -40,4 +40,4 @@ BENCHMARK_TEMPLATE(partial_tweets, simdjson_dom)->UseManualTime();
} // namespace partial_tweets } // namespace partial_tweets
#endif // SIMDJSON_EXCEPTIONS #endif // SIMDJSON_EXCEPTIONS
+2 -2
View File
@@ -13,12 +13,12 @@ struct simdjson_ondemand {
ondemand::parser parser{}; ondemand::parser parser{};
simdjson_really_inline uint64_t nullable_int(ondemand::value value) { simdjson_inline uint64_t nullable_int(ondemand::value value) {
if (value.is_null()) { return 0; } if (value.is_null()) { return 0; }
return value; return value;
} }
simdjson_really_inline twitter_user<std::string_view> read_user(ondemand::object user) { simdjson_inline twitter_user<std::string_view> read_user(ondemand::object user) {
return { user.find_field("id"), user.find_field("screen_name") }; return { user.find_field("id"), user.find_field("screen_name") };
} }
+2 -2
View File
@@ -32,7 +32,7 @@ struct tweet {
uint64_t retweet_count{}; uint64_t retweet_count{};
uint64_t favorite_count{}; uint64_t favorite_count{};
template<typename OtherStringType> template<typename OtherStringType>
simdjson_really_inline bool operator==(const tweet<OtherStringType> &other) const { simdjson_inline bool operator==(const tweet<OtherStringType> &other) const {
return created_at == other.created_at && return created_at == other.created_at &&
id == other.id && id == other.id &&
result == other.result && result == other.result &&
@@ -42,7 +42,7 @@ struct tweet {
favorite_count == other.favorite_count; favorite_count == other.favorite_count;
} }
template<typename OtherStringType> template<typename OtherStringType>
simdjson_really_inline bool operator!=(const tweet<OtherStringType> &other) const { return !(*this == other); } simdjson_inline bool operator!=(const tweet<OtherStringType> &other) const { return !(*this == other); }
}; };
template<typename StringType> template<typename StringType>
+6 -6
View File
@@ -9,24 +9,24 @@ namespace partial_tweets {
struct yyjson_base { struct yyjson_base {
using StringType=std::string_view; using StringType=std::string_view;
simdjson_really_inline std::string_view get_string_view(yyjson_val *obj, std::string_view key) { simdjson_inline std::string_view get_string_view(yyjson_val *obj, std::string_view key) {
auto val = yyjson_obj_getn(obj, key.data(), key.length()); auto val = yyjson_obj_getn(obj, key.data(), key.length());
if (!yyjson_is_str(val)) { throw "field is not uint64 or null!"; } if (!yyjson_is_str(val)) { throw "field is not uint64 or null!"; }
return { yyjson_get_str(val), yyjson_get_len(val) }; return { yyjson_get_str(val), yyjson_get_len(val) };
} }
simdjson_really_inline uint64_t get_uint64(yyjson_val *obj, std::string_view key) { simdjson_inline uint64_t get_uint64(yyjson_val *obj, std::string_view key) {
auto val = yyjson_obj_getn(obj, key.data(), key.length()); auto val = yyjson_obj_getn(obj, key.data(), key.length());
if (!yyjson_is_uint(val)) { throw "field is not uint64 or null!"; } if (!yyjson_is_uint(val)) { throw "field is not uint64 or null!"; }
return yyjson_get_uint(val); return yyjson_get_uint(val);
} }
simdjson_really_inline uint64_t get_nullable_uint64(yyjson_val *obj, std::string_view key) { simdjson_inline uint64_t get_nullable_uint64(yyjson_val *obj, std::string_view key) {
auto val = yyjson_obj_getn(obj, key.data(), key.length()); auto val = yyjson_obj_getn(obj, key.data(), key.length());
if (!yyjson_is_uint(val)) { } if (!yyjson_is_uint(val)) { }
auto type = yyjson_get_type(val); auto type = yyjson_get_type(val);
if (type != YYJSON_TYPE_NUM && type != YYJSON_TYPE_NULL ) { throw "field is not uint64 or null!"; } if (type != YYJSON_TYPE_NUM && type != YYJSON_TYPE_NULL ) { throw "field is not uint64 or null!"; }
return yyjson_get_uint(val); return yyjson_get_uint(val);
} }
simdjson_really_inline partial_tweets::twitter_user<std::string_view> get_user(yyjson_val *obj, std::string_view key) { simdjson_inline partial_tweets::twitter_user<std::string_view> get_user(yyjson_val *obj, std::string_view key) {
auto user = yyjson_obj_getn(obj, key.data(), key.length()); auto user = yyjson_obj_getn(obj, key.data(), key.length());
if (!yyjson_is_obj(user)) { throw "missing twitter user field!"; } if (!yyjson_is_obj(user)) { throw "missing twitter user field!"; }
return { get_uint64(user, "id"), get_string_view(user, "screen_name") }; return { get_uint64(user, "id"), get_string_view(user, "screen_name") };
@@ -66,14 +66,14 @@ struct yyjson : yyjson_base {
} }
}; };
BENCHMARK_TEMPLATE(partial_tweets, yyjson)->UseManualTime(); BENCHMARK_TEMPLATE(partial_tweets, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson_base { struct yyjson_insitu : yyjson_base {
bool run(simdjson::padded_string &json, std::vector<tweet<std::string_view>> &result) { bool run(simdjson::padded_string &json, std::vector<tweet<std::string_view>> &result) {
return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result); return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), result);
} }
}; };
BENCHMARK_TEMPLATE(partial_tweets, yyjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(partial_tweets, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace partial_tweets } // namespace partial_tweets
#endif // SIMDJSON_COMPETITION_YYJSON #endif // SIMDJSON_COMPETITION_YYJSON
+1 -1
View File
@@ -48,7 +48,7 @@ struct nlohmann_json_sax {
} }
bool number_unsigned(number_unsigned_t val) override { bool number_unsigned(number_unsigned_t val) override {
if (values & key_rt && !(values & found_rt)) { // retweet_count if (values & key_rt && !(values & found_rt)) { // retweet_count
rt = val; rt = int(val);
values &= ~(key_rt); values &= ~(key_rt);
values |= (found_rt); values |= (found_rt);
if (rt <= max_rt && rt >= result.retweet_count) { // Check if current tweet has more retweet than previous top tweet if (rt <= max_rt && rt >= result.retweet_count) { // Check if current tweet has more retweet than previous top tweet
+3 -3
View File
@@ -56,14 +56,14 @@ struct rapidjson : rapidjson_base {
} }
}; };
BENCHMARK_TEMPLATE(top_tweet, rapidjson)->UseManualTime(); BENCHMARK_TEMPLATE(top_tweet, rapidjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct rapidjson_insitu : rapidjson_base { struct rapidjson_insitu : rapidjson_base {
bool run(simdjson::padded_string &json, int64_t max_retweet_count, top_tweet_result<StringType> &result) { bool run(simdjson::padded_string &json, int64_t max_retweet_count, top_tweet_result<StringType> &result) {
return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag>(json.data()), max_retweet_count, result); return rapidjson_base::run(doc.ParseInsitu<kParseValidateEncodingFlag|kParseInsituFlag>(json.data()), max_retweet_count, result);
} }
}; };
BENCHMARK_TEMPLATE(top_tweet, rapidjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(top_tweet, rapidjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace top_tweet } // namespace top_tweet
#endif // SIMDJSON_COMPETITION_RAPIDJSON #endif // SIMDJSON_COMPETITION_RAPIDJSON
+1
View File
@@ -11,6 +11,7 @@ struct sajson {
size_t ast_buffer_size{0}; size_t ast_buffer_size{0};
size_t *ast_buffer{nullptr}; size_t *ast_buffer{nullptr};
~sajson() { free(ast_buffer); }
bool run(simdjson::padded_string &json, int32_t max_retweet_count, top_tweet_result<StringType> &result) { bool run(simdjson::padded_string &json, int32_t max_retweet_count, top_tweet_result<StringType> &result) {
if (!ast_buffer) { if (!ast_buffer) {
+3 -3
View File
@@ -13,13 +13,13 @@ struct top_tweet_result {
StringType screen_name{}; StringType screen_name{};
StringType text{}; StringType text{};
template<typename OtherStringType> template<typename OtherStringType>
simdjson_really_inline bool operator==(const top_tweet_result<OtherStringType> &other) const { simdjson_inline bool operator==(const top_tweet_result<OtherStringType> &other) const {
return retweet_count == other.retweet_count && return retweet_count == other.retweet_count &&
screen_name == other.screen_name && screen_name == other.screen_name &&
text == other.text; text == other.text;
} }
template<typename OtherStringType> template<typename OtherStringType>
simdjson_really_inline bool operator!=(const top_tweet_result<OtherStringType> &other) const { return !(*this == other); } simdjson_inline bool operator!=(const top_tweet_result<OtherStringType> &other) const { return !(*this == other); }
}; };
template<typename StringType> template<typename StringType>
@@ -60,7 +60,7 @@ struct runner : public file_runner<I> {
struct simdjson_dom; struct simdjson_dom;
template<typename I> simdjson_really_inline static void top_tweet(benchmark::State &state) { template<typename I> simdjson_inline static void top_tweet(benchmark::State &state) {
json_benchmark::run_json_benchmark<runner<I>, runner<simdjson_dom>>(state); json_benchmark::run_json_benchmark<runner<I>, runner<simdjson_dom>>(state);
} }
+2 -2
View File
@@ -55,14 +55,14 @@ struct yyjson : yyjson_base {
} }
}; };
BENCHMARK_TEMPLATE(top_tweet, yyjson)->UseManualTime(); BENCHMARK_TEMPLATE(top_tweet, yyjson)->UseManualTime();
#if SIMDJSON_COMPETITION_ONDEMAND_INSITU
struct yyjson_insitu : yyjson_base { struct yyjson_insitu : yyjson_base {
bool run(simdjson::padded_string &json, int64_t max_retweet_count, top_tweet_result<StringType> &result) { bool run(simdjson::padded_string &json, int64_t max_retweet_count, top_tweet_result<StringType> &result) {
return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), max_retweet_count, result); return yyjson_base::run(yyjson_read_opts(json.data(), json.size(), YYJSON_READ_INSITU, 0, 0), max_retweet_count, result);
} }
}; };
BENCHMARK_TEMPLATE(top_tweet, yyjson_insitu)->UseManualTime(); BENCHMARK_TEMPLATE(top_tweet, yyjson_insitu)->UseManualTime();
#endif // SIMDJSON_COMPETITION_ONDEMAND_INSITU
} // namespace top_tweet } // namespace top_tweet
#endif // SIMDJSON_COMPETITION_YYJSON #endif // SIMDJSON_COMPETITION_YYJSON
+4 -4
View File
@@ -70,11 +70,11 @@ int main() {}
target_include_directories(jsmn SYSTEM PUBLIC "${jsmn_SOURCE_DIR}") target_include_directories(jsmn SYSTEM PUBLIC "${jsmn_SOURCE_DIR}")
target_compile_definitions(jsmn INTERFACE SIMDJSON_COMPETITION_JSMN) target_compile_definitions(jsmn INTERFACE SIMDJSON_COMPETITION_JSMN)
message(STATUS "Importing json (nlohmann/json@v3.9.1)") message(STATUS "Importing json (nlohmann/json@v3.10.5)")
set(nlohmann_json_SOURCE_DIR "${dep_root}/json") set(nlohmann_json_SOURCE_DIR "${dep_root}/json")
if(NOT EXISTS "${nlohmann_json_SOURCE_DIR}") if(NOT EXISTS "${nlohmann_json_SOURCE_DIR}")
file(DOWNLOAD file(DOWNLOAD
"https://github.com/nlohmann/json/releases/download/v3.9.1/json.hpp" "https://github.com/nlohmann/json/releases/download/v3.10.5/json.hpp"
"${nlohmann_json_SOURCE_DIR}/nlohmann/json.hpp") "${nlohmann_json_SOURCE_DIR}/nlohmann/json.hpp")
endif() endif()
add_library(nlohmann_json INTERFACE) add_library(nlohmann_json INTERFACE)
@@ -91,7 +91,7 @@ int main() {}
target_include_directories(jsoncpp SYSTEM PUBLIC "${jsoncpp_SOURCE_DIR}") target_include_directories(jsoncpp SYSTEM PUBLIC "${jsoncpp_SOURCE_DIR}")
target_compile_definitions(jsoncpp INTERFACE SIMDJSON_COMPETITION_JSONCPP) target_compile_definitions(jsoncpp INTERFACE SIMDJSON_COMPETITION_JSONCPP)
import_dependency(rapidjson Tencent/rapidjson b32cd94) import_dependency(rapidjson Tencent/rapidjson f54b0e4)
add_library(rapidjson INTERFACE) add_library(rapidjson INTERFACE)
target_compile_definitions(rapidjson INTERFACE RAPIDJSON_HAS_STDSTRING) target_compile_definitions(rapidjson INTERFACE RAPIDJSON_HAS_STDSTRING)
target_include_directories(rapidjson SYSTEM INTERFACE target_include_directories(rapidjson SYSTEM INTERFACE
@@ -114,7 +114,7 @@ int main() {}
"${ujson4c_SOURCE_DIR}/3rdparty") "${ujson4c_SOURCE_DIR}/3rdparty")
target_compile_definitions(ujson4c INTERFACE SIMDJSON_COMPETITION_UJSON4C) target_compile_definitions(ujson4c INTERFACE SIMDJSON_COMPETITION_UJSON4C)
import_dependency(yyjson ibireme/yyjson aa33ec5) import_dependency(yyjson ibireme/yyjson c385651)
add_library(yyjson STATIC "${yyjson_SOURCE_DIR}/src/yyjson.c") add_library(yyjson STATIC "${yyjson_SOURCE_DIR}/src/yyjson.c")
target_include_directories(yyjson SYSTEM PUBLIC "${yyjson_SOURCE_DIR}/src") target_include_directories(yyjson SYSTEM PUBLIC "${yyjson_SOURCE_DIR}/src")
target_compile_definitions(yyjson INTERFACE SIMDJSON_COMPETITION_YYJSON) target_compile_definitions(yyjson INTERFACE SIMDJSON_COMPETITION_YYJSON)
+192 -90
View File
@@ -79,6 +79,150 @@ license you like.
/// to prevent private header inclusion. /// to prevent private header inclusion.
#define JSON_IS_AMALGAMATION #define JSON_IS_AMALGAMATION
// //////////////////////////////////////////////////////////////////////
// Beginning of content of file: include/json/version.h
// //////////////////////////////////////////////////////////////////////
#ifndef JSON_VERSION_H_INCLUDED
#define JSON_VERSION_H_INCLUDED
// Note: version must be updated in three places when doing a release. This
// annoying process ensures that amalgamate, CMake, and meson all report the
// correct version.
// 1. /meson.build
// 2. /include/json/version.h
// 3. /CMakeLists.txt
// IMPORTANT: also update the SOVERSION!!
#define JSONCPP_VERSION_STRING "1.9.5"
#define JSONCPP_VERSION_MAJOR 1
#define JSONCPP_VERSION_MINOR 9
#define JSONCPP_VERSION_PATCH 5
#define JSONCPP_VERSION_QUALIFIER
#define JSONCPP_VERSION_HEXA \
((JSONCPP_VERSION_MAJOR << 24) | (JSONCPP_VERSION_MINOR << 16) | \
(JSONCPP_VERSION_PATCH << 8))
#ifdef JSONCPP_USING_SECURE_MEMORY
#undef JSONCPP_USING_SECURE_MEMORY
#endif
#define JSONCPP_USING_SECURE_MEMORY 0
// If non-zero, the library zeroes any memory that it has allocated before
// it frees its memory.
#endif // JSON_VERSION_H_INCLUDED
// //////////////////////////////////////////////////////////////////////
// End of content of file: include/json/version.h
// //////////////////////////////////////////////////////////////////////
// //////////////////////////////////////////////////////////////////////
// Beginning of content of file: include/json/allocator.h
// //////////////////////////////////////////////////////////////////////
// Copyright 2007-2010 Baptiste Lepilleur and The JsonCpp Authors
// Distributed under MIT license, or public domain if desired and
// recognized in your jurisdiction.
// See file LICENSE for detail or copy at http://jsoncpp.sourceforge.net/LICENSE
#ifndef JSON_ALLOCATOR_H_INCLUDED
#define JSON_ALLOCATOR_H_INCLUDED
#include <cstring>
#include <memory>
#pragma pack(push, 8)
namespace Json {
template <typename T> class SecureAllocator {
public:
// Type definitions
using value_type = T;
using pointer = T*;
using const_pointer = const T*;
using reference = T&;
using const_reference = const T&;
using size_type = std::size_t;
using difference_type = std::ptrdiff_t;
/**
* Allocate memory for N items using the standard allocator.
*/
pointer allocate(size_type n) {
// allocate using "global operator new"
return static_cast<pointer>(::operator new(n * sizeof(T)));
}
/**
* Release memory which was allocated for N items at pointer P.
*
* The memory block is filled with zeroes before being released.
*/
void deallocate(pointer p, size_type n) {
// memset_s is used because memset may be optimized away by the compiler
memset_s(p, n * sizeof(T), 0, n * sizeof(T));
// free using "global operator delete"
::operator delete(p);
}
/**
* Construct an item in-place at pointer P.
*/
template <typename... Args> void construct(pointer p, Args&&... args) {
// construct using "placement new" and "perfect forwarding"
::new (static_cast<void*>(p)) T(std::forward<Args>(args)...);
}
size_type max_size() const { return size_t(-1) / sizeof(T); }
pointer address(reference x) const { return std::addressof(x); }
const_pointer address(const_reference x) const { return std::addressof(x); }
/**
* Destroy an item in-place at pointer P.
*/
void destroy(pointer p) {
// destroy using "explicit destructor"
p->~T();
}
// Boilerplate
SecureAllocator() {}
template <typename U> SecureAllocator(const SecureAllocator<U>&) {}
template <typename U> struct rebind { using other = SecureAllocator<U>; };
};
template <typename T, typename U>
bool operator==(const SecureAllocator<T>&, const SecureAllocator<U>&) {
return true;
}
template <typename T, typename U>
bool operator!=(const SecureAllocator<T>&, const SecureAllocator<U>&) {
return false;
}
} // namespace Json
#pragma pack(pop)
#endif // JSON_ALLOCATOR_H_INCLUDED
// //////////////////////////////////////////////////////////////////////
// End of content of file: include/json/allocator.h
// //////////////////////////////////////////////////////////////////////
// ////////////////////////////////////////////////////////////////////// // //////////////////////////////////////////////////////////////////////
// Beginning of content of file: include/json/config.h // Beginning of content of file: include/json/config.h
// ////////////////////////////////////////////////////////////////////// // //////////////////////////////////////////////////////////////////////
@@ -99,47 +243,38 @@ license you like.
#include <string> #include <string>
#include <type_traits> #include <type_traits>
/// If defined, indicates that json library is embedded in CppTL library.
//# define JSON_IN_CPPTL 1
/// If defined, indicates that json may leverage CppTL library
//# define JSON_USE_CPPTL 1
/// If defined, indicates that cpptl vector based map should be used instead of
/// std::map
/// as Value container.
//# define JSON_USE_CPPTL_SMALLMAP 1
// If non-zero, the library uses exceptions to report bad input instead of C // If non-zero, the library uses exceptions to report bad input instead of C
// assertion macros. The default is to use exceptions. // assertion macros. The default is to use exceptions.
#ifndef JSON_USE_EXCEPTION #ifndef JSON_USE_EXCEPTION
#define JSON_USE_EXCEPTION 1 #define JSON_USE_EXCEPTION 1
#endif #endif
// Temporary, tracked for removal with issue #982.
#ifndef JSON_USE_NULLREF
#define JSON_USE_NULLREF 1
#endif
/// If defined, indicates that the source file is amalgamated /// If defined, indicates that the source file is amalgamated
/// to prevent private header inclusion. /// to prevent private header inclusion.
/// Remarks: it is automatically defined in the generated amalgamated header. /// Remarks: it is automatically defined in the generated amalgamated header.
// #define JSON_IS_AMALGAMATION // #define JSON_IS_AMALGAMATION
#ifdef JSON_IN_CPPTL // Export macros for DLL visibility
#include <cpptl/config.h> #if defined(JSON_DLL_BUILD)
#ifndef JSON_USE_CPPTL
#define JSON_USE_CPPTL 1
#endif
#endif
#ifdef JSON_IN_CPPTL
#define JSON_API CPPTL_API
#elif defined(JSON_DLL_BUILD)
#if defined(_MSC_VER) || defined(__MINGW32__) #if defined(_MSC_VER) || defined(__MINGW32__)
#define JSON_API __declspec(dllexport) #define JSON_API __declspec(dllexport)
#define JSONCPP_DISABLE_DLL_INTERFACE_WARNING #define JSONCPP_DISABLE_DLL_INTERFACE_WARNING
#elif defined(__GNUC__) || defined(__clang__)
#define JSON_API __attribute__((visibility("default")))
#endif // if defined(_MSC_VER) #endif // if defined(_MSC_VER)
#elif defined(JSON_DLL) #elif defined(JSON_DLL)
#if defined(_MSC_VER) || defined(__MINGW32__) #if defined(_MSC_VER) || defined(__MINGW32__)
#define JSON_API __declspec(dllimport) #define JSON_API __declspec(dllimport)
#define JSONCPP_DISABLE_DLL_INTERFACE_WARNING #define JSONCPP_DISABLE_DLL_INTERFACE_WARNING
#endif // if defined(_MSC_VER) #endif // if defined(_MSC_VER)
#endif // ifdef JSON_IN_CPPTL #endif // ifdef JSON_DLL_BUILD
#if !defined(JSON_API) #if !defined(JSON_API)
#define JSON_API #define JSON_API
#endif #endif
@@ -152,8 +287,8 @@ license you like.
#if defined(_MSC_VER) && _MSC_VER < 1900 #if defined(_MSC_VER) && _MSC_VER < 1900
// As recommended at // As recommended at
// https://stackoverflow.com/questions/2915672/snprintf-and-visual-studio-2010 // https://stackoverflow.com/questions/2915672/snprintf-and-visual-studio-2010
extern JSON_API int extern JSON_API int msvc_pre1900_c99_snprintf(char* outBuf, size_t size,
msvc_pre1900_c99_snprintf(char* outBuf, size_t size, const char* format, ...); const char* format, ...);
#define jsoncpp_snprintf msvc_pre1900_c99_snprintf #define jsoncpp_snprintf msvc_pre1900_c99_snprintf
#else #else
#define jsoncpp_snprintf std::snprintf #define jsoncpp_snprintf std::snprintf
@@ -164,69 +299,30 @@ msvc_pre1900_c99_snprintf(char* outBuf, size_t size, const char* format, ...);
// Storages, and 64 bits integer support is disabled. // Storages, and 64 bits integer support is disabled.
// #define JSON_NO_INT64 1 // #define JSON_NO_INT64 1
#if defined(_MSC_VER) // MSVC
#define JSONCPP_DEPRECATED(message) __declspec(deprecated(message))
#endif // defined(_MSC_VER)
// JSONCPP_OVERRIDE is maintained for backwards compatibility of external tools. // JSONCPP_OVERRIDE is maintained for backwards compatibility of external tools.
// C++11 should be used directly in JSONCPP. // C++11 should be used directly in JSONCPP.
#define JSONCPP_OVERRIDE override #define JSONCPP_OVERRIDE override
#if __cplusplus >= 201103L
#define JSONCPP_NOEXCEPT noexcept
#define JSONCPP_OP_EXPLICIT explicit
#elif defined(_MSC_VER) && _MSC_VER < 1900
#define JSONCPP_NOEXCEPT throw()
#define JSONCPP_OP_EXPLICIT explicit
#elif defined(_MSC_VER) && _MSC_VER >= 1900
#define JSONCPP_NOEXCEPT noexcept
#define JSONCPP_OP_EXPLICIT explicit
#else
#define JSONCPP_NOEXCEPT throw()
#define JSONCPP_OP_EXPLICIT
#endif
#ifndef JSON_HAS_RVALUE_REFERENCES
#if defined(_MSC_VER)
#define JSON_HAS_RVALUE_REFERENCES 1
#endif // MSVC >= 2013
#ifdef __clang__
#if __has_feature(cxx_rvalue_references)
#define JSON_HAS_RVALUE_REFERENCES 1
#endif // has_feature
#elif defined __GNUC__ // not clang (gcc comes later since clang emulates gcc)
#if defined(__GXX_EXPERIMENTAL_CXX0X__) || (__cplusplus >= 201103L)
#define JSON_HAS_RVALUE_REFERENCES 1
#endif // GXX_EXPERIMENTAL
#endif // __clang__ || __GNUC__
#endif // not defined JSON_HAS_RVALUE_REFERENCES
#ifndef JSON_HAS_RVALUE_REFERENCES
#define JSON_HAS_RVALUE_REFERENCES 0
#endif
#ifdef __clang__ #ifdef __clang__
#if __has_extension(attribute_deprecated_with_message) #if __has_extension(attribute_deprecated_with_message)
#define JSONCPP_DEPRECATED(message) __attribute__((deprecated(message))) #define JSONCPP_DEPRECATED(message) __attribute__((deprecated(message)))
#endif #endif
#elif defined __GNUC__ // not clang (gcc comes later since clang emulates gcc) #elif defined(__GNUC__) // not clang (gcc comes later since clang emulates gcc)
#if (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 5)) #if (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 5))
#define JSONCPP_DEPRECATED(message) __attribute__((deprecated(message))) #define JSONCPP_DEPRECATED(message) __attribute__((deprecated(message)))
#elif (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1)) #elif (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1))
#define JSONCPP_DEPRECATED(message) __attribute__((__deprecated__)) #define JSONCPP_DEPRECATED(message) __attribute__((__deprecated__))
#endif // GNUC version #endif // GNUC version
#endif // __clang__ || __GNUC__ #elif defined(_MSC_VER) // MSVC (after clang because clang on Windows emulates
// MSVC)
#define JSONCPP_DEPRECATED(message) __declspec(deprecated(message))
#endif // __clang__ || __GNUC__ || _MSC_VER
#if !defined(JSONCPP_DEPRECATED) #if !defined(JSONCPP_DEPRECATED)
#define JSONCPP_DEPRECATED(message) #define JSONCPP_DEPRECATED(message)
#endif // if !defined(JSONCPP_DEPRECATED) #endif // if !defined(JSONCPP_DEPRECATED)
#if __GNUC__ >= 6 #if defined(__clang__) || (defined(__GNUC__) && (__GNUC__ >= 6))
#define JSON_USE_INT64_DOUBLE_CONVERSION 1 #define JSON_USE_INT64_DOUBLE_CONVERSION 1
#endif #endif
@@ -238,37 +334,37 @@ msvc_pre1900_c99_snprintf(char* outBuf, size_t size, const char* format, ...);
#endif // if !defined(JSON_IS_AMALGAMATION) #endif // if !defined(JSON_IS_AMALGAMATION)
namespace Json { namespace Json {
typedef int Int; using Int = int;
typedef unsigned int UInt; using UInt = unsigned int;
#if defined(JSON_NO_INT64) #if defined(JSON_NO_INT64)
typedef int LargestInt; using LargestInt = int;
typedef unsigned int LargestUInt; using LargestUInt = unsigned int;
#undef JSON_HAS_INT64 #undef JSON_HAS_INT64
#else // if defined(JSON_NO_INT64) #else // if defined(JSON_NO_INT64)
// For Microsoft Visual use specific types as long long is not supported // For Microsoft Visual use specific types as long long is not supported
#if defined(_MSC_VER) // Microsoft Visual Studio #if defined(_MSC_VER) // Microsoft Visual Studio
typedef __int64 Int64; using Int64 = __int64;
typedef unsigned __int64 UInt64; using UInt64 = unsigned __int64;
#else // if defined(_MSC_VER) // Other platforms, use long long #else // if defined(_MSC_VER) // Other platforms, use long long
typedef int64_t Int64; using Int64 = int64_t;
typedef uint64_t UInt64; using UInt64 = uint64_t;
#endif // if defined(_MSC_VER) #endif // if defined(_MSC_VER)
typedef Int64 LargestInt; using LargestInt = Int64;
typedef UInt64 LargestUInt; using LargestUInt = UInt64;
#define JSON_HAS_INT64 #define JSON_HAS_INT64
#endif // if defined(JSON_NO_INT64) #endif // if defined(JSON_NO_INT64)
template <typename T> template <typename T>
using Allocator = typename std::conditional<JSONCPP_USING_SECURE_MEMORY, using Allocator =
SecureAllocator<T>, typename std::conditional<JSONCPP_USING_SECURE_MEMORY, SecureAllocator<T>,
std::allocator<T>>::type; std::allocator<T>>::type;
using String = std::basic_string<char, std::char_traits<char>, Allocator<char>>; using String = std::basic_string<char, std::char_traits<char>, Allocator<char>>;
using IStringStream = std::basic_istringstream<String::value_type, using IStringStream =
String::traits_type, std::basic_istringstream<String::value_type, String::traits_type,
String::allocator_type>; String::allocator_type>;
using OStringStream = std::basic_ostringstream<String::value_type, using OStringStream =
String::traits_type, std::basic_ostringstream<String::value_type, String::traits_type,
String::allocator_type>; String::allocator_type>;
using IStream = std::istream; using IStream = std::istream;
using OStream = std::ostream; using OStream = std::ostream;
} // namespace Json } // namespace Json
@@ -310,17 +406,23 @@ using JSONCPP_OSTREAM = Json::OStream;
namespace Json { namespace Json {
// writer.h // writer.h
class StreamWriter;
class StreamWriterBuilder;
class Writer;
class FastWriter; class FastWriter;
class StyledWriter; class StyledWriter;
class StyledStreamWriter;
// reader.h // reader.h
class Reader; class Reader;
class CharReader;
class CharReaderBuilder;
// features.h // json_features.h
class Features; class Features;
// value.h // value.h
typedef unsigned int ArrayIndex; using ArrayIndex = unsigned int;
class StaticString; class StaticString;
class Path; class Path;
class PathArgument; class PathArgument;
+541 -561
View File
File diff suppressed because it is too large Load Diff
+508 -604
View File
File diff suppressed because it is too large Load Diff
+95 -43
View File
@@ -3,30 +3,35 @@ The Basics
An overview of what you need to know to use simdjson, with examples. An overview of what you need to know to use simdjson, with examples.
* [Requirements](#requirements) - [The Basics](#the-basics)
* [Including simdjson](#including-simdjson) - [Requirements](#requirements)
* [Using simdjson with package managers](#using-simdjson-with-package-managers) - [Including simdjson](#including-simdjson)
* [Using simdjson as a CMake dependency](#using-simdjson-as-a-cmake-dependency) - [Using simdjson with package managers](#using-simdjson-with-package-managers)
* [Versions](#versions) - [Using simdjson as a CMake dependency](#using-simdjson-as-a-cmake-dependency)
* [The Basics: Loading and Parsing JSON Documents](#the-basics-loading-and-parsing-json-documents) - [Versions](#versions)
* [Documents are Iterators](#documents-are-iterators) - [The Basics: Loading and Parsing JSON Documents](#the-basics-loading-and-parsing-json-documents)
* [C++11 Support and string_view](#c11-support-and-string_view) - [Documents are Iterators](#documents-are-iterators)
* [Using the Parsed JSON](#using-the-parsed-json) - [Parser, Document and JSON Scope](#parser-document-and-json-scope)
* [Minifying JSON strings without parsing](#minifying-json-strings-without-parsing) - [C++11 Support and string_view](#c11-support-and-string_view)
* [UTF-8 validation (alone)](#utf-8-validation-alone) - [Using the Parsed JSON](#using-the-parsed-json)
* [JSON Pointer](#json-pointer) - [Using the Parsed JSON: Additional examples](#using-the-parsed-json-additional-examples)
* [Error Handling](#error-handling) - [Minifying JSON strings without parsing](#minifying-json-strings-without-parsing)
* [Error Handling Example without Exceptions](#error-handling-examples-without-exceptions) - [UTF-8 validation (alone)](#utf-8-validation-alone)
* [Disabling Exceptions](#disabling-exceptions) - [JSON Pointer](#json-pointer)
* [Exceptions](#exceptions) - [Error Handling](#error-handling)
* [Current location in document](#current-location-in-document) - [Error Handling Examples without Exceptions](#error-handling-examples-without-exceptions)
* [Rewinding](#rewinding) - [Disabling Exceptions](#disabling-exceptions)
* [Direct Access to the Raw String](#direct-access-to-the-raw-string) - [Exceptions](#exceptions)
* [Newline-Delimited JSON (ndjson) and JSON lines](#newline-delimited-json-ndjson-and-json-lines) - [Current location in document](#current-location-in-document)
* [Parsing Numbers Inside Strings](#parsing-numbers-inside-strings) - [Rewinding](#rewinding)
* [Dynamic Number Types](#dynamic-number-types) - [Direct Access to the Raw String](#direct-access-to-the-raw-string)
* [Thread Safety](#thread-safety) - [Newline-Delimited JSON (ndjson) and JSON lines](#newline-delimited-json-ndjson-and-json-lines)
* [Standard Compliance](#standard-compliance) - [Parsing Numbers Inside Strings](#parsing-numbers-inside-strings)
- [Dynamic Number Types](#dynamic-number-types)
- [Raw Strings](#raw-strings)
- [Thread Safety](#thread-safety)
- [Standard Compliance](#standard-compliance)
- [Backwards Compatibility](#backwards-compatibility)
Requirements Requirements
@@ -240,27 +245,34 @@ transcode the UTF-8 strings produced by the simdjson library to other formats. S
Using the Parsed JSON Using the Parsed JSON
--------------------- ---------------------
We recommend that you first compile and run your code in Debug mode: under Visual Studio,
it means having the `_DEBUG` macro defined, and, for other compilers, it means leaving
We recommend that you first compile and run your code in Debug mode (with `NDEBUG` the `__OPTIMIZE__` macro undefined. The simdjson code will set `SIMDJSON_DEVELOPMENT_CHECKS=1`.
undefined). When you do so, the simdjson library runs additional sanity tests on Alternatively, you can set the macro `SIMDJSON_DEVELOPMENT_CHECKS` to 1 prior to including
your code to help ensure that you are using the library in a safe manner. Once the `simdjson.h` header to enable these additional checks: just make sure you remove the
your code has been tested, you can then run it in Release mode (with `NDEBUG` definition once your code has been tested. When `SIMDJSON_DEVELOPMENT_CHECKS` is set to 1, the
defined) for best performance. Alternatively, you can set the macro simdjson library runs additional (expensive) tests on your code to help ensure that you are
`SIMDJSON_DEVELOPMENT_CHECKS` to 1 prior to including the `simdjson.h` header using the library in a safe manner. Once your code has been tested, you can then run it in
to enable these additional checks: just make sure you remove the definition once your Release mode: under Visual Studio, it means having the `_DEBUG` macro undefined, and, for other
code has been tested. compilers, it means setting `__OPTIMIZE__` to a positive integer. You can also forcefully
disable these checks by setting `SIMDJSON_DEVELOPMENT_CHECKS` to 0. Once your code is tested, we
further encourage you to define `NDEBUG` in your Release builds to disable additional runtime
testing and get the best performance.
Once you have a document (`simdjson::ondemand::document`), you can navigate it with Once you have a document (`simdjson::ondemand::document`), you can navigate it with
idiomatic C++ iterators, operators and casts. Besides the document instances and idiomatic C++ iterators, operators and casts. Besides the document instances and
native types (`double`, `uint64_t`, `int64_t`, `bool`), we also access native types (`double`, `uint64_t`, `int64_t`, `bool`), we also access
Unicode (UTF-8) strings (`std::string_view`), objects (`simdjson::ondemand::object`) Unicode (UTF-8) strings (`std::string_view`), objects (`simdjson::ondemand::object`)
and arrays (`simdjson::ondemand::array`). and arrays (`simdjson::ondemand::array`).
We also have a generic type (`simdjson::ondemand::value`) which represents a potential We also have a generic ephemeral type (`simdjson::ondemand::value`) which represents a potential
array or object, or scalar type (`double`, `uint64_t`, `int64_t`, `bool`, `null`, string) inside array or object, or scalar type (`double`, `uint64_t`, `int64_t`, `bool`, `null`, string) inside
an array or an object. Both generic types (`simdjson::ondemand::document` and an array or an object. Both generic types (`simdjson::ondemand::document` and
`simdjson::ondemand::value`) have a `type()` method returning a `json_type` value describing the `simdjson::ondemand::value`) have a `type()` method returning a `json_type` value describing the
value (`json_type::array`, `json_type::object`, `json_type::number`, `json_type::string`, `json_type::boolean`, `json_type::null`). value (`json_type::array`, `json_type::object`, `json_type::number`, `json_type::string`,
`json_type::boolean`, `json_type::null`). A generic value (`simdjson::ondemand::value`)
is only valid temporarily, as soon as you access other values, other keys in objects, etc.
it becomes invalid: you should therefore consume the value immediately by converting it to a
scalar type, an array or an object.
Advanced users who need to determine the number types (integer or float) dynamically, Advanced users who need to determine the number types (integer or float) dynamically,
should review our section [dynamic number types](#dynamic-number-types). Indeed, should review our section [dynamic number types](#dynamic-number-types). Indeed,
@@ -295,10 +307,12 @@ support for users who avoid exceptions. See [the simdjson error handling documen
comparison. For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do comparison. For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do
not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. If you consume an not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. If you consume an
object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]`, your code object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]`, your code
is in error. Furthermore, you can only consume one field at a time, on the same object. Thus is in error. Furthermore, you can only consume one field at a time, on the same object. The
if you have retrieved `content["bids"].get_array()` and you later call `content["asks"].get_array()`, then the value instance you get from `content["bids"]` becomes invalid when you call `content["asks"]`.
first array should no longer be accessed: it would be unsafe to do so. You can detect such mistakes by first If you have retrieved `content["bids"].get_array()` and you later call
compiling and running the code in Debug mode: an OUT_OF_ORDER_ITERATION error is generated. `content["asks"].get_array()`, then the first array should no longer be accessed: it would be
unsafe to do so. You can detect such mistakes by first compiling and running the code in
Debug mode: an OUT_OF_ORDER_ITERATION error is generated.
> NOTE: JSON allows you to escape characters in keys. E.g., the key `"date"` may be written as > NOTE: JSON allows you to escape characters in keys. E.g., the key `"date"` may be written as
> `"\u0064\u0061\u0074\u0065"`. By default, simdjson does *not* unescape keys when matching by default. > `"\u0064\u0061\u0074\u0065"`. By default, simdjson does *not* unescape keys when matching by default.
@@ -419,7 +433,8 @@ support for users who avoid exceptions. See [the simdjson error handling documen
* **Counting elements in arrays:** Sometimes it is useful to scan an array to determine its length prior to parsing it. * **Counting elements in arrays:** Sometimes it is useful to scan an array to determine its length prior to parsing it.
For this purpose, `array` instances have a `count_elements` method. Users should be For this purpose, `array` instances have a `count_elements` method. Users should be
aware that the `count_elements` method can be costly since it requires scanning the aware that the `count_elements` method can be costly since it requires scanning the
whole array. You may use it as follows if your document is itself an array: whole array. You should only call `count_elements` as a last resort as it may
require scanning the document twice or more. You may use it as follows if your document is itself an array:
```C++ ```C++
auto cars_json = R"( [ 40.1, 39.9, 37.7, 40.4 ] )"_padded; auto cars_json = R"( [ 40.1, 39.9, 37.7, 40.4 ] )"_padded;
@@ -446,7 +461,8 @@ support for users who avoid exceptions. See [the simdjson error handling documen
parsing it. parsing it.
For this purpose, `object` instances have a `count_fields` method. Again, users should be For this purpose, `object` instances have a `count_fields` method. Again, users should be
aware that the `count_fields` method can be costly since it requires scanning the aware that the `count_fields` method can be costly since it requires scanning the
whole objects. You may use it as follows if your document is itself an object: whole objects. You should only call `count_fields` as a last resort as it may
require scanning the document twice or more. You may use it as follows if your document is itself an object:
```C++ ```C++
ondemand::parser parser; ondemand::parser parser;
@@ -467,7 +483,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
``` ```
* **Tree Walking and JSON Element Types:** Sometimes you don't necessarily have a document * **Tree Walking and JSON Element Types:** Sometimes you don't necessarily have a document
with a known type, and are trying to generically inspect or walk over JSON elements. To do that, you can use iterators and the `type()` method. You can also represent arbitrary JSON values with with a known type, and are trying to generically inspect or walk over JSON elements. To do that, you can use iterators and the `type()` method. You can also represent arbitrary JSON values with
`ondemand::value` instances: it can represent anything except a scalar document (lone number, string, null or Boolean). You can check for scalar documents with the method `scalar()`. `ondemand::value` instances: it can represent anything except a scalar document (lone number, string, null or Boolean). You can check for scalar documents with the method `scalar()`. You may also access [raw strings](#raw-strings).
For example, the following is a quick and dirty recursive function that verbosely prints the JSON document as JSON. This example also illustrates lifecycle requirements: the `document` instance holds the iterator. The document must remain in scope while you are accessing instances of `value`, `object` and `array`. For example, the following is a quick and dirty recursive function that verbosely prints the JSON document as JSON. This example also illustrates lifecycle requirements: the `document` instance holds the iterator. The document must remain in scope while you are accessing instances of `value`, `object` and `array`.
```c++ ```c++
void recursive_print_json(ondemand::value element) { void recursive_print_json(ondemand::value element) {
@@ -1445,6 +1461,42 @@ It will output:
9999999999999999999 negative: 0 is_integer: 1 large 64-bit integer: 9999999999999999999 large 64-bit integer: 9999999999999999999 9999999999999999999 negative: 0 is_integer: 1 large 64-bit integer: 9999999999999999999 large 64-bit integer: 9999999999999999999
``` ```
Raw Strings
-----------
It is sometimes useful to have access to a raw (unescaped) string: we make available a
minimalist `raw_json_string` data type which contains a pointer inside the string in the
original document, right after the quote. It is accessible via `get_raw_json_string()` on a
string instance and returned by the `key()` method on an object's field instance. It is always
optional: replacing `get_raw_json_string()` with `get_string()` and `key()` by
`unescaped_key()` returns an `string_view` instance of the unescaped string.
You can quickly compare a `raw_json_string` instance with a target string. You may also
unescape the `raw_json_string` on your own string buffer: `parser.unescape(mystr, ptr)`
advances the provided pointer `ptr` and returns a string_view instance on the newly serialized
string upon success, otherwise it returns an error. When unescaping to your own string buffer,
you should ensure that you have sufficient memory space: the total size of the strings plus
`simdjson::SIMDJSON_PADDING` bytes. The following example illustrates how we can unescape
JSON string to a user-provided buffer:
```C++
auto json = R"( {"name": "Jack The Ripper \u0033"} )"_padded;
// We create a buffer large enough to store all strings we need:
std::unique_ptr<uint8_t[]> buffer(new uint8_t[json.size() + simdjson::SIMDJSON_PADDING]);
uint8_t * ptr = buffer.get();
ondemand::parser parser;
ondemand::document doc = parser.iterate(json);
// We store our strings as 'string_view' instances in a vector:
std::vector<std::string_view> mystrings;
for (auto key_value : doc.get_object()) {
std::string_view keysv = parser.unescape(key_value.key(), ptr);// writes 'name'
mystrings.push_back(keysv);
std::string_view valuesv = parser.unescape(key_value.value().get_raw_json_string(), ptr);
// writes 'Jack The Ripper 3', escaping the \u0033
mystrings.push_back(valuesv);
}
```
Thread Safety Thread Safety
------------- -------------
+2 -1
View File
@@ -17,6 +17,7 @@ different version of the JSON parser for different CPU architectures, often with
algorithms to take better advantage of a given CPU! algorithms to take better advantage of a given CPU!
The current implementations are: The current implementations are:
* icelake: AVX-512F, AVX-512VBMI, etc.
* haswell: AVX2 (2013 Intel Haswell or later) * haswell: AVX2 (2013 Intel Haswell or later)
* westmere: SSE4.2 (2010 Westmere or later). * westmere: SSE4.2 (2010 Westmere or later).
* arm64: 64-bit ARMv8-A NEON * arm64: 64-bit ARMv8-A NEON
@@ -28,7 +29,7 @@ compiles *all* the implementations into the executable. On Intel, it will includ
(haswell, westmere and fallback), on ARM it will include 2 (arm64 and fallback), and on PPC it will include 2 (ppc64 and fallback). (haswell, westmere and fallback), on ARM it will include 2 (arm64 and fallback), and on PPC it will include 2 (ppc64 and fallback).
If you know more about where you're going to run and want to save the space, you can disable any of If you know more about where you're going to run and want to save the space, you can disable any of
these implementations at compile time with `-DSIMDJSON_IMPLEMENTATION_X=0` (where X is HASWELL, these implementations at compile time with `-DSIMDJSON_IMPLEMENTATION_X=0` (where X is ICELAKE, HASWELL,
WESTMERE, ARM64, PPC64 and FALLBACK). WESTMERE, ARM64, PPC64 and FALLBACK).
The simdjson library automatically sets header flags for each implementation as it compiles; there The simdjson library automatically sets header flags for each implementation as it compiles; there
+7 -7
View File
@@ -9,7 +9,7 @@ namespace {
// but the algorithms do not end up using the returned value. // but the algorithms do not end up using the returned value.
// Sadly, sanitizers are not smart enough to figure it out. // Sadly, sanitizers are not smart enough to figure it out.
SIMDJSON_NO_SANITIZE_UNDEFINED SIMDJSON_NO_SANITIZE_UNDEFINED
simdjson_really_inline int trailing_zeroes(uint64_t input_num) { simdjson_inline int trailing_zeroes(uint64_t input_num) {
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
unsigned long ret; unsigned long ret;
// Search the mask data from least significant bit (LSB) // Search the mask data from least significant bit (LSB)
@@ -22,12 +22,12 @@ simdjson_really_inline int trailing_zeroes(uint64_t input_num) {
} }
/* result might be undefined when input_num is zero */ /* result might be undefined when input_num is zero */
simdjson_really_inline uint64_t clear_lowest_bit(uint64_t input_num) { simdjson_inline uint64_t clear_lowest_bit(uint64_t input_num) {
return input_num & (input_num-1); return input_num & (input_num-1);
} }
/* result might be undefined when input_num is zero */ /* result might be undefined when input_num is zero */
simdjson_really_inline int leading_zeroes(uint64_t input_num) { simdjson_inline int leading_zeroes(uint64_t input_num) {
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
unsigned long leading_zero = 0; unsigned long leading_zero = 0;
// Search the mask data from most significant bit (MSB) // Search the mask data from most significant bit (MSB)
@@ -42,7 +42,7 @@ simdjson_really_inline int leading_zeroes(uint64_t input_num) {
} }
/* result might be undefined when input_num is zero */ /* result might be undefined when input_num is zero */
simdjson_really_inline int count_ones(uint64_t input_num) { simdjson_inline int count_ones(uint64_t input_num) {
return vaddv_u8(vcnt_u8(vcreate_u8(input_num))); return vaddv_u8(vcnt_u8(vcreate_u8(input_num)));
} }
@@ -61,7 +61,7 @@ simdjson_really_inline int count_ones(uint64_t input_num) {
#define SIMDJSON_PREFER_REVERSE_BITS 1 #define SIMDJSON_PREFER_REVERSE_BITS 1
/* reverse the bits */ /* reverse the bits */
simdjson_really_inline uint64_t reverse_bits(uint64_t input_num) { simdjson_inline uint64_t reverse_bits(uint64_t input_num) {
uint64_t rev_bits; uint64_t rev_bits;
__asm("rbit %0, %1" : "=r"(rev_bits) : "r"(input_num)); __asm("rbit %0, %1" : "=r"(rev_bits) : "r"(input_num));
return rev_bits; return rev_bits;
@@ -74,13 +74,13 @@ simdjson_really_inline uint64_t reverse_bits(uint64_t input_num) {
* of such undefined behavior is never used. * of such undefined behavior is never used.
**/ **/
SIMDJSON_NO_SANITIZE_UNDEFINED SIMDJSON_NO_SANITIZE_UNDEFINED
simdjson_really_inline uint64_t zero_leading_bit(uint64_t rev_bits, int leading_zeroes) { simdjson_inline uint64_t zero_leading_bit(uint64_t rev_bits, int leading_zeroes) {
return rev_bits ^ (uint64_t(0x8000000000000000) >> leading_zeroes); return rev_bits ^ (uint64_t(0x8000000000000000) >> leading_zeroes);
} }
#endif #endif
simdjson_really_inline bool add_overflow(uint64_t value1, uint64_t value2, uint64_t *result) { simdjson_inline bool add_overflow(uint64_t value1, uint64_t value2, uint64_t *result) {
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
*result = value1 + value2; *result = value1 + value2;
return *result < value1; return *result < value1;
+1 -1
View File
@@ -10,7 +10,7 @@ namespace {
// //
// For example, prefix_xor(00100100) == 00011100 // For example, prefix_xor(00100100) == 00011100
// //
simdjson_really_inline uint64_t prefix_xor(uint64_t bitmask) { simdjson_inline uint64_t prefix_xor(uint64_t bitmask) {
///////////// /////////////
// We could do this with PMULL, but it is apparently slow. // We could do this with PMULL, but it is apparently slow.
// //
+1 -1
View File
@@ -14,7 +14,7 @@ using namespace simdjson::dom;
class implementation final : public simdjson::implementation { class implementation final : public simdjson::implementation {
public: public:
simdjson_really_inline implementation() : simdjson::implementation("arm64", "ARM NEON", internal::instruction_set::NEON) {} simdjson_inline implementation() : simdjson::implementation("arm64", "ARM NEON", internal::instruction_set::NEON) {}
simdjson_warn_unused error_code create_dom_parser_implementation( simdjson_warn_unused error_code create_dom_parser_implementation(
size_t capacity, size_t capacity,
size_t max_length, size_t max_length,
+2
View File
@@ -5,4 +5,6 @@
// you use visual studio or other compilers. // you use visual studio or other compilers.
#include <arm_neon.h> #include <arm_neon.h>
static_assert(sizeof(uint8x16_t) <= simdjson::SIMDJSON_PADDING, "insufficient padding for arm64");
#endif // SIMDJSON_ARM64_INTRINSICS_H #endif // SIMDJSON_ARM64_INTRINSICS_H
+1 -1
View File
@@ -7,7 +7,7 @@ namespace {
// we don't have SSE, so let us use a scalar function // we don't have SSE, so let us use a scalar function
// credit: https://johnnylee-sde.github.io/Fast-numeric-string-to-int/ // credit: https://johnnylee-sde.github.io/Fast-numeric-string-to-int/
static simdjson_really_inline uint32_t parse_eight_digits_unrolled(const uint8_t *chars) { static simdjson_inline uint32_t parse_eight_digits_unrolled(const uint8_t *chars) {
uint64_t val; uint64_t val;
std::memcpy(&val, chars, sizeof(uint64_t)); std::memcpy(&val, chars, sizeof(uint64_t));
val = (val & 0x0F0F0F0F0F0F0F0F) * 2561 >> 8; val = (val & 0x0F0F0F0F0F0F0F0F) * 2561 >> 8;
+95 -95
View File
@@ -28,7 +28,7 @@ namespace {
* You should not use this function except for compile-time constants: * You should not use this function except for compile-time constants:
* it is not efficient. * it is not efficient.
*/ */
simdjson_really_inline uint8x16_t make_uint8x16_t(uint8_t x1, uint8_t x2, uint8_t x3, uint8_t x4, simdjson_inline uint8x16_t make_uint8x16_t(uint8_t x1, uint8_t x2, uint8_t x3, uint8_t x4,
uint8_t x5, uint8_t x6, uint8_t x7, uint8_t x8, uint8_t x5, uint8_t x6, uint8_t x7, uint8_t x8,
uint8_t x9, uint8_t x10, uint8_t x11, uint8_t x12, uint8_t x9, uint8_t x10, uint8_t x11, uint8_t x12,
uint8_t x13, uint8_t x14, uint8_t x15, uint8_t x16) { uint8_t x13, uint8_t x14, uint8_t x15, uint8_t x16) {
@@ -57,7 +57,7 @@ simdjson_really_inline uint8x16_t make_uint8x16_t(uint8_t x1, uint8_t x2, uint
return x; return x;
} }
simdjson_really_inline uint8x8_t make_uint8x8_t(uint8_t x1, uint8_t x2, uint8_t x3, uint8_t x4, simdjson_inline uint8x8_t make_uint8x8_t(uint8_t x1, uint8_t x2, uint8_t x3, uint8_t x4,
uint8_t x5, uint8_t x6, uint8_t x7, uint8_t x8) { uint8_t x5, uint8_t x6, uint8_t x7, uint8_t x8) {
uint8x8_t x{}; uint8x8_t x{};
x = vset_lane_u8(x1, x, 0); x = vset_lane_u8(x1, x, 0);
@@ -72,7 +72,7 @@ simdjson_really_inline uint8x8_t make_uint8x8_t(uint8_t x1, uint8_t x2, uint8_
} }
// We have to do the same work for make_int8x16_t // We have to do the same work for make_int8x16_t
simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x3, int8_t x4, simdjson_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x3, int8_t x4,
int8_t x5, int8_t x6, int8_t x7, int8_t x8, int8_t x5, int8_t x6, int8_t x7, int8_t x8,
int8_t x9, int8_t x10, int8_t x11, int8_t x12, int8_t x9, int8_t x10, int8_t x11, int8_t x12,
int8_t x13, int8_t x14, int8_t x15, int8_t x16) { int8_t x13, int8_t x14, int8_t x15, int8_t x16) {
@@ -118,24 +118,24 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
static const int SIZE = sizeof(value); static const int SIZE = sizeof(value);
// Conversion from/to SIMD register // Conversion from/to SIMD register
simdjson_really_inline base_u8(const uint8x16_t _value) : value(_value) {} simdjson_inline base_u8(const uint8x16_t _value) : value(_value) {}
simdjson_really_inline operator const uint8x16_t&() const { return this->value; } simdjson_inline operator const uint8x16_t&() const { return this->value; }
simdjson_really_inline operator uint8x16_t&() { return this->value; } simdjson_inline operator uint8x16_t&() { return this->value; }
// Bit operations // Bit operations
simdjson_really_inline simd8<T> operator|(const simd8<T> other) const { return vorrq_u8(*this, other); } simdjson_inline simd8<T> operator|(const simd8<T> other) const { return vorrq_u8(*this, other); }
simdjson_really_inline simd8<T> operator&(const simd8<T> other) const { return vandq_u8(*this, other); } simdjson_inline simd8<T> operator&(const simd8<T> other) const { return vandq_u8(*this, other); }
simdjson_really_inline simd8<T> operator^(const simd8<T> other) const { return veorq_u8(*this, other); } simdjson_inline simd8<T> operator^(const simd8<T> other) const { return veorq_u8(*this, other); }
simdjson_really_inline simd8<T> bit_andnot(const simd8<T> other) const { return vbicq_u8(*this, other); } simdjson_inline simd8<T> bit_andnot(const simd8<T> other) const { return vbicq_u8(*this, other); }
simdjson_really_inline simd8<T> operator~() const { return *this ^ 0xFFu; } simdjson_inline simd8<T> operator~() const { return *this ^ 0xFFu; }
simdjson_really_inline simd8<T>& operator|=(const simd8<T> other) { auto this_cast = static_cast<simd8<T>*>(this); *this_cast = *this_cast | other; return *this_cast; } simdjson_inline simd8<T>& operator|=(const simd8<T> other) { auto this_cast = static_cast<simd8<T>*>(this); *this_cast = *this_cast | other; return *this_cast; }
simdjson_really_inline simd8<T>& operator&=(const simd8<T> other) { auto this_cast = static_cast<simd8<T>*>(this); *this_cast = *this_cast & other; return *this_cast; } simdjson_inline simd8<T>& operator&=(const simd8<T> other) { auto this_cast = static_cast<simd8<T>*>(this); *this_cast = *this_cast & other; return *this_cast; }
simdjson_really_inline simd8<T>& operator^=(const simd8<T> other) { auto this_cast = static_cast<simd8<T>*>(this); *this_cast = *this_cast ^ other; return *this_cast; } simdjson_inline simd8<T>& operator^=(const simd8<T> other) { auto this_cast = static_cast<simd8<T>*>(this); *this_cast = *this_cast ^ other; return *this_cast; }
friend simdjson_really_inline Mask operator==(const simd8<T> lhs, const simd8<T> rhs) { return vceqq_u8(lhs, rhs); } friend simdjson_inline Mask operator==(const simd8<T> lhs, const simd8<T> rhs) { return vceqq_u8(lhs, rhs); }
template<int N=1> template<int N=1>
simdjson_really_inline simd8<T> prev(const simd8<T> prev_chunk) const { simdjson_inline simd8<T> prev(const simd8<T> prev_chunk) const {
return vextq_u8(prev_chunk, *this, 16 - N); return vextq_u8(prev_chunk, *this, 16 - N);
} }
}; };
@@ -146,17 +146,17 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
typedef uint16_t bitmask_t; typedef uint16_t bitmask_t;
typedef uint32_t bitmask2_t; typedef uint32_t bitmask2_t;
static simdjson_really_inline simd8<bool> splat(bool _value) { return vmovq_n_u8(uint8_t(-(!!_value))); } static simdjson_inline simd8<bool> splat(bool _value) { return vmovq_n_u8(uint8_t(-(!!_value))); }
simdjson_really_inline simd8(const uint8x16_t _value) : base_u8<bool>(_value) {} simdjson_inline simd8(const uint8x16_t _value) : base_u8<bool>(_value) {}
// False constructor // False constructor
simdjson_really_inline simd8() : simd8(vdupq_n_u8(0)) {} simdjson_inline simd8() : simd8(vdupq_n_u8(0)) {}
// Splat constructor // Splat constructor
simdjson_really_inline simd8(bool _value) : simd8(splat(_value)) {} simdjson_inline simd8(bool _value) : simd8(splat(_value)) {}
// We return uint32_t instead of uint16_t because that seems to be more efficient for most // We return uint32_t instead of uint16_t because that seems to be more efficient for most
// purposes (cutting it down to uint16_t costs performance in some compilers). // purposes (cutting it down to uint16_t costs performance in some compilers).
simdjson_really_inline uint32_t to_bitmask() const { simdjson_inline uint32_t to_bitmask() const {
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
const uint8x16_t bit_mask = make_uint8x16_t(0x01, 0x02, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80, const uint8x16_t bit_mask = make_uint8x16_t(0x01, 0x02, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80,
0x01, 0x02, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80); 0x01, 0x02, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80);
@@ -170,26 +170,26 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
tmp = vpaddq_u8(tmp, tmp); tmp = vpaddq_u8(tmp, tmp);
return vgetq_lane_u16(vreinterpretq_u16_u8(tmp), 0); return vgetq_lane_u16(vreinterpretq_u16_u8(tmp), 0);
} }
simdjson_really_inline bool any() const { return vmaxvq_u8(*this) != 0; } simdjson_inline bool any() const { return vmaxvq_u8(*this) != 0; }
}; };
// Unsigned bytes // Unsigned bytes
template<> template<>
struct simd8<uint8_t>: base_u8<uint8_t> { struct simd8<uint8_t>: base_u8<uint8_t> {
static simdjson_really_inline uint8x16_t splat(uint8_t _value) { return vmovq_n_u8(_value); } static simdjson_inline uint8x16_t splat(uint8_t _value) { return vmovq_n_u8(_value); }
static simdjson_really_inline uint8x16_t zero() { return vdupq_n_u8(0); } static simdjson_inline uint8x16_t zero() { return vdupq_n_u8(0); }
static simdjson_really_inline uint8x16_t load(const uint8_t* values) { return vld1q_u8(values); } static simdjson_inline uint8x16_t load(const uint8_t* values) { return vld1q_u8(values); }
simdjson_really_inline simd8(const uint8x16_t _value) : base_u8<uint8_t>(_value) {} simdjson_inline simd8(const uint8x16_t _value) : base_u8<uint8_t>(_value) {}
// Zero constructor // Zero constructor
simdjson_really_inline simd8() : simd8(zero()) {} simdjson_inline simd8() : simd8(zero()) {}
// Array constructor // Array constructor
simdjson_really_inline simd8(const uint8_t values[16]) : simd8(load(values)) {} simdjson_inline simd8(const uint8_t values[16]) : simd8(load(values)) {}
// Splat constructor // Splat constructor
simdjson_really_inline simd8(uint8_t _value) : simd8(splat(_value)) {} simdjson_inline simd8(uint8_t _value) : simd8(splat(_value)) {}
// Member-by-member initialization // Member-by-member initialization
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
simdjson_really_inline simd8( simdjson_inline simd8(
uint8_t v0, uint8_t v1, uint8_t v2, uint8_t v3, uint8_t v4, uint8_t v5, uint8_t v6, uint8_t v7, uint8_t v0, uint8_t v1, uint8_t v2, uint8_t v3, uint8_t v4, uint8_t v5, uint8_t v6, uint8_t v7,
uint8_t v8, uint8_t v9, uint8_t v10, uint8_t v11, uint8_t v12, uint8_t v13, uint8_t v14, uint8_t v15 uint8_t v8, uint8_t v9, uint8_t v10, uint8_t v11, uint8_t v12, uint8_t v13, uint8_t v14, uint8_t v15
) : simd8(make_uint8x16_t( ) : simd8(make_uint8x16_t(
@@ -197,7 +197,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
v8, v9, v10,v11,v12,v13,v14,v15 v8, v9, v10,v11,v12,v13,v14,v15
)) {} )) {}
#else #else
simdjson_really_inline simd8( simdjson_inline simd8(
uint8_t v0, uint8_t v1, uint8_t v2, uint8_t v3, uint8_t v4, uint8_t v5, uint8_t v6, uint8_t v7, uint8_t v0, uint8_t v1, uint8_t v2, uint8_t v3, uint8_t v4, uint8_t v5, uint8_t v6, uint8_t v7,
uint8_t v8, uint8_t v9, uint8_t v10, uint8_t v11, uint8_t v12, uint8_t v13, uint8_t v14, uint8_t v15 uint8_t v8, uint8_t v9, uint8_t v10, uint8_t v11, uint8_t v12, uint8_t v13, uint8_t v14, uint8_t v15
) : simd8(uint8x16_t{ ) : simd8(uint8x16_t{
@@ -207,7 +207,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
#endif #endif
// Repeat 16 values as many times as necessary (usually for lookup tables) // Repeat 16 values as many times as necessary (usually for lookup tables)
simdjson_really_inline static simd8<uint8_t> repeat_16( simdjson_inline static simd8<uint8_t> repeat_16(
uint8_t v0, uint8_t v1, uint8_t v2, uint8_t v3, uint8_t v4, uint8_t v5, uint8_t v6, uint8_t v7, uint8_t v0, uint8_t v1, uint8_t v2, uint8_t v3, uint8_t v4, uint8_t v5, uint8_t v6, uint8_t v7,
uint8_t v8, uint8_t v9, uint8_t v10, uint8_t v11, uint8_t v12, uint8_t v13, uint8_t v14, uint8_t v15 uint8_t v8, uint8_t v9, uint8_t v10, uint8_t v11, uint8_t v12, uint8_t v13, uint8_t v14, uint8_t v15
) { ) {
@@ -218,44 +218,44 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
} }
// Store to array // Store to array
simdjson_really_inline void store(uint8_t dst[16]) const { return vst1q_u8(dst, *this); } simdjson_inline void store(uint8_t dst[16]) const { return vst1q_u8(dst, *this); }
// Saturated math // Saturated math
simdjson_really_inline simd8<uint8_t> saturating_add(const simd8<uint8_t> other) const { return vqaddq_u8(*this, other); } simdjson_inline simd8<uint8_t> saturating_add(const simd8<uint8_t> other) const { return vqaddq_u8(*this, other); }
simdjson_really_inline simd8<uint8_t> saturating_sub(const simd8<uint8_t> other) const { return vqsubq_u8(*this, other); } simdjson_inline simd8<uint8_t> saturating_sub(const simd8<uint8_t> other) const { return vqsubq_u8(*this, other); }
// Addition/subtraction are the same for signed and unsigned // Addition/subtraction are the same for signed and unsigned
simdjson_really_inline simd8<uint8_t> operator+(const simd8<uint8_t> other) const { return vaddq_u8(*this, other); } simdjson_inline simd8<uint8_t> operator+(const simd8<uint8_t> other) const { return vaddq_u8(*this, other); }
simdjson_really_inline simd8<uint8_t> operator-(const simd8<uint8_t> other) const { return vsubq_u8(*this, other); } simdjson_inline simd8<uint8_t> operator-(const simd8<uint8_t> other) const { return vsubq_u8(*this, other); }
simdjson_really_inline simd8<uint8_t>& operator+=(const simd8<uint8_t> other) { *this = *this + other; return *this; } simdjson_inline simd8<uint8_t>& operator+=(const simd8<uint8_t> other) { *this = *this + other; return *this; }
simdjson_really_inline simd8<uint8_t>& operator-=(const simd8<uint8_t> other) { *this = *this - other; return *this; } simdjson_inline simd8<uint8_t>& operator-=(const simd8<uint8_t> other) { *this = *this - other; return *this; }
// Order-specific operations // Order-specific operations
simdjson_really_inline uint8_t max_val() const { return vmaxvq_u8(*this); } simdjson_inline uint8_t max_val() const { return vmaxvq_u8(*this); }
simdjson_really_inline uint8_t min_val() const { return vminvq_u8(*this); } simdjson_inline uint8_t min_val() const { return vminvq_u8(*this); }
simdjson_really_inline simd8<uint8_t> max_val(const simd8<uint8_t> other) const { return vmaxq_u8(*this, other); } simdjson_inline simd8<uint8_t> max_val(const simd8<uint8_t> other) const { return vmaxq_u8(*this, other); }
simdjson_really_inline simd8<uint8_t> min_val(const simd8<uint8_t> other) const { return vminq_u8(*this, other); } simdjson_inline simd8<uint8_t> min_val(const simd8<uint8_t> other) const { return vminq_u8(*this, other); }
simdjson_really_inline simd8<bool> operator<=(const simd8<uint8_t> other) const { return vcleq_u8(*this, other); } simdjson_inline simd8<bool> operator<=(const simd8<uint8_t> other) const { return vcleq_u8(*this, other); }
simdjson_really_inline simd8<bool> operator>=(const simd8<uint8_t> other) const { return vcgeq_u8(*this, other); } simdjson_inline simd8<bool> operator>=(const simd8<uint8_t> other) const { return vcgeq_u8(*this, other); }
simdjson_really_inline simd8<bool> operator<(const simd8<uint8_t> other) const { return vcltq_u8(*this, other); } simdjson_inline simd8<bool> operator<(const simd8<uint8_t> other) const { return vcltq_u8(*this, other); }
simdjson_really_inline simd8<bool> operator>(const simd8<uint8_t> other) const { return vcgtq_u8(*this, other); } simdjson_inline simd8<bool> operator>(const simd8<uint8_t> other) const { return vcgtq_u8(*this, other); }
// Same as >, but instead of guaranteeing all 1's == true, false = 0 and true = nonzero. For ARM, returns all 1's. // Same as >, but instead of guaranteeing all 1's == true, false = 0 and true = nonzero. For ARM, returns all 1's.
simdjson_really_inline simd8<uint8_t> gt_bits(const simd8<uint8_t> other) const { return simd8<uint8_t>(*this > other); } simdjson_inline simd8<uint8_t> gt_bits(const simd8<uint8_t> other) const { return simd8<uint8_t>(*this > other); }
// Same as <, but instead of guaranteeing all 1's == true, false = 0 and true = nonzero. For ARM, returns all 1's. // Same as <, but instead of guaranteeing all 1's == true, false = 0 and true = nonzero. For ARM, returns all 1's.
simdjson_really_inline simd8<uint8_t> lt_bits(const simd8<uint8_t> other) const { return simd8<uint8_t>(*this < other); } simdjson_inline simd8<uint8_t> lt_bits(const simd8<uint8_t> other) const { return simd8<uint8_t>(*this < other); }
// Bit-specific operations // Bit-specific operations
simdjson_really_inline simd8<bool> any_bits_set(simd8<uint8_t> bits) const { return vtstq_u8(*this, bits); } simdjson_inline simd8<bool> any_bits_set(simd8<uint8_t> bits) const { return vtstq_u8(*this, bits); }
simdjson_really_inline bool any_bits_set_anywhere() const { return this->max_val() != 0; } simdjson_inline bool any_bits_set_anywhere() const { return this->max_val() != 0; }
simdjson_really_inline bool any_bits_set_anywhere(simd8<uint8_t> bits) const { return (*this & bits).any_bits_set_anywhere(); } simdjson_inline bool any_bits_set_anywhere(simd8<uint8_t> bits) const { return (*this & bits).any_bits_set_anywhere(); }
template<int N> template<int N>
simdjson_really_inline simd8<uint8_t> shr() const { return vshrq_n_u8(*this, N); } simdjson_inline simd8<uint8_t> shr() const { return vshrq_n_u8(*this, N); }
template<int N> template<int N>
simdjson_really_inline simd8<uint8_t> shl() const { return vshlq_n_u8(*this, N); } simdjson_inline simd8<uint8_t> shl() const { return vshlq_n_u8(*this, N); }
// Perform a lookup assuming the value is between 0 and 16 (undefined behavior for out of range values) // Perform a lookup assuming the value is between 0 and 16 (undefined behavior for out of range values)
template<typename L> template<typename L>
simdjson_really_inline simd8<L> lookup_16(simd8<L> lookup_table) const { simdjson_inline simd8<L> lookup_16(simd8<L> lookup_table) const {
return lookup_table.apply_lookup_16_to(*this); return lookup_table.apply_lookup_16_to(*this);
} }
@@ -268,7 +268,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
// signature simd8<L> compress(uint16_t mask) would be // signature simd8<L> compress(uint16_t mask) would be
// sensible, but the AVX ISA makes this kind of approach difficult. // sensible, but the AVX ISA makes this kind of approach difficult.
template<typename L> template<typename L>
simdjson_really_inline void compress(uint16_t mask, L * output) const { simdjson_inline void compress(uint16_t mask, L * output) const {
using internal::thintable_epi8; using internal::thintable_epi8;
using internal::BitsSetTable256mul2; using internal::BitsSetTable256mul2;
using internal::pshufb_combine_table; using internal::pshufb_combine_table;
@@ -305,7 +305,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
// Copies all bytes corresponding to a 0 in the low half of the mask (interpreted as a // Copies all bytes corresponding to a 0 in the low half of the mask (interpreted as a
// bitset) to output1, then those corresponding to a 0 in the high half to output2. // bitset) to output1, then those corresponding to a 0 in the high half to output2.
template<typename L> template<typename L>
simdjson_really_inline void compress_halves(uint16_t mask, L *output1, L *output2) const { simdjson_inline void compress_halves(uint16_t mask, L *output1, L *output2) const {
using internal::thintable_epi8; using internal::thintable_epi8;
uint8_t mask1 = uint8_t(mask); // least significant 8 bits uint8_t mask1 = uint8_t(mask); // least significant 8 bits
uint8_t mask2 = uint8_t(mask >> 8); // most significant 8 bits uint8_t mask2 = uint8_t(mask >> 8); // most significant 8 bits
@@ -324,7 +324,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
} }
template<typename L> template<typename L>
simdjson_really_inline simd8<L> lookup_16( simdjson_inline simd8<L> lookup_16(
L replace0, L replace1, L replace2, L replace3, L replace0, L replace1, L replace2, L replace3,
L replace4, L replace5, L replace6, L replace7, L replace4, L replace5, L replace6, L replace7,
L replace8, L replace9, L replace10, L replace11, L replace8, L replace9, L replace10, L replace11,
@@ -338,7 +338,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
} }
template<typename T> template<typename T>
simdjson_really_inline simd8<uint8_t> apply_lookup_16_to(const simd8<T> original) { simdjson_inline simd8<uint8_t> apply_lookup_16_to(const simd8<T> original) {
return vqtbl1q_u8(*this, simd8<uint8_t>(original)); return vqtbl1q_u8(*this, simd8<uint8_t>(original));
} }
}; };
@@ -348,24 +348,24 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
struct simd8<int8_t> { struct simd8<int8_t> {
int8x16_t value; int8x16_t value;
static simdjson_really_inline simd8<int8_t> splat(int8_t _value) { return vmovq_n_s8(_value); } static simdjson_inline simd8<int8_t> splat(int8_t _value) { return vmovq_n_s8(_value); }
static simdjson_really_inline simd8<int8_t> zero() { return vdupq_n_s8(0); } static simdjson_inline simd8<int8_t> zero() { return vdupq_n_s8(0); }
static simdjson_really_inline simd8<int8_t> load(const int8_t values[16]) { return vld1q_s8(values); } static simdjson_inline simd8<int8_t> load(const int8_t values[16]) { return vld1q_s8(values); }
// Conversion from/to SIMD register // Conversion from/to SIMD register
simdjson_really_inline simd8(const int8x16_t _value) : value{_value} {} simdjson_inline simd8(const int8x16_t _value) : value{_value} {}
simdjson_really_inline operator const int8x16_t&() const { return this->value; } simdjson_inline operator const int8x16_t&() const { return this->value; }
simdjson_really_inline operator int8x16_t&() { return this->value; } simdjson_inline operator int8x16_t&() { return this->value; }
// Zero constructor // Zero constructor
simdjson_really_inline simd8() : simd8(zero()) {} simdjson_inline simd8() : simd8(zero()) {}
// Splat constructor // Splat constructor
simdjson_really_inline simd8(int8_t _value) : simd8(splat(_value)) {} simdjson_inline simd8(int8_t _value) : simd8(splat(_value)) {}
// Array constructor // Array constructor
simdjson_really_inline simd8(const int8_t* values) : simd8(load(values)) {} simdjson_inline simd8(const int8_t* values) : simd8(load(values)) {}
// Member-by-member initialization // Member-by-member initialization
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
simdjson_really_inline simd8( simdjson_inline simd8(
int8_t v0, int8_t v1, int8_t v2, int8_t v3, int8_t v4, int8_t v5, int8_t v6, int8_t v7, int8_t v0, int8_t v1, int8_t v2, int8_t v3, int8_t v4, int8_t v5, int8_t v6, int8_t v7,
int8_t v8, int8_t v9, int8_t v10, int8_t v11, int8_t v12, int8_t v13, int8_t v14, int8_t v15 int8_t v8, int8_t v9, int8_t v10, int8_t v11, int8_t v12, int8_t v13, int8_t v14, int8_t v15
) : simd8(make_int8x16_t( ) : simd8(make_int8x16_t(
@@ -373,7 +373,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
v8, v9, v10,v11,v12,v13,v14,v15 v8, v9, v10,v11,v12,v13,v14,v15
)) {} )) {}
#else #else
simdjson_really_inline simd8( simdjson_inline simd8(
int8_t v0, int8_t v1, int8_t v2, int8_t v3, int8_t v4, int8_t v5, int8_t v6, int8_t v7, int8_t v0, int8_t v1, int8_t v2, int8_t v3, int8_t v4, int8_t v5, int8_t v6, int8_t v7,
int8_t v8, int8_t v9, int8_t v10, int8_t v11, int8_t v12, int8_t v13, int8_t v14, int8_t v15 int8_t v8, int8_t v9, int8_t v10, int8_t v11, int8_t v12, int8_t v13, int8_t v14, int8_t v15
) : simd8(int8x16_t{ ) : simd8(int8x16_t{
@@ -382,7 +382,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
}) {} }) {}
#endif #endif
// Repeat 16 values as many times as necessary (usually for lookup tables) // Repeat 16 values as many times as necessary (usually for lookup tables)
simdjson_really_inline static simd8<int8_t> repeat_16( simdjson_inline static simd8<int8_t> repeat_16(
int8_t v0, int8_t v1, int8_t v2, int8_t v3, int8_t v4, int8_t v5, int8_t v6, int8_t v7, int8_t v0, int8_t v1, int8_t v2, int8_t v3, int8_t v4, int8_t v5, int8_t v6, int8_t v7,
int8_t v8, int8_t v9, int8_t v10, int8_t v11, int8_t v12, int8_t v13, int8_t v14, int8_t v15 int8_t v8, int8_t v9, int8_t v10, int8_t v11, int8_t v12, int8_t v13, int8_t v14, int8_t v15
) { ) {
@@ -393,7 +393,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
} }
// Store to array // Store to array
simdjson_really_inline void store(int8_t dst[16]) const { return vst1q_s8(dst, *this); } simdjson_inline void store(int8_t dst[16]) const { return vst1q_s8(dst, *this); }
// Explicit conversion to/from unsigned // Explicit conversion to/from unsigned
// //
@@ -401,35 +401,35 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
// In theory, we could check this occurrence with std::same_as and std::enabled_if but it is C++14 // In theory, we could check this occurrence with std::same_as and std::enabled_if but it is C++14
// and relatively ugly and hard to read. // and relatively ugly and hard to read.
#ifndef SIMDJSON_REGULAR_VISUAL_STUDIO #ifndef SIMDJSON_REGULAR_VISUAL_STUDIO
simdjson_really_inline explicit simd8(const uint8x16_t other): simd8(vreinterpretq_s8_u8(other)) {} simdjson_inline explicit simd8(const uint8x16_t other): simd8(vreinterpretq_s8_u8(other)) {}
#endif #endif
simdjson_really_inline explicit operator simd8<uint8_t>() const { return vreinterpretq_u8_s8(this->value); } simdjson_inline explicit operator simd8<uint8_t>() const { return vreinterpretq_u8_s8(this->value); }
// Math // Math
simdjson_really_inline simd8<int8_t> operator+(const simd8<int8_t> other) const { return vaddq_s8(*this, other); } simdjson_inline simd8<int8_t> operator+(const simd8<int8_t> other) const { return vaddq_s8(*this, other); }
simdjson_really_inline simd8<int8_t> operator-(const simd8<int8_t> other) const { return vsubq_s8(*this, other); } simdjson_inline simd8<int8_t> operator-(const simd8<int8_t> other) const { return vsubq_s8(*this, other); }
simdjson_really_inline simd8<int8_t>& operator+=(const simd8<int8_t> other) { *this = *this + other; return *this; } simdjson_inline simd8<int8_t>& operator+=(const simd8<int8_t> other) { *this = *this + other; return *this; }
simdjson_really_inline simd8<int8_t>& operator-=(const simd8<int8_t> other) { *this = *this - other; return *this; } simdjson_inline simd8<int8_t>& operator-=(const simd8<int8_t> other) { *this = *this - other; return *this; }
// Order-sensitive comparisons // Order-sensitive comparisons
simdjson_really_inline simd8<int8_t> max_val(const simd8<int8_t> other) const { return vmaxq_s8(*this, other); } simdjson_inline simd8<int8_t> max_val(const simd8<int8_t> other) const { return vmaxq_s8(*this, other); }
simdjson_really_inline simd8<int8_t> min_val(const simd8<int8_t> other) const { return vminq_s8(*this, other); } simdjson_inline simd8<int8_t> min_val(const simd8<int8_t> other) const { return vminq_s8(*this, other); }
simdjson_really_inline simd8<bool> operator>(const simd8<int8_t> other) const { return vcgtq_s8(*this, other); } simdjson_inline simd8<bool> operator>(const simd8<int8_t> other) const { return vcgtq_s8(*this, other); }
simdjson_really_inline simd8<bool> operator<(const simd8<int8_t> other) const { return vcltq_s8(*this, other); } simdjson_inline simd8<bool> operator<(const simd8<int8_t> other) const { return vcltq_s8(*this, other); }
simdjson_really_inline simd8<bool> operator==(const simd8<int8_t> other) const { return vceqq_s8(*this, other); } simdjson_inline simd8<bool> operator==(const simd8<int8_t> other) const { return vceqq_s8(*this, other); }
template<int N=1> template<int N=1>
simdjson_really_inline simd8<int8_t> prev(const simd8<int8_t> prev_chunk) const { simdjson_inline simd8<int8_t> prev(const simd8<int8_t> prev_chunk) const {
return vextq_s8(prev_chunk, *this, 16 - N); return vextq_s8(prev_chunk, *this, 16 - N);
} }
// Perform a lookup assuming no value is larger than 16 // Perform a lookup assuming no value is larger than 16
template<typename L> template<typename L>
simdjson_really_inline simd8<L> lookup_16(simd8<L> lookup_table) const { simdjson_inline simd8<L> lookup_16(simd8<L> lookup_table) const {
return lookup_table.apply_lookup_16_to(*this); return lookup_table.apply_lookup_16_to(*this);
} }
template<typename L> template<typename L>
simdjson_really_inline simd8<L> lookup_16( simdjson_inline simd8<L> lookup_16(
L replace0, L replace1, L replace2, L replace3, L replace0, L replace1, L replace2, L replace3,
L replace4, L replace5, L replace6, L replace7, L replace4, L replace5, L replace6, L replace7,
L replace8, L replace9, L replace10, L replace11, L replace8, L replace9, L replace10, L replace11,
@@ -443,7 +443,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
} }
template<typename T> template<typename T>
simdjson_really_inline simd8<int8_t> apply_lookup_16_to(const simd8<T> original) { simdjson_inline simd8<int8_t> apply_lookup_16_to(const simd8<T> original) {
return vqtbl1q_s8(*this, simd8<uint8_t>(original)); return vqtbl1q_s8(*this, simd8<uint8_t>(original));
} }
}; };
@@ -458,22 +458,22 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
simd8x64<T>& operator=(const simd8<T>& other) = delete; // no assignment allowed simd8x64<T>& operator=(const simd8<T>& other) = delete; // no assignment allowed
simd8x64() = delete; // no default constructor allowed simd8x64() = delete; // no default constructor allowed
simdjson_really_inline simd8x64(const simd8<T> chunk0, const simd8<T> chunk1, const simd8<T> chunk2, const simd8<T> chunk3) : chunks{chunk0, chunk1, chunk2, chunk3} {} simdjson_inline simd8x64(const simd8<T> chunk0, const simd8<T> chunk1, const simd8<T> chunk2, const simd8<T> chunk3) : chunks{chunk0, chunk1, chunk2, chunk3} {}
simdjson_really_inline simd8x64(const T ptr[64]) : chunks{simd8<T>::load(ptr), simd8<T>::load(ptr+16), simd8<T>::load(ptr+32), simd8<T>::load(ptr+48)} {} simdjson_inline simd8x64(const T ptr[64]) : chunks{simd8<T>::load(ptr), simd8<T>::load(ptr+16), simd8<T>::load(ptr+32), simd8<T>::load(ptr+48)} {}
simdjson_really_inline void store(T ptr[64]) const { simdjson_inline void store(T ptr[64]) const {
this->chunks[0].store(ptr+sizeof(simd8<T>)*0); this->chunks[0].store(ptr+sizeof(simd8<T>)*0);
this->chunks[1].store(ptr+sizeof(simd8<T>)*1); this->chunks[1].store(ptr+sizeof(simd8<T>)*1);
this->chunks[2].store(ptr+sizeof(simd8<T>)*2); this->chunks[2].store(ptr+sizeof(simd8<T>)*2);
this->chunks[3].store(ptr+sizeof(simd8<T>)*3); this->chunks[3].store(ptr+sizeof(simd8<T>)*3);
} }
simdjson_really_inline simd8<T> reduce_or() const { simdjson_inline simd8<T> reduce_or() const {
return (this->chunks[0] | this->chunks[1]) | (this->chunks[2] | this->chunks[3]); return (this->chunks[0] | this->chunks[1]) | (this->chunks[2] | this->chunks[3]);
} }
simdjson_really_inline uint64_t compress(uint64_t mask, T * output) const { simdjson_inline uint64_t compress(uint64_t mask, T * output) const {
uint64_t popcounts = vget_lane_u64(vreinterpret_u64_u8(vcnt_u8(vcreate_u8(~mask))), 0); uint64_t popcounts = vget_lane_u64(vreinterpret_u64_u8(vcnt_u8(vcreate_u8(~mask))), 0);
// compute the prefix sum of the popcounts of each byte // compute the prefix sum of the popcounts of each byte
uint64_t offsets = popcounts * 0x0101010101010101; uint64_t offsets = popcounts * 0x0101010101010101;
@@ -484,7 +484,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
return offsets >> 56; return offsets >> 56;
} }
simdjson_really_inline uint64_t to_bitmask() const { simdjson_inline uint64_t to_bitmask() const {
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
const uint8x16_t bit_mask = make_uint8x16_t( const uint8x16_t bit_mask = make_uint8x16_t(
0x01, 0x02, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80, 0x01, 0x02, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80,
@@ -504,7 +504,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
return vgetq_lane_u64(vreinterpretq_u64_u8(sum0), 0); return vgetq_lane_u64(vreinterpretq_u64_u8(sum0), 0);
} }
simdjson_really_inline uint64_t eq(const T m) const { simdjson_inline uint64_t eq(const T m) const {
const simd8<T> mask = simd8<T>::splat(m); const simd8<T> mask = simd8<T>::splat(m);
return simd8x64<bool>( return simd8x64<bool>(
this->chunks[0] == mask, this->chunks[0] == mask,
@@ -514,7 +514,7 @@ simdjson_really_inline int8x16_t make_int8x16_t(int8_t x1, int8_t x2, int8_t x
).to_bitmask(); ).to_bitmask();
} }
simdjson_really_inline uint64_t lteq(const T m) const { simdjson_inline uint64_t lteq(const T m) const {
const simd8<T> mask = simd8<T>::splat(m); const simd8<T> mask = simd8<T>::splat(m);
return simd8x64<bool>( return simd8x64<bool>(
this->chunks[0] <= mask, this->chunks[0] <= mask,
+6 -8
View File
@@ -15,18 +15,18 @@ using namespace simd;
struct backslash_and_quote { struct backslash_and_quote {
public: public:
static constexpr uint32_t BYTES_PROCESSED = 32; static constexpr uint32_t BYTES_PROCESSED = 32;
simdjson_really_inline static backslash_and_quote copy_and_find(const uint8_t *src, uint8_t *dst); simdjson_inline static backslash_and_quote copy_and_find(const uint8_t *src, uint8_t *dst);
simdjson_really_inline bool has_quote_first() { return ((bs_bits - 1) & quote_bits) != 0; } simdjson_inline bool has_quote_first() { return ((bs_bits - 1) & quote_bits) != 0; }
simdjson_really_inline bool has_backslash() { return bs_bits != 0; } simdjson_inline bool has_backslash() { return bs_bits != 0; }
simdjson_really_inline int quote_index() { return trailing_zeroes(quote_bits); } simdjson_inline int quote_index() { return trailing_zeroes(quote_bits); }
simdjson_really_inline int backslash_index() { return trailing_zeroes(bs_bits); } simdjson_inline int backslash_index() { return trailing_zeroes(bs_bits); }
uint32_t bs_bits; uint32_t bs_bits;
uint32_t quote_bits; uint32_t quote_bits;
}; // struct backslash_and_quote }; // struct backslash_and_quote
simdjson_really_inline backslash_and_quote backslash_and_quote::copy_and_find(const uint8_t *src, uint8_t *dst) { simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uint8_t *src, uint8_t *dst) {
// this can read up to 31 bytes beyond the buffer size, but we require // this can read up to 31 bytes beyond the buffer size, but we require
// SIMDJSON_PADDING of padding // SIMDJSON_PADDING of padding
static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "backslash and quote finder must process fewer than SIMDJSON_PADDING bytes"); static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "backslash and quote finder must process fewer than SIMDJSON_PADDING bytes");
@@ -48,6 +48,4 @@ simdjson_really_inline backslash_and_quote backslash_and_quote::copy_and_find(co
} // namespace SIMDJSON_IMPLEMENTATION } // namespace SIMDJSON_IMPLEMENTATION
} // namespace simdjson } // namespace simdjson
#include "simdjson/generic/stringparsing.h"
#endif // SIMDJSON_ARM64_STRINGPARSING_H #endif // SIMDJSON_ARM64_STRINGPARSING_H
+37 -6
View File
@@ -37,12 +37,12 @@ constexpr size_t SIMDJSON_MAXSIZE_BYTES = 0xFFFFFFFF;
/** /**
* The amount of padding needed in a buffer to parse JSON. * The amount of padding needed in a buffer to parse JSON.
* *
* the input buf should be readable up to buf + SIMDJSON_PADDING * The input buf should be readable up to buf + SIMDJSON_PADDING
* this is a stopgap; there should be a better description of the * this is a stopgap; there should be a better description of the
* main loop and its behavior that abstracts over this * main loop and its behavior that abstracts over this
* See https://github.com/simdjson/simdjson/issues/174 * See https://github.com/simdjson/simdjson/issues/174
*/ */
constexpr size_t SIMDJSON_PADDING = 32; constexpr size_t SIMDJSON_PADDING = 64;
/** /**
* By default, simdjson supports this many nested objects and arrays. * By default, simdjson supports this many nested objects and arrays.
@@ -168,6 +168,18 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
#endif // MSC_VER #endif // MSC_VER
#if defined(simdjson_inline)
// Prefer the user's definition of simdjson_inline; don't define it ourselves.
#elif defined(__GNUC__) && !defined(__OPTIMIZE__)
// If optimizations are disabled, forcing inlining can lead to significant
// code bloat and high compile times. Don't use simdjson_really_inline for
// unoptimized builds.
#define simdjson_inline inline
#else
// Force inlining for most simdjson functions.
#define simdjson_inline simdjson_really_inline
#endif
#if defined(SIMDJSON_VISUAL_STUDIO) #if defined(SIMDJSON_VISUAL_STUDIO)
/** /**
* Windows users need to do some extra work when building * Windows users need to do some extra work when building
@@ -266,11 +278,30 @@ namespace std {
/// If EXPR is an error, returns it. /// If EXPR is an error, returns it.
#define SIMDJSON_TRY(EXPR) { auto _err = (EXPR); if (_err) { return _err; } } #define SIMDJSON_TRY(EXPR) { auto _err = (EXPR); if (_err) { return _err; } }
// Unless the programmer has already set SIMDJSON_DEVELOPMENT_CHECKS,
// we want to set it under debug builds. We detect a debug build
// under Visual Studio when the _DEBUG macro is set. Under the other
// compilers, we use the fact that they define __OPTIMIZE__ whenever
// they allow optimizations.
// It is possible that this could miss some cases where SIMDJSON_DEVELOPMENT_CHECKS
// is helpful, but the programmer can set the macro SIMDJSON_DEVELOPMENT_CHECKS.
// It could also wrongly set SIMDJSON_DEVELOPMENT_CHECKS (e.g., if the programmer
// sets _DEBUG in a release build under Visual Studio, or if some compiler fails to
// set the __OPTIMIZE__ macro).
#ifndef SIMDJSON_DEVELOPMENT_CHECKS #ifndef SIMDJSON_DEVELOPMENT_CHECKS
#ifndef NDEBUG #ifdef _MSC_VER
#define SIMDJSON_DEVELOPMENT_CHECKS // Visual Studio seems to set _DEBUG for debug builds.
#endif #ifdef _DEBUG
#endif #define SIMDJSON_DEVELOPMENT_CHECKS 1
#endif // _DEBUG
#else // _MSC_VER
// All other compilers appear to set __OPTIMIZE__ to a positive integer
// when the compiler is optimizing.
#ifndef __OPTIMIZE__
#define SIMDJSON_DEVELOPMENT_CHECKS 1
#endif // __OPTIMIZE__
#endif // _MSC_VER
#endif // SIMDJSON_DEVELOPMENT_CHECKS
// The SIMDJSON_CHECK_EOF macro is a feature flag for the "don't require padding" // The SIMDJSON_CHECK_EOF macro is a feature flag for the "don't require padding"
// feature. // feature.
+6 -6
View File
@@ -12,11 +12,11 @@ namespace simdjson {
// //
// simdjson_result<dom::array> inline implementation // simdjson_result<dom::array> inline implementation
// //
simdjson_really_inline simdjson_result<dom::array>::simdjson_result() noexcept simdjson_inline simdjson_result<dom::array>::simdjson_result() noexcept
: internal::simdjson_result_base<dom::array>() {} : internal::simdjson_result_base<dom::array>() {}
simdjson_really_inline simdjson_result<dom::array>::simdjson_result(dom::array value) noexcept simdjson_inline simdjson_result<dom::array>::simdjson_result(dom::array value) noexcept
: internal::simdjson_result_base<dom::array>(std::forward<dom::array>(value)) {} : internal::simdjson_result_base<dom::array>(std::forward<dom::array>(value)) {}
simdjson_really_inline simdjson_result<dom::array>::simdjson_result(error_code error) noexcept simdjson_inline simdjson_result<dom::array>::simdjson_result(error_code error) noexcept
: internal::simdjson_result_base<dom::array>(error) {} : internal::simdjson_result_base<dom::array>(error) {}
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
@@ -50,8 +50,8 @@ namespace dom {
// //
// array inline implementation // array inline implementation
// //
simdjson_really_inline array::array() noexcept : tape{} {} simdjson_inline array::array() noexcept : tape{} {}
simdjson_really_inline array::array(const internal::tape_ref &_tape) noexcept : tape{_tape} {} simdjson_inline array::array(const internal::tape_ref &_tape) noexcept : tape{_tape} {}
inline array::iterator array::begin() const noexcept { inline array::iterator array::begin() const noexcept {
return internal::tape_ref(tape.doc, tape.json_index + 1); return internal::tape_ref(tape.doc, tape.json_index + 1);
} }
@@ -116,7 +116,7 @@ inline simdjson_result<element> array::at(size_t index) const noexcept {
// //
// array::iterator inline implementation // array::iterator inline implementation
// //
simdjson_really_inline array::iterator::iterator(const internal::tape_ref &_tape) noexcept : tape{_tape} { } simdjson_inline array::iterator::iterator(const internal::tape_ref &_tape) noexcept : tape{_tape} { }
inline element array::iterator::operator*() const noexcept { inline element array::iterator::operator*() const noexcept {
return element(tape); return element(tape);
} }
+6 -6
View File
@@ -22,7 +22,7 @@ class element;
class array { class array {
public: public:
/** Create a new, invalid array */ /** Create a new, invalid array */
simdjson_really_inline array() noexcept; simdjson_inline array() noexcept;
class iterator { class iterator {
public: public:
@@ -62,7 +62,7 @@ public:
iterator(const iterator&) noexcept = default; iterator(const iterator&) noexcept = default;
iterator& operator=(const iterator&) noexcept = default; iterator& operator=(const iterator&) noexcept = default;
private: private:
simdjson_really_inline iterator(const internal::tape_ref &tape) noexcept; simdjson_inline iterator(const internal::tape_ref &tape) noexcept;
internal::tape_ref tape; internal::tape_ref tape;
friend class array; friend class array;
}; };
@@ -133,7 +133,7 @@ public:
inline simdjson_result<element> at(size_t index) const noexcept; inline simdjson_result<element> at(size_t index) const noexcept;
private: private:
simdjson_really_inline array(const internal::tape_ref &tape) noexcept; simdjson_inline array(const internal::tape_ref &tape) noexcept;
internal::tape_ref tape; internal::tape_ref tape;
friend class element; friend class element;
friend struct simdjson_result<element>; friend struct simdjson_result<element>;
@@ -148,9 +148,9 @@ private:
template<> template<>
struct simdjson_result<dom::array> : public internal::simdjson_result_base<dom::array> { struct simdjson_result<dom::array> : public internal::simdjson_result_base<dom::array> {
public: public:
simdjson_really_inline simdjson_result() noexcept; ///< @private simdjson_inline simdjson_result() noexcept; ///< @private
simdjson_really_inline simdjson_result(dom::array value) noexcept; ///< @private simdjson_inline simdjson_result(dom::array value) noexcept; ///< @private
simdjson_really_inline simdjson_result(error_code error) noexcept; ///< @private simdjson_inline simdjson_result(error_code error) noexcept; ///< @private
inline simdjson_result<dom::element> at_pointer(std::string_view json_pointer) const noexcept; inline simdjson_result<dom::element> at_pointer(std::string_view json_pointer) const noexcept;
inline simdjson_result<dom::element> at(size_t index) const noexcept; inline simdjson_result<dom::element> at(size_t index) const noexcept;
+19 -19
View File
@@ -81,7 +81,7 @@ inline void stage1_worker::run(document_stream * ds, dom::parser * stage1, size_
} }
#endif #endif
simdjson_really_inline document_stream::document_stream( simdjson_inline document_stream::document_stream(
dom::parser &_parser, dom::parser &_parser,
const uint8_t *_buf, const uint8_t *_buf,
size_t _len, size_t _len,
@@ -103,7 +103,7 @@ simdjson_really_inline document_stream::document_stream(
#endif #endif
} }
simdjson_really_inline document_stream::document_stream() noexcept simdjson_inline document_stream::document_stream() noexcept
: parser{nullptr}, : parser{nullptr},
buf{nullptr}, buf{nullptr},
len{0}, len{0},
@@ -115,31 +115,31 @@ simdjson_really_inline document_stream::document_stream() noexcept
{ {
} }
simdjson_really_inline document_stream::~document_stream() noexcept { simdjson_inline document_stream::~document_stream() noexcept {
#ifdef SIMDJSON_THREADS_ENABLED #ifdef SIMDJSON_THREADS_ENABLED
worker.reset(); worker.reset();
#endif #endif
} }
simdjson_really_inline document_stream::iterator::iterator() noexcept simdjson_inline document_stream::iterator::iterator() noexcept
: stream{nullptr}, finished{true} { : stream{nullptr}, finished{true} {
} }
simdjson_really_inline document_stream::iterator document_stream::begin() noexcept { simdjson_inline document_stream::iterator document_stream::begin() noexcept {
start(); start();
// If there are no documents, we're finished. // If there are no documents, we're finished.
return iterator(this, error == EMPTY); return iterator(this, error == EMPTY);
} }
simdjson_really_inline document_stream::iterator document_stream::end() noexcept { simdjson_inline document_stream::iterator document_stream::end() noexcept {
return iterator(this, true); return iterator(this, true);
} }
simdjson_really_inline document_stream::iterator::iterator(document_stream* _stream, bool is_end) noexcept simdjson_inline document_stream::iterator::iterator(document_stream* _stream, bool is_end) noexcept
: stream{_stream}, finished{is_end} { : stream{_stream}, finished{is_end} {
} }
simdjson_really_inline document_stream::iterator::reference document_stream::iterator::operator*() noexcept { simdjson_inline document_stream::iterator::reference document_stream::iterator::operator*() noexcept {
// Note that in case of error, we do not yet mark // Note that in case of error, we do not yet mark
// the iterator as "finished": this detection is done // the iterator as "finished": this detection is done
// in the operator++ function since it is possible // in the operator++ function since it is possible
@@ -149,7 +149,7 @@ simdjson_really_inline document_stream::iterator::reference document_stream::ite
return stream->parser->doc.root(); return stream->parser->doc.root();
} }
simdjson_really_inline document_stream::iterator& document_stream::iterator::operator++() noexcept { simdjson_inline document_stream::iterator& document_stream::iterator::operator++() noexcept {
// If there is an error, then we want the iterator // If there is an error, then we want the iterator
// to be finished, no matter what. (E.g., we do not // to be finished, no matter what. (E.g., we do not
// keep generating documents with errors, or go beyond // keep generating documents with errors, or go beyond
@@ -176,7 +176,7 @@ simdjson_really_inline document_stream::iterator& document_stream::iterator::ope
return *this; return *this;
} }
simdjson_really_inline bool document_stream::iterator::operator!=(const document_stream::iterator &other) const noexcept { simdjson_inline bool document_stream::iterator::operator!=(const document_stream::iterator &other) const noexcept {
return finished != other.finished; return finished != other.finished;
} }
@@ -207,11 +207,11 @@ inline void document_stream::start() noexcept {
next(); next();
} }
simdjson_really_inline size_t document_stream::iterator::current_index() const noexcept { simdjson_inline size_t document_stream::iterator::current_index() const noexcept {
return stream->doc_index; return stream->doc_index;
} }
simdjson_really_inline std::string_view document_stream::iterator::source() const noexcept { simdjson_inline std::string_view document_stream::iterator::source() const noexcept {
const char* start = reinterpret_cast<const char*>(stream->buf) + current_index(); const char* start = reinterpret_cast<const char*>(stream->buf) + current_index();
bool object_or_array = ((*start == '[') || (*start == '{')); bool object_or_array = ((*start == '[') || (*start == '{'));
if(object_or_array) { if(object_or_array) {
@@ -304,32 +304,32 @@ inline void document_stream::start_stage1_thread() noexcept {
} // namespace dom } // namespace dom
simdjson_really_inline simdjson_result<dom::document_stream>::simdjson_result() noexcept simdjson_inline simdjson_result<dom::document_stream>::simdjson_result() noexcept
: simdjson_result_base() { : simdjson_result_base() {
} }
simdjson_really_inline simdjson_result<dom::document_stream>::simdjson_result(error_code error) noexcept simdjson_inline simdjson_result<dom::document_stream>::simdjson_result(error_code error) noexcept
: simdjson_result_base(error) { : simdjson_result_base(error) {
} }
simdjson_really_inline simdjson_result<dom::document_stream>::simdjson_result(dom::document_stream &&value) noexcept simdjson_inline simdjson_result<dom::document_stream>::simdjson_result(dom::document_stream &&value) noexcept
: simdjson_result_base(std::forward<dom::document_stream>(value)) { : simdjson_result_base(std::forward<dom::document_stream>(value)) {
} }
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
simdjson_really_inline dom::document_stream::iterator simdjson_result<dom::document_stream>::begin() noexcept(false) { simdjson_inline dom::document_stream::iterator simdjson_result<dom::document_stream>::begin() noexcept(false) {
if (error()) { throw simdjson_error(error()); } if (error()) { throw simdjson_error(error()); }
return first.begin(); return first.begin();
} }
simdjson_really_inline dom::document_stream::iterator simdjson_result<dom::document_stream>::end() noexcept(false) { simdjson_inline dom::document_stream::iterator simdjson_result<dom::document_stream>::end() noexcept(false) {
if (error()) { throw simdjson_error(error()); } if (error()) { throw simdjson_error(error()); }
return first.end(); return first.end();
} }
#else // SIMDJSON_EXCEPTIONS #else // SIMDJSON_EXCEPTIONS
#ifndef SIMDJSON_DISABLE_DEPRECATED_API #ifndef SIMDJSON_DISABLE_DEPRECATED_API
simdjson_really_inline dom::document_stream::iterator simdjson_result<dom::document_stream>::begin() noexcept { simdjson_inline dom::document_stream::iterator simdjson_result<dom::document_stream>::begin() noexcept {
first.error = error(); first.error = error();
return first.begin(); return first.begin();
} }
simdjson_really_inline dom::document_stream::iterator simdjson_result<dom::document_stream>::end() noexcept { simdjson_inline dom::document_stream::iterator simdjson_result<dom::document_stream>::end() noexcept {
first.error = error(); first.error = error();
return first.end(); return first.end();
} }
+20 -20
View File
@@ -80,13 +80,13 @@ public:
* error = parser.parse_many(json).get(docs); * error = parser.parse_many(json).get(docs);
* ``` * ```
*/ */
simdjson_really_inline document_stream() noexcept; simdjson_inline document_stream() noexcept;
/** Move one document_stream to another. */ /** Move one document_stream to another. */
simdjson_really_inline document_stream(document_stream &&other) noexcept = default; simdjson_inline document_stream(document_stream &&other) noexcept = default;
/** Move one document_stream to another. */ /** Move one document_stream to another. */
simdjson_really_inline document_stream &operator=(document_stream &&other) noexcept = default; simdjson_inline document_stream &operator=(document_stream &&other) noexcept = default;
simdjson_really_inline ~document_stream() noexcept; simdjson_inline ~document_stream() noexcept;
/** /**
* Returns the input size in bytes. * Returns the input size in bytes.
*/ */
@@ -125,11 +125,11 @@ public:
/** /**
* Default constructor. * Default constructor.
*/ */
simdjson_really_inline iterator() noexcept; simdjson_inline iterator() noexcept;
/** /**
* Get the current document (or error). * Get the current document (or error).
*/ */
simdjson_really_inline reference operator*() noexcept; simdjson_inline reference operator*() noexcept;
/** /**
* Advance to the next document (prefix). * Advance to the next document (prefix).
*/ */
@@ -138,7 +138,7 @@ public:
* Check if we're at the end yet. * Check if we're at the end yet.
* @param other the end iterator to compare to. * @param other the end iterator to compare to.
*/ */
simdjson_really_inline bool operator!=(const iterator &other) const noexcept; simdjson_inline bool operator!=(const iterator &other) const noexcept;
/** /**
* @private * @private
* *
@@ -154,7 +154,7 @@ public:
* may change in future versions of simdjson: we find the API somewhat * may change in future versions of simdjson: we find the API somewhat
* awkward and we would like to offer something friendlier. * awkward and we would like to offer something friendlier.
*/ */
simdjson_really_inline size_t current_index() const noexcept; simdjson_inline size_t current_index() const noexcept;
/** /**
* @private * @private
* *
@@ -174,10 +174,10 @@ public:
* may change in future versions of simdjson: we find the API somewhat * may change in future versions of simdjson: we find the API somewhat
* awkward and we would like to offer something friendlier. * awkward and we would like to offer something friendlier.
*/ */
simdjson_really_inline std::string_view source() const noexcept; simdjson_inline std::string_view source() const noexcept;
private: private:
simdjson_really_inline iterator(document_stream *s, bool finished) noexcept; simdjson_inline iterator(document_stream *s, bool finished) noexcept;
/** The document_stream we're iterating through. */ /** The document_stream we're iterating through. */
document_stream* stream; document_stream* stream;
/** Whether we're finished or not. */ /** Whether we're finished or not. */
@@ -188,11 +188,11 @@ public:
/** /**
* Start iterating the documents in the stream. * Start iterating the documents in the stream.
*/ */
simdjson_really_inline iterator begin() noexcept; simdjson_inline iterator begin() noexcept;
/** /**
* The end of the stream, for iterator comparison purposes. * The end of the stream, for iterator comparison purposes.
*/ */
simdjson_really_inline iterator end() noexcept; simdjson_inline iterator end() noexcept;
private: private:
@@ -208,7 +208,7 @@ private:
* @param len is the length of the raw byte buffer in bytes * @param len is the length of the raw byte buffer in bytes
* @param batch_size is the size of the windows (must be strictly greater or equal to the largest JSON document) * @param batch_size is the size of the windows (must be strictly greater or equal to the largest JSON document)
*/ */
simdjson_really_inline document_stream( simdjson_inline document_stream(
dom::parser &parser, dom::parser &parser,
const uint8_t *buf, const uint8_t *buf,
size_t len, size_t len,
@@ -302,19 +302,19 @@ private:
template<> template<>
struct simdjson_result<dom::document_stream> : public internal::simdjson_result_base<dom::document_stream> { struct simdjson_result<dom::document_stream> : public internal::simdjson_result_base<dom::document_stream> {
public: public:
simdjson_really_inline simdjson_result() noexcept; ///< @private simdjson_inline simdjson_result() noexcept; ///< @private
simdjson_really_inline simdjson_result(error_code error) noexcept; ///< @private simdjson_inline simdjson_result(error_code error) noexcept; ///< @private
simdjson_really_inline simdjson_result(dom::document_stream &&value) noexcept; ///< @private simdjson_inline simdjson_result(dom::document_stream &&value) noexcept; ///< @private
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
simdjson_really_inline dom::document_stream::iterator begin() noexcept(false); simdjson_inline dom::document_stream::iterator begin() noexcept(false);
simdjson_really_inline dom::document_stream::iterator end() noexcept(false); simdjson_inline dom::document_stream::iterator end() noexcept(false);
#else // SIMDJSON_EXCEPTIONS #else // SIMDJSON_EXCEPTIONS
#ifndef SIMDJSON_DISABLE_DEPRECATED_API #ifndef SIMDJSON_DISABLE_DEPRECATED_API
[[deprecated("parse_many() and load_many() may return errors. Use document_stream stream; error = parser.parse_many().get(doc); instead.")]] [[deprecated("parse_many() and load_many() may return errors. Use document_stream stream; error = parser.parse_many().get(doc); instead.")]]
simdjson_really_inline dom::document_stream::iterator begin() noexcept; simdjson_inline dom::document_stream::iterator begin() noexcept;
[[deprecated("parse_many() and load_many() may return errors. Use document_stream stream; error = parser.parse_many().get(doc); instead.")]] [[deprecated("parse_many() and load_many() may return errors. Use document_stream stream; error = parser.parse_many().get(doc); instead.")]]
simdjson_really_inline dom::document_stream::iterator end() noexcept; simdjson_inline dom::document_stream::iterator end() noexcept;
#endif // SIMDJSON_DISABLE_DEPRECATED_API #endif // SIMDJSON_DISABLE_DEPRECATED_API
#endif // SIMDJSON_EXCEPTIONS #endif // SIMDJSON_EXCEPTIONS
}; // struct simdjson_result<dom::document_stream> }; // struct simdjson_result<dom::document_stream>
+46 -46
View File
@@ -12,11 +12,11 @@ namespace simdjson {
// //
// simdjson_result<dom::element> inline implementation // simdjson_result<dom::element> inline implementation
// //
simdjson_really_inline simdjson_result<dom::element>::simdjson_result() noexcept simdjson_inline simdjson_result<dom::element>::simdjson_result() noexcept
: internal::simdjson_result_base<dom::element>() {} : internal::simdjson_result_base<dom::element>() {}
simdjson_really_inline simdjson_result<dom::element>::simdjson_result(dom::element &&value) noexcept simdjson_inline simdjson_result<dom::element>::simdjson_result(dom::element &&value) noexcept
: internal::simdjson_result_base<dom::element>(std::forward<dom::element>(value)) {} : internal::simdjson_result_base<dom::element>(std::forward<dom::element>(value)) {}
simdjson_really_inline simdjson_result<dom::element>::simdjson_result(error_code error) noexcept simdjson_inline simdjson_result<dom::element>::simdjson_result(error_code error) noexcept
: internal::simdjson_result_base<dom::element>(error) {} : internal::simdjson_result_base<dom::element>(error) {}
inline simdjson_result<dom::element_type> simdjson_result<dom::element>::type() const noexcept { inline simdjson_result<dom::element_type> simdjson_result<dom::element>::type() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
@@ -24,101 +24,101 @@ inline simdjson_result<dom::element_type> simdjson_result<dom::element>::type()
} }
template<typename T> template<typename T>
simdjson_really_inline bool simdjson_result<dom::element>::is() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is() const noexcept {
return !error() && first.is<T>(); return !error() && first.is<T>();
} }
template<typename T> template<typename T>
simdjson_really_inline simdjson_result<T> simdjson_result<dom::element>::get() const noexcept { simdjson_inline simdjson_result<T> simdjson_result<dom::element>::get() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get<T>(); return first.get<T>();
} }
template<typename T> template<typename T>
simdjson_warn_unused simdjson_really_inline error_code simdjson_result<dom::element>::get(T &value) const noexcept { simdjson_warn_unused simdjson_inline error_code simdjson_result<dom::element>::get(T &value) const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get<T>(value); return first.get<T>(value);
} }
simdjson_really_inline simdjson_result<dom::array> simdjson_result<dom::element>::get_array() const noexcept { simdjson_inline simdjson_result<dom::array> simdjson_result<dom::element>::get_array() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_array(); return first.get_array();
} }
simdjson_really_inline simdjson_result<dom::object> simdjson_result<dom::element>::get_object() const noexcept { simdjson_inline simdjson_result<dom::object> simdjson_result<dom::element>::get_object() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_object(); return first.get_object();
} }
simdjson_really_inline simdjson_result<const char *> simdjson_result<dom::element>::get_c_str() const noexcept { simdjson_inline simdjson_result<const char *> simdjson_result<dom::element>::get_c_str() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_c_str(); return first.get_c_str();
} }
simdjson_really_inline simdjson_result<size_t> simdjson_result<dom::element>::get_string_length() const noexcept { simdjson_inline simdjson_result<size_t> simdjson_result<dom::element>::get_string_length() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_string_length(); return first.get_string_length();
} }
simdjson_really_inline simdjson_result<std::string_view> simdjson_result<dom::element>::get_string() const noexcept { simdjson_inline simdjson_result<std::string_view> simdjson_result<dom::element>::get_string() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_string(); return first.get_string();
} }
simdjson_really_inline simdjson_result<int64_t> simdjson_result<dom::element>::get_int64() const noexcept { simdjson_inline simdjson_result<int64_t> simdjson_result<dom::element>::get_int64() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_int64(); return first.get_int64();
} }
simdjson_really_inline simdjson_result<uint64_t> simdjson_result<dom::element>::get_uint64() const noexcept { simdjson_inline simdjson_result<uint64_t> simdjson_result<dom::element>::get_uint64() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_uint64(); return first.get_uint64();
} }
simdjson_really_inline simdjson_result<double> simdjson_result<dom::element>::get_double() const noexcept { simdjson_inline simdjson_result<double> simdjson_result<dom::element>::get_double() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_double(); return first.get_double();
} }
simdjson_really_inline simdjson_result<bool> simdjson_result<dom::element>::get_bool() const noexcept { simdjson_inline simdjson_result<bool> simdjson_result<dom::element>::get_bool() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.get_bool(); return first.get_bool();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_array() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_array() const noexcept {
return !error() && first.is_array(); return !error() && first.is_array();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_object() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_object() const noexcept {
return !error() && first.is_object(); return !error() && first.is_object();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_string() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_string() const noexcept {
return !error() && first.is_string(); return !error() && first.is_string();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_int64() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_int64() const noexcept {
return !error() && first.is_int64(); return !error() && first.is_int64();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_uint64() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_uint64() const noexcept {
return !error() && first.is_uint64(); return !error() && first.is_uint64();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_double() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_double() const noexcept {
return !error() && first.is_double(); return !error() && first.is_double();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_number() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_number() const noexcept {
return !error() && first.is_number(); return !error() && first.is_number();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_bool() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_bool() const noexcept {
return !error() && first.is_bool(); return !error() && first.is_bool();
} }
simdjson_really_inline bool simdjson_result<dom::element>::is_null() const noexcept { simdjson_inline bool simdjson_result<dom::element>::is_null() const noexcept {
return !error() && first.is_null(); return !error() && first.is_null();
} }
simdjson_really_inline simdjson_result<dom::element> simdjson_result<dom::element>::operator[](std::string_view key) const noexcept { simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::operator[](std::string_view key) const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first[key]; return first[key];
} }
simdjson_really_inline simdjson_result<dom::element> simdjson_result<dom::element>::operator[](const char *key) const noexcept { simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::operator[](const char *key) const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first[key]; return first[key];
} }
simdjson_really_inline simdjson_result<dom::element> simdjson_result<dom::element>::at_pointer(const std::string_view json_pointer) const noexcept { simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::at_pointer(const std::string_view json_pointer) const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.at_pointer(json_pointer); return first.at_pointer(json_pointer);
} }
#ifndef SIMDJSON_DISABLE_DEPRECATED_API #ifndef SIMDJSON_DISABLE_DEPRECATED_API
[[deprecated("For standard compliance, use at_pointer instead, and prefix your pointers with a slash '/', see RFC6901 ")]] [[deprecated("For standard compliance, use at_pointer instead, and prefix your pointers with a slash '/', see RFC6901 ")]]
simdjson_really_inline simdjson_result<dom::element> simdjson_result<dom::element>::at(const std::string_view json_pointer) const noexcept { simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::at(const std::string_view json_pointer) const noexcept {
SIMDJSON_PUSH_DISABLE_WARNINGS SIMDJSON_PUSH_DISABLE_WARNINGS
SIMDJSON_DISABLE_DEPRECATED_WARNING SIMDJSON_DISABLE_DEPRECATED_WARNING
if (error()) { return error(); } if (error()) { return error(); }
@@ -126,51 +126,51 @@ SIMDJSON_DISABLE_DEPRECATED_WARNING
SIMDJSON_POP_DISABLE_WARNINGS SIMDJSON_POP_DISABLE_WARNINGS
} }
#endif // SIMDJSON_DISABLE_DEPRECATED_API #endif // SIMDJSON_DISABLE_DEPRECATED_API
simdjson_really_inline simdjson_result<dom::element> simdjson_result<dom::element>::at(size_t index) const noexcept { simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::at(size_t index) const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.at(index); return first.at(index);
} }
simdjson_really_inline simdjson_result<dom::element> simdjson_result<dom::element>::at_key(std::string_view key) const noexcept { simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::at_key(std::string_view key) const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.at_key(key); return first.at_key(key);
} }
simdjson_really_inline simdjson_result<dom::element> simdjson_result<dom::element>::at_key_case_insensitive(std::string_view key) const noexcept { simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::at_key_case_insensitive(std::string_view key) const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.at_key_case_insensitive(key); return first.at_key_case_insensitive(key);
} }
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
simdjson_really_inline simdjson_result<dom::element>::operator bool() const noexcept(false) { simdjson_inline simdjson_result<dom::element>::operator bool() const noexcept(false) {
return get<bool>(); return get<bool>();
} }
simdjson_really_inline simdjson_result<dom::element>::operator const char *() const noexcept(false) { simdjson_inline simdjson_result<dom::element>::operator const char *() const noexcept(false) {
return get<const char *>(); return get<const char *>();
} }
simdjson_really_inline simdjson_result<dom::element>::operator std::string_view() const noexcept(false) { simdjson_inline simdjson_result<dom::element>::operator std::string_view() const noexcept(false) {
return get<std::string_view>(); return get<std::string_view>();
} }
simdjson_really_inline simdjson_result<dom::element>::operator uint64_t() const noexcept(false) { simdjson_inline simdjson_result<dom::element>::operator uint64_t() const noexcept(false) {
return get<uint64_t>(); return get<uint64_t>();
} }
simdjson_really_inline simdjson_result<dom::element>::operator int64_t() const noexcept(false) { simdjson_inline simdjson_result<dom::element>::operator int64_t() const noexcept(false) {
return get<int64_t>(); return get<int64_t>();
} }
simdjson_really_inline simdjson_result<dom::element>::operator double() const noexcept(false) { simdjson_inline simdjson_result<dom::element>::operator double() const noexcept(false) {
return get<double>(); return get<double>();
} }
simdjson_really_inline simdjson_result<dom::element>::operator dom::array() const noexcept(false) { simdjson_inline simdjson_result<dom::element>::operator dom::array() const noexcept(false) {
return get<dom::array>(); return get<dom::array>();
} }
simdjson_really_inline simdjson_result<dom::element>::operator dom::object() const noexcept(false) { simdjson_inline simdjson_result<dom::element>::operator dom::object() const noexcept(false) {
return get<dom::object>(); return get<dom::object>();
} }
simdjson_really_inline dom::array::iterator simdjson_result<dom::element>::begin() const noexcept(false) { simdjson_inline dom::array::iterator simdjson_result<dom::element>::begin() const noexcept(false) {
if (error()) { throw simdjson_error(error()); } if (error()) { throw simdjson_error(error()); }
return first.begin(); return first.begin();
} }
simdjson_really_inline dom::array::iterator simdjson_result<dom::element>::end() const noexcept(false) { simdjson_inline dom::array::iterator simdjson_result<dom::element>::end() const noexcept(false) {
if (error()) { throw simdjson_error(error()); } if (error()) { throw simdjson_error(error()); }
return first.end(); return first.end();
} }
@@ -182,8 +182,8 @@ namespace dom {
// //
// element inline implementation // element inline implementation
// //
simdjson_really_inline element::element() noexcept : tape{} {} simdjson_inline element::element() noexcept : tape{} {}
simdjson_really_inline element::element(const internal::tape_ref &_tape) noexcept : tape{_tape} { } simdjson_inline element::element(const internal::tape_ref &_tape) noexcept : tape{_tape} { }
inline element_type element::type() const noexcept { inline element_type element::type() const noexcept {
auto tape_type = tape.tape_ref_type(); auto tape_type = tape.tape_ref_type();
@@ -290,12 +290,12 @@ inline simdjson_result<object> element::get_object() const noexcept {
} }
template<typename T> template<typename T>
simdjson_warn_unused simdjson_really_inline error_code element::get(T &value) const noexcept { simdjson_warn_unused simdjson_inline error_code element::get(T &value) const noexcept {
return get<T>().get(value); return get<T>().get(value);
} }
// An element-specific version prevents recursion with simdjson_result::get<element>(value) // An element-specific version prevents recursion with simdjson_result::get<element>(value)
template<> template<>
simdjson_warn_unused simdjson_really_inline error_code element::get<element>(element &value) const noexcept { simdjson_warn_unused simdjson_inline error_code element::get<element>(element &value) const noexcept {
value = element(tape); value = element(tape);
return SUCCESS; return SUCCESS;
} }
@@ -305,7 +305,7 @@ inline void element::tie(T &value, error_code &error) && noexcept {
} }
template<typename T> template<typename T>
simdjson_really_inline bool element::is() const noexcept { simdjson_inline bool element::is() const noexcept {
auto result = get<T>(); auto result = get<T>();
return !result.error(); return !result.error();
} }
+47 -47
View File
@@ -40,10 +40,10 @@ enum class element_type {
class element { class element {
public: public:
/** Create a new, invalid element. */ /** Create a new, invalid element. */
simdjson_really_inline element() noexcept; simdjson_inline element() noexcept;
/** The type of this element. */ /** The type of this element. */
simdjson_really_inline element_type type() const noexcept; simdjson_inline element_type type() const noexcept;
/** /**
* Cast this element to an array. * Cast this element to an array.
@@ -195,7 +195,7 @@ public:
* @tparam T bool, double, uint64_t, int64_t, std::string_view, const char *, dom::array, dom::object * @tparam T bool, double, uint64_t, int64_t, std::string_view, const char *, dom::array, dom::object
*/ */
template<typename T> template<typename T>
simdjson_really_inline bool is() const noexcept; simdjson_inline bool is() const noexcept;
/** /**
* Get the value as the provided type (T). * Get the value as the provided type (T).
@@ -240,7 +240,7 @@ public:
* @returns The error that occurred, or SUCCESS if there was no error. * @returns The error that occurred, or SUCCESS if there was no error.
*/ */
template<typename T> template<typename T>
simdjson_warn_unused simdjson_really_inline error_code get(T &value) const noexcept; simdjson_warn_unused simdjson_inline error_code get(T &value) const noexcept;
/** /**
* Get the value as the provided type (T), setting error if it's not the given type. * Get the value as the provided type (T), setting error if it's not the given type.
@@ -464,7 +464,7 @@ public:
inline bool dump_raw_tape(std::ostream &out) const noexcept; inline bool dump_raw_tape(std::ostream &out) const noexcept;
private: private:
simdjson_really_inline element(const internal::tape_ref &tape) noexcept; simdjson_inline element(const internal::tape_ref &tape) noexcept;
internal::tape_ref tape; internal::tape_ref tape;
friend class document; friend class document;
friend class object; friend class object;
@@ -481,59 +481,59 @@ private:
template<> template<>
struct simdjson_result<dom::element> : public internal::simdjson_result_base<dom::element> { struct simdjson_result<dom::element> : public internal::simdjson_result_base<dom::element> {
public: public:
simdjson_really_inline simdjson_result() noexcept; ///< @private simdjson_inline simdjson_result() noexcept; ///< @private
simdjson_really_inline simdjson_result(dom::element &&value) noexcept; ///< @private simdjson_inline simdjson_result(dom::element &&value) noexcept; ///< @private
simdjson_really_inline simdjson_result(error_code error) noexcept; ///< @private simdjson_inline simdjson_result(error_code error) noexcept; ///< @private
simdjson_really_inline simdjson_result<dom::element_type> type() const noexcept; simdjson_inline simdjson_result<dom::element_type> type() const noexcept;
template<typename T> template<typename T>
simdjson_really_inline bool is() const noexcept; simdjson_inline bool is() const noexcept;
template<typename T> template<typename T>
simdjson_really_inline simdjson_result<T> get() const noexcept; simdjson_inline simdjson_result<T> get() const noexcept;
template<typename T> template<typename T>
simdjson_warn_unused simdjson_really_inline error_code get(T &value) const noexcept; simdjson_warn_unused simdjson_inline error_code get(T &value) const noexcept;
simdjson_really_inline simdjson_result<dom::array> get_array() const noexcept; simdjson_inline simdjson_result<dom::array> get_array() const noexcept;
simdjson_really_inline simdjson_result<dom::object> get_object() const noexcept; simdjson_inline simdjson_result<dom::object> get_object() const noexcept;
simdjson_really_inline simdjson_result<const char *> get_c_str() const noexcept; simdjson_inline simdjson_result<const char *> get_c_str() const noexcept;
simdjson_really_inline simdjson_result<size_t> get_string_length() const noexcept; simdjson_inline simdjson_result<size_t> get_string_length() const noexcept;
simdjson_really_inline simdjson_result<std::string_view> get_string() const noexcept; simdjson_inline simdjson_result<std::string_view> get_string() const noexcept;
simdjson_really_inline simdjson_result<int64_t> get_int64() const noexcept; simdjson_inline simdjson_result<int64_t> get_int64() const noexcept;
simdjson_really_inline simdjson_result<uint64_t> get_uint64() const noexcept; simdjson_inline simdjson_result<uint64_t> get_uint64() const noexcept;
simdjson_really_inline simdjson_result<double> get_double() const noexcept; simdjson_inline simdjson_result<double> get_double() const noexcept;
simdjson_really_inline simdjson_result<bool> get_bool() const noexcept; simdjson_inline simdjson_result<bool> get_bool() const noexcept;
simdjson_really_inline bool is_array() const noexcept; simdjson_inline bool is_array() const noexcept;
simdjson_really_inline bool is_object() const noexcept; simdjson_inline bool is_object() const noexcept;
simdjson_really_inline bool is_string() const noexcept; simdjson_inline bool is_string() const noexcept;
simdjson_really_inline bool is_int64() const noexcept; simdjson_inline bool is_int64() const noexcept;
simdjson_really_inline bool is_uint64() const noexcept; simdjson_inline bool is_uint64() const noexcept;
simdjson_really_inline bool is_double() const noexcept; simdjson_inline bool is_double() const noexcept;
simdjson_really_inline bool is_number() const noexcept; simdjson_inline bool is_number() const noexcept;
simdjson_really_inline bool is_bool() const noexcept; simdjson_inline bool is_bool() const noexcept;
simdjson_really_inline bool is_null() const noexcept; simdjson_inline bool is_null() const noexcept;
simdjson_really_inline simdjson_result<dom::element> operator[](std::string_view key) const noexcept; simdjson_inline simdjson_result<dom::element> operator[](std::string_view key) const noexcept;
simdjson_really_inline simdjson_result<dom::element> operator[](const char *key) const noexcept; simdjson_inline simdjson_result<dom::element> operator[](const char *key) const noexcept;
simdjson_really_inline simdjson_result<dom::element> at_pointer(const std::string_view json_pointer) const noexcept; simdjson_inline simdjson_result<dom::element> at_pointer(const std::string_view json_pointer) const noexcept;
[[deprecated("For standard compliance, use at_pointer instead, and prefix your pointers with a slash '/', see RFC6901 ")]] [[deprecated("For standard compliance, use at_pointer instead, and prefix your pointers with a slash '/', see RFC6901 ")]]
simdjson_really_inline simdjson_result<dom::element> at(const std::string_view json_pointer) const noexcept; simdjson_inline simdjson_result<dom::element> at(const std::string_view json_pointer) const noexcept;
simdjson_really_inline simdjson_result<dom::element> at(size_t index) const noexcept; simdjson_inline simdjson_result<dom::element> at(size_t index) const noexcept;
simdjson_really_inline simdjson_result<dom::element> at_key(std::string_view key) const noexcept; simdjson_inline simdjson_result<dom::element> at_key(std::string_view key) const noexcept;
simdjson_really_inline simdjson_result<dom::element> at_key_case_insensitive(std::string_view key) const noexcept; simdjson_inline simdjson_result<dom::element> at_key_case_insensitive(std::string_view key) const noexcept;
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
simdjson_really_inline operator bool() const noexcept(false); simdjson_inline operator bool() const noexcept(false);
simdjson_really_inline explicit operator const char*() const noexcept(false); simdjson_inline explicit operator const char*() const noexcept(false);
simdjson_really_inline operator std::string_view() const noexcept(false); simdjson_inline operator std::string_view() const noexcept(false);
simdjson_really_inline operator uint64_t() const noexcept(false); simdjson_inline operator uint64_t() const noexcept(false);
simdjson_really_inline operator int64_t() const noexcept(false); simdjson_inline operator int64_t() const noexcept(false);
simdjson_really_inline operator double() const noexcept(false); simdjson_inline operator double() const noexcept(false);
simdjson_really_inline operator dom::array() const noexcept(false); simdjson_inline operator dom::array() const noexcept(false);
simdjson_really_inline operator dom::object() const noexcept(false); simdjson_inline operator dom::object() const noexcept(false);
simdjson_really_inline dom::array::iterator begin() const noexcept(false); simdjson_inline dom::array::iterator begin() const noexcept(false);
simdjson_really_inline dom::array::iterator end() const noexcept(false); simdjson_inline dom::array::iterator end() const noexcept(false);
#endif // SIMDJSON_EXCEPTIONS #endif // SIMDJSON_EXCEPTIONS
}; };
+6 -6
View File
@@ -12,11 +12,11 @@ namespace simdjson {
// //
// simdjson_result<dom::object> inline implementation // simdjson_result<dom::object> inline implementation
// //
simdjson_really_inline simdjson_result<dom::object>::simdjson_result() noexcept simdjson_inline simdjson_result<dom::object>::simdjson_result() noexcept
: internal::simdjson_result_base<dom::object>() {} : internal::simdjson_result_base<dom::object>() {}
simdjson_really_inline simdjson_result<dom::object>::simdjson_result(dom::object value) noexcept simdjson_inline simdjson_result<dom::object>::simdjson_result(dom::object value) noexcept
: internal::simdjson_result_base<dom::object>(std::forward<dom::object>(value)) {} : internal::simdjson_result_base<dom::object>(std::forward<dom::object>(value)) {}
simdjson_really_inline simdjson_result<dom::object>::simdjson_result(error_code error) noexcept simdjson_inline simdjson_result<dom::object>::simdjson_result(error_code error) noexcept
: internal::simdjson_result_base<dom::object>(error) {} : internal::simdjson_result_base<dom::object>(error) {}
inline simdjson_result<dom::element> simdjson_result<dom::object>::operator[](std::string_view key) const noexcept { inline simdjson_result<dom::element> simdjson_result<dom::object>::operator[](std::string_view key) const noexcept {
@@ -62,8 +62,8 @@ namespace dom {
// //
// object inline implementation // object inline implementation
// //
simdjson_really_inline object::object() noexcept : tape{} {} simdjson_inline object::object() noexcept : tape{} {}
simdjson_really_inline object::object(const internal::tape_ref &_tape) noexcept : tape{_tape} { } simdjson_inline object::object(const internal::tape_ref &_tape) noexcept : tape{_tape} { }
inline object::iterator object::begin() const noexcept { inline object::iterator object::begin() const noexcept {
return internal::tape_ref(tape.doc, tape.json_index + 1); return internal::tape_ref(tape.doc, tape.json_index + 1);
} }
@@ -149,7 +149,7 @@ inline simdjson_result<element> object::at_key_case_insensitive(std::string_view
// //
// object::iterator inline implementation // object::iterator inline implementation
// //
simdjson_really_inline object::iterator::iterator(const internal::tape_ref &_tape) noexcept : tape{_tape} { } simdjson_inline object::iterator::iterator(const internal::tape_ref &_tape) noexcept : tape{_tape} { }
inline const key_value_pair object::iterator::operator*() const noexcept { inline const key_value_pair object::iterator::operator*() const noexcept {
return key_value_pair(key(), value()); return key_value_pair(key(), value());
} }

Some files were not shown because too many files have changed in this diff Show More