Compare commits

...

79 Commits

Author SHA1 Message Date
Daniel Lemire 92174c927d New release 2023-06-15 09:21:07 -04:00
Ng Yong Xiang b399c0165e Add comma separated value parsing as an option in iterate_many (#2016)
* Add comma separated value parsing

* Fix failing tests

* Make tests work for exceptions

* Fix test

* Fix try catch making test fail

---------

Co-authored-by: Yong Xiang Ng <yxng@drwholdings.com>
2023-06-15 09:11:14 -04:00
Daniel Lemire 445ae024ba fix: bug identified by Yagiz Nizipli (#2019) 2023-06-15 09:10:13 -04:00
Daniel Lemire 65f9822542 Adding tests for issue 2017 (#2018) 2023-06-07 09:57:28 -04:00
Daniel Lemire 74bb7b2533 Adding popcnt 2023-06-05 09:04:36 -04:00
Daniel Lemire fd97288561 Adding popcnt. 2023-06-02 20:33:15 -04:00
Daniel Lemire 1da0d8ac65 Making sure popcnt is included 2023-06-02 20:32:54 -04:00
Daniel Lemire 85661a2901 Some systems release without NDEBUG set, let us not assert in these cases. (#2014) 2023-06-01 16:18:36 -04:00
Daniel Lemire 3fcdb65d88 Update README.md 2023-05-26 09:26:17 -04:00
Daniel Lemire 8b00d91a2c More documentation. (#2011) 2023-05-26 09:23:21 -04:00
Daniel Lemire 7db3f97ba8 Adding CXX 20 to CI (#2005)
* Adding CXX 20 to CI

* side-stepping new CXX 20 guard.

* Going another way

* Saving.

* Explicit.

* Saving...

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-05-26 09:23:02 -04:00
Daniel Lemire 412a8f7c4d Removing iso header. (#2009)
* Removing iso header.

* Fix.

* Another one.

* Caught another one.

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-05-23 16:19:21 -04:00
Daniel Lemire d4b31c9feb Update README.md 2023-05-23 16:16:26 -04:00
Daniel Lemire ecdcf276b2 Update README.md 2023-05-23 16:15:13 -04:00
Daniel Lemire 6d92563280 Fixing issue 1943 (#2006)
* Fixing issue 1943

* Tweaking.
2023-05-23 11:15:22 -04:00
Daniel Lemire 0998233d58 More documentation. (#2007) 2023-05-23 11:15:07 -04:00
Daniel Lemire cb735818ec Document key queries. (#2008) 2023-05-23 11:14:55 -04:00
Daniel Lemire 41ee548e3f Adding godbolt examples. (#2010)
* Adding godbolt examples.

* Updating the examples.
2023-05-23 11:14:41 -04:00
Ng Yong Xiang e9d5d5088a Introduce logging levels for conditional logging and log key for NO_SUCH_FIELD error (#2004)
* Add info and error logging

* Add tests for error logging

* Add logging for missing field

* Update docs for logging usage

* Fix style and pass by ref for string format args

* Make log_level explicit and simplify get log level from env

* Make log level int32_t

* Format enum class

* Fix ci

* Move enum to header

* Fix compilation for noexception build in test case

* Disable warnings and putenv

---------

Co-authored-by: Yong Xiang Ng <yxng@drwholdings.com>
2023-05-22 12:58:55 -04:00
Daniel Lemire 78151e80e5 Update README.md 2023-05-16 12:49:49 -04:00
Daniel Lemire 083a5d7351 Update README.md 2023-05-15 15:03:11 -04:00
Daniel Lemire 25fcc8b432 Version bump. 2023-05-14 17:18:01 -04:00
Daniel Lemire a9c120f779 Directly check altivec macro. (#2002) 2023-05-14 17:17:10 -04:00
Daniel Lemire b1487e34c3 Adding more PPC64 testing. (#2001) 2023-05-14 16:11:49 -04:00
Daniel Lemire 2a68a32012 Checking OS support for AVX-512 (#1998)
* Adding support for AVX-512 on macOS.

* Fix.

* Fix.

* Minor fix

* Setting the variable to zero.

* Fixing include

* Checking if the OS supports AVX-512

* Tweaking.

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>
2023-05-12 19:25:02 -04:00
Daniel Lemire 21a2ccaeb8 Another fun one 2023-05-05 22:12:59 -04:00
Daniel Lemire 6cbb6ac89b On ARM processors, rapidjson may struggle. Let us help it out. (#1996) 2023-05-05 21:42:37 -04:00
Dirk Stolle e9f0f1bb9b Mention IceLake in documentation about implementation selection. (#1995)
Since the creation of the IceLake implementation there are now
four available implementations on Intel/AMD x64 processors.
2023-04-28 21:29:01 -04:00
Serge Aleynikov 7ef1ebd4fb Add Erlang bindings (#1994) 2023-04-28 09:54:49 -04:00
Daniel Lemire 8de540e267 Update README.md 2023-04-27 14:27:05 -04:00
Daniel Lemire ebc51fbd1a Update bug_report.md 2023-04-24 08:05:26 -04:00
Daniel Lemire 84a31c1615 Update bug_report.md 2023-04-23 12:06:47 -04:00
Daniel Lemire f77ac963f4 Trying the undefined behaviour sanitizer (#1988)
* Trying the undefined behaviour sanitizer

* Trimming quiet
2023-04-23 11:50:05 -04:00
pjuhasz ae3d912f8d Add link to Perl binding (JSON::SIMD) (#1991)
Co-authored-by: Peter Juhasz <juhasz.peter@uhusystems.com>
2023-04-23 10:37:56 -04:00
Daniel Lemire abf3d7d440 Slightly safer linux counters 2023-04-21 13:26:49 -04:00
Daniel Lemire 3c313b8530 Removing old file. 2023-04-13 11:52:02 -04:00
Dirk Stolle f435fddda1 update string_view lite to version 1.7.0 (#1984)
This is the header as seen for the tag v1.7.0,
commit 7b1867b7c08f9ac010fb8b9dae79ff8c8359bafb, on
<https://github.com/martinmoene/string-view-lite>.
2023-04-11 13:58:38 -04:00
Daniel Lemire 14d927128b New release 2023-04-08 16:13:48 -04:00
Daniel Lemire 48152a0326 Fix for issue 1981 (#1982)
* Fix

* Rewording.

* Doing it differently.
2023-04-08 16:11:40 -04:00
Daniel Lemire b5cf53232c More documentation. 2023-04-06 10:52:18 -04:00
Daniel Lemire b3d0b732e6 Update README.md 2023-04-04 14:22:12 -04:00
Daniel Lemire 3c29f40f2f Trying to verify issue 1979 (#1980) 2023-04-04 11:03:37 -04:00
Daniel Lemire 4a89d79984 Guarding the benchmarks. 2023-04-01 18:39:11 -04:00
Daniel Lemire 9cb639dcd7 Update basics.md 2023-03-30 12:35:38 -04:00
Daniel Lemire 4699dd0ce2 Update basics.md 2023-03-30 10:49:19 -04:00
Daniel Lemire cfd337bb50 This PR adds an 'at_end()' method. (#1978)
* This PR adds an 'at_end()' method.

* Adding 1111 }

* Tweaking test.
2023-03-30 10:47:13 -04:00
Daniel Lemire 9bef628982 It is no longer necessary to check separately for trailing content. 2023-03-24 20:04:04 -04:00
Daniel Lemire 59025bc8b1 Avoiding the issue where doc.get_value().get_object() differs from doc.get_object() in how errors are reported. (#1975)
* Avoiding the issue where doc.get_value().get_object() differs from doc.get_object() in how errors are reported.

* Minor tweaks
2023-03-24 12:44:18 -04:00
Daniel Lemire ffe96dde19 Update HACKING.md 2023-03-22 09:02:24 -04:00
Daniel Lemire f9664cf184 Update HACKING.md 2023-03-22 08:56:12 -04:00
Daniel Lemire 830323f383 Update HACKING.md 2023-03-22 08:55:26 -04:00
Daniel Lemire eb8e15d9c0 Update HACKING.md 2023-03-22 08:53:44 -04:00
Daniel Lemire 3571f3a8d0 Additional documentation. 2023-03-18 16:38:16 -04:00
Daniel Lemire 4b13020d65 Patch release 2023-03-13 21:44:34 -04:00
Daniel Lemire 75c6c950e5 Ok. 2023-03-13 21:26:32 -04:00
Daniel Lemire 5ad7698af1 ... 2023-03-13 17:07:25 -04:00
Daniel Lemire f6f359d3eb Very silly. 2023-03-13 17:05:53 -04:00
Daniel Lemire 185e157e0c Silencing the stupid compiler warnings. 2023-03-13 16:06:10 -04:00
Daniel Lemire 3f3f6cde7c Bad compiler 2023-03-13 13:33:20 -04:00
Daniel Lemire 005054916d Bad compiler 2023-03-13 13:02:01 -04:00
Daniel Lemire 338224849d Disabling memory sanitizer with one function. Might help with issue 1965 (#1966)
* Might help with issue 1965

* Fix macro

* Adding test.

* Stupid compiler

* Silly compiler

* Stupid compilers

* Unnecessary fixes

* Update developer-options.cmake
2023-03-13 12:34:47 -04:00
Ashot Vardanian ada52641b4 Fix: Defining maximal DOM capacity (#1970) 2023-03-13 12:33:37 -04:00
Daniel Lemire ede9d57f22 Adding tests with _GLIBCXX_ASSERTIONS (#1969)
* Allowing users to specify -D_GLIBCXX_ASSERTIONS

* Adding CI tests for GLIBCXX_ASSERTIONS

* Adding curly brackets.
2023-03-13 12:32:43 -04:00
Daniel Lemire 7bf33f6c8a Bad compiler 2023-03-13 12:29:01 -04:00
Daniel Lemire d62b789cb3 More curly. 2023-03-13 10:49:15 -04:00
Daniel Lemire 33dbd44098 Better way to put -Og in debug mode (#1967)
* Better way to put -Og in debug mode

* Silly compiler

* Stupid compiler.

* Stupid compiler.

* Silly compiler
2023-03-13 10:11:33 -04:00
Daniel Lemire e0dcf8adc9 Silly compiler 2023-03-12 20:57:45 -04:00
Daniel Lemire 34dcd33a88 Patch. 2023-03-09 11:24:06 -05:00
Daniel Lemire 17a1a8e187 Adding -Og to debug builds. (#1964)
* Adding -Og to debug builds.

* Stupid compiler.

* bad, bad, bad compiler
2023-03-09 11:14:42 -05:00
Daniel Lemire 77ad00b63c Producing a minor patch release 2023-03-09 11:13:01 -05:00
Daniel Lemire 25e7f1765a Various tweaks. 2023-03-09 10:42:23 -05:00
Daniel Lemire 8d01255ceb Update basics.md 2023-03-09 08:27:32 -05:00
Daniel Lemire 8ec016b093 Update README.md 2023-03-08 15:30:52 -05:00
Daniel Lemire 9e726b9856 Removing obsolete badge 2023-03-08 14:50:37 -05:00
Daniel Lemire 312f1f26d3 Now obsolete. 2023-03-08 14:50:06 -05:00
Daniel Lemire ba50571d66 Merge branch 'master' of github.com:simdjson/simdjson 2023-03-06 16:05:10 -05:00
Daniel Lemire 4a587cb898 Adding guards. 2023-03-06 16:04:42 -05:00
Daniel Lemire bdf9123c92 Update README.md 2023-03-06 11:08:56 -05:00
Dominik Lohmann 6e4c29b7c5 Fix export component for simdjsonTargets.cmake (#1962)
This fixes a bug that caused simdjsonTargets.cmake not to be included in
CPack-generated packages, which—unlike `cmake --install`—does not
pick up this mislabeled install component.

I git-grepped through the code base, after this change all components
are either `simdjson_Development` or `simdjson_Runtime`.
2023-03-04 17:34:40 -05:00
94 changed files with 2196 additions and 893 deletions
+8 -6
View File
@@ -12,24 +12,26 @@ Before submitting an issue, please ensure that you have read the documentation:
* Basics is an overview of how to use simdjson and its APIs: https://github.com/simdjson/simdjson/blob/master/doc/basics.md * Basics is an overview of how to use simdjson and its APIs: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
* Performance shows some more advanced scenarios and how to tune for them: https://github.com/simdjson/simdjson/blob/master/doc/performance.md * Performance shows some more advanced scenarios and how to tune for them: https://github.com/simdjson/simdjson/blob/master/doc/performance.md
* Contributing: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md * Contributing: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md
* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017). * We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017). If you wish to support features that are not part of RFC 8259, then you should not refer to your issue as a bug.
**Describe the bug** **Describe the bug**
A clear and concise description of what the bug is. A clear and concise description of what the bug is. A bug is a failure to build with normal compiler settings or a misbehaviour: when running the code, you get a result that differs from the expected result from our documentation.
A compiler or static-analyzer warning is not a bug. A compiler or static-analyzer warning is not a bug. It is possible with tools such as Visual Studio to require that rarely enabled warnings are considered errors. Do not report such cases as bugs. We do accept pull requests if you want to silence warnings issued by code analyzers, however.
We are committed to providing good documentation. We accept the lack of documentation or a misleading documentation as a bug (a 'documentation bug'). We are committed to providing good documentation. We accept the lack of documentation or a misleading documentation as a bug (a 'documentation bug').
An unexpected poor software performance can be accepted as a bug (a 'performance bug').
We accept the identification of an issue by a sanitizer or some checker tool (e.g., valgrind) as a bug, but you must first ensure that it is not a false positive. We accept the identification of an issue by a sanitizer or some checker tool (e.g., valgrind) as a bug, but you must first ensure that it is not a false positive.
We recommend that you run your tests using different optimization levels. We recommend that you run your tests using different optimization levels. In particular, we recommend your run tests with the simdjson library and you code compiled in debug mode. The simdjson then sets the SIMDJSON_DEVELOPMENT_CHECKS macro to 1, and this triggers additional checks on your code and on the internals of the library. If possible, we recommend that you run tests with sanitizers (e.g., see [No more leaks with sanitize flags in gcc and clang](https://lemire.me/blog/2016/04/20/no-more-leaks-with-sanitize-flags-in-gcc-and-clang/)). You can compile the library with sanitizers for debugging purposes (e.g., set SIMDJSON_SANITIZE to ON using CMake), but you should also turn on sanitizers on your own code. You may also use tools like valgrind or the commercial equivalent.
Before reporting a bug, please ensure that you have read our documentation. Before reporting a bug, please ensure that you have read our documentation.
**To Reproduce** **To Reproduce**
Steps to reproduce the behaviour: provide a code sample if possible. Steps to reproduce the behaviour: provide a code sample if possible. Please provide a complete test with data. Remember that a bug is either a failure to build or an unexpected result when running the code.
If we cannot reproduce the issue, then we cannot address it. Note that a stack trace from your own program is not enough. A sample of your source code is insufficient: please provide a complete test for us to reproduce the issue. Please reduce the issue: use as small and as simple an example of the bug as possible. If we cannot reproduce the issue, then we cannot address it. Note that a stack trace from your own program is not enough. A sample of your source code is insufficient: please provide a complete test for us to reproduce the issue. Please reduce the issue: use as small and as simple an example of the bug as possible.
@@ -51,6 +53,6 @@ We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux syste
**Indicate whether you are willing or able to provide a bug fix as a pull request** **Indicate whether you are willing or able to provide a bug fix as a pull request**
If you plan to contribute to simdjson, please read our If you plan to contribute to simdjson, please read our guide:
* CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our * CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
* HACKING guide: https://github.com/simdjson/simdjson/blob/master/HACKING.md * HACKING guide: https://github.com/simdjson/simdjson/blob/master/HACKING.md
+29
View File
@@ -0,0 +1,29 @@
name: Ubuntu ppc64le (GCC 11)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: uraimo/run-on-arch-action@v2
name: Test
id: runcmd
with:
arch: ppc64le
distro: ubuntu_latest
githubToken: ${{ github.token }}
install: |
apt-get update -q -y
apt-get install -y cmake make g++
run: |
cmake -DCMAKE_BUILD_TYPE=Release -B build
cmake --build build -j=2
ctest --output-on-failure --test-dir build
-29
View File
@@ -1,29 +0,0 @@
name: Performance check on Ubuntu 18.04 CI (GCC 7)
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake
run: |
mkdir build &&
cd build &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_ENABLE_DOM_CHECKPERF=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . --target checkperf &&
ctest --output-on-failure -R checkperf ubuntu18-checkperf.yml
-25
View File
@@ -1,25 +0,0 @@
name: Ubuntu 18.04 CI (GCC 7) with Thread Sanitizer
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake
run: |
mkdir build &&
cd build &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE_THREADS=ON .. &&
cmake --build . --target document_stream_tests --target ondemand_document_stream_tests --target parse_many_test &&
ctest --output-on-failure -R parse_many_test &&
ctest --output-on-failure -R document_stream_tests
-25
View File
@@ -1,25 +0,0 @@
name: Ubuntu 18.04 CI (GCC 7)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake
run: |
mkdir build &&
cd build &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
ctest -j --output-on-failure -LE explicitonly &&
make install &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json
+9 -2
View File
@@ -14,10 +14,17 @@ jobs:
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake - name: Use cmake with address sanitizer
run: | run: |
mkdir builddebug && mkdir builddebug &&
cd builddebug && cd builddebug &&
cmake -DSIMDJSON_SANITIZE=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. && cmake -DSIMDJSON_SANITIZE=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . && cmake --build . &&
ctest -j --output-on-failure -LE explicitonly ctest -j --output-on-failure -LE explicitonly
- name: Use cmake with undefined sanitizer
run: |
mkdir builddebugundefsani &&
cd builddebugundefsani &&
cmake -DSIMDJSON_SANITIZE_UNDEFINED=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . &&
ctest -j --output-on-failure -LE explicitonly
+33
View File
@@ -0,0 +1,33 @@
name: Ubuntu 22.04 CI (GCC 12, CXX 20)
on: [push, pull_request]
jobs:
ubuntu-build:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
with:
path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: Use cmake
run: |
mkdir builddebug &&
cd builddebug &&
CXX=g++-12 cmake -DSIMDJSON_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake --build . &&
ctest -j --output-on-failure -LE explicitonly &&
cd .. &&
mkdir build &&
cd build &&
CXX=g++-12 cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake --build . &&
ctest -j --output-on-failure -LE explicitonly &&
cmake --install . &&
echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json &&
cd ../tests/installation_tests/find &&
mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX:PATH=../../../build/destination .. && cmake --build .
@@ -1,5 +1,4 @@
name: Ubuntu 18.04 CI (LLVM 7) name: Ubuntu 22.04 CI GCC 12 with GLIBCXX_ASSERTIONS
on: [push, pull_request] on: [push, pull_request]
jobs: jobs:
@@ -7,25 +6,19 @@ jobs:
if: >- if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') && ! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]') ! contains(toJSON(github.event.commits.*.message), '[skip github]')
runs-on: ubuntu-18.04 runs-on: ubuntu-22.04
env:
CC: clang-7
CXX: clang++-7
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
- uses: actions/cache@v3 - uses: actions/cache@v3
with: with:
path: dependencies/.cache path: dependencies/.cache
key: ${{ hashFiles('dependencies/CMakeLists.txt') }} key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
- name: install clang 7 - name: Install gcc12
run: | run: sudo apt-get install -y g++-12
sudo apt update
sudo apt install clang-7
- name: Use cmake - name: Use cmake
run: | run: |
mkdir build && mkdir build &&
cd build && cd build &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. && CXX=g++-12 cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GLIBCXX_ASSERTIONS=ON -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON .. &&
cmake --build . && cmake --build . &&
ctest -j --output-on-failure -LE explicitonly ctest . -E avoid_
+44
View File
@@ -0,0 +1,44 @@
name: VS17-CI CXX20
on: [push, pull_request]
jobs:
ci:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
name: windows-vs17
runs-on: windows-latest
strategy:
fail-fast: false
matrix:
include:
- {gen: Visual Studio 17 2022, arch: Win32, shared: ON}
- {gen: Visual Studio 17 2022, arch: Win32, shared: OFF}
- {gen: Visual Studio 17 2022, arch: x64, shared: ON}
- {gen: Visual Studio 17 2022, arch: x64, shared: OFF}
steps:
- name: checkout
uses: actions/checkout@v2
- name: Configure
run: |
cmake -DSIMDJSON_CXX_STANDARD=20 -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
- name: Build Debug
run: cmake --build build --config Debug --verbose
- name: Build Release
run: cmake --build build --config Release --verbose
- name: Run Release tests
run: |
cd build
ctest -C Release -LE explicitonly --output-on-failure
- name: Run Debug tests
run: |
cd build
ctest -C Debug -LE explicitonly --output-on-failure
- name: Install
run: |
cmake --install build --config Release
- name: Test Installation
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -B build_install_test tests/installation_tests/find
cmake --build build_install_test --config Release
+2 -1
View File
@@ -96,6 +96,7 @@
"future": "cpp", "future": "cpp",
"queue": "cpp", "queue": "cpp",
"shared_mutex": "cpp", "shared_mutex": "cpp",
"ranges": "cpp" "ranges": "cpp",
"span": "cpp"
} }
} }
+22 -5
View File
@@ -3,7 +3,7 @@ cmake_minimum_required(VERSION 3.14)
project( project(
simdjson simdjson
# The version number is modified by tools/release.py # The version number is modified by tools/release.py
VERSION 3.1.3 VERSION 3.2.0
DESCRIPTION "Parsing gigabytes of JSON per second" DESCRIPTION "Parsing gigabytes of JSON per second"
HOMEPAGE_URL "https://simdjson.org/" HOMEPAGE_URL "https://simdjson.org/"
LANGUAGES CXX C LANGUAGES CXX C
@@ -20,8 +20,8 @@ string(
# ---- Options, variables ---- # ---- Options, variables ----
# These version numbers are modified by tools/release.py # These version numbers are modified by tools/release.py
set(SIMDJSON_LIB_VERSION "15.0.0" CACHE STRING "simdjson library version") set(SIMDJSON_LIB_VERSION "16.0.0" CACHE STRING "simdjson library version")
set(SIMDJSON_LIB_SOVERSION "15" CACHE STRING "simdjson library soversion") set(SIMDJSON_LIB_SOVERSION "16" CACHE STRING "simdjson library soversion")
option(SIMDJSON_ENABLE_THREADS "Link with thread support" ON) option(SIMDJSON_ENABLE_THREADS "Link with thread support" ON)
@@ -95,6 +95,21 @@ if(
) )
endif() endif()
# GCC and Clang have horrendous Debug builds when using SIMD.
# A common fix is to use '-Og' instead.
# bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
if(
(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" OR
CMAKE_CXX_COMPILER_ID STREQUAL "Clang" OR
CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
)
message(STATUS "Adding -Og to compile flag")
simdjson_add_props(
target_compile_options PRIVATE
$<$<CONFIG:DEBUG>:-Og>
)
endif()
if(SIMDJSON_ENABLE_THREADS) if(SIMDJSON_ENABLE_THREADS)
find_package(Threads REQUIRED) find_package(Threads REQUIRED)
simdjson_add_props(target_link_libraries PUBLIC Threads::Threads) simdjson_add_props(target_link_libraries PUBLIC Threads::Threads)
@@ -149,7 +164,7 @@ install(
EXPORT simdjsonTargets EXPORT simdjsonTargets
NAMESPACE simdjson:: NAMESPACE simdjson::
DESTINATION "${SIMDJSON_INSTALL_CMAKEDIR}" DESTINATION "${SIMDJSON_INSTALL_CMAKEDIR}"
COMPONENT example_Development COMPONENT simdjson_Development
) )
# pkg-config # pkg-config
@@ -236,7 +251,9 @@ add_subdirectory(singleheader)
# #
add_subdirectory(tests) add_subdirectory(tests)
add_subdirectory(examples) add_subdirectory(examples)
add_subdirectory(benchmark) if(CMAKE_SIZEOF_VOID_P EQUAL 8) # we only include the benchmarks on 64-bit systems.
add_subdirectory(benchmark)
endif()
add_subdirectory(fuzz) add_subdirectory(fuzz)
# #
+1 -1
View File
@@ -38,7 +38,7 @@ PROJECT_NAME = simdjson
# could be handy for archiving the generated documentation or if some version # could be handy for archiving the generated documentation or if some version
# control system is used. # control system is used.
PROJECT_NUMBER = "3.1.3" PROJECT_NUMBER = "3.2.0"
# Using the PROJECT_BRIEF tag one can provide an optional one line description # Using the PROJECT_BRIEF tag one can provide an optional one line description
# for a project that appears at the top of each page and should give viewer a # for a project that appears at the top of each page and should give viewer a
+16 -3
View File
@@ -1,3 +1,4 @@
Hacking simdjson Hacking simdjson
================ ================
@@ -6,6 +7,16 @@ Here is wisdom about how to build, test and run simdjson from within the reposit
If you plan to contribute to simdjson, please read our [CONTRIBUTING](https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md) guide. If you plan to contribute to simdjson, please read our [CONTRIBUTING](https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md) guide.
- [Design notes](#design-notes)
- [Developer mode](#developer-mode)
- [Directory Structure and Source](#directory-structure-and-source)
- [Runtime Dispatching](#runtime-dispatching)
- [Regenerating Single-Header Files](#regenerating-single-header-files)
- [Usage (CMake on 64-bit platforms like Linux, FreeBSD or macOS)](#usage-cmake-on-64-bit-platforms-like-linux-freebsd-or-macos)
- [Usage (CMake on 64-bit Windows using Visual Studio 2019)](#usage-cmake-on-64-bit-windows-using-visual-studio-2019)
- [Various References](#various-references)
Design notes Design notes
------------------------------ ------------------------------
@@ -193,7 +204,7 @@ point it gets included (but only once per header). singleheader/simdjson.cpp is
src/simdjson.cpp the same way, except files under generic/ may be included and copy/pasted multiple src/simdjson.cpp the same way, except files under generic/ may be included and copy/pasted multiple
times. times.
### Usage (CMake on 64-bit platforms like Linux, FreeBSD or macOS) ## Usage (CMake on 64-bit platforms like Linux, FreeBSD or macOS)
Requirements: In addition to git, we require a recent version of CMake as well as bash. Requirements: In addition to git, we require a recent version of CMake as well as bash.
@@ -256,7 +267,9 @@ Note that the name of directory (`build`) is arbitrary, you can name it as you w
### Usage (CMake on 64-bit Windows using Visual Studio 2019) ## Usage (CMake on 64-bit Windows using Visual Studio 2019 or better)
Recent versions of Visual Studio support CMake natively, [please refer to the Visual Studio documentation](https://learn.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-170).
We assume you have a common 64-bit Windows PC with at least Visual Studio 2019. We assume you have a common 64-bit Windows PC with at least Visual Studio 2019.
@@ -285,7 +298,7 @@ Furthermore, if you have installed LLVM clang on Windows, for example as a compo
- `cmake --build . -config Release` - `cmake --build . -config Release`
### Various References ## Various References
- [How to implement atoi using SIMD?](https://stackoverflow.com/questions/35127060/how-to-implement-atoi-using-simd) - [How to implement atoi using SIMD?](https://stackoverflow.com/questions/35127060/how-to-implement-atoi-using-simd)
- [Parsing JSON is a Minefield 💣](http://seriot.ch/parsing_json.php) - [Parsing JSON is a Minefield 💣](http://seriot.ch/parsing_json.php)
+37 -22
View File
@@ -1,7 +1,7 @@
![Ubuntu 18.04 CI](https://github.com/simdjson/simdjson/workflows/Ubuntu%2018.04%20CI%20(GCC%207)/badge.svg)
[![Ubuntu 20.04 CI](https://github.com/simdjson/simdjson/workflows/Ubuntu%2020.04%20CI%20(GCC%209)/badge.svg)](https://simdjson.org/plots.html) [![Ubuntu 20.04 CI](https://github.com/simdjson/simdjson/workflows/Ubuntu%2020.04%20CI%20(GCC%209)/badge.svg)](https://simdjson.org/plots.html)
![VS16-CI](https://github.com/simdjson/simdjson/workflows/VS16-CI/badge.svg) ![VS16-CI](https://github.com/simdjson/simdjson/workflows/VS16-CI/badge.svg)
[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/simdjson.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:simdjson)
![MinGW64-CI](https://github.com/simdjson/simdjson/workflows/MinGW64-CI/badge.svg) ![MinGW64-CI](https://github.com/simdjson/simdjson/workflows/MinGW64-CI/badge.svg)
[![][license img]][license] [![][license img]][license]
@@ -28,16 +28,38 @@ This library is part of the [Awesome Modern C++](https://awesomecpp.com) list.
Table of Contents Table of Contents
----------------- -----------------
* [Real-world usage](#real-world-usage)
* [Quick Start](#quick-start) * [Quick Start](#quick-start)
* [Documentation](#documentation) * [Documentation](#documentation)
* [Performance results](#performance-results) * [Performance results](#performance-results)
* [Real-world usage](#real-world-usage)
* [Bindings and Ports of simdjson](#bindings-and-ports-of-simdjson) * [Bindings and Ports of simdjson](#bindings-and-ports-of-simdjson)
* [About simdjson](#about-simdjson) * [About simdjson](#about-simdjson)
* [Funding](#funding) * [Funding](#funding)
* [Contributing to simdjson](#contributing-to-simdjson) * [Contributing to simdjson](#contributing-to-simdjson)
* [License](#license) * [License](#license)
Real-world usage
----------------
- [ClickHouse](https://github.com/ClickHouse/ClickHouse)
- [Facebook/Meta Velox](https://velox-lib.io)
- [milvus](https://github.com/milvus-io/milvus)
- [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer)
- [Shopify HeapProfiler](https://github.com/Shopify/heap-profiler)
- [StarRocks](https://github.com/StarRocks/starrocks)
- [Microsoft FishStore](https://github.com/microsoft/FishStore)
- [Intel PCM](https://github.com/intel/pcm)
- [WatermelonDB](https://github.com/Nozbe/WatermelonDB)
- [Apache Doris](https://github.com/apache/doris)
- [Dgraph](https://github.com/dgraph-io/dgraph)
- [UJRPC](https://github.com/unum-cloud/ujrpc)
- [fastgltf](https://github.com/spnda/fastgltf)
- [vast](https://github.com/tenzir/vast)
- [ada-url](https://github.com/ada-url/ada)
If you are planning to use simdjson in a product, please work from one of our releases.
Quick Start Quick Start
----------- -----------
@@ -64,13 +86,13 @@ int main(void) {
ondemand::document tweets = parser.iterate(json); ondemand::document tweets = parser.iterate(json);
std::cout << uint64_t(tweets["search_metadata"]["count"]) << " results." << std::endl; std::cout << uint64_t(tweets["search_metadata"]["count"]) << " results." << std::endl;
} }
```
```
3. `c++ -o quickstart quickstart.cpp simdjson.cpp` 3. `c++ -o quickstart quickstart.cpp simdjson.cpp`
4. `./quickstart` 4. `./quickstart`
```
```
100 results. 100 results.
``` ```
Documentation Documentation
------------- -------------
@@ -81,7 +103,13 @@ Usage documentation is available:
* [Performance](doc/performance.md) shows some more advanced scenarios and how to tune for them. * [Performance](doc/performance.md) shows some more advanced scenarios and how to tune for them.
* [Implementation Selection](doc/implementation-selection.md) describes runtime CPU detection and * [Implementation Selection](doc/implementation-selection.md) describes runtime CPU detection and
how you can work with it. how you can work with it.
* [API](https://simdjson.org/api/1.0.0/annotated.html) contains the automatically generated API documentation. * [API](https://simdjson.github.io/simdjson/) contains the automatically generated API documentation.
Some users may want to browse code along with the compiled assembly. You want to check out the following lists of examples:
* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/98Kx9Kqjn)
* [simdjson examples with errors without exceptions](https://godbolt.org/z/PKG7GdbPo)
Performance results Performance results
------------------- -------------------
@@ -110,21 +138,6 @@ For NDJSON files, we can exceed 3 GB/s with [our multithreaded parsing function
Real-world usage
----------------
- [ClickHouse](https://github.com/ClickHouse/ClickHouse)
- [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer)
- [Shopify HeapProfiler](https://github.com/Shopify/heap-profiler)
- [StarRocks](https://github.com/StarRocks/starrocks)
- [Microsoft FishStore](https://github.com/microsoft/FishStore)
- [Intel PCM](https://github.com/intel/pcm)
- [WatermelonDB](https://github.com/Nozbe/WatermelonDB)
- [Apache Doris](https://github.com/apache/doris)
- [Dgraph](https://github.com/dgraph-io/dgraph)
If you are planning to use simdjson in a product, please work from one of our releases.
Bindings and Ports of simdjson Bindings and Ports of simdjson
------------------------------ ------------------------------
@@ -145,10 +158,12 @@ We distinguish between "bindings" (which just wrap the C++ code) and a port to a
- [simdjson-go](https://github.com/minio/simdjson-go): Go port using Golang assembly. - [simdjson-go](https://github.com/minio/simdjson-go): Go port using Golang assembly.
- [rcppsimdjson](https://github.com/eddelbuettel/rcppsimdjson): R bindings. - [rcppsimdjson](https://github.com/eddelbuettel/rcppsimdjson): R bindings.
- [simdjson_erlang](https://github.com/ChomperT/simdjson_erlang): erlang bindings. - [simdjson_erlang](https://github.com/ChomperT/simdjson_erlang): erlang bindings.
- [simdjsone](https://github.com/saleyn/simdjsone): erlang bindings.
- [lua-simdjson](https://github.com/FourierTransformer/lua-simdjson): lua bindings. - [lua-simdjson](https://github.com/FourierTransformer/lua-simdjson): lua bindings.
- [hermes-json](https://hackage.haskell.org/package/hermes-json): haskell bindings. - [hermes-json](https://hackage.haskell.org/package/hermes-json): haskell bindings.
- [simdjzon](https://github.com/travisstaloch/simdjzon): zig port. - [simdjzon](https://github.com/travisstaloch/simdjzon): zig port.
- [JSON-Simd](https://github.com/rawleyfowler/JSON-simd): Raku bindings. - [JSON-Simd](https://github.com/rawleyfowler/JSON-simd): Raku bindings.
- [JSON::SIMD](https://metacpan.org/pod/JSON::SIMD): Perl bindings; fully-featured JSON module that uses simdjson for decoding.
About simdjson About simdjson
-------------- --------------
-69
View File
@@ -1,69 +0,0 @@
# 0.5
## Highlights
Performance
* Faster and simpler UTF-8 validation with the lookup4 algorithm https://github.com/simdjson/simdjson/pull/993
* We improved the performance of simdjson under Visual Studio by about 25%. Users will still get better performance with clang-cl (+30%) but the gap has been reduced. https://github.com/simdjson/simdjson/pull/1031
Code usability
* In `parse_many`, when parsing streams of JSON documetns, we give to the users runtime control as to whether threads are used (via the parser.threaded attribute). https://github.com/simdjson/simdjson/issues/925
* Prefixed public macros to avoid name clashes with other libraries. https://github.com/simdjson/simdjson/issues/1035
* Better documentation regarding package managers (brew, MSYS2, conan, apt, vcpkg, FreeBSD package manager, etc.).
* Better documentation regarding CMake usage.
Standards
* We improved standard compliance with respect to both the JSON RFC 8259 and JSON Pointer RFC 6901. We added the at_pointer method to nodes for standard-compliant JSON Pointer queries. The legacy `at(std::string_view)` method remains but is deprecated since it is not standard-compliant as per RFC 6901.
* We removed computed GOTOs without sacrificing performance thus improving the C++ standard compliance (since computed GOTOs are compiler-specific extensions).
* Better support for C++20 https://github.com/simdjson/simdjson/pull/1050
# 0.4
## Highlights
- Test coverage has been greatly improved and we have resolved many static-analysis warnings on different systems.
- We added a fast (8GB/s) minifier that works directly on JSON strings.
- We added fast (10GB/s) UTF-8 validator that works directly on strings (any strings, including non-JSON).
- The array and object elements have a constant-time size() method.
- Performance improvements to the API (type(), get<>()).
- The parse_many function (ndjson) has been entirely reworked. It now uses a single secondary thread instead of several new threads.
- We have introduced a faster UTF-8 validation algorithm (lookup3) for all kernels (ARM, x64 SSE, x64 AVX).
- C++11 support for older compilers and systems.
- FreeBSD support (and tests).
- We support the clang front-end compiler (clangcl) under Visual Studio.
- It is now possible to target ARM platforms under Visual Studio.
- The simdjson library will never abort or print to standard output/error.
# 0.3
## Highlights
- **Multi-Document Parsing:** Read a bundle of JSON documents (ndjson) 2-4x faster than doing it
individually. [API docs](https://github.com/simdjson/simdjson/blob/master/doc/basics.md#newline-delimited-json-ndjson-and-json-lines) / [Design Details](https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md)
- **Simplified API:** The API has been completely revamped for ease of use, including a new JSON
navigation API and fluent support for error code *and* exception styles of error handling with a
single API. [Docs](https://github.com/simdjson/simdjson/blob/master/doc/basics.md#the-basics-loading-and-parsing-json-documents)
- **Exact Float Parsing:** Now simdjson parses floats flawlessly *without* any performance loss,
thanks to [great work by @michaeleisel and @lemire](https://github.com/simdjson/simdjson/pull/558).
[Blog Post](https://lemire.me/blog/2020/03/10/fast-float-parsing-in-practice/)
- **Even Faster:** The fastest parser got faster! With a [shiny new UTF-8 validator](https://github.com/simdjson/simdjson/pull/387)
and meticulously refactored SIMD core, simdjson 0.3 is 15% faster than before, running at 2.5 GB/s
(where 0.2 ran at 2.2 GB/s).
## Minor Highlights
- Fallback implementation: simdjson now has a non-SIMD fallback implementation, and can run even on
very old 64-bit machines.
- Automatic allocation: as part of API simplification, the parser no longer has to be preallocated--
it will adjust automatically when it encounters larger files.
- Runtime selection API: We've exposed simdjson's runtime CPU detection and implementation selection
as an API, so you can tell what implementation we detected and test with other implementations.
- Error handling your way: Whether you use exceptions or check error codes, simdjson lets you handle
errors in your style. APIs that can fail return simdjson_result<T>, letting you check the error
code before using the result. But if you are more comfortable with exceptions, skip the error code
and cast straight to T, and exceptions will be thrown automatically if an error happens. Use the
same API either way!
- Error chaining: We also worked to keep non-exception error-handling short and sweet. Instead of
having to check the error code after every single operation, now you can *chain* JSON navigation
calls like looking up an object field or array element, or casting to a string, so that you only
have to check the error code once at the very end.
+3 -3
View File
@@ -521,7 +521,7 @@ static void twitter_image_sizes(State& state) {
set<tuple<uint64_t, uint64_t>> image_sizes; set<tuple<uint64_t, uint64_t>> image_sizes;
for (dom::object tweet : doc["statuses"]) { for (dom::object tweet : doc["statuses"]) {
dom::array media; dom::array media;
if (not (error = tweet["entities"]["media"].get(media))) { if (! (error = tweet["entities"]["media"].get(media))) {
for (dom::object image : media) { for (dom::object image : media) {
for (auto size : image["sizes"].get_object()) { for (auto size : image["sizes"].get_object()) {
image_sizes.emplace(size.value["w"], size.value["h"]); image_sizes.emplace(size.value["w"], size.value["h"]);
@@ -564,7 +564,7 @@ static void error_code_twitter_default_profile(State& state) noexcept {
for (dom::element tweet : tweets) { for (dom::element tweet : tweets) {
dom::object user; dom::object user;
if ((error = tweet["user"].get(user))) { return; } if ((error = tweet["user"].get(user))) { return; }
bool default_profile; bool default_profile{};
if ((error = user["default_profile"].get(default_profile))) { return; } if ((error = user["default_profile"].get(default_profile))) { return; }
if (default_profile) { if (default_profile) {
std::string_view screen_name; std::string_view screen_name;
@@ -638,7 +638,7 @@ static void error_code_twitter_image_sizes(State& state) noexcept {
if ((error = doc["statuses"].get(statuses))) { return; } if ((error = doc["statuses"].get(statuses))) { return; }
for (dom::element tweet : statuses) { for (dom::element tweet : statuses) {
dom::array images; dom::array images;
if (not (error = tweet["entities"]["media"].get(images))) { if (! (error = tweet["entities"]["media"].get(images))) {
for (dom::element image : images) { for (dom::element image : images) {
dom::object sizes; dom::object sizes;
if ((error = image["sizes"].get(sizes))) { return; } if ((error = image["sizes"].get(sizes))) { return; }
+1 -1
View File
@@ -520,7 +520,7 @@ struct benchmarker {
double freqall = (all_stages_without_allocation.best.cycles() / all_stages_without_allocation.best.elapsed_sec()) / 1000000000.0; double freqall = (all_stages_without_allocation.best.cycles() / all_stages_without_allocation.best.elapsed_sec()) / 1000000000.0;
double freqmin = min(freq1, freq2); double freqmin = min(freq1, freq2);
double freqmax = max(freq1, freq2); double freqmax = max(freq1, freq2);
if((freqall < 0.95 * freqmin) or (freqall > 1.05 * freqmax)) { if((freqall < 0.95 * freqmin) || (freqall > 1.05 * freqmax)) {
printf("\nWarning: The processor frequency fluctuates in an expected way!!!\n" printf("\nWarning: The processor frequency fluctuates in an expected way!!!\n"
"Range for stage 1 and stage 2 : [%.3f GHz, %.3f GHz], overall: %.3f GHz.\n", "Range for stage 1 and stage 2 : [%.3f GHz, %.3f GHz], overall: %.3f GHz.\n",
freqmin, freqmax, freqall); freqmin, freqmax, freqall);
+2 -2
View File
@@ -134,7 +134,7 @@ struct event_collector {
#if defined(__linux__) #if defined(__linux__)
LinuxEvents<PERF_TYPE_HARDWARE> linux_events; LinuxEvents<PERF_TYPE_HARDWARE> linux_events;
event_collector(bool quiet = false) : linux_events(vector<int>{ event_collector(simdjson_unused bool quiet = false) : linux_events(vector<int>{
#if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS #if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
PERF_COUNT_HW_CPU_CYCLES, PERF_COUNT_HW_CPU_CYCLES,
PERF_COUNT_HW_INSTRUCTIONS, PERF_COUNT_HW_INSTRUCTIONS,
@@ -145,7 +145,7 @@ struct event_collector {
PERF_COUNT_HW_CACHE_REFERENCES, PERF_COUNT_HW_CACHE_REFERENCES,
PERF_COUNT_HW_CACHE_MISSES PERF_COUNT_HW_CACHE_MISSES
#endif #endif
}, quiet) {} }) {}
bool has_events() { bool has_events() {
return linux_events.is_working(); return linux_events.is_working();
} }
+2 -3
View File
@@ -100,9 +100,8 @@ simdjson2msgpack::to_msgpack(const simdjson::padded_string &json,
recursive_processor_ref(val); recursive_processor_ref(val);
#endif #endif
} }
if (doc.current_location().error() == simdjson::SUCCESS) { if (!doc.at_end()) {
// Example of error detection - this won't be reached on twitter.json in the benchmark. throw "There are unexpectedly tokens after the end of the json in the json2msgpack sample data";
throw "There are unexpectedly tokens after the end of the json in the json2msgpack sample data";
} }
return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf)); return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
} }
+19 -34
View File
@@ -1,22 +1,13 @@
// https://github.com/WojciechMula/toys/blob/master/000helpers/linux-perf-events.h
#pragma once #pragma once
#ifdef __linux__ #ifdef __linux__
#ifdef __has_include
#if __has_include(<asm/unistd.h>)
#include <asm/unistd.h> // for __NR_perf_event_open #include <asm/unistd.h> // for __NR_perf_event_open
#else
#warning "Header asm/unistd.h cannot be found though it is a linux system. Are linux headers missing?"
#endif
#else // no __has_include
// Please insure that linux headers have been installed.
#include <asm/unistd.h> // for __NR_perf_event_open
#endif
#include <linux/perf_event.h> // for perf event constants #include <linux/perf_event.h> // for perf event constants
#include <sys/ioctl.h> // for ioctl #include <sys/ioctl.h> // for ioctl
#include <unistd.h> // for syscall #include <unistd.h> // for syscall
#include <cerrno> // for errno #include <cerrno> // for errno
#include <cstring> // for std::memset #include <cstring> // for memset
#include <stdexcept> #include <stdexcept>
#include <iostream> #include <iostream>
@@ -28,13 +19,11 @@ template <int TYPE = PERF_TYPE_HARDWARE> class LinuxEvents {
perf_event_attr attribs{}; perf_event_attr attribs{};
size_t num_events{}; size_t num_events{};
std::vector<uint64_t> temp_result_vec{}; std::vector<uint64_t> temp_result_vec{};
std::vector<uint64_t> result{}; std::vector<uint64_t> ids{};
std::vector<int> fds{};
bool quiet;
public: public:
explicit LinuxEvents(std::vector<int> config_vec, bool _quiet=false) : fd(0), working(true), quiet{_quiet} { explicit LinuxEvents(std::vector<int> config_vec) : fd(0), working(true) {
std::memset(&attribs, 0, sizeof(attribs)); memset(&attribs, 0, sizeof(attribs));
attribs.type = TYPE; attribs.type = TYPE;
attribs.size = sizeof(attribs); attribs.size = sizeof(attribs);
attribs.disabled = 1; attribs.disabled = 1;
@@ -49,7 +38,7 @@ public:
int group = -1; // no group int group = -1; // no group
num_events = config_vec.size(); num_events = config_vec.size();
result.resize(config_vec.size()); ids.resize(config_vec.size());
uint32_t i = 0; uint32_t i = 0;
for (auto config : config_vec) { for (auto config : config_vec) {
attribs.config = config; attribs.config = config;
@@ -57,22 +46,17 @@ public:
if (_fd == -1) { if (_fd == -1) {
report_error("perf_event_open"); report_error("perf_event_open");
} }
fd = _fd; // fd tracks the last _fd value. ioctl(_fd, PERF_EVENT_IOC_ID, &ids[i++]);
fds.push_back(fd);
ioctl(fd, PERF_EVENT_IOC_ID, &result[i++]);
if (group == -1) { if (group == -1) {
group = fd; group = _fd;
fd = _fd;
} }
} }
temp_result_vec.resize(num_events * 2 + 1); temp_result_vec.resize(num_events * 2 + 1);
} }
~LinuxEvents() { ~LinuxEvents() { if (fd != -1) { close(fd); } }
for (auto tfd : fds) {
if (tfd != -1) { close(tfd); }
}
}
inline void start() { inline void start() {
if (fd != -1) { if (fd != -1) {
@@ -97,10 +81,16 @@ public:
} }
} }
// our actual results are in slots 1,3,5, ... of this structure // our actual results are in slots 1,3,5, ... of this structure
// we really should be checking our result obtained earlier to be safe
for (uint32_t i = 1; i < temp_result_vec.size(); i += 2) { for (uint32_t i = 1; i < temp_result_vec.size(); i += 2) {
results[i / 2] = temp_result_vec[i]; results[i / 2] = temp_result_vec[i];
} }
for (uint32_t i = 2; i < temp_result_vec.size(); i += 2) {
if(ids[i/2-1] != temp_result_vec[i]) {
report_error("event mismatch");
}
}
} }
bool is_working() { bool is_working() {
@@ -108,13 +98,8 @@ public:
} }
private: private:
void report_error(const std::string &context) { void report_error(const std::string &) {
if (!quiet) {
if (working) {
std::cerr << (context + ": " + std::string(strerror(errno))) << std::endl;
}
}
working = false; working = false;
} }
}; };
#endif #endif
+26 -1
View File
@@ -50,6 +50,23 @@ undefined behavior.")
endif() endif()
endif() endif()
option(SIMDJSON_SANITIZE_MEMORY "Sanitize memory" OFF)
if(SIMDJSON_SANITIZE_MEMORY)
message(STATUS "Setting the memory sanitizer.")
add_compile_options(
-fsanitize=memory -fno-sanitize-recover=all
)
link_libraries(
-fsanitize=memory -fno-sanitize-recover=all
)
# Ubuntu bug for GCC 5.0+ (safe for all versions)
if(CMAKE_COMPILER_IS_GNUCC)
link_libraries(-fuse-ld=gold)
endif()
endif()
if(SIMDJSON_SANITIZE_THREADS) if(SIMDJSON_SANITIZE_THREADS)
message(STATUS "Setting both the thread sanitizer \ message(STATUS "Setting both the thread sanitizer \
and the undefined-behavior sanitizer.") and the undefined-behavior sanitizer.")
@@ -73,6 +90,8 @@ if(NOT is_multi_config AND NOT CMAKE_BUILD_TYPE)
if(SIMDJSON_SANITIZE OR SIMDJSON_SANITIZE_UNDEFINED) if(SIMDJSON_SANITIZE OR SIMDJSON_SANITIZE_UNDEFINED)
message(STATUS "No build type selected and you have enabled the sanitizer, \ message(STATUS "No build type selected and you have enabled the sanitizer, \
default to Debug. Consider setting CMAKE_BUILD_TYPE.") default to Debug. Consider setting CMAKE_BUILD_TYPE.")
message(STATUS "Setting debug optimization flag to -O1 to help sanitizer.")
set(CMAKE_CXX_FLAGS_DEBUG "-O1" CACHE STRING "" FORCE)
set(CMAKE_BUILD_TYPE Debug CACHE STRING "Choose the type of build." FORCE) set(CMAKE_BUILD_TYPE Debug CACHE STRING "Choose the type of build." FORCE)
else() else()
message(STATUS "No build type selected, default to Release") message(STATUS "No build type selected, default to Release")
@@ -91,7 +110,8 @@ endif()
# We compile tools, tests, etc. with C++ 17. Override yourself if you need on a # We compile tools, tests, etc. with C++ 17. Override yourself if you need on a
# target. # target.
set(CMAKE_CXX_STANDARD 17) set(SIMDJSON_CXX_STANDARD 17 CACHE STRING "the C++ standard to use for simdjson")
set(CMAKE_CXX_STANDARD ${SIMDJSON_CXX_STANDARD})
set(CMAKE_CXX_STANDARD_REQUIRED ON) set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF) set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_MACOSX_RPATH OFF) set(CMAKE_MACOSX_RPATH OFF)
@@ -144,6 +164,11 @@ else()
) )
endif() endif()
option(SIMDJSON_GLIBCXX_ASSERTIONS "Set _GLIBCXX_ASSERTIONS" OFF)
if (SIMDJSON_GLIBCXX_ASSERTIONS)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_ASSERTIONS")
endif()
# #
# Other optional flags # Other optional flags
# #
+9 -1
View File
@@ -15,7 +15,7 @@ if(SIMDJSON_GOOGLE_BENCHMARKS)
set_off(BENCHMARK_ENABLE_INSTALL) set_off(BENCHMARK_ENABLE_INSTALL)
set_off(BENCHMARK_ENABLE_WERROR) set_off(BENCHMARK_ENABLE_WERROR)
import_dependency(google_benchmarks google/benchmark d572f47) import_dependency(google_benchmarks google/benchmark v1.7.1)
add_dependency(google_benchmarks) add_dependency(google_benchmarks)
endif() endif()
@@ -94,6 +94,14 @@ int main() {}
import_dependency(rapidjson Tencent/rapidjson f54b0e4) import_dependency(rapidjson Tencent/rapidjson f54b0e4)
add_library(rapidjson INTERFACE) add_library(rapidjson INTERFACE)
target_compile_definitions(rapidjson INTERFACE RAPIDJSON_HAS_STDSTRING) target_compile_definitions(rapidjson INTERFACE RAPIDJSON_HAS_STDSTRING)
include (TestBigEndian)
TEST_BIG_ENDIAN(IS_BIG_ENDIAN)
if(IS_BIG_ENDIAN)
target_compile_definitions(rapidjson INTERFACE RAPIDJSON_ENDIAN=1)
else()
target_compile_definitions(rapidjson INTERFACE RAPIDJSON_ENDIAN=0)
endif()
target_compile_definitions(rapidjson INTERFACE RAPIDJSON_HAS_STDSTRING)
target_include_directories(rapidjson SYSTEM INTERFACE target_include_directories(rapidjson SYSTEM INTERFACE
"${rapidjson_SOURCE_DIR}/include") "${rapidjson_SOURCE_DIR}/include")
target_compile_definitions(rapidjson INTERFACE SIMDJSON_COMPETITION_RAPIDJSON) target_compile_definitions(rapidjson INTERFACE SIMDJSON_COMPETITION_RAPIDJSON)
+199 -49
View File
@@ -23,16 +23,18 @@ An overview of what you need to know to use simdjson, with examples.
- [Disabling Exceptions](#disabling-exceptions) - [Disabling Exceptions](#disabling-exceptions)
- [Exceptions](#exceptions) - [Exceptions](#exceptions)
- [Current location in document](#current-location-in-document) - [Current location in document](#current-location-in-document)
- [Checking for trailing content](#checking-for-trailing-content)
- [Rewinding](#rewinding) - [Rewinding](#rewinding)
- [Direct Access to the Raw String](#direct-access-to-the-raw-string)
- [Newline-Delimited JSON (ndjson) and JSON lines](#newline-delimited-json-ndjson-and-json-lines) - [Newline-Delimited JSON (ndjson) and JSON lines](#newline-delimited-json-ndjson-and-json-lines)
- [Parsing Numbers Inside Strings](#parsing-numbers-inside-strings) - [Parsing Numbers Inside Strings](#parsing-numbers-inside-strings)
- [Dynamic Number Types](#dynamic-number-types) - [Dynamic Number Types](#dynamic-number-types)
- [Raw Strings](#raw-strings) - [Raw Strings](#raw-strings)
- [General Direct Access to the Raw JSON String](#general-direct-access-to-the-raw-json-string)
- [Thread Safety](#thread-safety) - [Thread Safety](#thread-safety)
- [Standard Compliance](#standard-compliance) - [Standard Compliance](#standard-compliance)
- [Backwards Compatibility](#backwards-compatibility) - [Backwards Compatibility](#backwards-compatibility)
- [Examples](#examples) - [Examples](#examples)
- [Performance Tips](#performance-tips)
Requirements Requirements
@@ -143,6 +145,12 @@ As required by the standard, your JSON document should be in a Unicode (UTF-8) s
string, from the beginning to the end, needs to be valid: we do not attempt to tolerate bad string, from the beginning to the end, needs to be valid: we do not attempt to tolerate bad
inputs before or after a document. inputs before or after a document.
For efficiency reasons, simdjson requires a string with a few bytes (`simdjson::SIMDJSON_PADDING`)
at the end, these bytes may be read but their content does not affect the parsing. In practice,
it means that the JSON inputs should be stored in a memory region with `simdjson::SIMDJSON_PADDING`
extra bytes at the end. You do not have to set these bytes to specific values though you may
want to if you want to avoid runtime warnings with some sanitizers.
The simdjson library offers a tree-like [API](https://en.wikipedia.org/wiki/API), which you can The simdjson library offers a tree-like [API](https://en.wikipedia.org/wiki/API), which you can
access by creating a `ondemand::parser` and calling the `iterate()` method. The iterate method access by creating a `ondemand::parser` and calling the `iterate()` method. The iterate method
quickly indexes the input string and may detect some errors. The following example illustrates quickly indexes the input string and may detect some errors. The following example illustrates
@@ -154,8 +162,7 @@ auto json = padded_string::load("twitter.json"); // load JSON file 'twitter.json
ondemand::document doc = parser.iterate(json); // position a pointer at the beginning of the JSON data ondemand::document doc = parser.iterate(json); // position a pointer at the beginning of the JSON data
``` ```
You can also create a padded string---for efficiency reasons, simdjson requires a string You can also create a padded string---and call `iterate()`:
with a few bytes (`simdjson::SIMDJSON_PADDING`) at the end---and calling `iterate()`:
```c++ ```c++
ondemand::parser parser; ondemand::parser parser;
@@ -175,11 +182,33 @@ ondemand::document doc = parser.iterate(json, strlen(json), sizeof(json));
The simdjson library will also accept `std::string` instances, as long as the `capacity()` of The simdjson library will also accept `std::string` instances, as long as the `capacity()` of
the string exceeds the `size()` by at least `SIMDJSON_PADDING`. You can increase the `capacity()` with the `reserve()` function of your strings. the string exceeds the `size()` by at least `SIMDJSON_PADDING`. You can increase the `capacity()` with the `reserve()` function of your strings.
You can copy your data directly on a `simdjson::padded_string` as follows:
```c++
const char * data = "my data"; // 7 bytes
simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer
```
Or as follows...
```c++
std::string data = "my data";
simdjson::padded_string my_padded_data(data); // copies to a padded buffer
```
We recommend against creating many `std::string` or many `std::padding_string` instances in your application to store your JSON data. We recommend against creating many `std::string` or many `std::padding_string` instances in your application to store your JSON data.
Consider reusing the same buffers and limiting memory allocations. Consider reusing the same buffers and limiting memory allocations.
By default, the simdjson library throws exceptions (`simdjson_error`) on errors. We omit `try`-`catch` clauses from our illustrating examples: if you omit `try`-`catch` in your code, an uncaught exception will halt your program. It is also possible to use simdjson without generating exceptions, and you may even build the library without exception support at all. See [Error Handling](#error-handling) for details. By default, the simdjson library throws exceptions (`simdjson_error`) on errors. We omit `try`-`catch` clauses from our illustrating examples: if you omit `try`-`catch` in your code, an uncaught exception will halt your program. It is also possible to use simdjson without generating exceptions, and you may even build the library without exception support at all. See [Error Handling](#error-handling) for details.
Some users may want to browse code along with the compiled assembly. You want to check out the following lists of examples:
* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/98Kx9Kqjn)
* [simdjson examples with errors without exceptions](https://godbolt.org/z/PKG7GdbPo)
Documents are Iterators Documents are Iterators
----------------------- -----------------------
@@ -301,20 +330,24 @@ support for users who avoid exceptions. See [the simdjson error handling documen
* **Validate What You Use:** When calling `iterate`, the document is quickly indexed. If it is * **Validate What You Use:** When calling `iterate`, the document is quickly indexed. If it is
not a valid Unicode (UTF-8) string or if there is an unclosed string, an error may be reported right away. not a valid Unicode (UTF-8) string or if there is an unclosed string, an error may be reported right away.
However, it is not fully validated. On Demand only fully validates the values you use and the However, it is not fully validated. On Demand only fully validates the values you use and the
structure leading to it. structure leading to it. It means that at every step as you traverse the document, you may encounter an error. You can handle errors either with exceptions or with error codes.
* **Extracting Values:** You can cast a JSON element to a native type: * **Extracting Values:** You can cast a JSON element to a native type:
`double(element)`. This works for `std::string_view`, double, uint64_t, int64_t, bool, `double(element)`. This works for `std::string_view`, double, uint64_t, int64_t, bool,
ondemand::object and ondemand::array. We also have explicit methods such as `get_string()`, `get_double()`, ondemand::object and ondemand::array. We also have explicit methods such as `get_string()`, `get_double()`,
`get_uint64()`, `get_int64()`, `get_bool()`, `get_object()` and `get_array()`. After a cast or an explicit method, `get_uint64()`, `get_int64()`, `get_bool()`, `get_object()` and `get_array()`. After a cast or an explicit method,
the number, string or boolean will be parsed, or the initial `[` or `{` will be verified. An exception is thrown if the number, string or boolean will be parsed, or the initial `{` or `[` will be verified for ondemand::object and ondemand::array. An exception is thrown if
the cast is not possible. The `get_string()` returns a valid UTF-8 string, after the cast is not possible. Importantly, when getting an ondemand::object or ondemand::array instance, its content is
not validated: you are only guaranteed that the corresponding initial character (`{` or `[`) is present. Thus,
for example, you could have an ondemand::object instance pointing at the invalid JSON `{ "this is not a valid object" }`: the validation occurs as you access the content.
The `get_string()` returns a valid UTF-8 string, after
unescaping characters as needed: unmatched surrogate pairs are treated as an error unless you unescaping characters as needed: unmatched surrogate pairs are treated as an error unless you
pass `true` (`get_string(true)`) as a parameter to get replacement characters where errors pass `true` (`get_string(true)`) as a parameter to get replacement characters where errors
occur. If you somehow need to access non-UTF-8 strings in a lossless manner occur. If you somehow need to access non-UTF-8 strings in a lossless manner
(e.g., if you strings contain unpaired surrogates), you may use the `get_wobbly_string()` function to get a string in the [WTF-8 format](https://simonsapin.github.io/wtf-8). (e.g., if you strings contain unpaired surrogates), you may use the `get_wobbly_string()` function to get a string in the [WTF-8 format](https://simonsapin.github.io/wtf-8).
Or you may pass `true` as a parameter to the
When calling `get_uint64()` and `get_int64()`, if the number does not fit in a corresponding When calling `get_uint64()` and `get_int64()`, if the number does not fit in a corresponding
64-bit integer type, it is also considered an error. 64-bit integer type, it is also considered an error. When parsing numbers or other scalar values, the library checks
that the value is followed by an expected character, thus you *may* get a number parsing error when accessing the digits
as an integer in the following strings: `{"number":12332a`, `{"number":12332\0`, `{"number":12332` (the digits appear at the end). We always abide by the [RFC 8259](https://www.tbray.org/ongoing/When/201x/2017/12/14/rfc8259.html) JSON specification so that, for example, numbers prefixed by the `+` sign are in error.
> IMPORTANT NOTE: values can only be parsed once. Since documents are *iterators*, once you have > IMPORTANT NOTE: values can only be parsed once. Since documents are *iterators*, once you have
> parsed a value (such as by casting to double), you cannot get at it again. It is an error to call > parsed a value (such as by casting to double), you cannot get at it again. It is an error to call
@@ -322,8 +355,8 @@ support for users who avoid exceptions. See [the simdjson error handling documen
* **Field Access:** To get the value of the "foo" field in an object, use `object["foo"]`. This will * **Field Access:** To get the value of the "foo" field in an object, use `object["foo"]`. This will
scan through the object looking for the field with the matching string, doing a character-by-character scan through the object looking for the field with the matching string, doing a character-by-character
comparison. For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do comparison. For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do
not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. If you consume an not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. Keep in mind that On Demand does not buffer or save the result of the parsing: if you repeatedly access `object["foo"]`, then it must repeatedly seek the key and parse the content. The library does not provide a distinct function to check if a key is present, instead we recommend you attempt to access the key: e.g., by doing `ondemand::value val{}; if(!object["foo"].get(val)) {...}`, you have that `val` contains the requested value inside the if clause. It is your responsability as a user to temporarily keep a reference to the value (`auto v = object["foo"]`), or to consume the content and store it in your own data structures. If you consume an
object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]`, your code object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]` then your code
is in error. Furthermore, you can only consume one field at a time, on the same object. The is in error. Furthermore, you can only consume one field at a time, on the same object. The
value instance you get from `content["bids"]` becomes invalid when you call `content["asks"]`. value instance you get from `content["bids"]` becomes invalid when you call `content["asks"]`.
If you have retrieved `content["bids"].get_array()` and you later call If you have retrieved `content["bids"].get_array()` and you later call
@@ -338,8 +371,9 @@ support for users who avoid exceptions. See [the simdjson error handling documen
> to support escaped keys, the method `unescaped_key()` provides the desired unescaped keys by > to support escaped keys, the method `unescaped_key()` provides the desired unescaped keys by
> parsing and writing out the unescaped keys to a string buffer and returning a `std::string_view` > parsing and writing out the unescaped keys to a string buffer and returning a `std::string_view`
> instance. You should expect a performance penalty when using `unescaped_key()`. > instance. You should expect a performance penalty when using `unescaped_key()`.
>
> ```c++ > ```c++
> auto json = R"({"k\u0065y": 1})"_padded; > auto json = R"({"k\u0065y": 1})"_padded;
> ondemand::parser parser; > ondemand::parser parser;
> auto doc = parser.iterate(json); > auto doc = parser.iterate(json);
> ondemand::object object = doc.get_object(); > ondemand::object object = doc.get_object();
@@ -446,7 +480,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
``` ```
This examples also show how we can string several operations and only check for the error once, a strategy we call *error chaining*. This examples also show how we can string several operations and only check for the error once, a strategy we call *error chaining*.
Though error chaining makes the code very compact, it also makes error reporting less precise: in this instance, you may get the Though error chaining makes the code very compact, it also makes error reporting less precise: in this instance, you may get the
same error whether the field "str", "123" or "abc" is missing. If you need to break down error handling per operation, avoid error chaining. same error whether the field "str", "123" or "abc" is missing. If you need to break down error handling per operation, avoid error chaining. Furthermore, you should be mindful that chaining that harm performance by encouraging redundancies: writing both `doc["str"]["123"]["abc"].get(value)` and `doc["str"]["123"]["zyw"].get(value)` in the same program may force multiple accesses to the same keys (`"str"` and `"123"`).
* **Counting elements in arrays:** Sometimes it is useful to scan an array to determine its length prior to parsing it. * **Counting elements in arrays:** Sometimes it is useful to scan an array to determine its length prior to parsing it.
For this purpose, `array` instances have a `count_elements` method. Users should be For this purpose, `array` instances have a `count_elements` method. Users should be
aware that the `count_elements` method can be costly since it requires scanning the aware that the `count_elements` method can be costly since it requires scanning the
@@ -1018,7 +1052,7 @@ bool parse() {
cout << "Make/Model: " << make << "/" << model << endl; cout << "Make/Model: " << make << "/" << model << endl;
// Casting a JSON element to an integer // Casting a JSON element to an integer
uint64_t year; uint64_t year{};
error = car["year"].get(year); error = car["year"].get(year);
if(error) { std::cerr << error << std::endl; return false; } if(error) { std::cerr << error << std::endl; return false; }
cout << "- This car is " << 2020 - year << " years old." << endl; cout << "- This car is " << 2020 - year << " years old." << endl;
@@ -1116,10 +1150,32 @@ int main(void) {
} }
``` ```
You can do handle errors gracefully as well...
```C++
#include <iostream>
#include "simdjson.h"
int main(void) {
simdjson::ondemand::parser parser;
simdjson::padded_string json_string;
simdjson::ondemand::document doc;
try {
json_string = padded_string::load("twitter.json");
doc = parser.iterate(json_string);
uint64_t identifier = doc["statuses"].at(0)["id"];
std::cout << identifier << std::endl;
} catch (simdjson::simdjson_error &error) {
std::cerr << "JSON error: " << error.what() << " near "
<< doc.current_location() << " in " << json_string << std::endl;
}
}
```
### Current location in document ### Current location in document
Sometimes, it might be helpful to know the current location in the document during iteration. This is especially useful when encountering errors. The `current_location()` method on a Sometimes, it might be helpful to know the current location in the document during iteration. This is especially useful when encountering errors. The `current_location()` method on a
`document` instances makes it easy to identify common JSON errors. Users can call the `current_location()` method on a validdocument instance to retrieve a `const char *` pointer to the current location in the document. This method also works even after an error has invalidated the document and the parser (e.g. `TAPE_ERROR`, `INCOMPLETE_ARRAY_OR_OBJECT`). `document` instances makes it easy to identify common JSON errors. Users can call the `current_location()` method on a valid document instance to retrieve a `const char *` pointer to the current location in the document. This method also works even after an error has invalidated the document and the parser (e.g. `TAPE_ERROR`, `INCOMPLETE_ARRAY_OR_OBJECT`).
When the input was a `padding_string` or another null-terminated source, then you may When the input was a `padding_string` or another null-terminated source, then you may
use the `const char *` pointer as a C string. As an example, consider the following use the `const char *` pointer as a C string. As an example, consider the following
example where we used the exception-free simdjson interface: example where we used the exception-free simdjson interface:
@@ -1132,6 +1188,12 @@ int64_t i;
auto error = doc["integer"].get_int64().get(i); // Expect to get integer from "integer" key, but get TAPE_ERROR auto error = doc["integer"].get_int64().get(i); // Expect to get integer from "integer" key, but get TAPE_ERROR
if (error) { if (error) {
std::cout << error << std::endl; // Prints TAPE_ERROR error message std::cout << error << std::endl; // Prints TAPE_ERROR error message
// Recover a pointer to the location of the first error:
const char * ptr;
doc.current_location().get(ptr);
// ptr points at 'false, "integer": -343} " which is the location of the error
//
// Because we pad simdjson::padded_string instances with null characters, you may also do the following:
std::cout<< doc.current_location() << std::endl; // Prints "false, "integer": -343} " (location of TAPE_ERROR) std::cout<< doc.current_location() << std::endl; // Prints "false, "integer": -343} " (location of TAPE_ERROR)
} }
``` ```
@@ -1208,6 +1270,32 @@ contains control characters that must be escaped and UNCLOSED_STRING if there
is an unclosed string in the document. We do not provide location information for these is an unclosed string in the document. We do not provide location information for these
errors. errors.
### Checking for trailing content
The parser validates all parsed content, but your code may exhaust the content while
not having processed the entire document. Thus, as a final optional step, you may
call `at_end()` on the document instance. If it returns `false`, then you may
conclude that you have trailing content and that your document is not valid JSON.
You may then use `doc.current_location()` to obtain a pointer to the start of the trailing
content.
```C++
auto json = R"([1, 2] foo ])"_padded;
ondemand::parser parser;
ondemand::document doc = parser.iterate(json);
ondemand::array array = doc.get_array();
for (uint64_t values : array) {
std::cout << values << std::endl;
}
if(!doc.at_end()) {
// In this instance, we will be left pointing at 'foo' since we have consumed the array [1,2].
std::cerr << "trailing content at byte index " << doc.current_location() - json.data() << std::endl;
}
```
The `at_end()` method is equivalent to `doc.current_location().error() == simdjson::SUCCESS` but
more convenient.
Rewinding Rewinding
---------- ----------
@@ -1246,41 +1334,6 @@ internal string buffer. Thus you should consume values only once
even if you can iterate through the array or object more than once. even if you can iterate through the array or object more than once.
If you unescape a string within an array more than once, you have unsafe code. If you unescape a string within an array more than once, you have unsafe code.
Direct Access to the Raw String
--------------------------------
The simdjson library makes explicit assumptions about types. For examples, numbers
must be integers (up to 64-bit integers) or binary64 floating-point numbers. Some users
have different needs. For example, some users might want to support big integers.
The library makes this possible by providing a `raw_json_token` method which returns
a `std::string_view` instance containing the value as a string which you may then
parse as you see fit.
```C++
simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"({"value":12321323213213213213213213213211223})"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::object obj = doc.get_object();
std::string_view token = obj["value"].raw_json_token();
// token has value 12321323213213213213213213213211223, it points inside the input string
```
The `raw_json_token` method even works when the JSON value is a string. In such cases, it
will return the complete string with the quotes and with eventual escaped sequences as in the
source document.
```C++
simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"({"value":"12321323213213213213213213213211223"})"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::object obj = doc.get_object();
string_view token = obj["value"].raw_json_token();
// token has value "12321323213213213213213213213211223", it points inside the input string
```
The `raw_json_token()` should be fast and free of allocation.
Newline-Delimited JSON (ndjson) and JSON lines Newline-Delimited JSON (ndjson) and JSON lines
---------------------------------------------- ----------------------------------------------
@@ -1585,6 +1638,78 @@ JSON string to a user-provided buffer:
} }
``` ```
General Direct Access to the Raw JSON String
--------------------------------
If your value is a string, the `raw_json_string` gives you direct access to the unprocess
string. The simdjson library allows you to have access to the raw underlying JSON
more generally.
The simdjson library makes explicit assumptions about types. For examples, numbers
must be integers (up to 64-bit integers) or binary64 floating-point numbers. Some users
have different needs. For example, some users might want to support big integers.
The library makes this possible by providing a `raw_json_token` method which returns
a `std::string_view` instance containing the value as a string which you may then
parse as you see fit.
```C++
simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"({"value":12321323213213213213213213213211223})"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::object obj = doc.get_object();
std::string_view token = obj["value"].raw_json_token();
// token has value 12321323213213213213213213213211223, it points inside the input string
```
The `raw_json_token` method even works when the JSON value is a string. In such cases, it
will return the complete string with the quotes and with eventual escaped sequences as in the
source document.
```C++
simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"({"value":"12321323213213213213213213213211223"})"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::object obj = doc.get_object();
string_view token = obj["value"].raw_json_token();
// token has value "12321323213213213213213213213211223", it points inside the input string
```
The `raw_json_token()` should be fast and free of allocation.
If you value is an array or an object, `raw_json_token()` returns effectively a single
character (`[`) or (`}`) which is not very useful. For arrays and objects, we have another
method called `raw_json()` which consumes (traverse) the array or the object.
```C++
simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"({"value":123})"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::object obj = doc.get_object();
string_view token = obj.raw_json(); // gives you `{"value":123}`
```
```C++
simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"([1,2,3])"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::array arr = doc.get_array();
string_view token = arr.raw_json(); // gives you `[1,2,3]`
```
Because `raw_json()` consumes to object or the array, if you want to both have
access to the raw string, and also use the array or object, you should call `reset()`.
```C++
simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"({"value":123})"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::object obj = doc.get_object();
string_view token = obj.raw_json(); // gives you `{"value":123}`
obj.reset(); // revise the object
uint64_t x = obj["value"]; // gives me 123
```
Thread Safety Thread Safety
------------- -------------
@@ -1812,3 +1937,28 @@ bool example() {
return true; return true;
} }
``` ```
Performance Tips
--------
- The On Demand front-end works best when doing a single pass over the input: avoid calling `count_elements`, `rewind` and similar methods.
- If you are familiar with assembly language, you may use the online tool godbolt to explore the compiled code. The following example may work: [https://godbolt.org/z/xE4GWs573](https://godbolt.org/z/xE4GWs573).
- Given a field `field` in an object, calling `field.key()` is often faster than `field.unescaped_key()` so if you do not need an unescaped `std::string_view` instance, prefer `field.key()`.
- For release builds, we recommend setting `NDEBUG` pre-processor directive when compiling the `simdjson` library. Importantly, using the optimization flags `-O2` or `-O3` under GCC and LLVM clang does not set the `NDEBUG` directive, you must set it manually (e.g., `-DNDEBUG`).
- For long streams of JSON documents, consider [`iterate_many`](iterate_many.md) and [`parse_many`](parse_many.md) for better performance.
- If possible, refer to each object and array in your code once. For example, the following code repeatedly refers to the `"data"` key to create an object...
```C++
std::string_view make = o["data"]["make"];
std::string_view model = o["data"]["model"];
std::string_view year = o["data"]["year"];
```
We expect that it is more efficient to access the `"data"` key once:
```C++
simdjson::ondemand::object data = o["data"];
std::string_view model = data["model"];
std::string_view year = data["year"];
std::string_view rating = data["rating"];
```
- To better understand the operation of your On Demand parser, and whether it is performing as well as you think it should be, there is a logger feature built in to simdjson! To use it, define the pre-processor directive `SIMDJSON_VERBOSE_LOGGING` prior to including the `simdjson.h` header, which enables logging in simdjson. Run your code. It may generate a lot of logging output; adding printouts from your application that show each section may be helpful. The logs output will show step-by-step information on state, buffer pointer position, depth, and key retrieval status. The log level is set to `INFO` by default, but can be modified with the environment variable `SIMDJSON_LOG_LEVEL`. Setting the environment variable `SIMDJSON_LOG_LEVEL` to `ERROR` will log only errors.
+21
View File
@@ -16,6 +16,7 @@ An overview of what you need to know to use simdjson, with examples.
* [Server Loops: Long-Running Processes and Memory Capacity](#server-loops-long-running-processes-and-memory-capacity) * [Server Loops: Long-Running Processes and Memory Capacity](#server-loops-long-running-processes-and-memory-capacity)
* [Best Use of the DOM API](#best-use-of-the-dom-api) * [Best Use of the DOM API](#best-use-of-the-dom-api)
* [Padding and Temporary Copies](#padding-and-temporary-copies) * [Padding and Temporary Copies](#padding-and-temporary-copies)
* [Performance Tips](#performance-tips)
DOM vs On Demand DOM vs On Demand
---------------------------------------------- ----------------------------------------------
@@ -44,6 +45,20 @@ dom::parser parser;
dom::element doc = parser.parse("[1,2,3]"_padded); // parse a string, the _padded suffix creates a simdjson::padded_string instance dom::element doc = parser.parse("[1,2,3]"_padded); // parse a string, the _padded suffix creates a simdjson::padded_string instance
``` ```
You can copy your data directly on a `simdjson::padded_string` as follows:
```c++
const char * data = "my data"; // 7 bytes
simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer
```
Or as follows...
```c++
std::string data = "my data";
simdjson::padded_string my_padded_data(data); // copies to a padded buffer
```
The parsed document resulting from the `parser.load` and `parser.parse` calls depends on the `parser` instance. Thus the `parser` instance must remain in scope. Furthermore, you must have at most one parsed document in play per `parser` instance. The parsed document resulting from the `parser.load` and `parser.parse` calls depends on the `parser` instance. Thus the `parser` instance must remain in scope. Furthermore, you must have at most one parsed document in play per `parser` instance.
You cannot copy a `parser` instance, you may only move it. You cannot copy a `parser` instance, you may only move it.
@@ -644,3 +659,9 @@ simdjson::dom::element element = parser.parse(padded_json_copy.get(), json_len,
```` ````
Setting the `realloc_if_needed` parameter `false` in this manner may lead to better performance since copies are avoided, but it requires that the user takes more responsibilities: the simdjson library cannot verify that the input buffer was padded with SIMDJSON_PADDING extra bytes. Setting the `realloc_if_needed` parameter `false` in this manner may lead to better performance since copies are avoided, but it requires that the user takes more responsibilities: the simdjson library cannot verify that the input buffer was padded with SIMDJSON_PADDING extra bytes.
Performance Tips
---------------------
- For release builds, we recommend setting `NDEBUG` pre-processor directive when compiling the `simdjson` library. Importantly, using the optimization flags `-O2` or `-O3` under GCC and LLVM clang does not set the `NDEBUG` directrive, you must set it manually (e.g., `-DNDEBUG`).
- For long streams of JSON documents, consider [`iterate_many`](iterate_many.md) and [`parse_many`](parse_many.md) for better performance.
+3 -2
View File
@@ -25,8 +25,9 @@ The current implementations are:
* fallback: A generic implementation that runs on any 64-bit processor. * fallback: A generic implementation that runs on any 64-bit processor.
In many cases, you don't know where your compiled binary is going to run, so simdjson automatically In many cases, you don't know where your compiled binary is going to run, so simdjson automatically
compiles *all* the implementations into the executable. On Intel, it will include 3 implementations compiles *all* the implementations into the executable. On Intel, it will include 4 implementations
(haswell, westmere and fallback), on ARM it will include 2 (arm64 and fallback), and on PPC it will include 2 (ppc64 and fallback). (icelake, haswell, westmere and fallback), on ARM it will include 2 (arm64 and fallback), and on PPC
it will include 2 (ppc64 and fallback).
If you know more about where you're going to run and want to save the space, you can disable any of If you know more about where you're going to run and want to save the space, you can disable any of
these implementations at compile time with `-DSIMDJSON_IMPLEMENTATION_X=0` (where X is ICELAKE, HASWELL, these implementations at compile time with `-DSIMDJSON_IMPLEMENTATION_X=0` (where X is ICELAKE, HASWELL,
+5
View File
@@ -237,3 +237,8 @@ This will print:
``` ```
Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet. Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
Comma separated documents
-----------
`iterate_many` also takes in an option to allow parsing of comma separated documents. In this mode, the entire buffer is processed in 1 batch and batch size will be increased to be as large as the JSON passed. Therefore, the capacity of the parser has to be sufficient to support the batch size set.
+1 -1
View File
@@ -752,7 +752,7 @@ On relevant systems, the On Demand API provides some support for runtime dispatc
Some users wish to run at the best possible speed. Under recent Intel and AMD processors, these users should take additional steps to verify that their code is well optimized. Some users wish to run at the best possible speed. Under recent Intel and AMD processors, these users should take additional steps to verify that their code is well optimized.
Given that the On Demand API offer limited runtime dispatching, it matters that your code is compiled against a specific CPU target. You should verify that the code is compiled against the target you expect. Thankfully, the simdjson library will tell you exactly what it detects as an implementation: `haswell` (AVX2 x64 processors), `westmere` (SSE4 x64 processors), `arm64` (64-bit ARM), `ppc64` (64-bit POWER), `fallback` (others). Under x64 processors, many programmers will want to target `haswell` whereas under ARM, most programmers will want to target `arm64` (and it should do so automatically). The `fallback` is probably only good for testing purposes, not for deployment. Given that the On Demand API offer limited runtime dispatching, it matters that your code is compiled against a specific CPU target. You should verify that the code is compiled against the target you expect. Thankfully, the simdjson library will tell you exactly what it detects as an implementation: `icelake` (AVX512 x64 processors), `haswell` (AVX2 x64 processors), `westmere` (SSE4 x64 processors), `arm64` (64-bit ARM), `ppc64` (64-bit POWER), `fallback` (others). Under x64 processors, many programmers will want to target `haswell` whereas under ARM, most programmers will want to target `arm64` (and it should do so automatically). The `fallback` is probably only good for testing purposes, not for deployment.
```C++ ```C++
std::cout << simdjson::builtin_implementation()->name() << std::endl; std::cout << simdjson::builtin_implementation()->name() << std::endl;
+16 -1
View File
@@ -3,7 +3,10 @@ Performance Notes
simdjson strives to be at its fastest *without tuning*, and generally achieves this. However, there simdjson strives to be at its fastest *without tuning*, and generally achieves this. However, there
are still some scenarios where tuning can enhance performance. are still some scenarios where tuning can enhance performance.
Once your code is tested, we
further encourage you to define `NDEBUG` in your Release builds to disable additional runtime
testing and get the best performance.
* [NDEBUG directive](#ndebug-directive)
* [Reusing the parser for maximum efficiency](#reusing-the-parser-for-maximum-efficiency) * [Reusing the parser for maximum efficiency](#reusing-the-parser-for-maximum-efficiency)
* [Reusing string buffers](#reusing-string-buffers) * [Reusing string buffers](#reusing-string-buffers)
* [Server Loops: Long-Running Processes and Memory Capacity](#server-loops-long-running-processes-and-memory-capacity) * [Server Loops: Long-Running Processes and Memory Capacity](#server-loops-long-running-processes-and-memory-capacity)
@@ -12,6 +15,18 @@ are still some scenarios where tuning can enhance performance.
* [Visual Studio](#visual-studio) * [Visual Studio](#visual-studio)
* [Power Usage and Downclocking](#power-usage-and-downclocking) * [Power Usage and Downclocking](#power-usage-and-downclocking)
NDEBUG directive
-------------
In C/C++, the `NDEBUG` pre-processor directive is not set by default. When it is not set, the simdjson library does
many additional checks that may impact negatively the performance. We recommend that, once your code
is well tested, you define `NDEBUG` directive in your Release builds. The `NDEBUG` directive should be defined
prior to including the `simdjson.h` header.
The `NDEBUG` directive is generally independent from optimization flags. For example, setting `-O3` under
GCC does not set the `NDEBUG` directive.
Reusing the parser for maximum efficiency Reusing the parser for maximum efficiency
----------------------------------------- -----------------------------------------
+4
View File
@@ -9,6 +9,10 @@ namespace {
// but the algorithms do not end up using the returned value. // but the algorithms do not end up using the returned value.
// Sadly, sanitizers are not smart enough to figure it out. // Sadly, sanitizers are not smart enough to figure it out.
SIMDJSON_NO_SANITIZE_UNDEFINED SIMDJSON_NO_SANITIZE_UNDEFINED
// This function can be used safely even if not all bytes have been
// initialized.
// See issue https://github.com/simdjson/simdjson/issues/1965
SIMDJSON_NO_SANITIZE_MEMORY
simdjson_inline int trailing_zeroes(uint64_t input_num) { simdjson_inline int trailing_zeroes(uint64_t input_num) {
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
unsigned long ret; unsigned long ret;
+1 -1
View File
@@ -226,7 +226,7 @@ inline error_code parser::ensure_capacity(document& target_document, size_t desi
} }
simdjson_inline void parser::set_max_capacity(size_t max_capacity) noexcept { simdjson_inline void parser::set_max_capacity(size_t max_capacity) noexcept {
if(max_capacity < MINIMAL_DOCUMENT_CAPACITY) { if(max_capacity > MINIMAL_DOCUMENT_CAPACITY) {
_max_capacity = max_capacity; _max_capacity = max_capacity;
} else { } else {
_max_capacity = MINIMAL_DOCUMENT_CAPACITY; _max_capacity = MINIMAL_DOCUMENT_CAPACITY;
+6 -2
View File
@@ -145,7 +145,9 @@ public:
* ### REQUIRED: Buffer Padding * ### REQUIRED: Buffer Padding
* *
* The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
* those bytes are initialized to, as long as they are allocated. * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
* using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
* SIMDJSON_PADDING bytes to avoid runtime warnings.
* *
* If realloc_if_needed is true (the default), it is assumed that the buffer does *not* have enough padding, * If realloc_if_needed is true (the default), it is assumed that the buffer does *not* have enough padding,
* and it is copied into an enlarged temporary buffer before parsing. Thus the following is safe: * and it is copied into an enlarged temporary buffer before parsing. Thus the following is safe:
@@ -408,7 +410,9 @@ public:
* ### REQUIRED: Buffer Padding * ### REQUIRED: Buffer Padding
* *
* The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
* those bytes are initialized to, as long as they are allocated. * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
* using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
* SIMDJSON_PADDING bytes to avoid runtime warnings.
* *
* ### Threads * ### Threads
* *
@@ -205,4 +205,8 @@ simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> simdj
if (error()) { return error(); } if (error()) { return error(); }
return first.at_pointer(json_pointer); return first.at_pointer(json_pointer);
} }
simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::array>::raw_json() noexcept {
if (error()) { return error(); }
return first.raw_json();
}
} // namespace simdjson } // namespace simdjson
@@ -186,6 +186,8 @@ public:
inline simdjson_result<bool> reset() & noexcept; inline simdjson_result<bool> reset() & noexcept;
simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> at(size_t index) noexcept; simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> at(size_t index) noexcept;
simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> at_pointer(std::string_view json_pointer) noexcept; simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> at_pointer(std::string_view json_pointer) noexcept;
simdjson_inline simdjson_result<std::string_view> raw_json() noexcept;
}; };
} // namespace simdjson } // namespace simdjson
@@ -20,7 +20,7 @@ inline std::string document::to_debug_string() noexcept {
return iter.to_string(); return iter.to_string();
} }
inline simdjson_result<const char *> document::current_location() noexcept { inline simdjson_result<const char *> document::current_location() const noexcept {
return iter.current_location(); return iter.current_location();
} }
@@ -28,6 +28,11 @@ inline int32_t document::current_depth() const noexcept {
return iter.depth(); return iter.depth();
} }
inline bool document::at_end() const noexcept {
return iter.at_end();
}
inline bool document::is_alive() noexcept { inline bool document::is_alive() noexcept {
return iter.is_alive(); return iter.is_alive();
} }
@@ -49,14 +54,24 @@ simdjson_inline simdjson_result<value> document::get_value() noexcept {
// gets called. // gets called.
iter.assert_at_document_depth(); iter.assert_at_document_depth();
switch (*iter.peek()) { switch (*iter.peek()) {
case '[': case '[': {
case '{': // The following lines check that the document ends with ].
auto value_iterator = get_root_value_iterator();
auto error = value_iterator.check_root_array();
if(error) { return error; }
return value(get_root_value_iterator()); return value(get_root_value_iterator());
}
case '{': {
// The following lines would check that the document ends with }.
auto value_iterator = get_root_value_iterator();
auto error = value_iterator.check_root_object();
if(error) { return error; }
return value(get_root_value_iterator());
}
default: default:
// Unfortunately, scalar documents are a special case in simdjson and they cannot // Unfortunately, scalar documents are a special case in simdjson and they cannot
// be safely converted to value instances. // be safely converted to value instances.
return SCALAR_DOCUMENT_AS_VALUE; return SCALAR_DOCUMENT_AS_VALUE;
// return value(get_root_value_iterator());
} }
} }
simdjson_inline simdjson_result<array> document::get_array() & noexcept { simdjson_inline simdjson_result<array> document::get_array() & noexcept {
@@ -497,6 +512,12 @@ simdjson_inline simdjson_result<const char *> simdjson_result<SIMDJSON_IMPLEMENT
return first.current_location(); return first.current_location();
} }
simdjson_inline bool simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::at_end() const noexcept {
if (error()) { return error(); }
return first.at_end();
}
simdjson_inline int32_t simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::current_depth() const noexcept { simdjson_inline int32_t simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::current_depth() const noexcept {
if (error()) { return error(); } if (error()) { return error(); }
return first.current_depth(); return first.current_depth();
+9 -1
View File
@@ -511,7 +511,14 @@ public:
/** /**
* Returns the current location in the document if in bounds. * Returns the current location in the document if in bounds.
*/ */
inline simdjson_result<const char *> current_location() noexcept; inline simdjson_result<const char *> current_location() const noexcept;
/**
* Returns true if this document has been fully parsed.
* If you have consumed the whole document and at_end() returns
* false, then there may be trailing content.
*/
inline bool at_end() const noexcept;
/** /**
* Returns the current depth in the document if in bounds. * Returns the current depth in the document if in bounds.
@@ -720,6 +727,7 @@ public:
simdjson_inline simdjson_result<bool> is_scalar() noexcept; simdjson_inline simdjson_result<bool> is_scalar() noexcept;
simdjson_inline simdjson_result<const char *> current_location() noexcept; simdjson_inline simdjson_result<const char *> current_location() noexcept;
simdjson_inline int32_t current_depth() const noexcept; simdjson_inline int32_t current_depth() const noexcept;
simdjson_inline bool at_end() const noexcept;
simdjson_inline bool is_negative() noexcept; simdjson_inline bool is_negative() noexcept;
simdjson_inline simdjson_result<bool> is_integer() noexcept; simdjson_inline simdjson_result<bool> is_integer() noexcept;
simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::number_type> get_number_type() noexcept; simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::number_type> get_number_type() noexcept;
@@ -84,12 +84,14 @@ simdjson_inline document_stream::document_stream(
ondemand::parser &_parser, ondemand::parser &_parser,
const uint8_t *_buf, const uint8_t *_buf,
size_t _len, size_t _len,
size_t _batch_size size_t _batch_size,
bool _allow_comma_separated
) noexcept ) noexcept
: parser{&_parser}, : parser{&_parser},
buf{_buf}, buf{_buf},
len{_len}, len{_len},
batch_size{_batch_size <= MINIMAL_BATCH_SIZE ? MINIMAL_BATCH_SIZE : _batch_size}, batch_size{_batch_size <= MINIMAL_BATCH_SIZE ? MINIMAL_BATCH_SIZE : _batch_size},
allow_comma_separated{_allow_comma_separated},
error{SUCCESS} error{SUCCESS}
#ifdef SIMDJSON_THREADS_ENABLED #ifdef SIMDJSON_THREADS_ENABLED
, use_thread(_parser.threaded) // we need to make a copy because _parser.threaded can change , use_thread(_parser.threaded) // we need to make a copy because _parser.threaded can change
@@ -107,6 +109,7 @@ simdjson_inline document_stream::document_stream() noexcept
buf{nullptr}, buf{nullptr},
len{0}, len{0},
batch_size{0}, batch_size{0},
allow_comma_separated{false},
error{UNINITIALIZED} error{UNINITIALIZED}
#ifdef SIMDJSON_THREADS_ENABLED #ifdef SIMDJSON_THREADS_ENABLED
, use_thread(false) , use_thread(false)
@@ -290,6 +293,8 @@ inline void document_stream::next_document() noexcept {
if (error) { return; } if (error) { return; }
// Always set depth=1 at the start of document // Always set depth=1 at the start of document
doc.iter._depth = 1; doc.iter._depth = 1;
// consume comma if comma separated is allowed
if (allow_comma_separated) { doc.iter.consume_character(','); }
// Resets the string buffer at the beginning, thus invalidating the strings. // Resets the string buffer at the beginning, thus invalidating the strings.
doc.iter._string_buf_loc = parser->string_buf.get(); doc.iter._string_buf_loc = parser->string_buf.get();
doc.iter._root = doc.iter.position(); doc.iter._root = doc.iter.position();
@@ -222,7 +222,8 @@ private:
ondemand::parser &parser, ondemand::parser &parser,
const uint8_t *buf, const uint8_t *buf,
size_t len, size_t len,
size_t batch_size size_t batch_size,
bool allow_comma_separated
) noexcept; ) noexcept;
/** /**
@@ -271,6 +272,7 @@ private:
const uint8_t *buf; const uint8_t *buf;
size_t len; size_t len;
size_t batch_size; size_t batch_size;
bool allow_comma_separated;
/** /**
* We are going to use just one document instance. The document owns * We are going to use just one document instance. The document owns
* the json_iterator. It implies that we only ever pass a reference * the json_iterator. It implies that we only ever pass a reference
@@ -208,7 +208,7 @@ inline std::string json_iterator::to_string() const noexcept {
+ std::string(" ]"); + std::string(" ]");
} }
inline simdjson_result<const char *> json_iterator::current_location() noexcept { inline simdjson_result<const char *> json_iterator::current_location() const noexcept {
if (!is_alive()) { // Unrecoverable error if (!is_alive()) { // Unrecoverable error
if (!at_root()) { if (!at_root()) {
return reinterpret_cast<const char *>(token.peek(-1)); return reinterpret_cast<const char *>(token.peek(-1));
@@ -337,6 +337,14 @@ simdjson_inline void json_iterator::reenter_child(token_position position, depth
_depth = child_depth; _depth = child_depth;
} }
simdjson_inline error_code json_iterator::consume_character(char c) noexcept {
if (*peek() == c) {
return_current_and_advance();
return SUCCESS;
}
return TAPE_ERROR;
}
#if SIMDJSON_DEVELOPMENT_CHECKS #if SIMDJSON_DEVELOPMENT_CHECKS
simdjson_inline token_position json_iterator::start_position(depth_t depth) const noexcept { simdjson_inline token_position json_iterator::start_position(depth_t depth) const noexcept {
@@ -358,19 +366,16 @@ simdjson_inline error_code json_iterator::optional_error(error_code _error, cons
return _error; return _error;
} }
template<int N>
simdjson_warn_unused simdjson_inline bool json_iterator::copy_to_buffer(const uint8_t *json, uint32_t max_len, uint8_t (&tmpbuf)[N]) noexcept { simdjson_warn_unused simdjson_inline bool json_iterator::copy_to_buffer(const uint8_t *json, uint32_t max_len, uint8_t *tmpbuf, size_t N) noexcept {
// This function is not expected to be called in performance-sensitive settings.
// Let us guard against silly cases: // Let us guard against silly cases:
if((N < max_len) || (N == 0)) { return false; } if((N < max_len) || (N == 0)) { return false; }
// Truncate whitespace to fit the buffer.
if (max_len > N-1) {
// if (jsoncharutils::is_not_structural_or_whitespace(json[N-1])) { return false; }
max_len = N-1;
}
// Copy to the buffer. // Copy to the buffer.
std::memcpy(tmpbuf, json, max_len); std::memcpy(tmpbuf, json, max_len);
tmpbuf[max_len] = ' '; if(N > max_len) { // We pad whatever remains with ' '.
std::memset(tmpbuf + max_len, ' ', N - max_len);
}
return true; return true;
} }
@@ -238,7 +238,12 @@ public:
*/ */
simdjson_inline error_code optional_error(error_code error, const char *message) noexcept; simdjson_inline error_code optional_error(error_code error, const char *message) noexcept;
template<int N> simdjson_warn_unused simdjson_inline bool copy_to_buffer(const uint8_t *json, uint32_t max_len, uint8_t (&tmpbuf)[N]) noexcept; /**
* Take an input in json containing max_len characters and attempt to copy it over to tmpbuf, a buffer with
* N bytes of capacity. It will return false if N is too small (smaller than max_len) of if it is zero.
* The buffer (tmpbuf) is padded with space characters.
*/
simdjson_warn_unused simdjson_inline bool copy_to_buffer(const uint8_t *json, uint32_t max_len, uint8_t *tmpbuf, size_t N) noexcept;
simdjson_inline token_position position() const noexcept; simdjson_inline token_position position() const noexcept;
/** /**
@@ -250,6 +255,7 @@ public:
simdjson_inline simdjson_result<std::string_view> unescape_wobbly(raw_json_string in) noexcept; simdjson_inline simdjson_result<std::string_view> unescape_wobbly(raw_json_string in) noexcept;
simdjson_inline void reenter_child(token_position position, depth_t child_depth) noexcept; simdjson_inline void reenter_child(token_position position, depth_t child_depth) noexcept;
simdjson_inline error_code consume_character(char c) noexcept;
#if SIMDJSON_DEVELOPMENT_CHECKS #if SIMDJSON_DEVELOPMENT_CHECKS
simdjson_inline token_position start_position(depth_t depth) const noexcept; simdjson_inline token_position start_position(depth_t depth) const noexcept;
simdjson_inline void set_start_position(depth_t depth, token_position position) noexcept; simdjson_inline void set_start_position(depth_t depth, token_position position) noexcept;
@@ -261,7 +267,7 @@ public:
/** /**
* Returns the current location in the document if in bounds. * Returns the current location in the document if in bounds.
*/ */
inline simdjson_result<const char *> current_location() noexcept; inline simdjson_result<const char *> current_location() const noexcept;
/** /**
* Updates this json iterator so that it is back at the beginning of the document, * Updates this json iterator so that it is back at the beginning of the document,
@@ -292,8 +298,8 @@ protected:
friend class raw_json_string; friend class raw_json_string;
friend class parser; friend class parser;
friend class value_iterator; friend class value_iterator;
friend simdjson_inline void logger::log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta) noexcept; friend simdjson_inline void logger::log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta, logger::log_level level) noexcept;
friend simdjson_inline void logger::log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail) noexcept; friend simdjson_inline void logger::log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail, logger::log_level level) noexcept;
}; // json_iterator }; // json_iterator
} // namespace ondemand } // namespace ondemand
+46 -13
View File
@@ -1,3 +1,4 @@
#include <memory>
namespace simdjson { namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION { namespace SIMDJSON_IMPLEMENTATION {
namespace ondemand { namespace ondemand {
@@ -18,36 +19,68 @@ static inline char printable_char(char c) {
} }
} }
static inline log_level get_log_level_from_env()
{
SIMDJSON_PUSH_DISABLE_WARNINGS
SIMDJSON_DISABLE_DEPRECATED_WARNING // Disable CRT_SECURE warning on MSVC: manually verified this is safe
char *lvl = getenv("SIMDJSON_LOG_LEVEL");
SIMDJSON_POP_DISABLE_WARNINGS
if (lvl && simdjson_strcasecmp(lvl, "ERROR") == 0) { return log_level::LOG_ERROR; }
return log_level::LOG_INFO;
}
static inline log_level log_threshold()
{
static log_level threshold = get_log_level_from_env();
return threshold;
}
static inline bool should_log(log_level level)
{
return level >= log_threshold();
}
template<typename... Args>
inline std::string string_format(const std::string& format, const Args&... args)
{
int size_s = std::snprintf(nullptr, 0, format.c_str(), args...) + 1;
auto size = static_cast<size_t>(size_s);
if (size <= 0) return std::string();
std::unique_ptr<char[]> buf(new char[size]);
std::snprintf(buf.get(), size, format.c_str(), args...);
return std::string(buf.get(), buf.get() + size - 1);
}
inline void log_event(const json_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept { inline void log_event(const json_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept {
log_line(iter, "", type, detail, delta, depth_delta); log_line(iter, "", type, detail, delta, depth_delta, log_level::LOG_INFO);
} }
inline void log_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail) noexcept { inline void log_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail) noexcept {
log_line(iter, index, depth, "", type, detail); log_line(iter, index, depth, "", type, detail, log_level::LOG_INFO);
} }
inline void log_value(const json_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept { inline void log_value(const json_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept {
log_line(iter, "", type, detail, delta, depth_delta); log_line(iter, "", type, detail, delta, depth_delta, log_level::LOG_INFO);
} }
inline void log_start_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail) noexcept { inline void log_start_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail) noexcept {
log_line(iter, index, depth, "+", type, detail); log_line(iter, index, depth, "+", type, detail, log_level::LOG_INFO);
if (LOG_ENABLED) { log_depth++; } if (LOG_ENABLED) { log_depth++; }
} }
inline void log_start_value(const json_iterator &iter, const char *type, int delta, int depth_delta) noexcept { inline void log_start_value(const json_iterator &iter, const char *type, int delta, int depth_delta) noexcept {
log_line(iter, "+", type, "", delta, depth_delta); log_line(iter, "+", type, "", delta, depth_delta, log_level::LOG_INFO);
if (LOG_ENABLED) { log_depth++; } if (LOG_ENABLED) { log_depth++; }
} }
inline void log_end_value(const json_iterator &iter, const char *type, int delta, int depth_delta) noexcept { inline void log_end_value(const json_iterator &iter, const char *type, int delta, int depth_delta) noexcept {
if (LOG_ENABLED) { log_depth--; } if (LOG_ENABLED) { log_depth--; }
log_line(iter, "-", type, "", delta, depth_delta); log_line(iter, "-", type, "", delta, depth_delta, log_level::LOG_INFO);
} }
inline void log_error(const json_iterator &iter, const char *error, const char *detail, int delta, int depth_delta) noexcept { inline void log_error(const json_iterator &iter, const char *error, const char *detail, int delta, int depth_delta) noexcept {
log_line(iter, "ERROR: ", error, detail, delta, depth_delta); log_line(iter, "ERROR: ", error, detail, delta, depth_delta, log_level::LOG_ERROR);
} }
inline void log_error(const json_iterator &iter, token_position index, depth_t depth, const char *error, const char *detail) noexcept { inline void log_error(const json_iterator &iter, token_position index, depth_t depth, const char *error, const char *detail) noexcept {
log_line(iter, index, depth, "ERROR: ", error, detail); log_line(iter, index, depth, "ERROR: ", error, detail, log_level::LOG_ERROR);
} }
inline void log_event(const value_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept { inline void log_event(const value_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept {
@@ -71,7 +104,7 @@ inline void log_error(const value_iterator &iter, const char *error, const char
} }
inline void log_headers() noexcept { inline void log_headers() noexcept {
if (LOG_ENABLED) { if (LOG_ENABLED && simdjson_unlikely(should_log(log_level::LOG_INFO))) {
// Technically a static variable is not thread-safe, but if you are using threads // Technically a static variable is not thread-safe, but if you are using threads
// and logging... well... // and logging... well...
static bool displayed_hint{false}; static bool displayed_hint{false};
@@ -121,11 +154,11 @@ inline void log_headers() noexcept {
} }
} }
inline void log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta) noexcept { inline void log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta, log_level level) noexcept {
log_line(iter, iter.position()+delta, depth_t(iter.depth()+depth_delta), title_prefix, title, detail); log_line(iter, iter.position()+delta, depth_t(iter.depth()+depth_delta), title_prefix, title, detail, level);
} }
inline void log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail) noexcept { inline void log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail, log_level level) noexcept {
if (LOG_ENABLED) { if (LOG_ENABLED && simdjson_unlikely(should_log(level))) {
const int indent = depth*2; const int indent = depth*2;
const auto buf = iter.token.buf; const auto buf = iter.token.buf;
printf("| %*s%s%-*s ", printf("| %*s%s%-*s ",
+9 -2
View File
@@ -7,6 +7,11 @@ class value_iterator;
namespace logger { namespace logger {
enum class log_level : int32_t {
LOG_INFO = 0,
LOG_ERROR = 1
};
#if SIMDJSON_VERBOSE_LOGGING #if SIMDJSON_VERBOSE_LOGGING
static constexpr const bool LOG_ENABLED = true; static constexpr const bool LOG_ENABLED = true;
#else #else
@@ -16,9 +21,11 @@ namespace logger {
// We do not want these functions to be 'really inlined' since real inlining is // We do not want these functions to be 'really inlined' since real inlining is
// for performance purposes and if you are using the loggers, you do not care about // for performance purposes and if you are using the loggers, you do not care about
// performance (or should not). // performance (or should not).
template<typename... Args>
static inline std::string string_format(const std::string& format, const Args&... args);
static inline void log_headers() noexcept; static inline void log_headers() noexcept;
static inline void log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail) noexcept; static inline void log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail, log_level level) noexcept;
static inline void log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta) noexcept; static inline void log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta, log_level level) noexcept;
static inline void log_event(const json_iterator &iter, const char *type, std::string_view detail="", int delta=0, int depth_delta=0) noexcept; static inline void log_event(const json_iterator &iter, const char *type, std::string_view detail="", int delta=0, int depth_delta=0) noexcept;
static inline void log_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail="") noexcept; static inline void log_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail="") noexcept;
static inline void log_value(const json_iterator &iter, const char *type, std::string_view detail="", int delta=-1, int depth_delta=0) noexcept; static inline void log_value(const json_iterator &iter, const char *type, std::string_view detail="", int delta=-1, int depth_delta=0) noexcept;
+28 -4
View File
@@ -5,13 +5,23 @@ namespace ondemand {
simdjson_inline simdjson_result<value> object::find_field_unordered(const std::string_view key) & noexcept { simdjson_inline simdjson_result<value> object::find_field_unordered(const std::string_view key) & noexcept {
bool has_value; bool has_value;
SIMDJSON_TRY( iter.find_field_unordered_raw(key).get(has_value) ); SIMDJSON_TRY( iter.find_field_unordered_raw(key).get(has_value) );
if (!has_value) { return NO_SUCH_FIELD; } if (!has_value) {
auto key_str = std::string(key.data(), key.size());
auto msg = logger::string_format("Cannot find key: %s", key_str.c_str());
logger::log_error(iter, msg.c_str());
return NO_SUCH_FIELD;
}
return value(iter.child()); return value(iter.child());
} }
simdjson_inline simdjson_result<value> object::find_field_unordered(const std::string_view key) && noexcept { simdjson_inline simdjson_result<value> object::find_field_unordered(const std::string_view key) && noexcept {
bool has_value; bool has_value;
SIMDJSON_TRY( iter.find_field_unordered_raw(key).get(has_value) ); SIMDJSON_TRY( iter.find_field_unordered_raw(key).get(has_value) );
if (!has_value) { return NO_SUCH_FIELD; } if (!has_value) {
auto key_str = std::string(key.data(), key.size());
auto msg = logger::string_format("Cannot find key: %s", key_str.c_str());
logger::log_error(iter, msg.c_str());
return NO_SUCH_FIELD;
}
return value(iter.child()); return value(iter.child());
} }
simdjson_inline simdjson_result<value> object::operator[](const std::string_view key) & noexcept { simdjson_inline simdjson_result<value> object::operator[](const std::string_view key) & noexcept {
@@ -23,13 +33,23 @@ simdjson_inline simdjson_result<value> object::operator[](const std::string_view
simdjson_inline simdjson_result<value> object::find_field(const std::string_view key) & noexcept { simdjson_inline simdjson_result<value> object::find_field(const std::string_view key) & noexcept {
bool has_value; bool has_value;
SIMDJSON_TRY( iter.find_field_raw(key).get(has_value) ); SIMDJSON_TRY( iter.find_field_raw(key).get(has_value) );
if (!has_value) { return NO_SUCH_FIELD; } if (!has_value) {
auto key_str = std::string(key.data(), key.size());
auto msg = logger::string_format("Cannot find key: %s", key_str.c_str());
logger::log_error(iter, msg.c_str());
return NO_SUCH_FIELD;
}
return value(iter.child()); return value(iter.child());
} }
simdjson_inline simdjson_result<value> object::find_field(const std::string_view key) && noexcept { simdjson_inline simdjson_result<value> object::find_field(const std::string_view key) && noexcept {
bool has_value; bool has_value;
SIMDJSON_TRY( iter.find_field_raw(key).get(has_value) ); SIMDJSON_TRY( iter.find_field_raw(key).get(has_value) );
if (!has_value) { return NO_SUCH_FIELD; } if (!has_value) {
auto key_str = std::string(key.data(), key.size());
auto msg = logger::string_format("Cannot find key: %s", key_str.c_str());
logger::log_error(iter, msg.c_str());
return NO_SUCH_FIELD;
}
return value(iter.child()); return value(iter.child());
} }
@@ -226,4 +246,8 @@ simdjson_inline simdjson_result<size_t> simdjson_result<SIMDJSON_IMPLEMENTATION
return first.count_fields(); return first.count_fields();
} }
simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::object>::raw_json() noexcept {
if (error()) { return error(); }
return first.raw_json();
}
} // namespace simdjson } // namespace simdjson
@@ -223,6 +223,7 @@ public:
inline simdjson_result<bool> reset() noexcept; inline simdjson_result<bool> reset() noexcept;
inline simdjson_result<bool> is_empty() noexcept; inline simdjson_result<bool> is_empty() noexcept;
inline simdjson_result<size_t> count_fields() & noexcept; inline simdjson_result<size_t> count_fields() & noexcept;
inline simdjson_result<std::string_view> raw_json() noexcept;
}; };
@@ -84,18 +84,19 @@ simdjson_warn_unused simdjson_inline simdjson_result<json_iterator> parser::iter
return json_iterator(reinterpret_cast<const uint8_t *>(json.data()), this); return json_iterator(reinterpret_cast<const uint8_t *>(json.data()), this);
} }
inline simdjson_result<document_stream> parser::iterate_many(const uint8_t *buf, size_t len, size_t batch_size) noexcept { inline simdjson_result<document_stream> parser::iterate_many(const uint8_t *buf, size_t len, size_t batch_size, bool allow_comma_separated) noexcept {
if(batch_size < MINIMAL_BATCH_SIZE) { batch_size = MINIMAL_BATCH_SIZE; } if(batch_size < MINIMAL_BATCH_SIZE) { batch_size = MINIMAL_BATCH_SIZE; }
return document_stream(*this, buf, len, batch_size); if(allow_comma_separated && batch_size < len) { batch_size = len; }
return document_stream(*this, buf, len, batch_size, allow_comma_separated);
} }
inline simdjson_result<document_stream> parser::iterate_many(const char *buf, size_t len, size_t batch_size) noexcept { inline simdjson_result<document_stream> parser::iterate_many(const char *buf, size_t len, size_t batch_size, bool allow_comma_separated) noexcept {
return iterate_many(reinterpret_cast<const uint8_t *>(buf), len, batch_size); return iterate_many(reinterpret_cast<const uint8_t *>(buf), len, batch_size, allow_comma_separated);
} }
inline simdjson_result<document_stream> parser::iterate_many(const std::string &s, size_t batch_size) noexcept { inline simdjson_result<document_stream> parser::iterate_many(const std::string &s, size_t batch_size, bool allow_comma_separated) noexcept {
return iterate_many(s.data(), s.length(), batch_size); return iterate_many(s.data(), s.length(), batch_size, allow_comma_separated);
} }
inline simdjson_result<document_stream> parser::iterate_many(const padded_string &s, size_t batch_size) noexcept { inline simdjson_result<document_stream> parser::iterate_many(const padded_string &s, size_t batch_size, bool allow_comma_separated) noexcept {
return iterate_many(s.data(), s.length(), batch_size); return iterate_many(s.data(), s.length(), batch_size, allow_comma_separated);
} }
simdjson_inline size_t parser::capacity() const noexcept { simdjson_inline size_t parser::capacity() const noexcept {
+15 -9
View File
@@ -78,7 +78,9 @@ public:
* ### REQUIRED: Buffer Padding * ### REQUIRED: Buffer Padding
* *
* The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
* those bytes are initialized to, as long as they are allocated. * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
* using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
* SIMDJSON_PADDING bytes to avoid runtime warnings.
* *
* @param json The JSON to parse. * @param json The JSON to parse.
* @param len The length of the JSON. * @param len The length of the JSON.
@@ -134,7 +136,9 @@ public:
* ### REQUIRED: Buffer Padding * ### REQUIRED: Buffer Padding
* *
* The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
* those bytes are initialized to, as long as they are allocated. * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
* using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
* SIMDJSON_PADDING bytes to avoid runtime warnings.
* *
* @param json The JSON to parse. * @param json The JSON to parse.
* *
@@ -188,7 +192,9 @@ public:
* ### REQUIRED: Buffer Padding * ### REQUIRED: Buffer Padding
* *
* The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
* those bytes are initialized to, as long as they are allocated. * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
* using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
* SIMDJSON_PADDING bytes to avoid runtime warnings.
* *
* ### Threads * ### Threads
* *
@@ -212,15 +218,15 @@ public:
* - other json errors if parsing fails. You should not rely on these errors to always the same for the * - other json errors if parsing fails. You should not rely on these errors to always the same for the
* same document: they may vary under runtime dispatch (so they may vary depending on your system and hardware). * same document: they may vary under runtime dispatch (so they may vary depending on your system and hardware).
*/ */
inline simdjson_result<document_stream> iterate_many(const uint8_t *buf, size_t len, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept; inline simdjson_result<document_stream> iterate_many(const uint8_t *buf, size_t len, size_t batch_size = DEFAULT_BATCH_SIZE, bool allow_comma_separated = false) noexcept;
/** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */ /** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */
inline simdjson_result<document_stream> iterate_many(const char *buf, size_t len, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept; inline simdjson_result<document_stream> iterate_many(const char *buf, size_t len, size_t batch_size = DEFAULT_BATCH_SIZE, bool allow_comma_separated = false) noexcept;
/** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */ /** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */
inline simdjson_result<document_stream> iterate_many(const std::string &s, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept; inline simdjson_result<document_stream> iterate_many(const std::string &s, size_t batch_size = DEFAULT_BATCH_SIZE, bool allow_comma_separated = false) noexcept;
inline simdjson_result<document_stream> iterate_many(const std::string &&s, size_t batch_size) = delete;// unsafe inline simdjson_result<document_stream> iterate_many(const std::string &&s, size_t batch_size, bool allow_comma_separated = false) = delete;// unsafe
/** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */ /** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */
inline simdjson_result<document_stream> iterate_many(const padded_string &s, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept; inline simdjson_result<document_stream> iterate_many(const padded_string &s, size_t batch_size = DEFAULT_BATCH_SIZE, bool allow_comma_separated = false) noexcept;
inline simdjson_result<document_stream> iterate_many(const padded_string &&s, size_t batch_size) = delete;// unsafe inline simdjson_result<document_stream> iterate_many(const padded_string &&s, size_t batch_size, bool allow_comma_separated = false) = delete;// unsafe
/** @private We do not want to allow implicit conversion from C string to std::string. */ /** @private We do not want to allow implicit conversion from C string to std::string. */
simdjson_result<document_stream> iterate_many(const char *buf, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept = delete; simdjson_result<document_stream> iterate_many(const char *buf, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept = delete;
@@ -116,8 +116,8 @@ protected:
friend class json_iterator; friend class json_iterator;
friend class value_iterator; friend class value_iterator;
friend class object; friend class object;
friend simdjson_inline void logger::log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta) noexcept; friend simdjson_inline void logger::log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta, logger::log_level lvl) noexcept;
friend simdjson_inline void logger::log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail) noexcept; friend simdjson_inline void logger::log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail, logger::log_level lvl) noexcept;
}; };
} // namespace ondemand } // namespace ondemand
@@ -34,12 +34,17 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::start
return true; return true;
} }
simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::started_root_object() noexcept { simdjson_warn_unused simdjson_inline error_code value_iterator::check_root_object() noexcept {
// When in streaming mode, we cannot expect peek_last() to be the last structural element of the // When in streaming mode, we cannot expect peek_last() to be the last structural element of the
// current document. It only works in the normal mode where we have indexed a single document. // current document. It only works in the normal mode where we have indexed a single document.
// Note that adding a check for 'streaming' is not expensive since we only have at most // Note that adding a check for 'streaming' is not expensive since we only have at most
// one root element. // one root element.
if ( ! _json_iter->streaming() ) { if ( ! _json_iter->streaming() ) {
// The following lines do not fully protect against garbage content within the
// object: e.g., `{"a":2} foo }`. Users concerned with garbage content should
// call `at_end()` on the document instance at the end of the processing to
// ensure that the processing has finished at the end.
//
if (*_json_iter->peek_last() != '}') { if (*_json_iter->peek_last() != '}') {
_json_iter->abandon(); _json_iter->abandon();
return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "missing } at end"); return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "missing } at end");
@@ -56,6 +61,12 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::start
return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "the document is unbalanced"); return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "the document is unbalanced");
} }
} }
return SUCCESS;
}
simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::started_root_object() noexcept {
auto error = check_root_object();
if(error) { return error; }
return started_object(); return started_object();
} }
@@ -419,12 +430,17 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::start
return true; return true;
} }
simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::started_root_array() noexcept { simdjson_warn_unused simdjson_inline error_code value_iterator::check_root_array() noexcept {
// When in streaming mode, we cannot expect peek_last() to be the last structural element of the // When in streaming mode, we cannot expect peek_last() to be the last structural element of the
// current document. It only works in the normal mode where we have indexed a single document. // current document. It only works in the normal mode where we have indexed a single document.
// Note that adding a check for 'streaming' is not expensive since we only have at most // Note that adding a check for 'streaming' is not expensive since we only have at most
// one root element. // one root element.
if ( ! _json_iter->streaming() ) { if ( ! _json_iter->streaming() ) {
// The following lines do not fully protect against garbage content within the
// array: e.g., `[1, 2] foo]`. Users concerned with garbage content should
// also call `at_end()` on the document instance at the end of the processing to
// ensure that the processing has finished at the end.
//
if (*_json_iter->peek_last() != ']') { if (*_json_iter->peek_last() != ']') {
_json_iter->abandon(); _json_iter->abandon();
return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "missing ] at end"); return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "missing ] at end");
@@ -441,6 +457,12 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::start
return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "the document is unbalanced"); return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "the document is unbalanced");
} }
} }
return SUCCESS;
}
simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::started_root_array() noexcept {
auto error = check_root_array();
if (error) { return error; }
return started_array(); return started_array();
} }
@@ -551,7 +573,7 @@ simdjson_inline simdjson_result<bool> value_iterator::is_root_integer(bool check
auto max_len = peek_start_length(); auto max_len = peek_start_length();
auto json = peek_root_scalar("is_root_integer"); auto json = peek_root_scalar("is_root_integer");
uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
return false; // if there are more than 20 characters, it cannot be represented as an integer. return false; // if there are more than 20 characters, it cannot be represented as an integer.
} }
auto answer = numberparsing::is_integer(tmpbuf); auto answer = numberparsing::is_integer(tmpbuf);
@@ -569,7 +591,7 @@ simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::number_type>
// 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest // 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest
// number: -0.<fraction>e-308. // number: -0.<fraction>e-308.
uint8_t tmpbuf[1074+8+1]; uint8_t tmpbuf[1074+8+1];
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 1074+8+1)) {
logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters"); logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters");
return NUMBER_ERROR; return NUMBER_ERROR;
} }
@@ -584,7 +606,7 @@ simdjson_inline simdjson_result<number> value_iterator::get_root_number(bool che
// 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest // 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest
// number: -0.<fraction>e-308. // number: -0.<fraction>e-308.
uint8_t tmpbuf[1074+8+1]; uint8_t tmpbuf[1074+8+1];
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 1074+8+1)) {
logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters"); logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters");
return NUMBER_ERROR; return NUMBER_ERROR;
} }
@@ -612,7 +634,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> value_iterator::g
auto max_len = peek_start_length(); auto max_len = peek_start_length();
auto json = peek_root_scalar("uint64"); auto json = peek_root_scalar("uint64");
uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters"); logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters");
return NUMBER_ERROR; return NUMBER_ERROR;
} }
@@ -627,7 +649,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> value_iterator::g
auto max_len = peek_start_length(); auto max_len = peek_start_length();
auto json = peek_root_scalar("uint64"); auto json = peek_root_scalar("uint64");
uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters"); logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters");
return NUMBER_ERROR; return NUMBER_ERROR;
} }
@@ -642,7 +664,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<int64_t> value_iterator::ge
auto max_len = peek_start_length(); auto max_len = peek_start_length();
auto json = peek_root_scalar("int64"); auto json = peek_root_scalar("int64");
uint8_t tmpbuf[20+1]; // -<19 digits> is the longest possible integer uint8_t tmpbuf[20+1]; // -<19 digits> is the longest possible integer
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters"); logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters");
return NUMBER_ERROR; return NUMBER_ERROR;
} }
@@ -658,7 +680,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<int64_t> value_iterator::ge
auto max_len = peek_start_length(); auto max_len = peek_start_length();
auto json = peek_root_scalar("int64"); auto json = peek_root_scalar("int64");
uint8_t tmpbuf[20+1]; // -<19 digits> is the longest possible integer uint8_t tmpbuf[20+1]; // -<19 digits> is the longest possible integer
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters"); logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters");
return NUMBER_ERROR; return NUMBER_ERROR;
} }
@@ -677,7 +699,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<double> value_iterator::get
// 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest // 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest
// number: -0.<fraction>e-308. // number: -0.<fraction>e-308.
uint8_t tmpbuf[1074+8+1]; uint8_t tmpbuf[1074+8+1];
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 1074+8+1)) {
logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters"); logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters");
return NUMBER_ERROR; return NUMBER_ERROR;
} }
@@ -696,7 +718,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<double> value_iterator::get
// 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest // 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest
// number: -0.<fraction>e-308. // number: -0.<fraction>e-308.
uint8_t tmpbuf[1074+8+1]; uint8_t tmpbuf[1074+8+1];
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 1074+8+1)) {
logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters"); logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters");
return NUMBER_ERROR; return NUMBER_ERROR;
} }
@@ -711,7 +733,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::get_r
auto max_len = peek_start_length(); auto max_len = peek_start_length();
auto json = peek_root_scalar("bool"); auto json = peek_root_scalar("bool");
uint8_t tmpbuf[5+1]; uint8_t tmpbuf[5+1];
if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { return incorrect_type_error("Not a boolean"); } if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 5+1)) { return incorrect_type_error("Not a boolean"); }
auto result = parse_bool(tmpbuf); auto result = parse_bool(tmpbuf);
if(result.error() == SUCCESS) { if(result.error() == SUCCESS) {
if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; } if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
@@ -924,11 +946,13 @@ simdjson_inline void value_iterator::move_at_container_start() noexcept {
} }
simdjson_inline simdjson_result<bool> value_iterator::reset_array() noexcept { simdjson_inline simdjson_result<bool> value_iterator::reset_array() noexcept {
if(error()) { return error(); }
move_at_container_start(); move_at_container_start();
return started_array(); return started_array();
} }
simdjson_inline simdjson_result<bool> value_iterator::reset_object() noexcept { simdjson_inline simdjson_result<bool> value_iterator::reset_object() noexcept {
if(error()) { return error(); }
move_at_container_start(); move_at_container_start();
return started_object(); return started_object();
} }
@@ -109,7 +109,14 @@ public:
* @error TAPE_ERROR if there is no matching } at end of document * @error TAPE_ERROR if there is no matching } at end of document
*/ */
simdjson_warn_unused simdjson_inline simdjson_result<bool> start_root_object() noexcept; simdjson_warn_unused simdjson_inline simdjson_result<bool> start_root_object() noexcept;
/**
* Checks whether an object could be started from the root. May be called by start_root_object.
*
* @returns SUCCESS if it is possible to safely start an object from the root (document level).
* @error INCORRECT_TYPE if there is no opening {
* @error TAPE_ERROR if there is no matching } at end of document
*/
simdjson_warn_unused simdjson_inline error_code check_root_object() noexcept;
/** /**
* Start an object iteration after the user has already checked and moved past the {. * Start an object iteration after the user has already checked and moved past the {.
* *
@@ -234,7 +241,14 @@ public:
* @error TAPE_ERROR if there is no matching ] at end of document * @error TAPE_ERROR if there is no matching ] at end of document
*/ */
simdjson_warn_unused simdjson_inline simdjson_result<bool> start_root_array() noexcept; simdjson_warn_unused simdjson_inline simdjson_result<bool> start_root_array() noexcept;
/**
* Checks whether an array could be started from the root. May be called by start_root_array.
*
* @returns SUCCESS if it is possible to safely start an array from the root (document level).
* @error INCORRECT_TYPE If there is no [.
* @error TAPE_ERROR if there is no matching ] at end of document
*/
simdjson_warn_unused simdjson_inline error_code check_root_array() noexcept;
/** /**
* Start an array iteration, after the user has already checked and moved past the [. * Start an array iteration, after the user has already checked and moved past the [.
* *
+1 -1
View File
@@ -9,7 +9,7 @@
#define SIMDJSON_TARGET_HASWELL #define SIMDJSON_TARGET_HASWELL
#define SIMDJSON_UNTARGET_HASWELL #define SIMDJSON_UNTARGET_HASWELL
#else #else
#define SIMDJSON_TARGET_HASWELL SIMDJSON_TARGET_REGION("avx2,bmi,pclmul,lzcnt") #define SIMDJSON_TARGET_HASWELL SIMDJSON_TARGET_REGION("avx2,bmi,pclmul,lzcnt,popcnt")
#define SIMDJSON_UNTARGET_HASWELL SIMDJSON_UNTARGET_REGION #define SIMDJSON_UNTARGET_HASWELL SIMDJSON_UNTARGET_REGION
#endif #endif
@@ -9,6 +9,10 @@ namespace {
// but the algorithms do not end up using the returned value. // but the algorithms do not end up using the returned value.
// Sadly, sanitizers are not smart enough to figure it out. // Sadly, sanitizers are not smart enough to figure it out.
SIMDJSON_NO_SANITIZE_UNDEFINED SIMDJSON_NO_SANITIZE_UNDEFINED
// This function can be used safely even if not all bytes have been
// initialized.
// See issue https://github.com/simdjson/simdjson/issues/1965
SIMDJSON_NO_SANITIZE_MEMORY
simdjson_inline int trailing_zeroes(uint64_t input_num) { simdjson_inline int trailing_zeroes(uint64_t input_num) {
#if SIMDJSON_REGULAR_VISUAL_STUDIO #if SIMDJSON_REGULAR_VISUAL_STUDIO
return (int)_tzcnt_u64(input_num); return (int)_tzcnt_u64(input_num);
+1 -1
View File
@@ -9,7 +9,7 @@
#define SIMDJSON_TARGET_ICELAKE #define SIMDJSON_TARGET_ICELAKE
#define SIMDJSON_UNTARGET_ICELAKE #define SIMDJSON_UNTARGET_ICELAKE
#else #else
#define SIMDJSON_TARGET_ICELAKE SIMDJSON_TARGET_REGION("avx512f,avx512dq,avx512cd,avx512bw,avx512vbmi,avx512vbmi2,avx512vl,avx2,bmi,pclmul,lzcnt") #define SIMDJSON_TARGET_ICELAKE SIMDJSON_TARGET_REGION("avx512f,avx512dq,avx512cd,avx512bw,avx512vbmi,avx512vbmi2,avx512vl,avx2,bmi,pclmul,lzcnt,popcnt")
#define SIMDJSON_UNTARGET_ICELAKE SIMDJSON_UNTARGET_REGION #define SIMDJSON_UNTARGET_ICELAKE SIMDJSON_UNTARGET_REGION
#endif #endif
@@ -9,6 +9,10 @@ namespace {
// but the algorithms do not end up using the returned value. // but the algorithms do not end up using the returned value.
// Sadly, sanitizers are not smart enough to figure it out. // Sadly, sanitizers are not smart enough to figure it out.
SIMDJSON_NO_SANITIZE_UNDEFINED SIMDJSON_NO_SANITIZE_UNDEFINED
// This function can be used safely even if not all bytes have been
// initialized.
// See issue https://github.com/simdjson/simdjson/issues/1965
SIMDJSON_NO_SANITIZE_MEMORY
simdjson_inline int trailing_zeroes(uint64_t input_num) { simdjson_inline int trailing_zeroes(uint64_t input_num) {
#if SIMDJSON_REGULAR_VISUAL_STUDIO #if SIMDJSON_REGULAR_VISUAL_STUDIO
return (int)_tzcnt_u64(input_num); return (int)_tzcnt_u64(input_num);
+2 -2
View File
@@ -78,9 +78,9 @@
#define SIMDJSON_CAN_ALWAYS_RUN_WESTMERE (SIMDJSON_IMPLEMENTATION_WESTMERE && SIMDJSON_IS_X86_64 && __SSE4_2__ && __PCLMUL__) #define SIMDJSON_CAN_ALWAYS_RUN_WESTMERE (SIMDJSON_IMPLEMENTATION_WESTMERE && SIMDJSON_IS_X86_64 && __SSE4_2__ && __PCLMUL__)
#ifndef SIMDJSON_IMPLEMENTATION_PPC64 #ifndef SIMDJSON_IMPLEMENTATION_PPC64
#define SIMDJSON_IMPLEMENTATION_PPC64 (SIMDJSON_IS_PPC64) #define SIMDJSON_IMPLEMENTATION_PPC64 (SIMDJSON_IS_PPC64 && SIMDJSON_IS_PPC64_VMX)
#endif #endif
#define SIMDJSON_CAN_ALWAYS_RUN_PPC64 SIMDJSON_IMPLEMENTATION_PPC64 && SIMDJSON_IS_PPC64 #define SIMDJSON_CAN_ALWAYS_RUN_PPC64 SIMDJSON_IMPLEMENTATION_PPC64 && SIMDJSON_IS_PPC64 && SIMDJSON_IS_PPC64_VMX
// Default Fallback to on unless a builtin implementation has already been selected. // Default Fallback to on unless a builtin implementation has already been selected.
#ifndef SIMDJSON_IMPLEMENTATION_FALLBACK #ifndef SIMDJSON_IMPLEMENTATION_FALLBACK
+46 -14
View File
@@ -57,7 +57,6 @@ POSSIBILITY OF SUCH DAMAGE.
namespace simdjson { namespace simdjson {
namespace internal { namespace internal {
enum instruction_set { enum instruction_set {
DEFAULT = 0x0, DEFAULT = 0x0,
NEON = 0x1, NEON = 0x1,
@@ -107,7 +106,10 @@ constexpr uint32_t cpuid_avx512cd_bit = 1 << 28; ///< @private bit 28 of EBX
constexpr uint32_t cpuid_avx512bw_bit = 1 << 30; ///< @private bit 30 of EBX for EAX=0x7 constexpr uint32_t cpuid_avx512bw_bit = 1 << 30; ///< @private bit 30 of EBX for EAX=0x7
constexpr uint32_t cpuid_avx512vl_bit = 1U << 31; ///< @private bit 31 of EBX for EAX=0x7 constexpr uint32_t cpuid_avx512vl_bit = 1U << 31; ///< @private bit 31 of EBX for EAX=0x7
constexpr uint32_t cpuid_avx512vbmi2_bit = 1 << 6; ///< @private bit 6 of ECX for EAX=0x7 constexpr uint32_t cpuid_avx512vbmi2_bit = 1 << 6; ///< @private bit 6 of ECX for EAX=0x7
constexpr uint64_t cpuid_avx256_saved = uint64_t(1) << 2; ///< @private bit 2 = AVX
constexpr uint64_t cpuid_avx512_saved = uint64_t(7) << 5; ///< @private bits 5,6,7 = opmask, ZMM_hi256, hi16_ZMM
constexpr uint32_t cpuid_sse42_bit = 1 << 20; ///< @private bit 20 of ECX for EAX=0x1 constexpr uint32_t cpuid_sse42_bit = 1 << 20; ///< @private bit 20 of ECX for EAX=0x1
constexpr uint32_t cpuid_osxsave = (uint32_t(1) << 26) | (uint32_t(1) << 27); ///< @private bits 26+27 of ECX for EAX=0x1
constexpr uint32_t cpuid_pclmulqdq_bit = 1 << 1; ///< @private bit 1 of ECX for EAX=0x1 constexpr uint32_t cpuid_pclmulqdq_bit = 1 << 1; ///< @private bit 1 of ECX for EAX=0x1
} }
@@ -117,7 +119,7 @@ static inline void cpuid(uint32_t *eax, uint32_t *ebx, uint32_t *ecx,
uint32_t *edx) { uint32_t *edx) {
#if defined(_MSC_VER) #if defined(_MSC_VER)
int cpu_info[4]; int cpu_info[4];
__cpuid(cpu_info, *eax); __cpuidex(cpu_info, *eax, *ecx);
*eax = cpu_info[0]; *eax = cpu_info[0];
*ebx = cpu_info[1]; *ebx = cpu_info[1];
*ecx = cpu_info[2]; *ecx = cpu_info[2];
@@ -135,10 +137,48 @@ static inline void cpuid(uint32_t *eax, uint32_t *ebx, uint32_t *ecx,
#endif #endif
} }
static inline uint64_t xgetbv() {
#if defined(_MSC_VER)
return _xgetbv(0);
#else
uint32_t xcr0_lo, xcr0_hi;
asm volatile("xgetbv\n\t" : "=a" (xcr0_lo), "=d" (xcr0_hi) : "c" (0));
return xcr0_lo | (uint64_t(xcr0_hi) << 32);
#endif
}
static inline uint32_t detect_supported_architectures() { static inline uint32_t detect_supported_architectures() {
uint32_t eax, ebx, ecx, edx; uint32_t eax, ebx, ecx, edx;
uint32_t host_isa = 0x0; uint32_t host_isa = 0x0;
// EBX for EAX=0x1
eax = 0x1;
ecx = 0x0;
cpuid(&eax, &ebx, &ecx, &edx);
if (ecx & cpuid_sse42_bit) {
host_isa |= instruction_set::SSE42;
} else {
return host_isa; // everything after is redundant
}
if (ecx & cpuid_pclmulqdq_bit) {
host_isa |= instruction_set::PCLMULQDQ;
}
if ((ecx & cpuid_osxsave) != cpuid_osxsave) {
return host_isa;
}
// xgetbv for checking if the OS saves registers
uint64_t xcr0 = xgetbv();
if ((xcr0 & cpuid_avx256_saved) == 0) {
return host_isa;
}
// ECX for EAX=0x7 // ECX for EAX=0x7
eax = 0x7; eax = 0x7;
ecx = 0x0; ecx = 0x0;
@@ -154,6 +194,10 @@ static inline uint32_t detect_supported_architectures() {
host_isa |= instruction_set::BMI2; host_isa |= instruction_set::BMI2;
} }
if (!((xcr0 & cpuid_avx512_saved) == cpuid_avx512_saved)) {
return host_isa;
}
if (ebx & cpuid_avx512f_bit) { if (ebx & cpuid_avx512f_bit) {
host_isa |= instruction_set::AVX512F; host_isa |= instruction_set::AVX512F;
} }
@@ -190,18 +234,6 @@ static inline uint32_t detect_supported_architectures() {
host_isa |= instruction_set::AVX512VBMI2; host_isa |= instruction_set::AVX512VBMI2;
} }
// EBX for EAX=0x1
eax = 0x1;
cpuid(&eax, &ebx, &ecx, &edx);
if (ecx & cpuid_sse42_bit) {
host_isa |= instruction_set::SSE42;
}
if (ecx & cpuid_pclmulqdq_bit) {
host_isa |= instruction_set::PCLMULQDQ;
}
return host_isa; return host_isa;
} }
#else // fallback #else // fallback
+65 -19
View File
@@ -12,7 +12,7 @@
#define NONSTD_SV_LITE_H_INCLUDED #define NONSTD_SV_LITE_H_INCLUDED
#define string_view_lite_MAJOR 1 #define string_view_lite_MAJOR 1
#define string_view_lite_MINOR 6 #define string_view_lite_MINOR 7
#define string_view_lite_PATCH 0 #define string_view_lite_PATCH 0
#define string_view_lite_VERSION nssv_STRINGIFY(string_view_lite_MAJOR) "." nssv_STRINGIFY(string_view_lite_MINOR) "." nssv_STRINGIFY(string_view_lite_PATCH) #define string_view_lite_VERSION nssv_STRINGIFY(string_view_lite_MAJOR) "." nssv_STRINGIFY(string_view_lite_MINOR) "." nssv_STRINGIFY(string_view_lite_PATCH)
@@ -72,7 +72,7 @@
// Control presence of exception handling (try and auto discover): // Control presence of exception handling (try and auto discover):
#ifndef nssv_CONFIG_NO_EXCEPTIONS #ifndef nssv_CONFIG_NO_EXCEPTIONS
# if _MSC_VER # if defined(_MSC_VER)
# include <cstddef> // for _HAS_EXCEPTIONS # include <cstddef> // for _HAS_EXCEPTIONS
# endif # endif
# if defined(__cpp_exceptions) || defined(__EXCEPTIONS) || (_HAS_EXCEPTIONS) # if defined(__cpp_exceptions) || defined(__EXCEPTIONS) || (_HAS_EXCEPTIONS)
@@ -82,7 +82,7 @@
# endif # endif
#endif #endif
// C++ language version detection (C++20 is speculative): // C++ language version detection (C++23 is speculative):
// Note: VC14.0/1900 (VS2015) lacks too much from C++14. // Note: VC14.0/1900 (VS2015) lacks too much from C++14.
#ifndef nssv_CPLUSPLUS #ifndef nssv_CPLUSPLUS
@@ -98,7 +98,8 @@
#define nssv_CPP11_OR_GREATER_ ( nssv_CPLUSPLUS >= 201103L ) #define nssv_CPP11_OR_GREATER_ ( nssv_CPLUSPLUS >= 201103L )
#define nssv_CPP14_OR_GREATER ( nssv_CPLUSPLUS >= 201402L ) #define nssv_CPP14_OR_GREATER ( nssv_CPLUSPLUS >= 201402L )
#define nssv_CPP17_OR_GREATER ( nssv_CPLUSPLUS >= 201703L ) #define nssv_CPP17_OR_GREATER ( nssv_CPLUSPLUS >= 201703L )
#define nssv_CPP20_OR_GREATER ( nssv_CPLUSPLUS >= 202000L ) #define nssv_CPP20_OR_GREATER ( nssv_CPLUSPLUS >= 202002L )
#define nssv_CPP23_OR_GREATER ( nssv_CPLUSPLUS >= 202300L )
// use C++17 std::string_view if available and requested: // use C++17 std::string_view if available and requested:
@@ -288,6 +289,8 @@ using std::operator<<;
#define nssv_HAVE_CONSTEXPR_11 nssv_CPP11_140 #define nssv_HAVE_CONSTEXPR_11 nssv_CPP11_140
#define nssv_HAVE_EXPLICIT_CONVERSION nssv_CPP11_140 #define nssv_HAVE_EXPLICIT_CONVERSION nssv_CPP11_140
#define nssv_HAVE_INLINE_NAMESPACE nssv_CPP11_140 #define nssv_HAVE_INLINE_NAMESPACE nssv_CPP11_140
#define nssv_HAVE_IS_DEFAULT nssv_CPP11_140
#define nssv_HAVE_IS_DELETE nssv_CPP11_140
#define nssv_HAVE_NOEXCEPT nssv_CPP11_140 #define nssv_HAVE_NOEXCEPT nssv_CPP11_140
#define nssv_HAVE_NULLPTR nssv_CPP11_100 #define nssv_HAVE_NULLPTR nssv_CPP11_100
#define nssv_HAVE_REF_QUALIFIER nssv_CPP11_140 #define nssv_HAVE_REF_QUALIFIER nssv_CPP11_140
@@ -469,6 +472,17 @@ nssv_DISABLE_MSVC_WARNINGS( 4455 26481 26472 )
namespace nonstd { namespace sv_lite { namespace nonstd { namespace sv_lite {
//
// basic_string_view declaration:
//
template
<
class CharT,
class Traits = std::char_traits<CharT>
>
class basic_string_view;
namespace detail { namespace detail {
// support constexpr comparison in C++14; // support constexpr comparison in C++14;
@@ -536,14 +550,33 @@ inline nssv_constexpr14 std::size_t length( CharT * s )
#endif // OPTIMIZE #endif // OPTIMIZE
} // namespace detail #if nssv_CPP11_OR_GREATER && ! nssv_CPP17_OR_GREATER
#if defined(__OPTIMIZE__)
template // gcc, clang provide __OPTIMIZE__
< // Expect tail call optimization to make search() non-recursive:
class CharT,
class Traits = std::char_traits<CharT> template< class CharT, class Traits = std::char_traits<CharT> >
> constexpr const CharT* search( basic_string_view<CharT, Traits> haystack, basic_string_view<CharT, Traits> needle )
class basic_string_view; {
return haystack.starts_with( needle ) ? haystack.begin() :
haystack.empty() ? haystack.end() : search( haystack.substr(1), needle );
}
#else // OPTIMIZE
// non-recursive:
template< class CharT, class Traits = std::char_traits<CharT> >
constexpr const CharT* search( basic_string_view<CharT, Traits> haystack, basic_string_view<CharT, Traits> needle )
{
return std::search( haystack.begin(), haystack.end(), needle.begin(), needle.end() );
}
#endif // OPTIMIZE
#endif // nssv_CPP11_OR_GREATER && ! nssv_CPP17_OR_GREATER
} // namespace detail
// //
// basic_string_view: // basic_string_view:
@@ -570,7 +603,7 @@ public:
typedef const_pointer iterator; typedef const_pointer iterator;
typedef const_pointer const_iterator; typedef const_pointer const_iterator;
typedef std::reverse_iterator< const_iterator > reverse_iterator; typedef std::reverse_iterator< const_iterator > reverse_iterator;
typedef std::reverse_iterator< const_iterator > const_reverse_iterator; typedef std::reverse_iterator< const_iterator > const_reverse_iterator;
typedef std::size_t size_type; typedef std::size_t size_type;
typedef std::ptrdiff_t difference_type; typedef std::ptrdiff_t difference_type;
@@ -607,6 +640,14 @@ public:
#endif #endif
{} {}
#if nssv_HAVE_NULLPTR
# if nssv_HAVE_IS_DELETE
nssv_constexpr basic_string_view( std::nullptr_t ) nssv_noexcept = delete;
# else
private: nssv_constexpr basic_string_view( std::nullptr_t ) nssv_noexcept; public:
# endif
#endif
// Assignment: // Assignment:
#if nssv_CPP11_OR_GREATER #if nssv_CPP11_OR_GREATER
@@ -804,25 +845,30 @@ public:
// find(), 4x: // find(), 4x:
nssv_constexpr14 size_type find( basic_string_view v, size_type pos = 0 ) const nssv_noexcept // (1) nssv_constexpr size_type find( basic_string_view v, size_type pos = 0 ) const nssv_noexcept // (1)
{ {
return assert( v.size() == 0 || v.data() != nssv_nullptr ) return assert( v.size() == 0 || v.data() != nssv_nullptr )
, pos >= size() , pos >= size()
? npos ? npos : to_pos(
: to_pos( std::search( cbegin() + pos, cend(), v.cbegin(), v.cend(), Traits::eq ) ); #if nssv_CPP11_OR_GREATER && ! nssv_CPP17_OR_GREATER
detail::search( substr(pos), v )
#else
std::search( cbegin() + pos, cend(), v.cbegin(), v.cend(), Traits::eq )
#endif
);
} }
nssv_constexpr14 size_type find( CharT c, size_type pos = 0 ) const nssv_noexcept // (2) nssv_constexpr size_type find( CharT c, size_type pos = 0 ) const nssv_noexcept // (2)
{ {
return find( basic_string_view( &c, 1 ), pos ); return find( basic_string_view( &c, 1 ), pos );
} }
nssv_constexpr14 size_type find( CharT const * s, size_type pos, size_type n ) const // (3) nssv_constexpr size_type find( CharT const * s, size_type pos, size_type n ) const // (3)
{ {
return find( basic_string_view( s, n ), pos ); return find( basic_string_view( s, n ), pos );
} }
nssv_constexpr14 size_type find( CharT const * s, size_type pos = 0 ) const // (4) nssv_constexpr size_type find( CharT const * s, size_type pos = 0 ) const // (4)
{ {
return find( basic_string_view( s ), pos ); return find( basic_string_view( s ), pos );
} }
@@ -1381,7 +1427,7 @@ Stream & write_to_stream( Stream & os, View const & sv )
{ {
typename Stream::sentry sentry( os ); typename Stream::sentry sentry( os );
if ( !os ) if ( !sentry )
return os; return os;
const std::streamsize length = static_cast<std::streamsize>( sv.length() ); const std::streamsize length = static_cast<std::streamsize>( sv.length() );
+3 -4
View File
@@ -35,9 +35,8 @@ inline char *allocate_padded_buffer(size_t length) noexcept {
if (padded_buffer == nullptr) { if (padded_buffer == nullptr) {
return nullptr; return nullptr;
} }
// We write zeroes in the padded region to avoid having uninitized // We write nulls in the padded region to avoid having uninitialized
// garbage. If nothing else, garbage getting read might trigger a // content which may trigger warning for some sanitizers
// warning in a memory checking.
std::memset(padded_buffer + length, 0, totalpaddedlength - length); std::memset(padded_buffer + length, 0, totalpaddedlength - length);
return padded_buffer; return padded_buffer;
} // allocate_padded_buffer() } // allocate_padded_buffer()
@@ -67,7 +66,7 @@ inline padded_string::padded_string(std::string_view sv_) noexcept
: viable_size(sv_.size()), data_ptr(internal::allocate_padded_buffer(sv_.size())) { : viable_size(sv_.size()), data_ptr(internal::allocate_padded_buffer(sv_.size())) {
if(simdjson_unlikely(!data_ptr)) { if(simdjson_unlikely(!data_ptr)) {
//allocation failed or zero size //allocation failed or zero size
viable_size=0; viable_size = 0;
return; return;
} }
if (sv_.size()) { if (sv_.size()) {
+21 -12
View File
@@ -32,19 +32,14 @@
#endif // __clang__ #endif // __clang__
#endif // _MSC_VER #endif // _MSC_VER
#if SIMDJSON_REGULAR_VISUAL_STUDIO
// https://en.wikipedia.org/wiki/C_alternative_tokens
// This header should have no effect, except maybe
// under Visual Studio.
#include <iso646.h>
#endif
#if defined(__x86_64__) || defined(_M_AMD64) #if defined(__x86_64__) || defined(_M_AMD64)
#define SIMDJSON_IS_X86_64 1 #define SIMDJSON_IS_X86_64 1
#elif defined(__aarch64__) || defined(_M_ARM64) #elif defined(__aarch64__) || defined(_M_ARM64)
#define SIMDJSON_IS_ARM64 1 #define SIMDJSON_IS_ARM64 1
#elif defined(__PPC64__) || defined(_M_PPC64) #elif defined(__PPC64__) || defined(_M_PPC64)
#define SIMDJSON_IS_PPC64 1 #if defined(__ALTIVEC__)
#define SIMDJSON_IS_PPC64_VMX 1
#endif // defined(__ALTIVEC__)
#else #else
#define SIMDJSON_IS_32BITS 1 #define SIMDJSON_IS_32BITS 1
@@ -148,6 +143,19 @@ use a 64-bit target such as x64, 64-bit ARM or 64-bit PPC.")
#define SIMDJSON_NO_SANITIZE_UNDEFINED #define SIMDJSON_NO_SANITIZE_UNDEFINED
#endif #endif
#if defined(__clang__) || defined(__GNUC__)
#if defined(__has_feature)
# if __has_feature(memory_sanitizer)
#define SIMDJSON_NO_SANITIZE_MEMORY __attribute__((no_sanitize("memory")))
# endif // if __has_feature(memory_sanitizer)
#endif // defined(__has_feature)
#endif
// make sure it is defined as 'nothing' if it is unapplicable.
#ifndef SIMDJSON_NO_SANITIZE_MEMORY
#define SIMDJSON_NO_SANITIZE_MEMORY
#endif
#if SIMDJSON_VISUAL_STUDIO #if SIMDJSON_VISUAL_STUDIO
// This is one case where we do not distinguish between // This is one case where we do not distinguish between
// regular visual studio and clang under visual studio. // regular visual studio and clang under visual studio.
@@ -162,8 +170,9 @@ use a 64-bit target such as x64, 64-bit ARM or 64-bit PPC.")
#define simdjson_strncasecmp strncasecmp #define simdjson_strncasecmp strncasecmp
#endif #endif
#ifdef NDEBUG #if defined(NDEBUG) || defined(__OPTIMIZE__) || (defined(_MSC_VER) && !defined(_DEBUG))
// If NDEBUG is set, or __OPTIMIZE__ is set, or we are under MSVC in release mode,
// then do away with asserts and use __assume.
#if SIMDJSON_VISUAL_STUDIO #if SIMDJSON_VISUAL_STUDIO
#define SIMDJSON_UNREACHABLE() __assume(0) #define SIMDJSON_UNREACHABLE() __assume(0)
#define SIMDJSON_ASSUME(COND) __assume(COND) #define SIMDJSON_ASSUME(COND) __assume(COND)
@@ -172,8 +181,8 @@ use a 64-bit target such as x64, 64-bit ARM or 64-bit PPC.")
#define SIMDJSON_ASSUME(COND) do { if (!(COND)) __builtin_unreachable(); } while (0) #define SIMDJSON_ASSUME(COND) do { if (!(COND)) __builtin_unreachable(); } while (0)
#endif #endif
#else // NDEBUG #else // defined(NDEBUG) || defined(__OPTIMIZE__) || (defined(_MSC_VER) && !defined(_DEBUG))
// This should only ever be enabled in debug mode.
#define SIMDJSON_UNREACHABLE() assert(0); #define SIMDJSON_UNREACHABLE() assert(0);
#define SIMDJSON_ASSUME(COND) assert(COND) #define SIMDJSON_ASSUME(COND) assert(COND)
+4
View File
@@ -9,6 +9,10 @@ namespace {
// but the algorithms do not end up using the returned value. // but the algorithms do not end up using the returned value.
// Sadly, sanitizers are not smart enough to figure it out. // Sadly, sanitizers are not smart enough to figure it out.
SIMDJSON_NO_SANITIZE_UNDEFINED SIMDJSON_NO_SANITIZE_UNDEFINED
// This function can be used safely even if not all bytes have been
// initialized.
// See issue https://github.com/simdjson/simdjson/issues/1965
SIMDJSON_NO_SANITIZE_MEMORY
simdjson_inline int trailing_zeroes(uint64_t input_num) { simdjson_inline int trailing_zeroes(uint64_t input_num) {
#if SIMDJSON_REGULAR_VISUAL_STUDIO #if SIMDJSON_REGULAR_VISUAL_STUDIO
unsigned long ret; unsigned long ret;
+3 -3
View File
@@ -4,7 +4,7 @@
#define SIMDJSON_SIMDJSON_VERSION_H #define SIMDJSON_SIMDJSON_VERSION_H
/** The version of simdjson being used (major.minor.revision) */ /** The version of simdjson being used (major.minor.revision) */
#define SIMDJSON_VERSION "3.1.3" #define SIMDJSON_VERSION "3.2.0"
namespace simdjson { namespace simdjson {
enum { enum {
@@ -15,11 +15,11 @@ enum {
/** /**
* The minor version (major.MINOR.revision) of simdjson being used. * The minor version (major.MINOR.revision) of simdjson being used.
*/ */
SIMDJSON_VERSION_MINOR = 1, SIMDJSON_VERSION_MINOR = 2,
/** /**
* The revision (major.minor.REVISION) of simdjson being used. * The revision (major.minor.REVISION) of simdjson being used.
*/ */
SIMDJSON_VERSION_REVISION = 3 SIMDJSON_VERSION_REVISION = 0
}; };
} // namespace simdjson } // namespace simdjson
+1 -1
View File
@@ -9,7 +9,7 @@
#define SIMDJSON_TARGET_WESTMERE #define SIMDJSON_TARGET_WESTMERE
#define SIMDJSON_UNTARGET_WESTMERE #define SIMDJSON_UNTARGET_WESTMERE
#else #else
#define SIMDJSON_TARGET_WESTMERE SIMDJSON_TARGET_REGION("sse4.2,pclmul") #define SIMDJSON_TARGET_WESTMERE SIMDJSON_TARGET_REGION("sse4.2,pclmul,popcnt")
#define SIMDJSON_UNTARGET_WESTMERE SIMDJSON_UNTARGET_REGION #define SIMDJSON_UNTARGET_WESTMERE SIMDJSON_UNTARGET_REGION
#endif #endif
@@ -9,6 +9,10 @@ namespace {
// but the algorithms do not end up using the returned value. // but the algorithms do not end up using the returned value.
// Sadly, sanitizers are not smart enough to figure it out. // Sadly, sanitizers are not smart enough to figure it out.
SIMDJSON_NO_SANITIZE_UNDEFINED SIMDJSON_NO_SANITIZE_UNDEFINED
// This function can be used safely even if not all bytes have been
// initialized.
// See issue https://github.com/simdjson/simdjson/issues/1965
SIMDJSON_NO_SANITIZE_MEMORY
simdjson_inline int trailing_zeroes(uint64_t input_num) { simdjson_inline int trailing_zeroes(uint64_t input_num) {
#if SIMDJSON_REGULAR_VISUAL_STUDIO #if SIMDJSON_REGULAR_VISUAL_STUDIO
unsigned long ret; unsigned long ret;
+2 -1
View File
@@ -112,7 +112,8 @@ def dofile(fid, prepath, filename):
print(f"// redefining SIMDJSON_IMPLEMENTATION to \"{current_implementation}\"\n// {line}", file=fid) print(f"// redefining SIMDJSON_IMPLEMENTATION to \"{current_implementation}\"\n// {line}", file=fid)
elif undefines_simdjson_implementation.search(line): elif undefines_simdjson_implementation.search(line):
# Don't include #undef SIMDJSON_IMPLEMENTATION since we're handling it ourselves # Don't include #undef SIMDJSON_IMPLEMENTATION since we're handling it ourselves
print(f"// {line}") # print(f"// {line}")
pass
else: else:
# copy the line, with SIMDJSON_IMPLEMENTATION replace to what it is currently defined to # copy the line, with SIMDJSON_IMPLEMENTATION replace to what it is currently defined to
print(uses_simdjson_implementation.sub(current_implementation+"\\1",line), file=fid) print(uses_simdjson_implementation.sub(current_implementation+"\\1",line), file=fid)
+32 -32
View File
@@ -1,4 +1,4 @@
/* auto-generated on 2023-03-02 09:39:35 -0500. Do not edit! */ /* auto-generated on 2023-06-15 09:11:14 -0400. Do not edit! */
/* begin file src/simdjson.cpp */ /* begin file src/simdjson.cpp */
#include "simdjson.h" #include "simdjson.h"
@@ -1558,37 +1558,37 @@ namespace simdjson {
namespace internal { namespace internal {
SIMDJSON_DLLIMPORTEXPORT const error_code_info error_codes[] { SIMDJSON_DLLIMPORTEXPORT const error_code_info error_codes[] {
{ SUCCESS, "No error" }, { SUCCESS, "SUCCESS: No error" },
{ CAPACITY, "This parser can't support a document that big" }, { CAPACITY, "CAPACITY: This parser can't support a document that big" },
{ MEMALLOC, "Error allocating memory, we're most likely out of memory" }, { MEMALLOC, "MEMALLOC: Error allocating memory, we're most likely out of memory" },
{ TAPE_ERROR, "The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc." }, { TAPE_ERROR, "TAPE_ERROR: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc." },
{ DEPTH_ERROR, "The JSON document was too deep (too many nested objects and arrays)" }, { DEPTH_ERROR, "DEPTH_ERROR: The JSON document was too deep (too many nested objects and arrays)" },
{ STRING_ERROR, "Problem while parsing a string" }, { STRING_ERROR, "STRING_ERROR: Problem while parsing a string" },
{ T_ATOM_ERROR, "Problem while parsing an atom starting with the letter 't'" }, { T_ATOM_ERROR, "T_ATOM_ERROR: Problem while parsing an atom starting with the letter 't'" },
{ F_ATOM_ERROR, "Problem while parsing an atom starting with the letter 'f'" }, { F_ATOM_ERROR, "F_ATOM_ERROR: Problem while parsing an atom starting with the letter 'f'" },
{ N_ATOM_ERROR, "Problem while parsing an atom starting with the letter 'n'" }, { N_ATOM_ERROR, "N_ATOM_ERROR: Problem while parsing an atom starting with the letter 'n'" },
{ NUMBER_ERROR, "Problem while parsing a number" }, { NUMBER_ERROR, "NUMBER_ERROR: Problem while parsing a number" },
{ UTF8_ERROR, "The input is not valid UTF-8" }, { UTF8_ERROR, "UTF8_ERROR: The input is not valid UTF-8" },
{ UNINITIALIZED, "Uninitialized" }, { UNINITIALIZED, "UNINITIALIZED: Uninitialized" },
{ EMPTY, "Empty: no JSON found" }, { EMPTY, "EMPTY: no JSON found" },
{ UNESCAPED_CHARS, "Within strings, some characters must be escaped, we found unescaped characters" }, { UNESCAPED_CHARS, "UNESCAPED_CHARS: Within strings, some characters must be escaped, we found unescaped characters" },
{ UNCLOSED_STRING, "A string is opened, but never closed." }, { UNCLOSED_STRING, "UNCLOSED_STRING: A string is opened, but never closed." },
{ UNSUPPORTED_ARCHITECTURE, "simdjson does not have an implementation supported by this CPU architecture (perhaps it's a non-SIMD CPU?)." }, { UNSUPPORTED_ARCHITECTURE, "UNSUPPORTED_ARCHITECTURE: simdjson does not have an implementation supported by this CPU architecture. Please report this error to the core team as it should never happen." },
{ INCORRECT_TYPE, "The JSON element does not have the requested type." }, { INCORRECT_TYPE, "INCORRECT_TYPE: The JSON element does not have the requested type." },
{ NUMBER_OUT_OF_RANGE, "The JSON number is too large or too small to fit within the requested type." }, { NUMBER_OUT_OF_RANGE, "NUMBER_OUT_OF_RANGE: The JSON number is too large or too small to fit within the requested type." },
{ INDEX_OUT_OF_BOUNDS, "Attempted to access an element of a JSON array that is beyond its length." }, { INDEX_OUT_OF_BOUNDS, "INDEX_OUT_OF_BOUNDS: Attempted to access an element of a JSON array that is beyond its length." },
{ NO_SUCH_FIELD, "The JSON field referenced does not exist in this object." }, { NO_SUCH_FIELD, "NO_SUCH_FIELD: The JSON field referenced does not exist in this object." },
{ IO_ERROR, "Error reading the file." }, { IO_ERROR, "IO_ERROR: Error reading the file." },
{ INVALID_JSON_POINTER, "Invalid JSON pointer syntax." }, { INVALID_JSON_POINTER, "INVALID_JSON_POINTER: Invalid JSON pointer syntax." },
{ INVALID_URI_FRAGMENT, "Invalid URI fragment syntax." }, { INVALID_URI_FRAGMENT, "INVALID_URI_FRAGMENT: Invalid URI fragment syntax." },
{ UNEXPECTED_ERROR, "Unexpected error, consider reporting this problem as you may have found a bug in simdjson" }, { UNEXPECTED_ERROR, "UNEXPECTED_ERROR: Unexpected error, consider reporting this problem as you may have found a bug in simdjson" },
{ PARSER_IN_USE, "Cannot parse a new document while a document is still in use." }, { PARSER_IN_USE, "PARSER_IN_USE: Cannot parse a new document while a document is still in use." },
{ OUT_OF_ORDER_ITERATION, "Objects and arrays can only be iterated when they are first encountered." }, { OUT_OF_ORDER_ITERATION, "OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered." },
{ INSUFFICIENT_PADDING, "simdjson requires the input JSON string to have at least SIMDJSON_PADDING extra bytes allocated, beyond the string's length. Consider using the simdjson::padded_string class if needed." }, { INSUFFICIENT_PADDING, "INSUFFICIENT_PADDING: simdjson requires the input JSON string to have at least SIMDJSON_PADDING extra bytes allocated, beyond the string's length. Consider using the simdjson::padded_string class if needed." },
{ INCOMPLETE_ARRAY_OR_OBJECT, "JSON document ended early in the middle of an object or array." }, { INCOMPLETE_ARRAY_OR_OBJECT, "INCOMPLETE_ARRAY_OR_OBJECT: JSON document ended early in the middle of an object or array." },
{ SCALAR_DOCUMENT_AS_VALUE, "A JSON document made of a scalar (number, Boolean, null or string) is treated as a value. Use get_bool(), get_double(), etc. on the document instead. "}, { SCALAR_DOCUMENT_AS_VALUE, "SCALAR_DOCUMENT_AS_VALUE: A JSON document made of a scalar (number, Boolean, null or string) is treated as a value. Use get_bool(), get_double(), etc. on the document instead. "},
{ OUT_OF_BOUNDS, "Attempted to access location outside of document."}, { OUT_OF_BOUNDS, "OUT_OF_BOUNDS: Attempt to access location outside of document."},
{ TRAILING_CONTENT, "Unexpected trailing content in the JSON input."} { TRAILING_CONTENT, "TRAILING_CONTENT: Unexpected trailing content in the JSON input."}
}; // error_messages[] }; // error_messages[]
} // namespace internal } // namespace internal
+407 -134
View File
File diff suppressed because it is too large Load Diff
+31 -31
View File
@@ -4,37 +4,37 @@ namespace simdjson {
namespace internal { namespace internal {
SIMDJSON_DLLIMPORTEXPORT const error_code_info error_codes[] { SIMDJSON_DLLIMPORTEXPORT const error_code_info error_codes[] {
{ SUCCESS, "No error" }, { SUCCESS, "SUCCESS: No error" },
{ CAPACITY, "This parser can't support a document that big" }, { CAPACITY, "CAPACITY: This parser can't support a document that big" },
{ MEMALLOC, "Error allocating memory, we're most likely out of memory" }, { MEMALLOC, "MEMALLOC: Error allocating memory, we're most likely out of memory" },
{ TAPE_ERROR, "The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc." }, { TAPE_ERROR, "TAPE_ERROR: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc." },
{ DEPTH_ERROR, "The JSON document was too deep (too many nested objects and arrays)" }, { DEPTH_ERROR, "DEPTH_ERROR: The JSON document was too deep (too many nested objects and arrays)" },
{ STRING_ERROR, "Problem while parsing a string" }, { STRING_ERROR, "STRING_ERROR: Problem while parsing a string" },
{ T_ATOM_ERROR, "Problem while parsing an atom starting with the letter 't'" }, { T_ATOM_ERROR, "T_ATOM_ERROR: Problem while parsing an atom starting with the letter 't'" },
{ F_ATOM_ERROR, "Problem while parsing an atom starting with the letter 'f'" }, { F_ATOM_ERROR, "F_ATOM_ERROR: Problem while parsing an atom starting with the letter 'f'" },
{ N_ATOM_ERROR, "Problem while parsing an atom starting with the letter 'n'" }, { N_ATOM_ERROR, "N_ATOM_ERROR: Problem while parsing an atom starting with the letter 'n'" },
{ NUMBER_ERROR, "Problem while parsing a number" }, { NUMBER_ERROR, "NUMBER_ERROR: Problem while parsing a number" },
{ UTF8_ERROR, "The input is not valid UTF-8" }, { UTF8_ERROR, "UTF8_ERROR: The input is not valid UTF-8" },
{ UNINITIALIZED, "Uninitialized" }, { UNINITIALIZED, "UNINITIALIZED: Uninitialized" },
{ EMPTY, "Empty: no JSON found" }, { EMPTY, "EMPTY: no JSON found" },
{ UNESCAPED_CHARS, "Within strings, some characters must be escaped, we found unescaped characters" }, { UNESCAPED_CHARS, "UNESCAPED_CHARS: Within strings, some characters must be escaped, we found unescaped characters" },
{ UNCLOSED_STRING, "A string is opened, but never closed." }, { UNCLOSED_STRING, "UNCLOSED_STRING: A string is opened, but never closed." },
{ UNSUPPORTED_ARCHITECTURE, "simdjson does not have an implementation supported by this CPU architecture (perhaps it's a non-SIMD CPU?)." }, { UNSUPPORTED_ARCHITECTURE, "UNSUPPORTED_ARCHITECTURE: simdjson does not have an implementation supported by this CPU architecture. Please report this error to the core team as it should never happen." },
{ INCORRECT_TYPE, "The JSON element does not have the requested type." }, { INCORRECT_TYPE, "INCORRECT_TYPE: The JSON element does not have the requested type." },
{ NUMBER_OUT_OF_RANGE, "The JSON number is too large or too small to fit within the requested type." }, { NUMBER_OUT_OF_RANGE, "NUMBER_OUT_OF_RANGE: The JSON number is too large or too small to fit within the requested type." },
{ INDEX_OUT_OF_BOUNDS, "Attempted to access an element of a JSON array that is beyond its length." }, { INDEX_OUT_OF_BOUNDS, "INDEX_OUT_OF_BOUNDS: Attempted to access an element of a JSON array that is beyond its length." },
{ NO_SUCH_FIELD, "The JSON field referenced does not exist in this object." }, { NO_SUCH_FIELD, "NO_SUCH_FIELD: The JSON field referenced does not exist in this object." },
{ IO_ERROR, "Error reading the file." }, { IO_ERROR, "IO_ERROR: Error reading the file." },
{ INVALID_JSON_POINTER, "Invalid JSON pointer syntax." }, { INVALID_JSON_POINTER, "INVALID_JSON_POINTER: Invalid JSON pointer syntax." },
{ INVALID_URI_FRAGMENT, "Invalid URI fragment syntax." }, { INVALID_URI_FRAGMENT, "INVALID_URI_FRAGMENT: Invalid URI fragment syntax." },
{ UNEXPECTED_ERROR, "Unexpected error, consider reporting this problem as you may have found a bug in simdjson" }, { UNEXPECTED_ERROR, "UNEXPECTED_ERROR: Unexpected error, consider reporting this problem as you may have found a bug in simdjson" },
{ PARSER_IN_USE, "Cannot parse a new document while a document is still in use." }, { PARSER_IN_USE, "PARSER_IN_USE: Cannot parse a new document while a document is still in use." },
{ OUT_OF_ORDER_ITERATION, "Objects and arrays can only be iterated when they are first encountered." }, { OUT_OF_ORDER_ITERATION, "OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered." },
{ INSUFFICIENT_PADDING, "simdjson requires the input JSON string to have at least SIMDJSON_PADDING extra bytes allocated, beyond the string's length. Consider using the simdjson::padded_string class if needed." }, { INSUFFICIENT_PADDING, "INSUFFICIENT_PADDING: simdjson requires the input JSON string to have at least SIMDJSON_PADDING extra bytes allocated, beyond the string's length. Consider using the simdjson::padded_string class if needed." },
{ INCOMPLETE_ARRAY_OR_OBJECT, "JSON document ended early in the middle of an object or array." }, { INCOMPLETE_ARRAY_OR_OBJECT, "INCOMPLETE_ARRAY_OR_OBJECT: JSON document ended early in the middle of an object or array." },
{ SCALAR_DOCUMENT_AS_VALUE, "A JSON document made of a scalar (number, Boolean, null or string) is treated as a value. Use get_bool(), get_double(), etc. on the document instead. "}, { SCALAR_DOCUMENT_AS_VALUE, "SCALAR_DOCUMENT_AS_VALUE: A JSON document made of a scalar (number, Boolean, null or string) is treated as a value. Use get_bool(), get_double(), etc. on the document instead. "},
{ OUT_OF_BOUNDS, "Attempted to access location outside of document."}, { OUT_OF_BOUNDS, "OUT_OF_BOUNDS: Attempt to access location outside of document."},
{ TRAILING_CONTENT, "Unexpected trailing content in the JSON input."} { TRAILING_CONTENT, "TRAILING_CONTENT: Unexpected trailing content in the JSON input."}
}; // error_messages[] }; // error_messages[]
} // namespace internal } // namespace internal
+12 -12
View File
@@ -59,28 +59,28 @@ private:
template<typename T> template<typename T>
bool cast_tester<T>::test_get(element element, T expected) { bool cast_tester<T>::test_get(element element, T expected) {
T actual; T actual{};
ASSERT_SUCCESS(element.get(actual)); ASSERT_SUCCESS(element.get(actual));
return assert_equal(actual, expected); return assert_equal(actual, expected);
} }
template<typename T> template<typename T>
bool cast_tester<T>::test_get(simdjson_result<element> element, T expected) { bool cast_tester<T>::test_get(simdjson_result<element> element, T expected) {
T actual; T actual{};
ASSERT_SUCCESS(element.get(actual)); ASSERT_SUCCESS(element.get(actual));
return assert_equal(actual, expected); return assert_equal(actual, expected);
} }
template<typename T> template<typename T>
bool cast_tester<T>::test_get_error(element element, error_code expected_error) { bool cast_tester<T>::test_get_error(element element, error_code expected_error) {
T actual; T actual{};
ASSERT_EQUAL(element.get(actual), expected_error); ASSERT_EQUAL(element.get(actual), expected_error);
return true; return true;
} }
template<typename T> template<typename T>
bool cast_tester<T>::test_get_error(simdjson_result<element> element, error_code expected_error) { bool cast_tester<T>::test_get_error(simdjson_result<element> element, error_code expected_error) {
T actual; T actual{};
ASSERT_EQUAL(element.get(actual), expected_error); ASSERT_EQUAL(element.get(actual), expected_error);
return true; return true;
} }
@@ -113,28 +113,28 @@ bool cast_tester<T>::test_get_t_error(simdjson_result<element> element, error_co
template<typename T> template<typename T>
bool cast_tester<T>::test_named_get(element element, T expected) { bool cast_tester<T>::test_named_get(element element, T expected) {
T actual; T actual{};
ASSERT_SUCCESS(named_get(element).get(actual)); ASSERT_SUCCESS(named_get(element).get(actual));
return assert_equal(actual, expected); return assert_equal(actual, expected);
} }
template<typename T> template<typename T>
bool cast_tester<T>::test_named_get(simdjson_result<element> element, T expected) { bool cast_tester<T>::test_named_get(simdjson_result<element> element, T expected) {
T actual; T actual{};
ASSERT_SUCCESS(named_get(element).get(actual)); ASSERT_SUCCESS(named_get(element).get(actual));
return assert_equal(actual, expected); return assert_equal(actual, expected);
} }
template<typename T> template<typename T>
bool cast_tester<T>::test_named_get_error(element element, error_code expected_error) { bool cast_tester<T>::test_named_get_error(element element, error_code expected_error) {
T actual; T actual{};
ASSERT_EQUAL(named_get(element).get(actual), expected_error); ASSERT_EQUAL(named_get(element).get(actual), expected_error);
return true; return true;
} }
template<typename T> template<typename T>
bool cast_tester<T>::test_named_get_error(simdjson_result<element> element, error_code expected_error) { bool cast_tester<T>::test_named_get_error(simdjson_result<element> element, error_code expected_error) {
T actual; T actual{};
ASSERT_EQUAL(named_get(element).get(actual), expected_error); ASSERT_EQUAL(named_get(element).get(actual), expected_error);
return true; return true;
} }
@@ -143,7 +143,7 @@ bool cast_tester<T>::test_named_get_error(simdjson_result<element> element, erro
template<typename T> template<typename T>
bool cast_tester<T>::test_implicit_cast(element element, T expected) { bool cast_tester<T>::test_implicit_cast(element element, T expected) {
T actual; T actual{};
try { try {
actual = element; actual = element;
} catch(simdjson_error &e) { } catch(simdjson_error &e) {
@@ -155,7 +155,7 @@ bool cast_tester<T>::test_implicit_cast(element element, T expected) {
template<typename T> template<typename T>
bool cast_tester<T>::test_implicit_cast(simdjson_result<element> element, T expected) { bool cast_tester<T>::test_implicit_cast(simdjson_result<element> element, T expected) {
T actual; T actual{};
try { try {
actual = element; actual = element;
} catch(simdjson_error &e) { } catch(simdjson_error &e) {
@@ -168,7 +168,7 @@ bool cast_tester<T>::test_implicit_cast(simdjson_result<element> element, T expe
template<typename T> template<typename T>
bool cast_tester<T>::test_implicit_cast_error(element element, error_code expected_error) { bool cast_tester<T>::test_implicit_cast_error(element element, error_code expected_error) {
try { try {
simdjson_unused T actual; simdjson_unused T actual{};
actual = element; actual = element;
return false; return false;
} catch(simdjson_error &e) { } catch(simdjson_error &e) {
@@ -180,7 +180,7 @@ bool cast_tester<T>::test_implicit_cast_error(element element, error_code expect
template<typename T> template<typename T>
bool cast_tester<T>::test_implicit_cast_error(simdjson_result<element> element, error_code expected_error) { bool cast_tester<T>::test_implicit_cast_error(simdjson_result<element> element, error_code expected_error) {
try { try {
simdjson_unused T actual; simdjson_unused T actual{};
actual = element; actual = element;
return false; return false;
} catch(simdjson_error &e) { } catch(simdjson_error &e) {
+32 -14
View File
@@ -9,7 +9,6 @@
#include <set> #include <set>
#include <sstream> #include <sstream>
#include <utility> #include <utility>
#include <ciso646>
#include <unistd.h> #include <unistd.h>
#include "simdjson.h" #include "simdjson.h"
@@ -67,6 +66,15 @@ namespace number_tests {
return true; return true;
} }
bool issue2017() {
TEST_START();
simdjson::dom::parser parser;
simdjson::padded_string docdata = R"({"score":0.8825149536132812})"_padded;
double score;
ASSERT_SUCCESS(parser.parse(docdata)["score"].get_double().get(score));
ASSERT_EQUAL(score, 0.8825149536132812);
TEST_SUCCEED();
}
bool small_integers() { bool small_integers() {
std::cout << __func__ << std::endl; std::cout << __func__ << std::endl;
@@ -74,7 +82,7 @@ namespace number_tests {
for (int m = 10; m < 20; m++) { for (int m = 10; m < 20; m++) {
for (int i = -1024; i < 1024; i++) { for (int i = -1024; i < 1024; i++) {
auto str = std::to_string(i); auto str = std::to_string(i);
int64_t actual; int64_t actual{};
ASSERT_SUCCESS(parser.parse(str).get(actual)); ASSERT_SUCCESS(parser.parse(str).get(actual));
if (actual != i) { if (actual != i) {
std::cerr << "JSON '" << str << "' parsed to " << actual << " instead of " << i << std::endl; std::cerr << "JSON '" << str << "' parsed to " << actual << " instead of " << i << std::endl;
@@ -378,7 +386,8 @@ namespace number_tests {
} }
bool run() { bool run() {
return truncated_borderline() && return issue2017() &&
truncated_borderline() &&
specific_tests() && specific_tests() &&
ground_truth() && ground_truth() &&
small_integers() && small_integers() &&
@@ -551,7 +560,7 @@ namespace parse_api_tests {
ASSERT_SUCCESS( parser.parse_many(empty_batches_ndjson, BATCH_SIZE*16).get(stream) ); ASSERT_SUCCESS( parser.parse_many(empty_batches_ndjson, BATCH_SIZE*16).get(stream) );
for (auto doc : stream) { for (auto doc : stream) {
count++; count++;
uint64_t val; uint64_t val{};
ASSERT_SUCCESS( doc.get(val) ); ASSERT_SUCCESS( doc.get(val) );
ASSERT_EQUAL( val, count ); ASSERT_EQUAL( val, count );
} }
@@ -956,6 +965,22 @@ namespace dom_api_tests {
return true; return true;
} }
bool issue1979() {
TEST_START();
auto json = R"({
"@avito-core/toggles:6.1.18": {
"add_model_review_from": true
}
})"_padded;
simdjson::dom::parser parser;
simdjson::dom::element doc;
ASSERT_SUCCESS(parser.parse(json).get(doc));
simdjson::dom::object main_object;
ASSERT_SUCCESS(doc.get_object().get(main_object));
ASSERT_SUCCESS(main_object["@avito-core/toggles:6.1.18"].get_object().error())
TEST_SUCCEED();
}
bool document_object_index() { bool document_object_index() {
std::cout << "Running " << __func__ << std::endl; std::cout << "Running " << __func__ << std::endl;
string json(R"({ "a": 1, "b": 2, "c/d": 3})"); string json(R"({ "a": 1, "b": 2, "c/d": 3})");
@@ -978,14 +1003,6 @@ namespace dom_api_tests {
simdjson::error_code error; simdjson::error_code error;
simdjson_unused element val; simdjson_unused element val;
// This is disabled, see https://github.com/simdjson/simdjson/issues/1243
//#ifndef _LIBCPP_VERSION // should work everywhere but with libc++, must include the <ciso646> header.
// std::tie(val,error) = object["d"];
// ASSERT_ERROR( error, NO_SUCH_FIELD );
// std::tie(std::ignore,error) = object["d"];
// ASSERT_ERROR( error, NO_SUCH_FIELD );
//#endif
// tie(val, error) = object["d"]; fails with "no viable overloaded '='" on Apple clang version 11.0.0 tie(val, error) = doc["d"];
object["d"].tie(val, error); object["d"].tie(val, error);
ASSERT_ERROR( error, NO_SUCH_FIELD ); ASSERT_ERROR( error, NO_SUCH_FIELD );
ASSERT_ERROR( object["d"].get(val), NO_SUCH_FIELD ); ASSERT_ERROR( object["d"].get(val), NO_SUCH_FIELD );
@@ -1023,7 +1040,7 @@ namespace dom_api_tests {
std::cout << "Running " << __func__ << std::endl; std::cout << "Running " << __func__ << std::endl;
// Prints the number of results in twitter.json // Prints the number of results in twitter.json
dom::parser parser; dom::parser parser;
uint64_t result_count; uint64_t result_count{};
ASSERT_SUCCESS( parser.load(TWITTER_JSON)["search_metadata"]["count"].get(result_count) ); ASSERT_SUCCESS( parser.load(TWITTER_JSON)["search_metadata"]["count"].get(result_count) );
ASSERT_EQUAL( result_count, 100 ); ASSERT_EQUAL( result_count, 100 );
return true; return true;
@@ -1039,7 +1056,7 @@ namespace dom_api_tests {
for (auto tweet : tweets) { for (auto tweet : tweets) {
object user; object user;
ASSERT_SUCCESS( tweet["user"].get(user) ); ASSERT_SUCCESS( tweet["user"].get(user) );
bool default_profile; bool default_profile{};
ASSERT_SUCCESS( user["default_profile"].get(default_profile) ); ASSERT_SUCCESS( user["default_profile"].get(default_profile) );
if (default_profile) { if (default_profile) {
std::string_view screen_name; std::string_view screen_name;
@@ -1254,6 +1271,7 @@ namespace dom_api_tests {
#if SIMDJSON_ENABLE_DEPRECATED_API #if SIMDJSON_ENABLE_DEPRECATED_API
ParsedJson_Iterator_test() && ParsedJson_Iterator_test() &&
#endif #endif
issue1979() &&
object_iterator() && object_iterator() &&
array_iterator() && array_iterator() &&
object_iterator_empty() && object_iterator_empty() &&
+3 -3
View File
@@ -634,7 +634,7 @@ namespace document_stream_tests {
ASSERT_SUCCESS( odparser.parse_many(json.data(), json.length(), 50).get(odstream) ); ASSERT_SUCCESS( odparser.parse_many(json.data(), json.length(), 50).get(odstream) );
for (auto doc: odstream) { for (auto doc: odstream) {
if(counter < 6) { if(counter < 6) {
int64_t val; int64_t val{};
ASSERT_SUCCESS(doc.at_pointer("/4").get(val)); ASSERT_SUCCESS(doc.at_pointer("/4").get(val));
ASSERT_EQUAL(val, 5); ASSERT_EQUAL(val, 5);
} else { } else {
@@ -797,7 +797,7 @@ namespace document_stream_tests {
simdjson::dom::document_stream stream; simdjson::dom::document_stream stream;
ASSERT_SUCCESS( parser.parse_many(str, batch_size).get(stream) ); ASSERT_SUCCESS( parser.parse_many(str, batch_size).get(stream) );
for (auto doc : stream) { for (auto doc : stream) {
int64_t keyid; int64_t keyid{};
ASSERT_SUCCESS( doc["id"].get(keyid) ); ASSERT_SUCCESS( doc["id"].get(keyid) );
ASSERT_EQUAL( keyid, int64_t(count) ); ASSERT_EQUAL( keyid, int64_t(count) );
@@ -837,7 +837,7 @@ namespace document_stream_tests {
simdjson::dom::document_stream stream; simdjson::dom::document_stream stream;
ASSERT_SUCCESS( parser.parse_many(str, batch_size).get(stream) ); ASSERT_SUCCESS( parser.parse_many(str, batch_size).get(stream) );
for (auto doc : stream) { for (auto doc : stream) {
int64_t keyid; int64_t keyid{};
ASSERT_SUCCESS( doc["id"].get(keyid) ); ASSERT_SUCCESS( doc["id"].get(keyid) );
ASSERT_EQUAL( keyid, int64_t(count) ); ASSERT_EQUAL( keyid, int64_t(count) );
+2 -2
View File
@@ -46,7 +46,7 @@ namespace parser_load {
ASSERT_SUCCESS(parser.parse_many(DOC).get(docs)); ASSERT_SUCCESS(parser.parse_many(DOC).get(docs));
for (auto doc : docs) { for (auto doc : docs) {
count++; count++;
uint64_t val; uint64_t val{};
auto error = doc.get(val); auto error = doc.get(val);
if (count == 3) { if (count == 3) {
ASSERT_ERROR(error, TAPE_ERROR); ASSERT_ERROR(error, TAPE_ERROR);
@@ -83,7 +83,7 @@ namespace parser_load {
ASSERT_SUCCESS(parser.parse_many(DOC).get(docs)); ASSERT_SUCCESS(parser.parse_many(DOC).get(docs));
for (auto doc : docs) { for (auto doc : docs) {
count++; count++;
uint64_t val; uint64_t val{};
auto error = doc.get(val); auto error = doc.get(val);
if (count == 3) { if (count == 3) {
ASSERT_ERROR(error, TAPE_ERROR); ASSERT_ERROR(error, TAPE_ERROR);
+2 -2
View File
@@ -33,12 +33,12 @@ static bool parse_and_validate(const std::string src, T expected) {
simdjson::dom::parser parser; simdjson::dom::parser parser;
if constexpr (std::is_same<int64_t, T>::value) { if constexpr (std::is_same<int64_t, T>::value) {
int64_t actual; int64_t actual{};
ASSERT_SUCCESS( parser.parse(pstr)["key"].get(actual) ); ASSERT_SUCCESS( parser.parse(pstr)["key"].get(actual) );
std::cout << std::boolalpha << "test: " << (expected == actual) << std::endl; std::cout << std::boolalpha << "test: " << (expected == actual) << std::endl;
ASSERT_EQUAL( expected, actual ); ASSERT_EQUAL( expected, actual );
} else { } else {
uint64_t actual; uint64_t actual{};
ASSERT_SUCCESS( parser.parse(pstr)["key"].get(actual) ); ASSERT_SUCCESS( parser.parse(pstr)["key"].get(actual) );
std::cout << std::boolalpha << "test: " << (expected == actual) << std::endl; std::cout << std::boolalpha << "test: " << (expected == actual) << std::endl;
ASSERT_EQUAL( expected, actual ); ASSERT_EQUAL( expected, actual );
+1 -1
View File
@@ -140,7 +140,7 @@ bool tester(int seed, size_t volume) {
std::vector<char> buffer(1024); // large buffer (can't overflow) std::vector<char> buffer(1024); // large buffer (can't overflow)
simdjson::dom::parser parser; simdjson::dom::parser parser;
RandomEngine rand(seed); RandomEngine rand(seed);
double result; double result{};
for (size_t i = 0; i < volume; i++) { for (size_t i = 0; i < volume; i++) {
if((i%100000) == 0) { std::cout << "."; std::cout.flush(); } if((i%100000) == 0) { std::cout << "."; std::cout.flush(); }
size_t length = build_random_string(rand, buffer.data()); size_t length = build_random_string(rand, buffer.data());
+4 -4
View File
@@ -41,7 +41,7 @@ void basics_error_2() {
cout << "Make/Model: " << make << "/" << model << endl; cout << "Make/Model: " << make << "/" << model << endl;
// Casting a JSON element to an integer // Casting a JSON element to an integer
uint64_t year; uint64_t year{};
if ((error = car["year"].get(year))) { cerr << error << endl; exit(1); } if ((error = car["year"].get(year))) { cerr << error << endl; exit(1); }
cout << "- This car is " << 2020 - year << "years old." << endl; cout << "- This car is " << 2020 - year << "years old." << endl;
@@ -81,13 +81,13 @@ void basics_error_3() {
dom::object innerobj; dom::object innerobj;
if ((error = key_value.value.get(innerobj))) { cerr << error << endl; exit(1); } if ((error = key_value.value.get(innerobj))) { cerr << error << endl; exit(1); }
double va, vb; double va{}, vb{};
if ((error = innerobj["a"].get(va))) { cerr << error << endl; exit(1); } if ((error = innerobj["a"].get(va))) { cerr << error << endl; exit(1); }
cout << "a: " << va << ", "; cout << "a: " << va << ", ";
if ((error = innerobj["b"].get(vb))) { cerr << error << endl; exit(1); } if ((error = innerobj["b"].get(vb))) { cerr << error << endl; exit(1); }
cout << "b: " << vb << ", "; cout << "b: " << vb << ", ";
int64_t vc; int64_t vc{};
if ((error = innerobj["c"].get(vc))) { cerr << error << endl; exit(1); } if ((error = innerobj["c"].get(vc))) { cerr << error << endl; exit(1); }
cout << "c: " << vc << endl; cout << "c: " << vc << endl;
} }
@@ -132,7 +132,7 @@ void basics_error_2_cpp17() {
cout << "Make/Model: " << make << "/" << model << endl; cout << "Make/Model: " << make << "/" << model << endl;
// Casting a JSON element to an integer // Casting a JSON element to an integer
uint64_t year; uint64_t year{};
if ((error = car["year"].get(year))) { cerr << error << endl; exit(1); } if ((error = car["year"].get(year))) { cerr << error << endl; exit(1); }
cout << "- This car is " << 2020 - year << "years old." << endl; cout << "- This car is " << 2020 - year << "years old." << endl;
-1
View File
@@ -60,7 +60,6 @@ static inline void write_utf8(unsigned codepoint, char *&end) {
*end++ = static_cast<char>(0x80 | ((codepoint >> 6) & 0x3F)); *end++ = static_cast<char>(0x80 | ((codepoint >> 6) & 0x3F));
*end++ = static_cast<char>(0x80 | (codepoint & 0x3F)); *end++ = static_cast<char>(0x80 | (codepoint & 0x3F));
} else { } else {
assert(codepoint < 0x200000);
*end++ = static_cast<char>(0xF0 | (codepoint >> 18)); *end++ = static_cast<char>(0xF0 | (codepoint >> 18));
*end++ = static_cast<char>(0x80 | ((codepoint >> 12) & 0x3F)); *end++ = static_cast<char>(0x80 | ((codepoint >> 12) & 0x3F));
*end++ = static_cast<char>(0x80 | ((codepoint >> 6) & 0x3F)); *end++ = static_cast<char>(0x80 | ((codepoint >> 6) & 0x3F));
-1
View File
@@ -1,5 +1,4 @@
#include <cinttypes> #include <cinttypes>
#include <ciso646>
#include <cmath> #include <cmath>
#include <cstdio> #include <cstdio>
#include <cstdlib> #include <cstdlib>
+2
View File
@@ -3,6 +3,7 @@ link_libraries(simdjson)
include_directories(..) include_directories(..)
add_subdirectory(compilation_failure_tests) add_subdirectory(compilation_failure_tests)
add_cpp_test(ondemand_log_tests LABELS ondemand acceptance per_implementation) add_cpp_test(ondemand_log_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_log_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_tostring_tests LABELS ondemand acceptance per_implementation) add_cpp_test(ondemand_tostring_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_active_tests LABELS ondemand acceptance per_implementation) add_cpp_test(ondemand_active_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_array_tests LABELS ondemand acceptance per_implementation) add_cpp_test(ondemand_array_tests LABELS ondemand acceptance per_implementation)
@@ -24,6 +25,7 @@ add_cpp_test(ondemand_readme_examples LABELS ondemand acceptance per_impl
add_cpp_test(ondemand_scalar_tests LABELS ondemand acceptance per_implementation) add_cpp_test(ondemand_scalar_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_twitter_tests LABELS ondemand acceptance per_implementation) add_cpp_test(ondemand_twitter_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_wrong_type_error_tests LABELS ondemand acceptance per_implementation) add_cpp_test(ondemand_wrong_type_error_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_iterate_many_csv LABELS ondemand acceptance per_implementation)
if(HAVE_POSIX_FORK AND HAVE_POSIX_WAIT) # assert tests use fork and wait, which aren't on MSVC if(HAVE_POSIX_FORK AND HAVE_POSIX_WAIT) # assert tests use fork and wait, which aren't on MSVC
add_cpp_test(ondemand_assert_out_of_order_values LABELS assert per_implementation explicitonly ondemand) add_cpp_test(ondemand_assert_out_of_order_values LABELS assert per_implementation explicitonly ondemand)
+32 -17
View File
@@ -6,6 +6,20 @@ using namespace simdjson;
namespace array_tests { namespace array_tests {
using namespace std; using namespace std;
using simdjson::ondemand::json_type; using simdjson::ondemand::json_type;
bool issue1977() {
TEST_START();
auto json = R"([1, 2] foo ])"_padded;
ondemand::parser parser;
ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(json).get(doc));
ondemand::array array;
ASSERT_SUCCESS(doc.get_array().get(array));
for (auto values : array) {
ASSERT_SUCCESS(values);
}
ASSERT_FALSE(doc.at_end());
TEST_SUCCEED();
}
bool issue1588() { bool issue1588() {
TEST_START(); TEST_START();
const auto json = R"({ const auto json = R"({
@@ -134,7 +148,7 @@ namespace array_tests {
ondemand::parser parser; ondemand::parser parser;
ondemand::document doc; ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(json).get(doc)); ASSERT_SUCCESS(parser.iterate(json).get(doc));
size_t count; size_t count{};
ASSERT_SUCCESS(doc.count_elements().get(count)); ASSERT_SUCCESS(doc.count_elements().get(count));
ondemand::array arr; ondemand::array arr;
ASSERT_SUCCESS(doc.get_array().get(arr)); ASSERT_SUCCESS(doc.get_array().get(arr));
@@ -182,7 +196,7 @@ namespace array_tests {
ASSERT_SUCCESS(doc.get_object().get(obj)); ASSERT_SUCCESS(doc.get_object().get(obj));
ondemand::value v; ondemand::value v;
ASSERT_SUCCESS(doc.find_field("test").get(v)); ASSERT_SUCCESS(doc.find_field("test").get(v));
size_t count; size_t count{};
ASSERT_SUCCESS(v.count_elements().get(count)); ASSERT_SUCCESS(v.count_elements().get(count));
ASSERT_EQUAL(count, 3); ASSERT_EQUAL(count, 3);
ASSERT_SUCCESS(doc.find_field("joe").get(v)); ASSERT_SUCCESS(doc.find_field("joe").get(v));
@@ -200,7 +214,7 @@ namespace array_tests {
ondemand::array array; ondemand::array array;
ASSERT_RESULT( doc_result.type(), json_type::array ); ASSERT_RESULT( doc_result.type(), json_type::array );
ASSERT_SUCCESS( doc_result.get_array().get(array) ); ASSERT_SUCCESS( doc_result.get_array().get(array) );
size_t count; size_t count{};
ASSERT_SUCCESS( array.count_elements().get(count) ); ASSERT_SUCCESS( array.count_elements().get(count) );
ASSERT_EQUAL(count, expected_value.size()); ASSERT_EQUAL(count, expected_value.size());
return true; return true;
@@ -209,13 +223,13 @@ namespace array_tests {
ondemand::array array; ondemand::array array;
ASSERT_RESULT( doc_result.type(), json_type::array ); ASSERT_RESULT( doc_result.type(), json_type::array );
ASSERT_SUCCESS( doc_result.get(array) ); ASSERT_SUCCESS( doc_result.get(array) );
size_t count; size_t count{};
ASSERT_SUCCESS( array.count_elements().get(count) ); ASSERT_SUCCESS( array.count_elements().get(count) );
ASSERT_EQUAL(count, expected_value.size()); ASSERT_EQUAL(count, expected_value.size());
size_t i = 0; size_t i = 0;
std::vector<uint64_t> receiver(count); std::vector<uint64_t> receiver(count);
for (auto value : array) { for (auto value : array) {
uint64_t actual; uint64_t actual{};
ASSERT_SUCCESS( value.get(actual) ); ASSERT_SUCCESS( value.get(actual) );
ASSERT_EQUAL(actual, expected_value[i]); ASSERT_EQUAL(actual, expected_value[i]);
receiver[i] = actual; receiver[i] = actual;
@@ -235,7 +249,7 @@ namespace array_tests {
ondemand::array array; ondemand::array array;
ASSERT_RESULT( doc_result.type(), json_type::array ); ASSERT_RESULT( doc_result.type(), json_type::array );
ASSERT_SUCCESS( doc_result.get_array().get(array) ); ASSERT_SUCCESS( doc_result.get_array().get(array) );
size_t count; size_t count{};
ASSERT_SUCCESS( array.count_elements().get(count) ); ASSERT_SUCCESS( array.count_elements().get(count) );
ASSERT_EQUAL(count, 0); ASSERT_EQUAL(count, 0);
return true; return true;
@@ -244,13 +258,13 @@ namespace array_tests {
ondemand::array array; ondemand::array array;
ASSERT_RESULT( doc_result.type(), json_type::array ); ASSERT_RESULT( doc_result.type(), json_type::array );
ASSERT_SUCCESS( doc_result.get(array) ); ASSERT_SUCCESS( doc_result.get(array) );
size_t count; size_t count{};
ASSERT_SUCCESS( array.count_elements().get(count) ); ASSERT_SUCCESS( array.count_elements().get(count) );
ASSERT_EQUAL(count, 0); ASSERT_EQUAL(count, 0);
size_t i = 0; size_t i = 0;
std::vector<uint64_t> receiver(count); std::vector<uint64_t> receiver(count);
for (auto value : array) { for (auto value : array) {
uint64_t actual; uint64_t actual{};
ASSERT_SUCCESS( value.get(actual) ); ASSERT_SUCCESS( value.get(actual) );
i++; i++;
} }
@@ -269,7 +283,7 @@ namespace array_tests {
ondemand::array array; ondemand::array array;
ASSERT_RESULT( doc_result.type(), json_type::array ); ASSERT_RESULT( doc_result.type(), json_type::array );
ASSERT_SUCCESS( doc_result.get(array) ); ASSERT_SUCCESS( doc_result.get(array) );
size_t count; size_t count{};
auto e = array.count_elements().get(count); auto e = array.count_elements().get(count);
if( e != TAPE_ERROR) { if( e != TAPE_ERROR) {
std::cout << e << "\n"; std::cout << e << "\n";
@@ -285,7 +299,7 @@ namespace array_tests {
TEST_START(); TEST_START();
auto empty = R"( [] )"_padded; auto empty = R"( [] )"_padded;
SUBTEST("ondemand::empty_doc_array", test_ondemand_doc(empty, [&](auto doc_result) { SUBTEST("ondemand::empty_doc_array", test_ondemand_doc(empty, [&](auto doc_result) {
size_t count; size_t count{};
ASSERT_RESULT( doc_result.type(), json_type::array ); ASSERT_RESULT( doc_result.type(), json_type::array );
ASSERT_SUCCESS( doc_result.count_elements().get(count) ); ASSERT_SUCCESS( doc_result.count_elements().get(count) );
ASSERT_EQUAL( count, 0 ); ASSERT_EQUAL( count, 0 );
@@ -293,7 +307,7 @@ namespace array_tests {
})); }));
auto basic = R"( [-1.234, 100000000000000, null, [1,2,3], {"t":true, "f":false}] )"_padded; auto basic = R"( [-1.234, 100000000000000, null, [1,2,3], {"t":true, "f":false}] )"_padded;
SUBTEST("ondemand::basic_doc_array", test_ondemand_doc(basic, [&](auto doc_result) { SUBTEST("ondemand::basic_doc_array", test_ondemand_doc(basic, [&](auto doc_result) {
size_t count; size_t count{};
ASSERT_RESULT( doc_result.type(), json_type::array ); ASSERT_RESULT( doc_result.type(), json_type::array );
ASSERT_SUCCESS( doc_result.count_elements().get(count) ); ASSERT_SUCCESS( doc_result.count_elements().get(count) );
ASSERT_EQUAL( count, 5 ); ASSERT_EQUAL( count, 5 );
@@ -331,7 +345,7 @@ namespace array_tests {
size_t i = 0; size_t i = 0;
for (auto value : array) { for (auto value : array) {
int64_t actual; int64_t actual{};
ASSERT_SUCCESS( value.get(actual) ); ASSERT_SUCCESS( value.get(actual) );
ASSERT_EQUAL(actual, expected_value[i]); ASSERT_EQUAL(actual, expected_value[i]);
i++; i++;
@@ -355,7 +369,7 @@ namespace array_tests {
ASSERT_SUCCESS( doc_result.get(array) ); ASSERT_SUCCESS( doc_result.get(array) );
i = 0; i = 0;
for (auto value : array) { for (auto value : array) {
int64_t actual; int64_t actual{};
ASSERT_SUCCESS( value.get(actual) ); ASSERT_SUCCESS( value.get(actual) );
container[i] = actual; container[i] = actual;
i++; i++;
@@ -379,7 +393,7 @@ namespace array_tests {
array.reset(); array.reset();
i = 0; i = 0;
for (auto value : array) { for (auto value : array) {
int64_t actual; int64_t actual{};
ASSERT_SUCCESS( value.get(actual) ); ASSERT_SUCCESS( value.get(actual) );
container[i] = actual; container[i] = actual;
i++; i++;
@@ -435,7 +449,7 @@ namespace array_tests {
TEST_SUCCEED(); TEST_SUCCEED();
} }
bool count_empty(simdjson::ondemand::array arr) { bool count_empty(simdjson::ondemand::array arr) {
size_t count; size_t count{};
ASSERT_SUCCESS(arr.count_elements().get(count)); ASSERT_SUCCESS(arr.count_elements().get(count));
ASSERT_EQUAL(count, 0); ASSERT_EQUAL(count, 0);
bool is_empty; bool is_empty;
@@ -466,7 +480,7 @@ namespace array_tests {
for (auto d : data) { for (auto d : data) {
simdjson::ondemand::array arr; simdjson::ondemand::array arr;
ASSERT_SUCCESS(d.get_array().get(arr)); ASSERT_SUCCESS(d.get_array().get(arr));
size_t count; size_t count{};
ASSERT_SUCCESS(arr.count_elements().get(count)); ASSERT_SUCCESS(arr.count_elements().get(count));
ASSERT_EQUAL(count, 4); ASSERT_EQUAL(count, 4);
} }
@@ -528,7 +542,7 @@ namespace array_tests {
size_t i=0; size_t i=0;
for (auto value : array) { for (auto value : array) {
int64_t actual; int64_t actual{};
ASSERT_SUCCESS( value.get(actual) ); ASSERT_SUCCESS( value.get(actual) );
ASSERT_EQUAL(actual, expected_value[i]); ASSERT_EQUAL(actual, expected_value[i]);
i++; i++;
@@ -830,6 +844,7 @@ namespace array_tests {
bool run() { bool run() {
return return
issue1977() &&
issue1876() && issue1876() &&
issue1742() && issue1742() &&
empty_rewind_convoluted() && empty_rewind_convoluted() &&
@@ -7,7 +7,7 @@ namespace document_stream_tests {
template <typename T> template <typename T>
bool process_doc(T &docref) { bool process_doc(T &docref) {
int64_t val; int64_t val{};
ASSERT_SUCCESS(docref.at_pointer("/4").get(val)); ASSERT_SUCCESS(docref.at_pointer("/4").get(val));
ASSERT_EQUAL(val, 5); ASSERT_EQUAL(val, 5);
return true; return true;
@@ -193,6 +193,20 @@ namespace document_stream_tests {
TEST_SUCCEED(); TEST_SUCCEED();
} }
bool issue1977() {
TEST_START();
std::string json = R"( 1111 })";
ondemand::parser odparser;
ondemand::document_stream odstream;
ASSERT_SUCCESS(odparser.iterate_many(json).get(odstream));
auto i = odstream.begin();
for (; i != odstream.end(); ++i) {
ASSERT_TRUE(false);
}
ASSERT_TRUE(i.current_index() == 0);
TEST_SUCCEED();
}
bool issue1683() { bool issue1683() {
TEST_START(); TEST_START();
@@ -567,7 +581,7 @@ namespace document_stream_tests {
size_t count{0}; size_t count{0};
ASSERT_SUCCESS( parser.iterate_many(str, batch_size).get(stream) ); ASSERT_SUCCESS( parser.iterate_many(str, batch_size).get(stream) );
for (auto doc : stream) { for (auto doc : stream) {
int64_t keyid; int64_t keyid{};
ASSERT_SUCCESS( doc["id"].get(keyid) ); ASSERT_SUCCESS( doc["id"].get(keyid) );
ASSERT_EQUAL( keyid, int64_t(count) ); ASSERT_EQUAL( keyid, int64_t(count) );
@@ -604,7 +618,7 @@ namespace document_stream_tests {
ASSERT_SUCCESS( odparser.iterate_many(json.data(), json.length(), 50).get(odstream) ); ASSERT_SUCCESS( odparser.iterate_many(json.data(), json.length(), 50).get(odstream) );
for (auto doc: odstream) { for (auto doc: odstream) {
if(counter < 6) { if(counter < 6) {
int64_t val; int64_t val{};
ASSERT_SUCCESS(doc.at_pointer("/4").get(val)); ASSERT_SUCCESS(doc.at_pointer("/4").get(val));
ASSERT_EQUAL(val, 5); ASSERT_EQUAL(val, 5);
} else { } else {
@@ -643,7 +657,7 @@ namespace document_stream_tests {
size_t count{0}; size_t count{0};
ASSERT_SUCCESS( parser.iterate_many(str, batch_size).get(stream) ); ASSERT_SUCCESS( parser.iterate_many(str, batch_size).get(stream) );
for (auto doc : stream) { for (auto doc : stream) {
int64_t keyid; int64_t keyid{};
ASSERT_SUCCESS( doc["id"].get(keyid) ); ASSERT_SUCCESS( doc["id"].get(keyid) );
ASSERT_EQUAL( keyid, int64_t(count) ); ASSERT_EQUAL( keyid, int64_t(count) );
@@ -789,6 +803,7 @@ namespace document_stream_tests {
bool run() { bool run() {
return return
issue1977() &&
string_with_trailing() && string_with_trailing() &&
uint64_with_trailing() && uint64_with_trailing() &&
int64_with_trailing() && int64_with_trailing() &&
@@ -19,9 +19,9 @@ namespace error_location_tests {
const char* c; const char* c;
// Must call current_location first because get_int64() will consume values // Must call current_location first because get_int64() will consume values
ASSERT_SUCCESS(doc.current_location().get(c)); ASSERT_SUCCESS(doc.current_location().get(c));
ASSERT_EQUAL(*c,expected[count]); ASSERT_EQUAL(*c, expected[count]);
ASSERT_SUCCESS(value.get_int64().get(i)); ASSERT_SUCCESS(value.get_int64().get(i));
ASSERT_EQUAL(i,expected_values[count]); ASSERT_EQUAL(i, expected_values[count]);
count++; count++;
} }
ASSERT_EQUAL(count,3); ASSERT_EQUAL(count,3);
@@ -64,12 +64,14 @@ namespace error_location_tests {
ASSERT_SUCCESS(doc.at_pointer("/a/2/1").get(i)); ASSERT_SUCCESS(doc.at_pointer("/a/2/1").get(i));
ASSERT_EQUAL(i, 4); ASSERT_EQUAL(i, 4);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, ",5]], \"b\": {\"c\": [1.2, 2.3]}} "); std::string expected = ",5]], \"b\": {\"c\": [1.2, 2.3]}} ";
ASSERT_EQUAL(std::string(ptr, expected.size()), expected);
double d; double d;
ASSERT_SUCCESS(doc.at_pointer("/b/c/1").get(d)); ASSERT_SUCCESS(doc.at_pointer("/b/c/1").get(d));
ASSERT_EQUAL(d, 2.3); ASSERT_EQUAL(d, 2.3);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "]}} "); expected = "]}} ";
ASSERT_EQUAL(std::string(ptr, expected.size()), expected);
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -83,11 +85,13 @@ namespace error_location_tests {
double d; double d;
ASSERT_ERROR(doc.at_pointer("/b/c/0").get(d), NUMBER_ERROR); ASSERT_ERROR(doc.at_pointer("/b/c/0").get(d), NUMBER_ERROR);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "1.2., 2.3]}} "); std::string expected = "1.2., 2.3]}} ";
ASSERT_EQUAL(std::string(ptr, expected.size()), expected);
uint64_t i; uint64_t i;
ASSERT_ERROR(doc.at_pointer("/a/2/1").get(i), TAPE_ERROR); ASSERT_ERROR(doc.at_pointer("/a/2/1").get(i), TAPE_ERROR);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "4,5]], \"b\": {\"c\": [1.2., 2.3]}} "); expected = "4,5]], \"b\": {\"c\": [1.2., 2.3]}} ";
ASSERT_EQUAL(std::string(ptr, expected.size()), expected);
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -100,7 +104,7 @@ namespace error_location_tests {
ASSERT_SUCCESS(parser.iterate(json).get(doc)); ASSERT_SUCCESS(parser.iterate(json).get(doc));
ASSERT_ERROR(doc["a"], INCORRECT_TYPE); ASSERT_ERROR(doc["a"], INCORRECT_TYPE);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "\xc3\x94\xc3\xb8\xe2\x84\xa6{\"a\":1, 3} "); ASSERT_EQUAL(std::string(ptr, 18), "\xc3\x94\xc3\xb8\xe2\x84\xa6{\"a\":1, 3} ");
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -115,7 +119,7 @@ namespace error_location_tests {
ASSERT_SUCCESS(doc.get_array().get(arr)); ASSERT_SUCCESS(doc.get_array().get(arr));
ASSERT_ERROR(arr.count_elements(), TAPE_ERROR); ASSERT_ERROR(arr.count_elements(), TAPE_ERROR);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr - 2, "] "); ASSERT_EQUAL(std::string(ptr - 2,2), "] ");
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -136,7 +140,7 @@ namespace error_location_tests {
} }
ASSERT_EQUAL(count, 1); ASSERT_EQUAL(count, 1);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "1.23, 2] "); ASSERT_EQUAL(std::string(ptr, strlen("1.23, 2] ")), "1.23, 2] ");
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -149,7 +153,7 @@ namespace error_location_tests {
ASSERT_SUCCESS(parser.iterate(json).get(doc)); ASSERT_SUCCESS(parser.iterate(json).get(doc));
ASSERT_ERROR(doc["b"], TAPE_ERROR); ASSERT_ERROR(doc["b"], TAPE_ERROR);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "3.5, \"b\":5} "); ASSERT_EQUAL(std::string(ptr, strlen("3.5, \"b\":5} ")), "3.5, \"b\":5} ");
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -164,7 +168,7 @@ namespace error_location_tests {
ASSERT_ERROR(val, INCOMPLETE_ARRAY_OR_OBJECT); ASSERT_ERROR(val, INCOMPLETE_ARRAY_OR_OBJECT);
} }
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "[1,2,3 "); ASSERT_EQUAL(std::string(ptr, strlen("[1,2,3 ")), "[1,2,3 ");
TEST_SUCCEED(); TEST_SUCCEED();
} }
+55
View File
@@ -6,6 +6,59 @@ using namespace simdjson;
namespace error_tests { namespace error_tests {
using namespace std; using namespace std;
bool badbadjson() {
TEST_START();
ondemand::parser parser;
auto json_string = R"({
"main": "therain"_"in_spain"_
})"_padded;
ondemand::document document;
auto error = parser.iterate(json_string).get(document);
if(error != simdjson::SUCCESS) {
return false;
}
ondemand::object obj;
error = document.get_object().get(obj);
if(error != simdjson::SUCCESS) {
return false;
}
std::string_view name_value{};
error = obj["name"].get_string().get(name_value);
ASSERT_ERROR(error,TAPE_ERROR);
// Check for "main" field
std::string_view main_value{};
error = obj["main"].get_string().get(main_value);
ASSERT_ERROR(error,TAPE_ERROR);
TEST_SUCCEED();
}
bool badbadjson2() {
TEST_START();
ondemand::parser parser;
auto json_string = R"({
"main": "therain"_"in_spain"_
})"_padded;
ondemand::document document;
auto error = parser.iterate(json_string).get(document);
if(error != simdjson::SUCCESS) {
return false;
}
ondemand::object obj;
error = document.get_object().get(obj);
if(error != simdjson::SUCCESS) {
return false;
}
std::string_view main_value{};
error = obj["main"].get_string().get(main_value);
ASSERT_ERROR(error, simdjson::SUCCESS);
std::string_view name_value{};
error = obj["name"].get_string().get(name_value);
ASSERT_ERROR(error,TAPE_ERROR);
TEST_SUCCEED();
}
bool issue1834() { bool issue1834() {
TEST_START(); TEST_START();
ondemand::parser parser; ondemand::parser parser;
@@ -313,6 +366,8 @@ namespace error_tests {
bool run() { bool run() {
return return
badbadjson() &&
badbadjson2() &&
issue1834() && issue1834() &&
issue1834_2() && issue1834_2() &&
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
@@ -0,0 +1,160 @@
#include "simdjson.h"
#include "test_ondemand.h"
#include <cstdint>
using namespace simdjson;
namespace iterate_many_csv_tests {
using namespace std;
bool normal() {
TEST_START();
auto json = R"( 1, 2, 3, 4, "a", "b", "c", {"hello": "world"} , [1, 2, 3])"_padded;
ondemand::parser parser;
ondemand::document_stream doc_stream;
ASSERT_SUCCESS(parser.iterate_many(json, json.size(), true).get(doc_stream));
for (auto doc : doc_stream)
{
ASSERT_SUCCESS(doc);
}
TEST_SUCCEED();
}
bool small_batch_size() {
TEST_START();
auto json = R"( 1, 2, 3, 4, "a", "b", "c", {"hello": "world"} , [1, 2, 3])"_padded;
ondemand::parser parser;
ondemand::document_stream doc_stream;
ASSERT_SUCCESS(parser.iterate_many(json, 32, true).get(doc_stream));
for (auto doc : doc_stream)
{
ASSERT_SUCCESS(doc);
}
TEST_SUCCEED();
}
bool trailing_comma() {
TEST_START();
auto json = R"(1,)"_padded;
ondemand::parser parser;
ondemand::document_stream doc_stream;
ASSERT_SUCCESS(parser.iterate_many(json, json.size(), true).get(doc_stream));
for (auto doc : doc_stream)
{
ASSERT_SUCCESS(doc);
}
TEST_SUCCEED();
}
bool check_parsed_values() {
TEST_START();
auto json = R"( 1 , "a" , [100, 1] , {"hello" : "world"} , )"_padded;
ondemand::parser parser;
ondemand::document_stream doc_stream;
ASSERT_SUCCESS(parser.iterate_many(json, json.size(), true).get(doc_stream));
auto begin = doc_stream.begin();
auto end = doc_stream.end();
int cnt = 0;
auto it = begin;
for (; it != end && cnt < 4; ++it, ++cnt) {
auto doc = *it;
switch (cnt)
{
case 0:
{
int64_t actual;
ASSERT_SUCCESS(doc.get_int64().get(actual));
ASSERT_EQUAL(actual, 1);
break;
}
case 1:
{
std::string_view sv;
ASSERT_SUCCESS(doc.get_string().get(sv));
ASSERT_EQUAL(sv, "a");
break;
}
case 2:
{
std::vector<int64_t> expected{100, 1};
ondemand::array arr;
ASSERT_SUCCESS(doc.get_array().get(arr));
size_t element_count;
ASSERT_SUCCESS(arr.count_elements().get(element_count));
ASSERT_EQUAL(element_count, 2);
int i = 0;
for (auto a : arr)
{
int64_t actual;
ASSERT_SUCCESS(a.get(actual));
ASSERT_EQUAL(actual, expected[i++]);
}
break;
}
case 3:
{
ondemand::object obj;
ASSERT_SUCCESS(doc.get_object().get(obj));
std::string_view sv;
obj.find_field("hello").get(sv);
ASSERT_EQUAL(sv, "world");
break;
}
default:
TEST_FAIL("Too many cases")
}
}
ASSERT_EQUAL(cnt, 4);
ASSERT_TRUE(!(it != end));
TEST_SUCCEED();
}
#if SIMDJSON_EXCEPTIONS
bool leading_comma() {
TEST_START();
auto json = R"(,1)"_padded;
ondemand::parser parser;
ondemand::document_stream doc_stream;
ASSERT_SUCCESS(parser.iterate_many(json, json.size(), true).get(doc_stream));
try {
auto begin = doc_stream.begin();
auto end = doc_stream.end();
for (auto it = begin; it != end; ++it) {}
} catch (simdjson_error& e) {
ASSERT_ERROR(e.error(), TAPE_ERROR);
}
TEST_SUCCEED();
}
#endif
bool run() {
return normal() &&
small_batch_size() &&
trailing_comma() &&
check_parsed_values() &&
#if SIMDJSON_EXCEPTIONS
leading_comma() &&
#endif
true;
}
}
int main(int argc, char *argv[]) {
return test_main(argc, argv, iterate_many_csv_tests::run);
}
@@ -195,7 +195,7 @@ namespace json_pointer_tests {
ASSERT_SUCCESS(parser.iterate(cars_json).get(cars)); ASSERT_SUCCESS(parser.iterate(cars_json).get(cars));
for (int i = 0; i < 3; i++) { for (int i = 0; i < 3; i++) {
double x; double x;
std::string json_pointer = "/" + std::to_string(i) + "/tire_pressure/1"; std::string json_pointer = std::string("/") + std::to_string(i) + std::string("/tire_pressure/1");
ASSERT_SUCCESS(cars.at_pointer(json_pointer).get(x)); ASSERT_SUCCESS(cars.at_pointer(json_pointer).get(x));
measured.push_back(x); measured.push_back(x);
} }
@@ -315,7 +315,7 @@ namespace json_pointer_tests {
std::vector<car_type> content; std::vector<car_type> content;
for (int i = 0; i < 3; i++) { for (int i = 0; i < 3; i++) {
ondemand::object obj; ondemand::object obj;
std::string json_pointer = "/" + std::to_string(i); std::string json_pointer = std::string("/") + std::to_string(i);
// Each successive at_pointer call invalidates // Each successive at_pointer call invalidates
// previously parsed values, strings, objects and array. // previously parsed values, strings, objects and array.
ASSERT_SUCCESS(cars.at_pointer(json_pointer).get(obj)); ASSERT_SUCCESS(cars.at_pointer(json_pointer).get(obj));
@@ -360,7 +360,7 @@ namespace json_pointer_tests {
ondemand::document cars = parser.iterate(cars_json); ondemand::document cars = parser.iterate(cars_json);
std::vector<car_type> content; std::vector<car_type> content;
for (int i = 0; i < 3; i++) { for (int i = 0; i < 3; i++) {
std::string json_pointer = "/" + std::to_string(i); std::string json_pointer = std::string("/") + std::to_string(i);
// Each successive at_pointer call invalidates // Each successive at_pointer call invalidates
// previously parsed values, strings, objects and array. // previously parsed values, strings, objects and array.
ondemand::object obj(cars.at_pointer(json_pointer).get_object()); ondemand::object obj(cars.at_pointer(json_pointer).get_object());
@@ -0,0 +1,67 @@
#define SIMDJSON_VERBOSE_LOGGING 1
#include "simdjson.h"
#include "test_ondemand.h"
#include <iostream>
#include <string>
#include <stdlib.h>
using namespace simdjson;
namespace log_error_tests {
#if SIMDJSON_EXCEPTIONS
using namespace std;
bool tape_error()
{
TEST_START();
auto json = R"( {"a", "hello"} )"_padded;
ondemand::parser parser;
try {
ondemand::document doc = parser.iterate(json);
std::cout << doc["a"] << std::endl;
TEST_FAIL("Should have thrown an exception!")
} catch (simdjson_error& e) {
ASSERT_ERROR(e.error(), TAPE_ERROR);
}
TEST_SUCCEED();
}
bool no_such_field()
{
TEST_START();
auto json = R"( {"a": "hello"} )"_padded;
ondemand::parser parser;
try {
ondemand::document doc = parser.iterate(json);
std::cout << doc["missing_key"] << std::endl;
TEST_FAIL("Should have thrown an exception!")
} catch (simdjson_error& e) {
ASSERT_ERROR(e.error(), NO_SUCH_FIELD);
}
TEST_SUCCEED();
}
#endif // SIMDJSON_EXCEPTIONS
bool run()
{
SIMDJSON_PUSH_DISABLE_WARNINGS
SIMDJSON_DISABLE_DEPRECATED_WARNING // Disable CRT_SECURE warning on MSVC: manually verified this is safe
std::string str = "SIMDJSON_LOG_LEVEL=ERROR";
putenv(str.data());
bool rc =
#if SIMDJSON_EXCEPTIONS
tape_error() &&
no_such_field() &&
#endif // #if SIMDJSON_EXCEPTIONS
true;
SIMDJSON_POP_DISABLE_WARNINGS
return rc;
}
}
int main(int argc, char *argv[]) {
return test_main(argc, argv, log_error_tests::run);
}
+32 -7
View File
@@ -5,6 +5,29 @@ using namespace simdjson;
namespace misc_tests { namespace misc_tests {
using namespace std; using namespace std;
bool issue1981_success() {
auto error_phrase = R"(false)"_padded;
TEST_START();
ondemand::parser parser;
ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(error_phrase).get(doc));
bool b;
ASSERT_SUCCESS( doc.get_bool().get(b));
ASSERT_FALSE(b);
TEST_SUCCEED();
}
bool issue1981_failure() {
auto error_phrase = R"(falseA)"_padded;
TEST_START();
ondemand::parser parser;
ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(error_phrase).get(doc));
bool b;
ASSERT_ERROR( doc.get_bool().get(b), INCORRECT_TYPE);
TEST_SUCCEED();
}
bool replacement_char() { bool replacement_char() {
auto fun_phrase = R"( ["I \u2665 Unicode. Even broken \ud800 Unicode." ])"_padded; auto fun_phrase = R"( ["I \u2665 Unicode. Even broken \ud800 Unicode." ])"_padded;
std::string_view expected_fun = "I \xe2\x99\xa5 Unicode. Even broken \xef\xbf\xbd Unicode."; std::string_view expected_fun = "I \xe2\x99\xa5 Unicode. Even broken \xef\xbf\xbd Unicode.";
@@ -67,7 +90,7 @@ namespace misc_tests {
ASSERT_SUCCESS(val.get_object().get(obj)); ASSERT_SUCCESS(val.get_object().get(obj));
ondemand::array arr; ondemand::array arr;
ASSERT_SUCCESS(obj["a"].get_array().get(arr)); ASSERT_SUCCESS(obj["a"].get_array().get(arr));
size_t count; size_t count{};
ASSERT_SUCCESS(arr.count_elements().get(count)); ASSERT_SUCCESS(arr.count_elements().get(count));
ASSERT_EQUAL(3,count); ASSERT_EQUAL(3,count);
TEST_SUCCEED(); TEST_SUCCEED();
@@ -110,7 +133,7 @@ namespace misc_tests {
ASSERT_SUCCESS(parser.iterate(json).get(doc)); ASSERT_SUCCESS(parser.iterate(json).get(doc));
std::string_view view; std::string_view view;
ASSERT_SUCCESS( doc.get_string().get(view)); ASSERT_SUCCESS( doc.get_string().get(view));
ASSERT_EQUAL(view, u8"wow:\uFFFF"); ASSERT_EQUAL(view, "wow:\xef\xbf\xbf");
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -513,9 +536,9 @@ namespace misc_tests {
return true; return true;
} }
simdjson_warn_unused bool test_raw_json_token(string_view json, string_view expected_token, int expected_start_index = 0) { simdjson_warn_unused bool test_raw_json_token(string_view json, string_view expected_token, int expected_start_index = 0) {
string title = "'"; string title("'");
title.append(json.data(), json.length()); title.append(json.data(), json.length());
title += "'"; title += std::string("'");
padded_string json_padded = json; padded_string json_padded = json;
SUBTEST(title, test_ondemand_doc(json_padded, [&](auto doc) { SUBTEST(title, test_ondemand_doc(json_padded, [&](auto doc) {
string_view token; string_view token;
@@ -529,11 +552,11 @@ namespace misc_tests {
// Test values // Test values
auto json_in_hash = string(R"({"a":)"); auto json_in_hash = string(R"({"a":)");
json_in_hash.append(json.data(), json.length()); json_in_hash.append(json.data(), json.length());
json_in_hash += "}"; json_in_hash += std::string("}");
json_padded = json_in_hash; json_padded = json_in_hash;
title = "'"; title = std::string("'");
title.append(json_in_hash.data(), json_in_hash.length()); title.append(json_in_hash.data(), json_in_hash.length());
title += "'"; title += std::string("'");
SUBTEST(title, test_ondemand_doc(json_padded, [&](auto doc) { SUBTEST(title, test_ondemand_doc(json_padded, [&](auto doc) {
string_view token; string_view token;
ASSERT_SUCCESS( doc["a"].raw_json_token().get(token) ); ASSERT_SUCCESS( doc["a"].raw_json_token().get(token) );
@@ -570,6 +593,8 @@ namespace misc_tests {
bool run() { bool run() {
return return
issue1981_success() &&
issue1981_failure() &&
replacement_char() && replacement_char() &&
wobbly_tests() && wobbly_tests() &&
issue_uffff() && issue_uffff() &&
+20 -5
View File
@@ -232,13 +232,13 @@ namespace number_tests {
for(simdjson_result<ondemand::value> valr : arr) { for(simdjson_result<ondemand::value> valr : arr) {
ondemand::value val; ondemand::value val;
ASSERT_SUCCESS(valr.get(val)); ASSERT_SUCCESS(valr.get(val));
ondemand::number_type nt; ondemand::number_type nt{};
ASSERT_SUCCESS(val.get_number_type().get(nt)); ASSERT_SUCCESS(val.get_number_type().get(nt));
ASSERT_EQUAL(expectedtypes[counter], nt); ASSERT_EQUAL(expectedtypes[counter], nt);
ondemand::number num; ondemand::number num;
ASSERT_SUCCESS(val.get_number().get(num)); ASSERT_SUCCESS(val.get_number().get(num));
ASSERT_EQUAL(is_negative[counter], val.is_negative()); ASSERT_EQUAL(is_negative[counter], val.is_negative());
bool intvalue; bool intvalue{};
ASSERT_SUCCESS(val.is_integer().get(intvalue)); ASSERT_SUCCESS(val.is_integer().get(intvalue));
ASSERT_EQUAL(is_integer[counter], intvalue); ASSERT_EQUAL(is_integer[counter], intvalue);
ondemand::number_type t = num.get_number_type(); ondemand::number_type t = num.get_number_type();
@@ -333,15 +333,17 @@ namespace number_tests {
TEST_SUCCEED(); TEST_SUCCEED();
} }
bool get_root_number_tests() { bool get_root_number_tests() {
TEST_START(); TEST_START();
ondemand::parser parser; ondemand::parser parser;
ondemand::document doc; ondemand::document doc;
padded_string docdata; padded_string docdata;
ondemand::number number; ondemand::number number;
ondemand::number_type nt; ondemand::number_type nt{};
bool intvalue; bool intvalue{};
docdata = R"(1.0)"_padded; docdata = R"(1.0)"_padded;
ASSERT_SUCCESS(parser.iterate(docdata).get(doc)); ASSERT_SUCCESS(parser.iterate(docdata).get(doc));
@@ -389,8 +391,21 @@ namespace number_tests {
TEST_SUCCEED(); TEST_SUCCEED();
} }
bool issue2017() {
TEST_START();
ondemand::parser parser;
ondemand::document doc;
padded_string docdata = R"({"score":0.8825149536132812})"_padded;
ASSERT_SUCCESS(parser.iterate(docdata).get(doc));
double score;
ASSERT_SUCCESS(doc["score"].get_double().get(score));
ASSERT_EQUAL(score, 0.8825149536132812);
TEST_SUCCEED();
}
bool run() { bool run() {
return issue_1898() && return issue2017() &&
issue_1898() &&
issue1878() && issue1878() &&
get_root_number_tests() && get_root_number_tests() &&
get_number_tests()&& get_number_tests()&&
@@ -10,7 +10,7 @@ namespace object_error_tests {
bool assert_iterate_object(T &&object, const char **expected_key, V *expected, size_t N, simdjson::error_code *expected_error, size_t N2) { bool assert_iterate_object(T &&object, const char **expected_key, V *expected, size_t N, simdjson::error_code *expected_error, size_t N2) {
size_t count = 0; size_t count = 0;
for (auto field : object) { for (auto field : object) {
V actual; V actual{};
auto actual_error = field.value().get(actual); auto actual_error = field.value().get(actual);
if (count >= N) { if (count >= N) {
ASSERT((count - N) < N2, "Extra error reported"); ASSERT((count - N) < N2, "Extra error reported");
+82 -2
View File
@@ -6,6 +6,38 @@ using namespace simdjson;
namespace object_tests { namespace object_tests {
using namespace std; using namespace std;
using simdjson::ondemand::json_type; using simdjson::ondemand::json_type;
bool issue1979() {
TEST_START();
auto json = R"({
"@avito-core/toggles:6.1.18": {
"add_model_review_from": true
}
})"_padded;
ondemand::parser parser;
ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(json).get(doc));
ondemand::object object;
ASSERT_SUCCESS(doc.get_object().get(object));
ASSERT_SUCCESS(object["@avito-core/toggles:6.1.18"].get_object().error());
TEST_SUCCEED();
}
bool issue1977() {
TEST_START();
auto json = R"({"1": 2} foo })"_padded;
ondemand::parser parser;
ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(json).get(doc));
ondemand::object object;
ASSERT_SUCCESS(doc.get_object().get(object));
for (auto values : object) {
ASSERT_SUCCESS(values);
}
ASSERT_FALSE(doc.at_end());
TEST_SUCCEED();
}
bool issue1745() { bool issue1745() {
TEST_START(); TEST_START();
auto json = R"({ auto json = R"({
@@ -225,6 +257,25 @@ namespace object_tests {
} }
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
bool issue1965() {
TEST_START();
std::string str = "{\"query\":\"ah\"}";
std::unique_ptr<char[]> buffer(new char[str.size() + simdjson::SIMDJSON_PADDING]);
memcpy(buffer.get(), str.data(), str.size());
simdjson::padded_string_view view(buffer.get(), str.size(), str.size() + simdjson::SIMDJSON_PADDING);
simdjson::ondemand::parser parser;
simdjson::ondemand::document doc = parser.iterate(view);
simdjson::ondemand::object root = doc.get_object();
simdjson::ondemand::value query = root.find_field("query");
simdjson::ondemand::raw_json_string raw = query.get_raw_json_string();
std::unique_ptr<uint8_t[]> dst_buffer(new uint8_t[3 + simdjson::SIMDJSON_PADDING]);
uint8_t * dst = dst_buffer.get();
std::string_view fieldstring = parser.unescape(raw, dst);
std::cout << fieldstring << std::endl;
TEST_SUCCEED();
}
bool issue1745_with_exceptions() { bool issue1745_with_exceptions() {
TEST_START(); TEST_START();
auto json = R"({ auto json = R"({
@@ -1098,6 +1149,29 @@ namespace object_tests {
TEST_SUCCEED(); TEST_SUCCEED();
} }
bool issue1974a() {
TEST_START();
padded_string bad_json = R"({"key":111)"_padded;
ondemand::parser parser;
ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(bad_json).get(doc));
ondemand::object object;
ASSERT_ERROR(doc.get_object().get(object), INCOMPLETE_ARRAY_OR_OBJECT);
TEST_SUCCEED();
}
bool issue1974b() {
TEST_START();
padded_string bad_json = R"({"key":111)"_padded;
ondemand::parser parser;
ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(bad_json).get(doc));
ondemand::value val;
ASSERT_ERROR(doc.get_value().get(val), INCOMPLETE_ARRAY_OR_OBJECT);
TEST_SUCCEED();
}
bool iterate_bad_doc_object_count() { bool iterate_bad_doc_object_count() {
TEST_START(); TEST_START();
padded_string bad_jsons[4] = {R"( {"a":5 "b":3} )"_padded, R"( {"a":5, 3} )"_padded, R"( {"a":5, "b": } )"_padded, R"( {"a":5, "b":3 )"_padded}; padded_string bad_jsons[4] = {R"( {"a":5 "b":3} )"_padded, R"( {"a":5, 3} )"_padded, R"( {"a":5, "b": } )"_padded, R"( {"a":5, "b":3 )"_padded};
@@ -1107,7 +1181,7 @@ namespace object_tests {
for (auto name : names) { for (auto name : names) {
SUBTEST("ondemand::" + name, test_ondemand_doc(bad_jsons[count], [&](auto doc_result) { SUBTEST("ondemand::" + name, test_ondemand_doc(bad_jsons[count], [&](auto doc_result) {
ASSERT_RESULT( doc_result.type(), json_type::object ); ASSERT_RESULT(doc_result.type(), json_type::object );
ASSERT_ERROR(doc_result.count_fields(), errors[count]); ASSERT_ERROR(doc_result.count_fields(), errors[count]);
return true; return true;
})); }));
@@ -1220,7 +1294,13 @@ namespace object_tests {
} }
bool run() { bool run() {
return return issue1979() &&
issue1977() &&
#if SIMDJSON_EXCEPTIONS
issue1965() &&
#endif
issue1974a() &&
issue1974b() &&
issue1876a() && issue1876a() &&
issue1876() && issue1876() &&
test_strager() && test_strager() &&
+246 -150
View File
@@ -5,98 +5,166 @@ using namespace std;
using namespace simdjson; using namespace simdjson;
using error_code=simdjson::error_code; using error_code=simdjson::error_code;
bool string1() {
const char * data = "my data"; // 7 bytes
simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer
std::cout << my_padded_data << std::endl;
return true;
}
bool string2() {
std::string data = "my data";
simdjson::padded_string my_padded_data(data); // copies to a padded buffer
std::cout << my_padded_data << std::endl;
return true;
}
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
bool gen_raw1() {
TEST_START();
simdjson::ondemand::parser parser;
simdjson::padded_string docdata = R"({"value":123})"_padded;
simdjson::ondemand::document doc = parser.iterate(docdata);
simdjson::ondemand::object obj = doc.get_object();
string_view token = obj.raw_json(); // gives you `{"value":123}`
ASSERT_EQUAL(token, R"({"value":123})");
TEST_SUCCEED();
}
bool number_tests() { bool gen_raw2() {
ondemand::parser parser; TEST_START();
padded_string docdata = R"([1.0, 3, 1, 3.1415,-13231232,9999999999999999999])"_padded; simdjson::ondemand::parser parser;
ondemand::document doc = parser.iterate(docdata); simdjson::padded_string docdata = R"([1,2,3])"_padded;
ondemand::array arr = doc.get_array(); simdjson::ondemand::document doc = parser.iterate(docdata);
for(ondemand::value val : arr) { simdjson::ondemand::array arr = doc.get_array();
std::cout << val << " "; string_view token = arr.raw_json(); // gives you `[1,2,3]`
std::cout << "negative: " << val.is_negative() << " "; ASSERT_EQUAL(token, R"([1,2,3])");
std::cout << "is_integer: " << val.is_integer() << " "; TEST_SUCCEED();
ondemand::number num = val.get_number(); }
ondemand::number_type t = num.get_number_type();
// direct computation without materializing the number: bool gen_raw3() {
ondemand::number_type dt = val.get_number_type(); TEST_START();
if(t != dt) { throw std::runtime_error("bug"); } simdjson::ondemand::parser parser;
switch(t) { simdjson::padded_string docdata = R"({"value":123})"_padded;
case ondemand::number_type::signed_integer: simdjson::ondemand::document doc = parser.iterate(docdata);
std::cout << "integer: " << int64_t(num) << " "; simdjson::ondemand::object obj = doc.get_object();
std::cout << "integer: " << num.get_int64() << std::endl; string_view token = obj.raw_json(); // gives you `{"value":123}`
break; ASSERT_EQUAL(token, R"({"value":123})");
case ondemand::number_type::unsigned_integer: obj.reset(); // revise the object
std::cout << "large 64-bit integer: " << uint64_t(num) << " "; uint64_t x = obj["value"]; // gives me 123
std::cout << "large 64-bit integer: " << num.get_uint64() << std::endl; ASSERT_EQUAL(x, 123);
break;
case ondemand::number_type::floating_point_number: TEST_SUCCEED();
std::cout << "float: " << double(num) << " "; }
std::cout << "float: " << num.get_double() << std::endl;
break; bool at_end() {
} TEST_START();
} auto json = R"([1, 2] foo ])"_padded;
return true; ondemand::parser parser;
ondemand::document doc = parser.iterate(json);
ondemand::array array = doc.get_array();
for (uint64_t values : array) {
std::cout << values << std::endl;
} }
if(!doc.at_end()) {
std::cerr << "trailing content at byte index " << doc.current_location() - json.data() << std::endl;
}
TEST_SUCCEED();
}
void recursive_print_json(ondemand::value element) { bool number_tests() {
bool add_comma; TEST_START();
switch (element.type()) { ondemand::parser parser;
case ondemand::json_type::array: padded_string docdata = R"([1.0, 3, 1, 3.1415,-13231232,9999999999999999999])"_padded;
cout << "["; ondemand::document doc = parser.iterate(docdata);
add_comma = false; ondemand::array arr = doc.get_array();
for (auto child : element.get_array()) { for(ondemand::value val : arr) {
if (add_comma) { std::cout << val << " ";
cout << ","; std::cout << "negative: " << val.is_negative() << " ";
} std::cout << "is_integer: " << val.is_integer() << " ";
// We need the call to value() to get ondemand::number num = val.get_number();
// an ondemand::value type. ondemand::number_type t = num.get_number_type();
recursive_print_json(child.value()); // direct computation without materializing the number:
add_comma = true; ondemand::number_type dt = val.get_number_type();
} if(t != dt) { throw std::runtime_error("bug"); }
cout << "]"; switch(t) {
break; case ondemand::number_type::signed_integer:
case ondemand::json_type::object: std::cout << "integer: " << int64_t(num) << " ";
cout << "{"; std::cout << "integer: " << num.get_int64() << std::endl;
add_comma = false; break;
for (auto field : element.get_object()) { case ondemand::number_type::unsigned_integer:
if (add_comma) { std::cout << "large 64-bit integer: " << uint64_t(num) << " ";
cout << ","; std::cout << "large 64-bit integer: " << num.get_uint64() << std::endl;
} break;
// key() returns the key as it appears in the raw case ondemand::number_type::floating_point_number:
// JSON document, if we want the unescaped key, std::cout << "float: " << double(num) << " ";
// we should do field.unescaped_key(). std::cout << "float: " << num.get_double() << std::endl;
cout << "\"" << field.key() << "\": "; break;
recursive_print_json(field.value());
add_comma = true;
}
cout << "}\n";
break;
case ondemand::json_type::number:
// assume it fits in a double
cout << element.get_double();
break;
case ondemand::json_type::string:
// get_string() would return escaped string, but
// we are happy with unescaped string.
cout << "\"" << element.get_raw_json_string() << "\"";
break;
case ondemand::json_type::boolean:
cout << element.get_bool();
break;
case ondemand::json_type::null:
// we check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
cout << "null";
}
break;
} }
}
TEST_SUCCEED();
}
bool recursive_print_json(ondemand::value element) {
TEST_START();
bool add_comma;
switch (element.type()) {
case ondemand::json_type::array:
cout << "[";
add_comma = false;
for (auto child : element.get_array()) {
if (add_comma) {
cout << ",";
}
// We need the call to value() to get
// an ondemand::value type.
recursive_print_json(child.value());
add_comma = true;
}
cout << "]";
break;
case ondemand::json_type::object:
cout << "{";
add_comma = false;
for (auto field : element.get_object()) {
if (add_comma) {
cout << ",";
}
// key() returns the key as it appears in the raw
// JSON document, if we want the unescaped key,
// we should do field.unescaped_key().
cout << "\"" << field.key() << "\": ";
recursive_print_json(field.value());
add_comma = true;
}
cout << "}\n";
break;
case ondemand::json_type::number:
// assume it fits in a double
cout << element.get_double();
break;
case ondemand::json_type::string:
// get_string() would return escaped string, but
// we are happy with unescaped string.
cout << "\"" << element.get_raw_json_string() << "\"";
break;
case ondemand::json_type::boolean:
cout << element.get_bool();
break;
case ondemand::json_type::null:
// we check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
cout << "null";
}
break;
}
TEST_SUCCEED();
} }
bool basics_treewalk() { bool basics_treewalk() {
TEST_START();
padded_string json[3] = {R"( [ padded_string json[3] = {R"( [
{ "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] }, { "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
{ "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] }, { "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
@@ -109,77 +177,82 @@ bool basics_treewalk() {
recursive_print_json(val); recursive_print_json(val);
std::cout << std::endl; std::cout << std::endl;
} }
return true; TEST_SUCCEED();
} }
void print_depth_space(ondemand::value element) { bool print_depth_space(ondemand::value element) {
TEST_START();
for(auto i = 0; i < element.current_depth(); i++) { for(auto i = 0; i < element.current_depth(); i++) {
cout << " "; cout << " ";
} }
TEST_SUCCEED();
} }
void recursive_print_json_breakline(ondemand::value element) { bool recursive_print_json_breakline(ondemand::value element) {
bool add_comma; TEST_START();
switch (element.type()) { bool add_comma;
case ondemand::json_type::array: switch (element.type()) {
cout << endl; case ondemand::json_type::array:
print_depth_space(element); cout << endl;
cout << "["; print_depth_space(element);
add_comma = false; cout << "[";
for (auto child : element.get_array()) { add_comma = false;
if (add_comma) { for (auto child : element.get_array()) {
print_depth_space(element); if (add_comma) {
cout << ","; print_depth_space(element);
} cout << ",";
// We need the call to value() to get
// an ondemand::value type.
recursive_print_json_breakline(child.value());
add_comma = true;
} }
cout << "]"; // We need the call to value() to get
break; // an ondemand::value type.
case ondemand::json_type::object: recursive_print_json_breakline(child.value());
cout << endl; add_comma = true;
print_depth_space(element);
cout << "{";
add_comma = false;
for (auto field : element.get_object()) {
if (add_comma) {
print_depth_space(element);
cout << ",";
}
// key() returns the key as it appears in the raw
// JSON document, if we want the unescaped key,
// we should do field.unescaped_key().
cout << "\"" << field.key() << "\": ";
recursive_print_json_breakline(field.value());
add_comma = true;
}
cout << "}\n";
break;
case ondemand::json_type::number:
// assume it fits in a double
cout << element.get_double();
break;
case ondemand::json_type::string:
// get_string() would return escaped string, but
// we are happy with unescaped string.
cout << "\"" << element.get_raw_json_string() << "\"";
break;
case ondemand::json_type::boolean:
cout << element.get_bool();
break;
case ondemand::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
cout << "null";
}
break;
} }
cout << "]";
break;
case ondemand::json_type::object:
cout << endl;
print_depth_space(element);
cout << "{";
add_comma = false;
for (auto field : element.get_object()) {
if (add_comma) {
print_depth_space(element);
cout << ",";
}
// key() returns the key as it appears in the raw
// JSON document, if we want the unescaped key,
// we should do field.unescaped_key().
cout << "\"" << field.key() << "\": ";
recursive_print_json_breakline(field.value());
add_comma = true;
}
cout << "}\n";
break;
case ondemand::json_type::number:
// assume it fits in a double
cout << element.get_double();
break;
case ondemand::json_type::string:
// get_string() would return escaped string, but
// we are happy with unescaped string.
cout << "\"" << element.get_raw_json_string() << "\"";
break;
case ondemand::json_type::boolean:
cout << element.get_bool();
break;
case ondemand::json_type::null:
// We check that the value is indeed null
// otherwise: an error is thrown.
if(element.is_null()) {
cout << "null";
}
break;
}
TEST_SUCCEED();
} }
bool basics_treewalk_breakline() { bool basics_treewalk_breakline() {
TEST_START();
padded_string json[3] = {R"( [ padded_string json[3] = {R"( [
{ "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] }, { "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
{ "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] }, { "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
@@ -192,7 +265,7 @@ bool basics_treewalk_breakline() {
recursive_print_json_breakline(val); recursive_print_json_breakline(val);
std::cout << std::endl; std::cout << std::endl;
} }
return true; TEST_SUCCEED();
} }
bool basics_1() { bool basics_1() {
@@ -576,7 +649,7 @@ bool using_the_parsed_json_no_exceptions() {
cout << "Make/Model: " << make << "/" << model << endl; cout << "Make/Model: " << make << "/" << model << endl;
// Casting a JSON element to an integer // Casting a JSON element to an integer
uint64_t year; uint64_t year{};
error = car["year"].get(year); error = car["year"].get(year);
if(error) { std::cerr << error << std::endl; return false; } if(error) { std::cerr << error << std::endl; return false; }
cout << "- This car is " << 2020 - year << " years old." << endl; cout << "- This car is " << 2020 - year << " years old." << endl;
@@ -695,7 +768,7 @@ bool json_pointer_multiple() {
ASSERT_SUCCESS(cars.count_elements().get(size)); ASSERT_SUCCESS(cars.count_elements().get(size));
double expected[] = {39.9, 31, 30}; double expected[] = {39.9, 31, 30};
for (size_t i = 0; i < size; i++) { for (size_t i = 0; i < size; i++) {
std::string json_pointer = "/" + std::to_string(i) + "/tire_pressure/1"; std::string json_pointer = std::string("/") + std::to_string(i) + std::string("/tire_pressure/1");
double x; double x;
ASSERT_SUCCESS(cars.at_pointer(json_pointer).get(x)); ASSERT_SUCCESS(cars.at_pointer(json_pointer).get(x));
ASSERT_EQUAL(x,expected[i]); ASSERT_EQUAL(x,expected[i]);
@@ -778,9 +851,9 @@ bool ndjson_basics_example() {
size_t count{0}; size_t count{0};
int64_t expected[3] = {1,2,3}; int64_t expected[3] = {1,2,3};
for (auto doc : docs) { for (auto doc : docs) {
int64_t actual; int64_t actual{};
ASSERT_SUCCESS( doc["foo"].get(actual) ); ASSERT_SUCCESS( doc["foo"].get(actual) );
ASSERT_EQUAL( actual,expected[count++] ); ASSERT_EQUAL( actual, expected[count++] );
} }
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -948,6 +1021,22 @@ int load_example_except() {
std::cout << identifier << std::endl; std::cout << identifier << std::endl;
return EXIT_SUCCESS; return EXIT_SUCCESS;
} }
int load_example_except_morecomplete(void) {
TEST_START();
simdjson::ondemand::parser parser;
simdjson::padded_string json_string;
simdjson::ondemand::document doc;
try {
json_string = padded_string::load("twitter.json");
doc = parser.iterate(json_string);
uint64_t identifier = doc["statuses"].at(0)["id"];
std::cout << identifier << std::endl;
} catch (simdjson::simdjson_error &error) {
std::cerr << "JSON error: " << error.what() << " near "
<< doc.current_location() << " in " << json_string << std::endl;
}
return EXIT_SUCCESS;
}
#endif #endif
bool test_load_example() { bool test_load_example() {
TEST_START(); TEST_START();
@@ -973,10 +1062,12 @@ bool current_location_tape_error() {
int64_t i; int64_t i;
ASSERT_ERROR(doc["integer"].get_int64().get(i), TAPE_ERROR); ASSERT_ERROR(doc["integer"].get_int64().get(i), TAPE_ERROR);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "false, \"integer\": -343} "); std::string expected = "false, \"integer\": -343} ";
ASSERT_EQUAL(std::string(ptr,expected.size()), expected);
TEST_SUCCEED(); TEST_SUCCEED();
} }
bool current_location_user_error() { bool current_location_user_error() {
TEST_START(); TEST_START();
auto json = R"( [1,2,3] )"_padded; auto json = R"( [1,2,3] )"_padded;
@@ -987,7 +1078,8 @@ bool current_location_user_error() {
int64_t i; int64_t i;
ASSERT_ERROR(doc["integer"].get_int64().get(i), INCORRECT_TYPE); ASSERT_ERROR(doc["integer"].get_int64().get(i), INCORRECT_TYPE);
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "[1,2,3] "); std::string expected = "[1,2,3] ";
ASSERT_EQUAL(std::string(ptr, expected.size()), expected);
TEST_SUCCEED(); TEST_SUCCEED();
} }
@@ -1021,7 +1113,8 @@ bool current_location_no_error() {
auto error = val.get_object().get(obj); auto error = val.get_object().get(obj);
if (!error) { if (!error) {
ASSERT_SUCCESS(doc.current_location().get(ptr)); ASSERT_SUCCESS(doc.current_location().get(ptr));
ASSERT_EQUAL(ptr, "\"key\": \"value\"}, true] "); std::string expected = "\"key\": \"value\"}, true] ";
ASSERT_EQUAL(std::string(ptr, expected.size()), expected);
} }
} }
TEST_SUCCEED(); TEST_SUCCEED();
@@ -1040,6 +1133,7 @@ struct ZuluBBox {
<< std::endl; << std::endl;
} }
}; };
#if SIMDJSON_EXCEPTIONS
bool example1956() { bool example1956() {
@@ -1200,10 +1294,12 @@ bool example1958() {
} }
return true; return true;
} }
#endif
bool run() { bool run() {
return true return true
#if SIMDJSON_EXCEPTIONS #if SIMDJSON_EXCEPTIONS
&& gen_raw1() && gen_raw2() && gen_raw3()
&& at_end()
&& example1956() && example1958() && example1956() && example1958()
// && basics_1() // Fails because twitter.json isn't in current directory. Compile test only. // && basics_1() // Fails because twitter.json isn't in current directory. Compile test only.
&& basics_treewalk() && basics_treewalk()
+8 -8
View File
@@ -18,7 +18,7 @@ namespace scalar_tests {
bool test_scalar_value(const padded_string &json, const T &expected, bool test_twice=true) { bool test_scalar_value(const padded_string &json, const T &expected, bool test_twice=true) {
std::cout << "- JSON: " << json << endl; std::cout << "- JSON: " << json << endl;
SUBTEST( "simdjson_result<document>", test_ondemand_doc(json, [&](auto doc_result) { SUBTEST( "simdjson_result<document>", test_ondemand_doc(json, [&](auto doc_result) {
T actual; T actual{};
ASSERT_RESULT( doc_result.type(), expected_json_type<T>() ); ASSERT_RESULT( doc_result.type(), expected_json_type<T>() );
ASSERT_SUCCESS( doc_result.get(actual) ); ASSERT_SUCCESS( doc_result.get(actual) );
ASSERT_EQUAL( actual, expected ); ASSERT_EQUAL( actual, expected );
@@ -33,7 +33,7 @@ namespace scalar_tests {
SUBTEST( "document", test_ondemand_doc(json, [&](auto doc_result) { SUBTEST( "document", test_ondemand_doc(json, [&](auto doc_result) {
ondemand::document doc; ondemand::document doc;
ASSERT_SUCCESS( std::move(doc_result).get(doc) ); ASSERT_SUCCESS( std::move(doc_result).get(doc) );
T actual; T actual{};
ASSERT_RESULT( doc.type(), expected_json_type<T>() ); ASSERT_RESULT( doc.type(), expected_json_type<T>() );
ASSERT_SUCCESS( doc.get(actual) ); ASSERT_SUCCESS( doc.get(actual) );
ASSERT_EQUAL( actual, expected ); ASSERT_EQUAL( actual, expected );
@@ -50,7 +50,7 @@ namespace scalar_tests {
padded_string whitespace_json = std::string(json) + " "; padded_string whitespace_json = std::string(json) + " ";
std::cout << "- JSON: " << whitespace_json << endl; std::cout << "- JSON: " << whitespace_json << endl;
SUBTEST( "simdjson_result<document>", test_ondemand_doc(whitespace_json, [&](auto doc_result) { SUBTEST( "simdjson_result<document>", test_ondemand_doc(whitespace_json, [&](auto doc_result) {
T actual; T actual{};
ASSERT_RESULT( doc_result.type(), expected_json_type<T>() ); ASSERT_RESULT( doc_result.type(), expected_json_type<T>() );
ASSERT_SUCCESS( doc_result.get(actual) ); ASSERT_SUCCESS( doc_result.get(actual) );
ASSERT_EQUAL( actual, expected ); ASSERT_EQUAL( actual, expected );
@@ -65,7 +65,7 @@ namespace scalar_tests {
SUBTEST( "document", test_ondemand_doc(whitespace_json, [&](auto doc_result) { SUBTEST( "document", test_ondemand_doc(whitespace_json, [&](auto doc_result) {
ondemand::document doc; ondemand::document doc;
ASSERT_SUCCESS( std::move(doc_result).get(doc) ); ASSERT_SUCCESS( std::move(doc_result).get(doc) );
T actual; T actual{};
ASSERT_RESULT( doc.type(), expected_json_type<T>() ); ASSERT_RESULT( doc.type(), expected_json_type<T>() );
ASSERT_SUCCESS( doc.get(actual) ); ASSERT_SUCCESS( doc.get(actual) );
ASSERT_EQUAL( actual, expected ); ASSERT_EQUAL( actual, expected );
@@ -85,7 +85,7 @@ namespace scalar_tests {
SUBTEST( "simdjson_result<value>", test_ondemand_doc(array_json, [&](auto doc_result) { SUBTEST( "simdjson_result<value>", test_ondemand_doc(array_json, [&](auto doc_result) {
int count = 0; int count = 0;
for (simdjson_result<ondemand::value> val_result : doc_result) { for (simdjson_result<ondemand::value> val_result : doc_result) {
T actual; T actual{};
ASSERT_RESULT( val_result.type(), expected_json_type<T>() ); ASSERT_RESULT( val_result.type(), expected_json_type<T>() );
ASSERT_SUCCESS( val_result.get(actual) ); ASSERT_SUCCESS( val_result.get(actual) );
ASSERT_EQUAL(actual, expected); ASSERT_EQUAL(actual, expected);
@@ -105,7 +105,7 @@ namespace scalar_tests {
for (simdjson_result<ondemand::value> val_result : doc_result) { for (simdjson_result<ondemand::value> val_result : doc_result) {
ondemand::value val; ondemand::value val;
ASSERT_SUCCESS( val_result.get(val) ); ASSERT_SUCCESS( val_result.get(val) );
T actual; T actual{};
ASSERT_RESULT( val.type(), expected_json_type<T>() ); ASSERT_RESULT( val.type(), expected_json_type<T>() );
ASSERT_SUCCESS( val.get(actual) ); ASSERT_SUCCESS( val.get(actual) );
ASSERT_EQUAL(actual, expected); ASSERT_EQUAL(actual, expected);
@@ -129,7 +129,7 @@ namespace scalar_tests {
SUBTEST( "simdjson_result<value>", test_ondemand_doc(whitespace_array_json, [&](auto doc_result) { SUBTEST( "simdjson_result<value>", test_ondemand_doc(whitespace_array_json, [&](auto doc_result) {
int count = 0; int count = 0;
for (simdjson_result<ondemand::value> val_result : doc_result) { for (simdjson_result<ondemand::value> val_result : doc_result) {
T actual; T actual{};
ASSERT_RESULT( val_result.type(), expected_json_type<T>() ); ASSERT_RESULT( val_result.type(), expected_json_type<T>() );
ASSERT_SUCCESS( val_result.get(actual) ); ASSERT_SUCCESS( val_result.get(actual) );
ASSERT_EQUAL(actual, expected); ASSERT_EQUAL(actual, expected);
@@ -150,7 +150,7 @@ namespace scalar_tests {
for (simdjson_result<ondemand::value> val_result : doc_result) { for (simdjson_result<ondemand::value> val_result : doc_result) {
ondemand::value val; ondemand::value val;
ASSERT_SUCCESS( val_result.get(val) ); ASSERT_SUCCESS( val_result.get(val) );
T actual; T actual{};
ASSERT_RESULT( val.type(), expected_json_type<T>() ); ASSERT_RESULT( val.type(), expected_json_type<T>() );
ASSERT_SUCCESS( val.get(actual) ); ASSERT_SUCCESS( val.get(actual) );
ASSERT_EQUAL(actual, expected); ASSERT_EQUAL(actual, expected);
@@ -1,5 +1,4 @@
#include <cinttypes> #include <cinttypes>
#include <ciso646>
#include <cmath> #include <cmath>
#include <cstdio> #include <cstdio>
#include <cstdlib> #include <cstdlib>
+1 -1
View File
@@ -56,7 +56,7 @@ namespace twitter_tests {
std::string_view screen_name; std::string_view screen_name;
ASSERT_SUCCESS( user["screen_name"].get(screen_name) ); ASSERT_SUCCESS( user["screen_name"].get(screen_name) );
bool default_profile; bool default_profile{};
ASSERT_SUCCESS( user["default_profile"].get(default_profile) ); ASSERT_SUCCESS( user["default_profile"].get(default_profile) );
if (default_profile) { if (default_profile) {
default_users.insert(screen_name); default_users.insert(screen_name);
+1 -1
View File
@@ -10,7 +10,7 @@
template<typename T, typename F> template<typename T, typename F>
bool test_ondemand(simdjson::ondemand::parser &parser, const simdjson::padded_string &json, const F& f) { bool test_ondemand(simdjson::ondemand::parser &parser, const simdjson::padded_string &json, const F& f) {
auto doc = parser.iterate(json); auto doc = parser.iterate(json);
T val; T val{};
ASSERT_SUCCESS( doc.get(val) ); ASSERT_SUCCESS( doc.get(val) );
return f(val); return f(val);
} }
+1
View File
@@ -114,6 +114,7 @@ simdjson_inline bool assert_iterate_error(T &arr, simdjson::error_code expected,
#define ASSERT_EQUAL(ACTUAL, EXPECTED) do { if (!::assert_equal ((ACTUAL), (EXPECTED), #ACTUAL)) { return false; } } while (0); #define ASSERT_EQUAL(ACTUAL, EXPECTED) do { if (!::assert_equal ((ACTUAL), (EXPECTED), #ACTUAL)) { return false; } } while (0);
#define ASSERT_RESULT(ACTUAL, EXPECTED) do { if (!::assert_result ((ACTUAL), (EXPECTED), #ACTUAL)) { return false; } } while (0); #define ASSERT_RESULT(ACTUAL, EXPECTED) do { if (!::assert_result ((ACTUAL), (EXPECTED), #ACTUAL)) { return false; } } while (0);
#define ASSERT_SUCCESS(ACTUAL) do { if (!::assert_success((ACTUAL), #ACTUAL)) { return false; } } while (0); #define ASSERT_SUCCESS(ACTUAL) do { if (!::assert_success((ACTUAL), #ACTUAL)) { return false; } } while (0);
#define ASSERT_FAILURE(ACTUAL) do { if (::assert_success((ACTUAL), #ACTUAL)) { return false; } } while (0);
#define ASSERT_ERROR(ACTUAL, EXPECTED) do { if (!::assert_error ((ACTUAL), (EXPECTED), #ACTUAL)) { return false; } } while (0); #define ASSERT_ERROR(ACTUAL, EXPECTED) do { if (!::assert_error ((ACTUAL), (EXPECTED), #ACTUAL)) { return false; } } while (0);
#define ASSERT_TRUE(ACTUAL) do { if (!::assert_true ((ACTUAL), #ACTUAL)) { return false; } } while (0); #define ASSERT_TRUE(ACTUAL) do { if (!::assert_true ((ACTUAL), #ACTUAL)) { return false; } } while (0);
#define ASSERT_FALSE(ACTUAL) do { if (!::assert_false ((ACTUAL), #ACTUAL)) { return false; } } while (0); #define ASSERT_FALSE(ACTUAL) do { if (!::assert_false ((ACTUAL), #ACTUAL)) { return false; } } while (0);
+2 -2
View File
@@ -123,7 +123,7 @@ void recurse(simdjson::dom::element element, stat_t &s, size_t depth) {
if (element.is<int64_t>()) { if (element.is<int64_t>()) {
s.integer_count++; // because an int can be sometimes represented as a double, we s.integer_count++; // because an int can be sometimes represented as a double, we
// to check whether it is an integer first!!! // to check whether it is an integer first!!!
int64_t v; int64_t v{};
error = element.get(v); error = element.get(v);
SIMDJSON_ASSUME(!error); SIMDJSON_ASSUME(!error);
if((v >= std::numeric_limits<int32_t>::min()) and (v <= std::numeric_limits<int32_t>::max()) ) { if((v >= std::numeric_limits<int32_t>::min()) and (v <= std::numeric_limits<int32_t>::max()) ) {
@@ -138,7 +138,7 @@ void recurse(simdjson::dom::element element, stat_t &s, size_t depth) {
} else if (element.is<double>()) { } else if (element.is<double>()) {
s.float_count++; s.float_count++;
} else if (element.is<bool>()) { } else if (element.is<bool>()) {
bool v; bool v{};
error = element.get(v); error = element.get(v);
SIMDJSON_ASSUME(!error); SIMDJSON_ASSUME(!error);
if (v) { if (v) {
+1 -10
View File
@@ -80,15 +80,6 @@ else :
atleastminor= (currentv[0] != newversion[0]) or (currentv[1] != newversion[1]) atleastminor= (currentv[0] != newversion[0]) or (currentv[1] != newversion[1])
if(atleastminor):
print(colored(0, 255, 0, "This is more than a revision."))
releasefile = maindir + os.sep + "RELEASES.md"
releasedata = open(releasefile).read()
pattern = re.compile("#\s+\d+\.\d+")
m = pattern.search(releasedata)
if(m == None):
print(colored(255, 0, 0, "You are preparing a new minor release and you have not yet updated RELEASES.md."))
sys.exit(-1)
versionfilerel = os.sep + "include" + os.sep + "simdjson" + os.sep + "simdjson_version.h" versionfilerel = os.sep + "include" + os.sep + "simdjson" + os.sep + "simdjson_version.h"
versionfile = maindir + versionfilerel versionfile = maindir + versionfilerel
@@ -177,7 +168,7 @@ readmefile = maindir + os.sep + "README.md"
readmedata = open(readmefile).read() readmedata = open(readmefile).read()
m = pattern.search(readmedata) m = pattern.search(readmedata)
if m == None: if m == None:
print(colored(255, 0, 0, 'I cannot find a link to the API documentation in your README?????')) print('I cannot find a link to the API documentation in your README')
else: else:
detectedreadme = m.group(1) detectedreadme = m.group(1)
print("found a link to your API documentation in the README file: "+detectedreadme+" ("+toversionstring(*newversion)+")") print("found a link to your API documentation in the README file: "+detectedreadme+" ("+toversionstring(*newversion)+")")