New release

* Add comma separated value parsing

* Fix failing tests

* Make tests work for exceptions

* Fix test

* Fix try catch making test fail

---------

Co-authored-by: Yong Xiang Ng <yxng@drwholdings.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -12,24 +12,26 @@ Before submitting an issue, please ensure that you have read the documentation:
 * Basics is an overview of how to use simdjson and its APIs: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
 * Performance shows some more advanced scenarios and how to tune for them: https://github.com/simdjson/simdjson/blob/master/doc/performance.md
 * Contributing: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md
-* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017).
+* We follow the [JSON specification as described by RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.txt) (T. Bray, 2017). If you wish to support features that are not part of RFC 8259, then you should not refer to your issue as a bug.
 
 
 **Describe the bug**
-A clear and concise description of what the bug is.
+A clear and concise description of what the bug is. A bug is a failure to build with normal compiler settings or a misbehaviour: when running the code, you get a result that differs from the expected result from our documentation.
 
-A compiler or static-analyzer warning is not a bug.
+A compiler or static-analyzer warning is not a bug. It is possible with tools such as Visual Studio to require that rarely enabled warnings are considered errors. Do not report such cases as bugs. We do accept pull requests if you want to silence warnings issued by code analyzers, however.
 
 We are committed to providing good documentation. We accept the lack of documentation or a misleading documentation as a bug (a 'documentation bug').
 
+An unexpected poor software performance can be accepted as a bug (a 'performance bug').
+
 We accept the identification of an issue by a sanitizer or some checker tool (e.g., valgrind) as a bug, but you must first ensure that it is not a false positive.
 
-We recommend that you run your tests using different optimization levels.
+We recommend that you run your tests using different optimization levels. In particular, we recommend your run tests with the simdjson library and you code compiled in debug mode. The simdjson then sets the SIMDJSON_DEVELOPMENT_CHECKS macro to 1, and this triggers additional checks on your code and on the internals of the library. If possible, we recommend that you run tests with sanitizers (e.g., see [No more leaks with sanitize flags in gcc and clang](https://lemire.me/blog/2016/04/20/no-more-leaks-with-sanitize-flags-in-gcc-and-clang/)).  You can compile the library with sanitizers for debugging purposes (e.g., set SIMDJSON_SANITIZE to ON using CMake), but you should also turn on sanitizers on your own code. You may also use tools like valgrind or the commercial equivalent.
 
 Before reporting a bug, please ensure that you have read our documentation.
 
 **To Reproduce**
-Steps to reproduce the behaviour: provide a code sample if possible.
+Steps to reproduce the behaviour: provide a code sample if possible. Please provide a complete test with data. Remember that a bug is either a failure to build or an unexpected result when running the code.
 
 If we cannot reproduce the issue, then we cannot address it. Note that a stack trace from your own program is not enough. A sample of your source code is insufficient: please provide a complete test for us to reproduce the issue. Please reduce the issue: use as small and as simple an example of the bug as possible.
 
@@ -51,6 +53,6 @@ We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux syste
 
 **Indicate whether you are willing or able to provide a bug fix as a pull request**
 
-If you plan to contribute to simdjson, please read our
+If you plan to contribute to simdjson, please read our guide:
 * CONTRIBUTING guide: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md and our
 * HACKING guide: https://github.com/simdjson/simdjson/blob/master/HACKING.md

.github/workflows/documentation.yml

@@ -0,0 +1,34 @@
+name: Doxygen GitHub Pages
+
+on:
+  push:
+    branches:
+      - master
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pages: write
+  id-token: write
+
+jobs:
+  deploy:
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Install Doxygen
+        run: sudo apt-get install doxygen graphviz -y
+      - run: mkdir docs
+      - name: Install theme
+        run: ./tools/prepare_doxygen.sh
+      - name: Generate Doxygen Documentation
+        run: doxygen
+      - name: Deploy to GitHub Pages
+        uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: doc/api/html

.github/workflows/fuzzers.yml

@@ -24,7 +24,7 @@ jobs:
       implementations: haswell westmere fallback
       UBSAN_OPTIONS: halt_on_error=1
       MAXLEN: -max_len=4000
-      CLANGVERSION: 11
+      CLANGVERSION: 15
       # which optimization level to use for the sanitizer build (see build_fuzzer.variants.sh)
       OPTLEVEL: -O3
 
@@ -32,7 +32,7 @@ jobs:
     - name: Install packages necessary for building
       run: |
         sudo apt update
-        sudo apt-get install --quiet ninja-build valgrind zip unzip
+        sudo apt-get install --quiet ninja-build valgrind zip unzip lsb-release wget software-properties-common gnupg
         wget https://apt.llvm.org/llvm.sh
         chmod +x llvm.sh
         sudo ./llvm.sh $CLANGVERSION
@@ -50,12 +50,12 @@ jobs:
         path: out/
         key: corpus-${{ github.run_id }}
         restore-keys: corpus-
-    
+
     - name: show statistics for the cached corpus
       run: |
         echo number of files in github action corpus cache:
         find out -type f |wc -l
-    
+
     - name: Create and prepare the initial seed corpus
       run: |
         fuzz/build_corpus.sh
@@ -92,7 +92,7 @@ jobs:
             export SIMDJSON_FORCE_IMPLEMENTATION=$implementation
             build-sanitizers$OPTLEVEL/fuzz/fuzz_$fuzzer out/$fuzzer $others seedcorpus -max_total_time=20 $MAXLEN
           done
-          echo now have $(ls out/$fuzzer |wc -l) files in corpus        
+          echo now have $(ls out/$fuzzer |wc -l) files in corpus
         done
 
     - name: Fuzz differential impl. fuzzers with sanitizer+asserts (good at detecting errors)
@@ -119,7 +119,7 @@ jobs:
 
     - name: Package the corpus into an artifact
       run: |
-        for fuzzer in $defaultimplfuzzers $implfuzzers; do      
+        for fuzzer in $defaultimplfuzzers $implfuzzers; do
           tar rf corpus.tar out/$fuzzer
         done
 
@@ -141,7 +141,7 @@ jobs:
       run: |
         for fuzzer in $defaultimplfuzzers $implfuzzers; do
           find out/$fuzzer -type f |sort|head -n200|xargs -n40 valgrind build-replay/fuzz/fuzz_$fuzzer 2>&1|tee valgrind-$fuzzer.txt
-        done 
+        done
 
     - name: Compress the valgrind output
       run: tar cf valgrind.tar valgrind-*.txt

.github/workflows/ppc64.yml

@@ -0,0 +1,29 @@
+name: Ubuntu ppc64le (GCC 11)
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: uraimo/run-on-arch-action@v2
+        name: Test
+        id: runcmd
+        with:
+          arch: ppc64le
+          distro: ubuntu_latest
+          githubToken: ${{ github.token }}
+          install: |
+            apt-get update -q -y
+            apt-get install -y cmake make g++
+          run: |
+            cmake -DCMAKE_BUILD_TYPE=Release -B build
+            cmake --build build -j=2
+            ctest --output-on-failure --test-dir build

.github/workflows/s390x.yml

@@ -0,0 +1,29 @@
+name: Ubuntu s390x (GCC 11)
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: uraimo/run-on-arch-action@v2
+        name: Test
+        id: runcmd
+        with:
+          arch: s390x
+          distro: ubuntu_latest
+          githubToken: ${{ github.token }}
+          install: |
+            apt-get update -q -y
+            apt-get install -y cmake make g++
+          run: |
+            cmake -DCMAKE_BUILD_TYPE=Release -B build
+            cmake --build build -j=2
+            ctest --output-on-failure --test-dir build

.github/workflows/ubuntu18-checkperf.yml

@@ -1,29 +0,0 @@
-name: Performance check on Ubuntu 18.04 CI (GCC 7)
-
-on:
-  push:
-    branches:
-      - master
-  pull_request:
-    branches:
-      - master
-
-jobs:
-  ubuntu-build:
-    if: >-
-      ! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
-      ! contains(toJSON(github.event.commits.*.message), '[skip github]')
-    runs-on: ubuntu-18.04
-    steps:
-      - uses: actions/checkout@v3
-      - uses: actions/cache@v3
-        with:
-          path: dependencies/.cache
-          key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
-      - name: Use cmake
-        run: |
-          mkdir build &&
-          cd build &&
-          cmake  -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_ENABLE_DOM_CHECKPERF=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination ..  &&
-          cmake --build . --target checkperf  &&
-          ctest --output-on-failure -R checkperf  ubuntu18-checkperf.yml

.github/workflows/ubuntu18.yml

@@ -1,25 +0,0 @@
-name: Ubuntu 18.04 CI (GCC 7)
-
-on: [push, pull_request]
-
-jobs:
-  ubuntu-build:
-    if: >-
-      ! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
-      ! contains(toJSON(github.event.commits.*.message), '[skip github]')
-    runs-on: ubuntu-18.04
-    steps:
-      - uses: actions/checkout@v3
-      - uses: actions/cache@v3
-        with:
-          path: dependencies/.cache
-          key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
-      - name: Use cmake
-        run: |
-          mkdir build &&
-          cd build &&
-          cmake  -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination ..  &&
-          cmake --build .   &&
-          ctest -j --output-on-failure -LE explicitonly   &&
-          make install  &&
-          echo -e '#include <simdjson.h>\nint main(int argc,char**argv) {simdjson::dom::parser parser;simdjson::dom::element tweets = parser.load(argv[1]); }' > tmp.cpp && c++ -Idestination/include -Ldestination/lib -std=c++17 -Wl,-rpath,destination/lib -o linkandrun tmp.cpp -lsimdjson && ./linkandrun jsonexamples/twitter.json

.github/workflows/ubuntu20-sani.yml

@@ -14,10 +14,17 @@ jobs:
         with:
           path: dependencies/.cache
           key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
-      - name: Use cmake
+      - name: Use cmake with address sanitizer
         run: |
           mkdir builddebug &&
           cd builddebug &&
           cmake -DSIMDJSON_SANITIZE=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF ..  &&
           cmake --build .   &&
-          ctest -j --output-on-failure -LE explicitonly
+          ctest -j --output-on-failure -LE explicitonly
+      - name: Use cmake with undefined sanitizer
+        run: |
+          mkdir builddebugundefsani &&
+          cd builddebugundefsani &&
+          cmake -DSIMDJSON_SANITIZE_UNDEFINED=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF ..  &&
+          cmake --build .   &&
+          ctest -j --output-on-failure -LE explicitonly

.github/workflows/ubuntu22-clang14.yml

@@ -1,4 +1,4 @@
-name: Ubuntu 18.04 CI (LLVM 7)
+name: Ubuntu 22.04 CI (CLANG 14)
 
 on: [push, pull_request]
 
@@ -7,25 +7,19 @@ jobs:
     if: >-
       ! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
       ! contains(toJSON(github.event.commits.*.message), '[skip github]')
-    runs-on: ubuntu-18.04
-
-    env:
-      CC: clang-7
-      CXX: clang++-7
+    runs-on: ubuntu-22.04
     steps:
       - uses: actions/checkout@v3
       - uses: actions/cache@v3
         with:
           path: dependencies/.cache
           key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
-      - name: install clang 7
-        run: |
-          sudo apt update
-          sudo apt install clang-7
+      - name: Install clang++-14
+        run: sudo apt-get install -y clang++-14
       - name: Use cmake
         run: |
           mkdir build &&
           cd build &&
-          cmake  -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination ..  &&
+          CXX=clang++-14 cmake -DSIMDJSON_DEVELOPER_MODE=ON ..  &&
           cmake --build .   &&
           ctest -j --output-on-failure -LE explicitonly

.github/workflows/ubuntu22-cxx20.yml

@@ -1,31 +1,30 @@
-name: Macos (Xcode 11)
+name: Ubuntu 22.04 CI (GCC 12, CXX 20)
 
 on: [push, pull_request]
 
 jobs:
-  macos-build:
+  ubuntu-build:
     if: >-
       ! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
       ! contains(toJSON(github.event.commits.*.message), '[skip github]')
-    runs-on: macos-latest
+    runs-on: ubuntu-22.04
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/cache@v3
+      - uses: actions/checkout@v2
+      - uses: actions/cache@v2
         with:
           path: dependencies/.cache
           key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
       - name: Use cmake
         run: |
-          xcversion select 11.7
           mkdir builddebug &&
           cd builddebug &&
-          cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF ..  &&
+          CXX=g++-12 cmake -DSIMDJSON_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF ..  &&
           cmake --build .   &&
           ctest -j --output-on-failure -LE explicitonly   &&
           cd .. &&
           mkdir build &&
           cd build &&
-          cmake  -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination ..  &&
+          CXX=g++-12 cmake -DSIMDJSON_CXX_STANDARD=20  -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination ..  &&
           cmake --build .   &&
           ctest -j --output-on-failure -LE explicitonly   &&
           cmake --install . &&

.github/workflows/ubuntu22-glibcxxassertions.yml

@@ -1,25 +1,24 @@
-name: Ubuntu 18.04 CI (GCC 7) with Thread Sanitizer
-
+name: Ubuntu 22.04 CI GCC 12 with GLIBCXX_ASSERTIONS
 on: [push, pull_request]
 
-
 jobs:
   ubuntu-build:
     if: >-
       ! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
       ! contains(toJSON(github.event.commits.*.message), '[skip github]')
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-22.04
     steps:
       - uses: actions/checkout@v3
       - uses: actions/cache@v3
         with:
           path: dependencies/.cache
           key: ${{ hashFiles('dependencies/CMakeLists.txt') }}
+      - name: Install gcc12
+        run: sudo apt-get install -y g++-12
       - name: Use cmake
         run: |
           mkdir build &&
           cd build &&
-          cmake  -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE_THREADS=ON ..  &&
-          cmake --build . --target document_stream_tests --target ondemand_document_stream_tests --target parse_many_test  &&
-          ctest --output-on-failure  -R parse_many_test  &&
-          ctest --output-on-failure  -R document_stream_tests
+          CXX=g++-12 cmake -DCMAKE_BUILD_TYPE=Debug  -DSIMDJSON_GLIBCXX_ASSERTIONS=ON -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON ..  &&
+          cmake --build .   &&
+          ctest . -E avoid_

.github/workflows/vs17-ci-cxx20.yml

@@ -0,0 +1,44 @@
+name: VS17-CI CXX20
+
+on: [push, pull_request]
+
+jobs:
+  ci:
+    if: >-
+      ! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
+      ! contains(toJSON(github.event.commits.*.message), '[skip github]')
+    name: windows-vs17
+    runs-on: windows-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - {gen: Visual Studio 17 2022, arch: Win32, shared: ON}
+          - {gen: Visual Studio 17 2022, arch: Win32, shared: OFF}
+          - {gen: Visual Studio 17 2022, arch: x64, shared: ON}
+          - {gen: Visual Studio 17 2022, arch: x64, shared: OFF}
+    steps:
+    - name: checkout
+      uses: actions/checkout@v2
+    - name: Configure
+      run: |
+        cmake -DSIMDJSON_CXX_STANDARD=20 -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
+    - name: Build Debug
+      run: cmake --build build --config Debug --verbose
+    - name: Build Release
+      run: cmake --build build --config Release --verbose
+    - name: Run Release tests
+      run: |
+        cd build
+        ctest -C Release  -LE explicitonly  --output-on-failure
+    - name: Run Debug tests
+      run: |
+        cd build
+        ctest -C Debug  -LE explicitonly  --output-on-failure
+    - name: Install
+      run: |
+        cmake --install build --config Release
+    - name: Test Installation
+      run: |
+        cmake -G "${{matrix.gen}}" -A ${{matrix.arch}}  -B build_install_test tests/installation_tests/find
+        cmake --build build_install_test --config Release

.vscode/settings.json

@@ -96,6 +96,7 @@
         "future": "cpp",
         "queue": "cpp",
         "shared_mutex": "cpp",
-        "ranges": "cpp"
+        "ranges": "cpp",
+        "span": "cpp"
     }
 }

CMakeLists.txt

@@ -3,7 +3,7 @@ cmake_minimum_required(VERSION 3.14)
 project(
     simdjson
     # The version number is modified by tools/release.py
-    VERSION 2.2.3
+    VERSION 3.2.0
     DESCRIPTION "Parsing gigabytes of JSON per second"
     HOMEPAGE_URL "https://simdjson.org/"
     LANGUAGES CXX C
@@ -20,8 +20,8 @@ string(
 # ---- Options, variables ----
 
 # These version numbers are modified by tools/release.py
-set(SIMDJSON_LIB_VERSION "13.0.0" CACHE STRING "simdjson library version")
-set(SIMDJSON_LIB_SOVERSION "13" CACHE STRING "simdjson library soversion")
+set(SIMDJSON_LIB_VERSION "16.0.0" CACHE STRING "simdjson library version")
+set(SIMDJSON_LIB_SOVERSION "16" CACHE STRING "simdjson library soversion")
 
 option(SIMDJSON_ENABLE_THREADS "Link with thread support" ON)
 
@@ -95,6 +95,21 @@ if(
   )
 endif()
 
+# GCC and Clang have horrendous Debug builds when using SIMD.
+# A common fix is to use '-Og' instead.
+# bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
+if(
+    (CMAKE_CXX_COMPILER_ID STREQUAL "GNU" OR
+        CMAKE_CXX_COMPILER_ID STREQUAL "Clang" OR
+        CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
+)
+  message(STATUS "Adding -Og to compile flag")
+  simdjson_add_props(
+      target_compile_options PRIVATE
+      $<$<CONFIG:DEBUG>:-Og>
+  )
+endif()
+
 if(SIMDJSON_ENABLE_THREADS)
   find_package(Threads REQUIRED)
   simdjson_add_props(target_link_libraries PUBLIC Threads::Threads)
@@ -149,7 +164,7 @@ install(
     EXPORT simdjsonTargets
     NAMESPACE simdjson::
     DESTINATION "${SIMDJSON_INSTALL_CMAKEDIR}"
-    COMPONENT example_Development
+    COMPONENT simdjson_Development
 )
 
 # pkg-config
@@ -188,11 +203,14 @@ endif()
 
 # ---- Developer mode extras ----
 
+if(is_top_project AND NOT SIMDJSON_DEVELOPER_MODE)
+  message(STATUS "Building only the library. Advanced users and contributors may want to turn SIMDJSON_DEVELOPER_MODE to ON, e.g., via -D SIMDJSON_DEVELOPER_MODE=ON.")
+elseif(SIMDJSON_DEVELOPER_MODE AND NOT is_top_project)
+  message(AUTHOR_WARNING "Developer mode in simdjson is intended for the developers of simdjson")
+endif()
+
 if(NOT SIMDJSON_DEVELOPER_MODE)
-  message(STATUS "Building only the library. Advanced users may want to turn SIMDJSON_DEVELOPER_MODE to ON, e.g., via -D SIMDJSON_DEVELOPER_MODE=ON.")
   return()
-elseif(NOT is_top_project)
-  message(AUTHOR_WARNING "Developer mode is intended for developers of simdjson")
 endif()
 
 simdjson_apply_props(simdjson-internal-flags)
@@ -233,7 +251,9 @@ add_subdirectory(singleheader)
 #
 add_subdirectory(tests)
 add_subdirectory(examples)
-add_subdirectory(benchmark)
+if(CMAKE_SIZEOF_VOID_P EQUAL 8) # we only include the benchmarks on 64-bit systems.
+  add_subdirectory(benchmark)
+endif()
 add_subdirectory(fuzz)
 
 #

Doxyfile

@@ -38,7 +38,7 @@ PROJECT_NAME           = simdjson
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER         = "2.2.3"
+PROJECT_NUMBER         = "3.2.0"
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a
@@ -829,7 +829,7 @@ WARN_LOGFILE           =
 # spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
 # Note: If this tag is empty the current directory is searched.
 
-INPUT                  = doc include
+INPUT                  = doc include/simdjson include/simdjson/dom include/simdjson/generic
 
 # This tag can be used to specify the character encoding of the source files
 # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
@@ -1246,7 +1246,10 @@ HTML_STYLESHEET        =
 # list). For an example see the documentation.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-HTML_EXTRA_STYLESHEET  =
+
+HTML_EXTRA_STYLESHEET  = theme/doxygen-awesome.css \
+                         theme/doxygen-awesome-sidebar-only.css \
+                         theme/doxygen-awesome-sidebar-only-darkmode-toggle.css
 
 # The HTML_EXTRA_FILES tag can be used to specify one or more extra images or
 # other source files which should be copied to the HTML output directory. Note
@@ -1256,7 +1259,10 @@ HTML_EXTRA_STYLESHEET  =
 # files will be copied as-is; there are no commands or markers available.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-HTML_EXTRA_FILES       =
+HTML_EXTRA_FILES       = theme/doxygen-awesome-darkmode-toggle.js \
+                         theme/doxygen-awesome-interactive-toc.js \
+                         theme/doxygen-awesome-fragment-copy-button.js \
+                         theme/doxygen-awesome-paragraph-link.js
 
 # The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen
 # will adjust the colors in the style sheet and background images according to
@@ -1543,7 +1549,7 @@ DISABLE_INDEX          = NO
 # The default value is: NO.
 # This tag requires that the tag GENERATE_HTML is set to YES.
 
-GENERATE_TREEVIEW      = NO
+GENERATE_TREEVIEW      = YES
 
 # The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that
 # doxygen will group on one line in the generated HTML documentation.

HACKING.md

@@ -1,3 +1,4 @@
+
 Hacking simdjson
 ================
 
@@ -6,6 +7,16 @@ Here is wisdom about how to build, test and run simdjson from within the reposit
 If you plan to contribute to simdjson, please read our [CONTRIBUTING](https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md) guide.
 
 
+  - [Design notes](#design-notes)
+  - [Developer mode](#developer-mode)
+  - [Directory Structure and Source](#directory-structure-and-source)
+  - [Runtime Dispatching](#runtime-dispatching)
+  - [Regenerating Single-Header Files](#regenerating-single-header-files)
+  - [Usage (CMake on 64-bit platforms like Linux, FreeBSD or macOS)](#usage-cmake-on-64-bit-platforms-like-linux-freebsd-or-macos)
+  - [Usage (CMake on 64-bit Windows using Visual Studio 2019)](#usage-cmake-on-64-bit-windows-using-visual-studio-2019)
+  - [Various References](#various-references)
+
+
 Design notes
 ------------------------------
 
@@ -193,7 +204,7 @@ point it gets included (but only once per header). singleheader/simdjson.cpp is
 src/simdjson.cpp the same way, except files under generic/ may be included and copy/pasted multiple
 times.
 
-### Usage (CMake on 64-bit platforms like Linux, FreeBSD or macOS)
+## Usage (CMake on 64-bit platforms like Linux, FreeBSD or macOS)
 
 Requirements: In addition to git, we require a recent version of CMake as well as bash.
 
@@ -256,7 +267,9 @@ Note that the name of directory (`build`) is arbitrary, you can name it as you w
 
 
 
-### Usage (CMake on 64-bit Windows using Visual Studio 2019)
+## Usage (CMake on 64-bit Windows using Visual Studio 2019 or better)
+
+Recent versions of Visual Studio support CMake natively, [please refer to the Visual Studio documentation](https://learn.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-170).
 
 We assume you have a common 64-bit Windows PC with at least Visual Studio 2019.
 
@@ -285,7 +298,7 @@ Furthermore, if you have installed LLVM clang on Windows, for example as a compo
 - `cmake --build . -config Release`
 
 
-### Various References
+## Various References
 
 - [How to implement atoi using SIMD?](https://stackoverflow.com/questions/35127060/how-to-implement-atoi-using-simd)
 - [Parsing JSON is a Minefield 💣](http://seriot.ch/parsing_json.php)

LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2018-2019 The simdjson authors 
+   Copyright 2018-2023 The simdjson authors
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -1,9 +1,11 @@
 
-![Ubuntu 18.04 CI](https://github.com/simdjson/simdjson/workflows/Ubuntu%2018.04%20CI%20(GCC%207)/badge.svg)
 [![Ubuntu 20.04 CI](https://github.com/simdjson/simdjson/workflows/Ubuntu%2020.04%20CI%20(GCC%209)/badge.svg)](https://simdjson.org/plots.html)
 ![VS16-CI](https://github.com/simdjson/simdjson/workflows/VS16-CI/badge.svg)
+[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/simdjson.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:simdjson)
 ![MinGW64-CI](https://github.com/simdjson/simdjson/workflows/MinGW64-CI/badge.svg)
-[![][license img]][license]  [![Doxygen Documentation](https://img.shields.io/badge/docs-doxygen-green.svg)](https://simdjson.org/api/2.0.0/index.html)
+[![][license img]][license]
+
+[![Doxygen Documentation](https://img.shields.io/badge/docs-doxygen-green.svg)](https://simdjson.github.io/simdjson/)
 
 simdjson : Parsing gigabytes of JSON per second
 ===============================================
@@ -26,16 +28,38 @@ This library is part of the [Awesome Modern C++](https://awesomecpp.com) list.
 Table of Contents
 -----------------
 
+* [Real-world usage](#real-world-usage)
 * [Quick Start](#quick-start)
 * [Documentation](#documentation)
 * [Performance results](#performance-results)
-* [Real-world usage](#real-world-usage)
 * [Bindings and Ports of simdjson](#bindings-and-ports-of-simdjson)
 * [About simdjson](#about-simdjson)
 * [Funding](#funding)
 * [Contributing to simdjson](#contributing-to-simdjson)
 * [License](#license)
 
+
+Real-world usage
+----------------
+
+- [ClickHouse](https://github.com/ClickHouse/ClickHouse)
+- [Facebook/Meta Velox](https://velox-lib.io)
+- [milvus](https://github.com/milvus-io/milvus)
+- [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer)
+- [Shopify HeapProfiler](https://github.com/Shopify/heap-profiler)
+- [StarRocks](https://github.com/StarRocks/starrocks)
+- [Microsoft FishStore](https://github.com/microsoft/FishStore)
+- [Intel PCM](https://github.com/intel/pcm)
+- [WatermelonDB](https://github.com/Nozbe/WatermelonDB)
+- [Apache Doris](https://github.com/apache/doris)
+- [Dgraph](https://github.com/dgraph-io/dgraph)
+- [UJRPC](https://github.com/unum-cloud/ujrpc)
+- [fastgltf](https://github.com/spnda/fastgltf)
+- [vast](https://github.com/tenzir/vast)
+- [ada-url](https://github.com/ada-url/ada)
+
+If you are planning to use simdjson in a product, please work from one of our releases.
+
 Quick Start
 -----------
 
@@ -62,13 +86,13 @@ int main(void) {
     ondemand::document tweets = parser.iterate(json);
     std::cout << uint64_t(tweets["search_metadata"]["count"]) << " results." << std::endl;
 }
-
-   ```
+```
 3. `c++ -o quickstart quickstart.cpp simdjson.cpp`
 4. `./quickstart`
-   ```
+
+  ```
    100 results.
-   ```
+  ```
 
 Documentation
 -------------
@@ -79,7 +103,13 @@ Usage documentation is available:
 * [Performance](doc/performance.md) shows some more advanced scenarios and how to tune for them.
 * [Implementation Selection](doc/implementation-selection.md) describes runtime CPU detection and
   how you can work with it.
-* [API](https://simdjson.org/api/1.0.0/annotated.html) contains the automatically generated API documentation.
+* [API](https://simdjson.github.io/simdjson/) contains the automatically generated API documentation.
+
+
+Some users may want to browse code along with the compiled assembly. You want to check out the following lists of examples:
+
+* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/98Kx9Kqjn)
+* [simdjson examples with errors without exceptions](https://godbolt.org/z/PKG7GdbPo)
 
 Performance results
 -------------------
@@ -108,17 +138,6 @@ For NDJSON files, we can exceed 3 GB/s with [our  multithreaded parsing function
 
 
 
-Real-world usage
-----------------
-
-- [ClickHouse](https://github.com/ClickHouse/ClickHouse)
-- [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer)
-- [Shopify HeapProfiler](https://github.com/Shopify/heap-profiler)
-- [StarRocks](https://github.com/StarRocks/starrocks)
-- [Microsoft FishStore](https://github.com/microsoft/FishStore)
-
-
-If you are planning to use simdjson in a product, please work from one of our releases.
 
 Bindings and Ports of simdjson
 ------------------------------
@@ -139,9 +158,12 @@ We distinguish between "bindings" (which just wrap the C++ code) and a port to a
 - [simdjson-go](https://github.com/minio/simdjson-go): Go port using Golang assembly.
 - [rcppsimdjson](https://github.com/eddelbuettel/rcppsimdjson): R bindings.
 - [simdjson_erlang](https://github.com/ChomperT/simdjson_erlang): erlang bindings.
+- [simdjsone](https://github.com/saleyn/simdjsone): erlang bindings.
 - [lua-simdjson](https://github.com/FourierTransformer/lua-simdjson): lua bindings.
 - [hermes-json](https://hackage.haskell.org/package/hermes-json): haskell bindings.
 - [simdjzon](https://github.com/travisstaloch/simdjzon): zig port.
+- [JSON-Simd](https://github.com/rawleyfowler/JSON-simd): Raku bindings.
+- [JSON::SIMD](https://metacpan.org/pod/JSON::SIMD): Perl bindings; fully-featured JSON module that uses simdjson for decoding.
 
 About simdjson
 --------------
@@ -184,7 +206,7 @@ License
 
 This code is made available under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html).
 
-Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it under the liberal (business-friendly) MIT license.
+Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it is under the liberal (business-friendly) MIT license.
 
 For compilers that do not support [C++17](https://en.wikipedia.org/wiki/C%2B%2B17), we bundle the string-view library which is published under the [Boost license](http://www.boost.org/LICENSE_1_0.txt). Like the Apache license, the Boost license is a permissive license allowing commercial redistribution.
 

RELEASES.md

@@ -1,69 +0,0 @@
-# 0.5
-
-## Highlights
-
-Performance
-* Faster and simpler UTF-8 validation with the lookup4 algorithm https://github.com/simdjson/simdjson/pull/993
-* We improved the performance of simdjson under Visual Studio by about 25%. Users will still get better performance with clang-cl (+30%) but the gap has been reduced. https://github.com/simdjson/simdjson/pull/1031
-
-Code usability
-* In `parse_many`, when parsing streams of JSON documetns, we give to the users runtime control as to whether threads are used (via the parser.threaded attribute). https://github.com/simdjson/simdjson/issues/925
-* Prefixed public macros to avoid name clashes with other libraries. https://github.com/simdjson/simdjson/issues/1035
-* Better documentation regarding package managers (brew, MSYS2, conan, apt, vcpkg, FreeBSD package manager, etc.).
-* Better documentation regarding  CMake usage.
-
-Standards
-* We improved standard compliance with respect to both the JSON RFC 8259 and JSON Pointer RFC 6901. We added the at_pointer method to nodes for standard-compliant JSON Pointer queries. The legacy `at(std::string_view)` method remains but is deprecated since it is not standard-compliant as per  RFC 6901.
-* We removed computed GOTOs without sacrificing performance thus improving the C++ standard compliance (since computed GOTOs are compiler-specific extensions).
-* Better support for C++20 https://github.com/simdjson/simdjson/pull/1050
-
-# 0.4
-
-## Highlights
-
-- Test coverage has been greatly improved and we have resolved many static-analysis warnings on different systems.
-- We added a fast (8GB/s) minifier that works directly on JSON strings.
-- We added fast (10GB/s) UTF-8 validator that works directly on strings (any strings, including non-JSON).
-- The array and object elements have a constant-time size() method.
-- Performance improvements to the API (type(), get<>()).
-- The parse_many function (ndjson) has been entirely reworked. It now uses a single secondary thread instead of several new threads.
-- We have introduced a faster UTF-8 validation algorithm (lookup3) for all kernels (ARM, x64 SSE, x64 AVX).
-- C++11 support for older compilers and systems.
-- FreeBSD support (and tests).
-- We support the clang front-end compiler (clangcl) under Visual Studio.
-- It is now possible to target ARM platforms under Visual Studio.
-- The simdjson library will never abort or print to standard output/error.
-
-# 0.3
-
-## Highlights
-
-- **Multi-Document Parsing:** Read a bundle of JSON documents (ndjson) 2-4x faster than doing it
-  individually. [API docs](https://github.com/simdjson/simdjson/blob/master/doc/basics.md#newline-delimited-json-ndjson-and-json-lines) / [Design Details](https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md)
-- **Simplified API:** The API has been completely revamped for ease of use, including a new JSON
-  navigation API and fluent support for error code *and* exception styles of error handling with a
-  single API. [Docs](https://github.com/simdjson/simdjson/blob/master/doc/basics.md#the-basics-loading-and-parsing-json-documents)
-- **Exact Float Parsing:** Now simdjson parses floats flawlessly *without* any performance loss,
-  thanks to [great work by @michaeleisel and @lemire](https://github.com/simdjson/simdjson/pull/558).
-  [Blog Post](https://lemire.me/blog/2020/03/10/fast-float-parsing-in-practice/)
-- **Even Faster:** The fastest parser got faster! With a [shiny new UTF-8 validator](https://github.com/simdjson/simdjson/pull/387)
-  and meticulously refactored SIMD core, simdjson 0.3 is 15% faster than before, running at 2.5 GB/s
-  (where 0.2 ran at 2.2 GB/s).
-
-## Minor Highlights
-
-- Fallback implementation: simdjson now has a non-SIMD fallback implementation, and can run even on
-  very old 64-bit machines.
-- Automatic allocation: as part of API simplification, the parser no longer has to be preallocated--
-  it will adjust automatically when it encounters larger files.
-- Runtime selection API: We've exposed simdjson's runtime CPU detection and implementation selection
-  as an API, so you can tell what implementation we detected and test with other implementations.
-- Error handling your way: Whether you use exceptions or check error codes, simdjson lets you handle
-  errors in your style. APIs that can fail return simdjson_result<T>, letting you check the error
-  code before using the result. But if you are more comfortable with exceptions, skip the error code
-  and cast straight to T, and exceptions will be thrown automatically if an error happens. Use the
-  same API either way!
-- Error chaining: We also worked to keep non-exception error-handling short and sweet. Instead of
-  having to check the error code after every single operation, now you can *chain* JSON navigation
-  calls like looking up an object field or array element, or casting to a string, so that you only
-  have to check the error code once at the very end.

benchmark/bench_dom_api.cpp

@@ -521,7 +521,7 @@ static void twitter_image_sizes(State& state) {
     set<tuple<uint64_t, uint64_t>> image_sizes;
     for (dom::object tweet : doc["statuses"]) {
       dom::array media;
-      if (not (error = tweet["entities"]["media"].get(media))) {
+      if (! (error = tweet["entities"]["media"].get(media))) {
         for (dom::object image : media) {
           for (auto size : image["sizes"].get_object()) {
              image_sizes.emplace(size.value["w"], size.value["h"]);
@@ -564,7 +564,7 @@ static void error_code_twitter_default_profile(State& state) noexcept {
     for (dom::element tweet : tweets) {
       dom::object user;
       if ((error = tweet["user"].get(user))) { return; }
-      bool default_profile;
+      bool default_profile{};
       if ((error = user["default_profile"].get(default_profile))) { return; }
       if (default_profile) {
         std::string_view screen_name;
@@ -638,7 +638,7 @@ static void error_code_twitter_image_sizes(State& state) noexcept {
     if ((error = doc["statuses"].get(statuses))) { return; }
     for (dom::element tweet : statuses) {
       dom::array images;
-      if (not (error = tweet["entities"]["media"].get(images))) {
+      if (! (error = tweet["entities"]["media"].get(images))) {
         for (dom::element image : images) {
           dom::object sizes;
           if ((error = image["sizes"].get(sizes))) { return; }

benchmark/benchmarker.h

@@ -520,7 +520,7 @@ struct benchmarker {
         double freqall = (all_stages_without_allocation.best.cycles() / all_stages_without_allocation.best.elapsed_sec()) / 1000000000.0;
         double freqmin = min(freq1, freq2);
         double freqmax = max(freq1, freq2);
-        if((freqall < 0.95 * freqmin) or (freqall > 1.05 * freqmax)) {
+        if((freqall < 0.95 * freqmin) || (freqall > 1.05 * freqmax)) {
           printf("\nWarning: The processor frequency fluctuates in an expected way!!!\n"
           "Range for stage 1 and stage 2 : [%.3f GHz, %.3f GHz], overall: %.3f GHz.\n",
           freqmin, freqmax, freqall);

benchmark/event_counter.h

@@ -134,7 +134,7 @@ struct event_collector {
 
 #if defined(__linux__)
   LinuxEvents<PERF_TYPE_HARDWARE> linux_events;
-  event_collector(bool quiet = false) : linux_events(vector<int>{
+  event_collector(simdjson_unused bool quiet = false) : linux_events(vector<int>{
   #if SIMDJSON_SIMPLE_PERFORMANCE_COUNTERS
     PERF_COUNT_HW_CPU_CYCLES,
     PERF_COUNT_HW_INSTRUCTIONS,
@@ -145,7 +145,7 @@ struct event_collector {
     PERF_COUNT_HW_CACHE_REFERENCES,
     PERF_COUNT_HW_CACHE_MISSES
   #endif
-  }, quiet) {}
+  }) {}
   bool has_events() {
     return linux_events.is_working();
   }

benchmark/json2msgpack/simdjson_ondemand.h

@@ -51,6 +51,7 @@ private:
   simdjson_inline void
   write_raw_string(simdjson::ondemand::raw_json_string rjs);
   inline void recursive_processor(simdjson::ondemand::value element);
+  inline void recursive_processor_ref(simdjson::ondemand::value& element);
 
   simdjson::ondemand::parser parser;
   uint8_t *buff{};
@@ -74,17 +75,33 @@ simdjson2msgpack::to_msgpack(const simdjson::padded_string &json,
       write_byte(0xc2 + doc.get_bool());
       break;
     case simdjson::ondemand::json_type::null:
-      write_byte(0xc0);
+      // We check that the value is indeed null
+      // otherwise: an error is thrown.
+      if(doc.is_null()) {
+        write_byte(0xc0);
+      }
       break;
     case simdjson::ondemand::json_type::array:
     case simdjson::ondemand::json_type::object:
     default:
       // impossible
-      break;
+      SIMDJSON_UNREACHABLE();
     }
   } else {
     simdjson::ondemand::value val = doc;
+#define SIMDJSON_GCC_COMPILER ((__GNUC__) && !(__clang__) && !(__INTEL_COMPILER))
+#if SIMDJSON_GCC_COMPILER
+    // the GCC compiler does well with by-value passing.
+    // GCC has superior recursive inlining:
+    // https://stackoverflow.com/questions/29186186/why-does-gcc-generate-a-faster-program-than-clang-in-this-recursive-fibonacci-co
+    // https://godbolt.org/z/TeK4doE51
     recursive_processor(val);
+#else
+    recursive_processor_ref(val);
+#endif
+  }
+  if (!doc.at_end()) {
+     throw "There are unexpectedly tokens after the end of the json in the json2msgpack sample data";
   }
   return std::string_view(reinterpret_cast<char *>(buf), size_t(buff - buf));
 }
@@ -156,7 +173,58 @@ void simdjson2msgpack::recursive_processor(simdjson::ondemand::value element) {
     write_byte(0xc2 + element.get_bool());
     break;
   case simdjson::ondemand::json_type::null:
-    write_byte(0xc0);
+    // We check that the value is indeed null
+    // otherwise: an error is thrown.
+    if(element.is_null()) {
+      write_byte(0xc0);
+    }
+    break;
+  default:
+    SIMDJSON_UNREACHABLE();
+  }
+}
+
+
+void simdjson2msgpack::recursive_processor_ref(simdjson::ondemand::value& element) {
+  switch (element.type()) {
+  case simdjson::ondemand::json_type::array: {
+    uint32_t counter = 0;
+    write_byte(0xdd);
+    uint8_t *location = skip_uint32();
+    for (auto child : element.get_array()) {
+      counter++;
+      simdjson::ondemand::value v = child.value();
+      recursive_processor_ref(v);
+    }
+    write_uint32_at(counter, location);
+  } break;
+  case simdjson::ondemand::json_type::object: {
+    uint32_t counter = 0;
+    write_byte(0xdf);
+    uint8_t *location = skip_uint32();
+    for (auto field : element.get_object()) {
+      counter++;
+      write_raw_string(field.key());
+      simdjson::ondemand::value v = field.value();
+      recursive_processor_ref(v);
+    }
+    write_uint32_at(counter, location);
+  } break;
+  case simdjson::ondemand::json_type::number:
+    write_double(element.get_double());
+    break;
+  case simdjson::ondemand::json_type::string:
+    write_raw_string(element.get_raw_json_string());
+    break;
+  case simdjson::ondemand::json_type::boolean:
+    write_byte(0xc2 + element.get_bool());
+    break;
+  case simdjson::ondemand::json_type::null:
+    // We check that the value is indeed null
+    // otherwise: an error is thrown.
+    if(element.is_null()) {
+      write_byte(0xc0);
+    }
     break;
   default:
     SIMDJSON_UNREACHABLE();

benchmark/linux/linux-perf-events.h

@@ -1,22 +1,13 @@
-// https://github.com/WojciechMula/toys/blob/master/000helpers/linux-perf-events.h
 #pragma once
 #ifdef __linux__
-#ifdef __has_include
-#if __has_include(<asm/unistd.h>)
+
 #include <asm/unistd.h>       // for __NR_perf_event_open
-#else
-#warning "Header asm/unistd.h cannot be found though it is a linux system. Are linux headers missing?"
-#endif
-#else // no __has_include
-// Please insure that linux headers have been installed.
-#include <asm/unistd.h>       // for __NR_perf_event_open
-#endif
 #include <linux/perf_event.h> // for perf event constants
 #include <sys/ioctl.h>        // for ioctl
 #include <unistd.h>           // for syscall
 
 #include <cerrno>  // for errno
-#include <cstring> // for std::memset
+#include <cstring> // for memset
 #include <stdexcept>
 
 #include <iostream>
@@ -28,13 +19,11 @@ template <int TYPE = PERF_TYPE_HARDWARE> class LinuxEvents {
   perf_event_attr attribs{};
   size_t num_events{};
   std::vector<uint64_t> temp_result_vec{};
-  std::vector<uint64_t> result{};
-  std::vector<int> fds{};
-  bool quiet;
+  std::vector<uint64_t> ids{};
 
 public:
-  explicit LinuxEvents(std::vector<int> config_vec, bool _quiet=false) : fd(0), working(true), quiet{_quiet} {
-    std::memset(&attribs, 0, sizeof(attribs));
+  explicit LinuxEvents(std::vector<int> config_vec) : fd(0), working(true) {
+    memset(&attribs, 0, sizeof(attribs));
     attribs.type = TYPE;
     attribs.size = sizeof(attribs);
     attribs.disabled = 1;
@@ -49,7 +38,7 @@ public:
 
     int group = -1; // no group
     num_events = config_vec.size();
-    result.resize(config_vec.size());
+    ids.resize(config_vec.size());
     uint32_t i = 0;
     for (auto config : config_vec) {
       attribs.config = config;
@@ -57,22 +46,17 @@ public:
       if (_fd == -1) {
         report_error("perf_event_open");
       }
-      fd = _fd; // fd tracks the last _fd value.
-      fds.push_back(fd);
-      ioctl(fd, PERF_EVENT_IOC_ID, &result[i++]);
+      ioctl(_fd, PERF_EVENT_IOC_ID, &ids[i++]);
       if (group == -1) {
-        group = fd;
+        group = _fd;
+        fd = _fd;
       }
     }
 
     temp_result_vec.resize(num_events * 2 + 1);
   }
 
-  ~LinuxEvents() {
-   for (auto tfd : fds) {
-      if (tfd != -1) { close(tfd); }
-    }
-  }
+  ~LinuxEvents() { if (fd != -1) { close(fd); } }
 
   inline void start() {
     if (fd != -1) {
@@ -97,10 +81,16 @@ public:
       }
     }
     // our actual results are in slots 1,3,5, ... of this structure
-    // we really should be checking our result obtained earlier to be safe
     for (uint32_t i = 1; i < temp_result_vec.size(); i += 2) {
       results[i / 2] = temp_result_vec[i];
     }
+    for (uint32_t i = 2; i < temp_result_vec.size(); i += 2) {
+      if(ids[i/2-1] != temp_result_vec[i]) {
+        report_error("event mismatch");
+      }
+    }
+
+
   }
 
   bool is_working() {
@@ -108,13 +98,8 @@ public:
   }
 
 private:
-  void report_error(const std::string &context) {
-    if (!quiet) {
-      if (working) {
-        std::cerr << (context + ": " + std::string(strerror(errno))) << std::endl;
-      }
-    }
+  void report_error(const std::string &) {
     working = false;
   }
 };
-#endif
+#endif

cmake/developer-options.cmake

@@ -50,6 +50,23 @@ undefined behavior.")
   endif()
 endif()
 
+option(SIMDJSON_SANITIZE_MEMORY "Sanitize memory" OFF)
+
+
+if(SIMDJSON_SANITIZE_MEMORY)
+  message(STATUS "Setting the memory sanitizer.")
+  add_compile_options(
+      -fsanitize=memory -fno-sanitize-recover=all
+  )
+  link_libraries(
+      -fsanitize=memory -fno-sanitize-recover=all
+  )
+  # Ubuntu bug for GCC 5.0+ (safe for all versions)
+  if(CMAKE_COMPILER_IS_GNUCC)
+    link_libraries(-fuse-ld=gold)
+  endif()
+endif()
+
 if(SIMDJSON_SANITIZE_THREADS)
   message(STATUS "Setting both the thread sanitizer \
 and the undefined-behavior sanitizer.")
@@ -73,6 +90,8 @@ if(NOT is_multi_config AND NOT CMAKE_BUILD_TYPE)
   if(SIMDJSON_SANITIZE OR SIMDJSON_SANITIZE_UNDEFINED)
     message(STATUS "No build type selected and you have enabled the sanitizer, \
 default to Debug. Consider setting CMAKE_BUILD_TYPE.")
+    message(STATUS "Setting debug optimization flag to -O1 to help sanitizer.")
+    set(CMAKE_CXX_FLAGS_DEBUG "-O1" CACHE STRING "" FORCE)
     set(CMAKE_BUILD_TYPE Debug CACHE STRING "Choose the type of build." FORCE)
   else()
     message(STATUS "No build type selected, default to Release")
@@ -91,7 +110,8 @@ endif()
 
 # We compile tools, tests, etc. with C++ 17. Override yourself if you need on a
 # target.
-set(CMAKE_CXX_STANDARD 17)
+set(SIMDJSON_CXX_STANDARD 17 CACHE STRING "the C++ standard to use for simdjson")
+set(CMAKE_CXX_STANDARD ${SIMDJSON_CXX_STANDARD})
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 set(CMAKE_CXX_EXTENSIONS OFF)
 set(CMAKE_MACOSX_RPATH OFF)
@@ -144,6 +164,11 @@ else()
   )
 endif()
 
+option(SIMDJSON_GLIBCXX_ASSERTIONS "Set _GLIBCXX_ASSERTIONS" OFF)
+if (SIMDJSON_GLIBCXX_ASSERTIONS)
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_ASSERTIONS")
+endif()
+
 #
 # Other optional flags
 #
@@ -187,6 +212,18 @@ else()
   message(STATUS "AVX-512 instructions are not allowed.")
 endif()
 
+option(
+    SIMDJSON_SKIPUTF8VALIDATION
+    "SKIP UTF8 VALIDATION."
+    OFF
+)
+if(SIMDJSON_SKIPUTF8VALIDATION)
+  add_compile_definitions(SIMDJSON_UTF8VALIDATION=0)
+  message(STATUS "SKIP UTF8 VALIDATION")
+else()
+  add_compile_definitions(SIMDJSON_UTF8VALIDATION=1)
+endif()
+
 include(CheckSymbolExists)
 check_symbol_exists(fork unistd.h HAVE_POSIX_FORK)
 check_symbol_exists(wait sys/wait.h HAVE_POSIX_WAIT)

dependencies/CMakeLists.txt

@@ -15,7 +15,7 @@ if(SIMDJSON_GOOGLE_BENCHMARKS)
   set_off(BENCHMARK_ENABLE_INSTALL)
   set_off(BENCHMARK_ENABLE_WERROR)
 
-  import_dependency(google_benchmarks google/benchmark f91b6b4)
+  import_dependency(google_benchmarks google/benchmark v1.7.1)
   add_dependency(google_benchmarks)
 endif()
 
@@ -94,6 +94,14 @@ int main() {}
   import_dependency(rapidjson Tencent/rapidjson f54b0e4)
   add_library(rapidjson INTERFACE)
   target_compile_definitions(rapidjson INTERFACE RAPIDJSON_HAS_STDSTRING)
+  include (TestBigEndian)
+  TEST_BIG_ENDIAN(IS_BIG_ENDIAN)
+  if(IS_BIG_ENDIAN)
+  target_compile_definitions(rapidjson INTERFACE RAPIDJSON_ENDIAN=1)
+  else()
+  target_compile_definitions(rapidjson INTERFACE RAPIDJSON_ENDIAN=0)
+  endif()
+  target_compile_definitions(rapidjson INTERFACE RAPIDJSON_HAS_STDSTRING)
   target_include_directories(rapidjson SYSTEM INTERFACE
           "${rapidjson_SOURCE_DIR}/include")
   target_compile_definitions(rapidjson INTERFACE SIMDJSON_COMPETITION_RAPIDJSON)

dependencies/import.cmake

@@ -40,7 +40,7 @@ macro(add_dependency NAME)
     message(FATAL_ERROR "Missing ${NAME}_SOURCE_DIR variable")
   endif()
 
-  add_subdirectory("${${NAME}_SOURCE_DIR}" "${PROJECT_BINARY_DIR}/_deps/${NAME}")
+  add_subdirectory("${${NAME}_SOURCE_DIR}" "${PROJECT_BINARY_DIR}/_deps/${NAME}" EXCLUDE_FROM_ALL)
 endmacro()
 
 function(set_off NAME)

doc/basics.md

@@ -12,7 +12,7 @@ An overview of what you need to know to use simdjson, with examples.
   - [The Basics: Loading and Parsing JSON Documents](#the-basics-loading-and-parsing-json-documents)
   - [Documents are Iterators](#documents-are-iterators)
     - [Parser, Document and JSON Scope](#parser-document-and-json-scope)
-  - [C++11 Support and string_view](#c11-support-and-string_view)
+  - [string_view](#string_view)
   - [Using the Parsed JSON](#using-the-parsed-json)
     - [Using the Parsed JSON: Additional examples](#using-the-parsed-json-additional-examples)
   - [Minifying JSON strings without parsing](#minifying-json-strings-without-parsing)
@@ -23,25 +23,30 @@ An overview of what you need to know to use simdjson, with examples.
     - [Disabling Exceptions](#disabling-exceptions)
     - [Exceptions](#exceptions)
     - [Current location in document](#current-location-in-document)
+    - [Checking for trailing content](#checking-for-trailing-content)
   - [Rewinding](#rewinding)
-  - [Direct Access to the Raw String](#direct-access-to-the-raw-string)
   - [Newline-Delimited JSON (ndjson) and JSON lines](#newline-delimited-json-ndjson-and-json-lines)
   - [Parsing Numbers Inside Strings](#parsing-numbers-inside-strings)
   - [Dynamic Number Types](#dynamic-number-types)
   - [Raw Strings](#raw-strings)
+  - [General Direct Access to the Raw JSON String](#general-direct-access-to-the-raw-json-string)
   - [Thread Safety](#thread-safety)
   - [Standard Compliance](#standard-compliance)
   - [Backwards Compatibility](#backwards-compatibility)
+  - [Examples](#examples)
+  - [Performance Tips](#performance-tips)
 
 
 Requirements
 ------------------
 
 - A recent compiler (LLVM clang 6 or better, GNU GCC 7.4 or better, Xcode 11 or better) on a 64-bit (PPC, ARM or x64 Intel/AMD) POSIX systems such as macOS, freeBSD or Linux. We require that the compiler supports the C++11 standard or better.
-- Visual Studio 2017 or better under 64-bit Windows. Users should target a 64-bit build (x64) instead of a 32-bit build (x86). We support the LLVM clang compiler under Visual Studio (clangcl) as well as as the regular Visual Studio compiler. We also support MinGW 64-bit under Windows.
+- Visual Studio 2017 or better under 64-bit Windows. Users should target a 64-bit build (x64 or ARM64) instead of a 32-bit build (x86). We support the LLVM clang compiler under Visual Studio (clangcl) as well as as the regular Visual Studio compiler. We also support MinGW 64-bit under Windows.
 
 
-Support for AVX-512 require a processor with AVX512-VBMI2 support (Ice Lake or better) under a 64-bit system and a recent compiler (LLVM clang 6 or better, GCC 8 or better, Visual Studio 2019 or better).
+Support for AVX-512 require a processor with AVX512-VBMI2 support (Ice Lake or better, AMD Zen 4 or better) under a 64-bit system and a recent compiler (LLVM clang 6 or better, GCC 8 or better, Visual Studio 2019 or better). You need a correspondingly recent assembler such as gas (2.30+) or nasm (2.14+): recent compilers usually come with recent assemblers. If you mix a recent compiler with an incompatible/old assembler (e.g., when using a recent compiler with an old Linux distribution), you may get errors at build time because the compiler produces instructions that the assembler does not recognize: you should update your assembler to match your compiler (e.g., upgrade binutils to version 2.30 or better under Linux) or use an older compiler matching the capabilities of your assembler.
+
+We test the library on a big-endian system (IBM s390x with Linux) .
 
 Including simdjson
 ------------------
@@ -61,7 +66,7 @@ c++ myproject.cpp simdjson.cpp
 ```
 
 Note:
-- Users on macOS and other platforms where default compilers do not provide C++11 compliant by default should request it with the appropriate flag (e.g., `c++ -std=c++17 myproject.cpp simdjson.cpp`).
+- Users on macOS and other platforms where default compilers do not provide C++11 compliant by default should request it with the appropriate flag (e.g., `c++ -std=c++11 myproject.cpp simdjson.cpp`).
 - The library relies on [runtime CPU detection](implementation-selection.md): avoid specifying an architecture at compile time (e.g., `-march-native`).
 
 Using simdjson with package managers
@@ -140,6 +145,12 @@ As required by the standard, your JSON document should be in a Unicode (UTF-8) s
 string, from the beginning to the end, needs to be valid: we do not attempt to tolerate bad
 inputs before or after a document.
 
+For efficiency reasons, simdjson requires a string with a few bytes (`simdjson::SIMDJSON_PADDING`)
+at the end, these bytes may be read but their content does not affect the parsing. In practice,
+it means that the JSON inputs should be stored in a memory region with `simdjson::SIMDJSON_PADDING`
+extra bytes at the end. You do not have to set these bytes to specific values though you may
+want to if you want to avoid runtime warnings with some sanitizers.
+
 The simdjson library offers a tree-like [API](https://en.wikipedia.org/wiki/API), which you can
 access by creating a `ondemand::parser` and calling the `iterate()` method. The iterate method
 quickly indexes the input string and may detect some errors. The following example illustrates
@@ -151,8 +162,7 @@ auto json = padded_string::load("twitter.json"); // load JSON file 'twitter.json
 ondemand::document doc = parser.iterate(json); // position a pointer at the beginning of the JSON data
 ```
 
-You can also create a padded string---for efficiency reasons, simdjson requires a string
-with a few bytes (`simdjson::SIMDJSON_PADDING`) at the end---and calling `iterate()`:
+You can also create a padded string---and call `iterate()`:
 
 ```c++
 ondemand::parser parser;
@@ -172,11 +182,33 @@ ondemand::document doc = parser.iterate(json, strlen(json), sizeof(json));
 The simdjson library will also accept `std::string` instances, as long as the `capacity()` of
 the string exceeds the `size()` by at least `SIMDJSON_PADDING`. You can increase the `capacity()` with the `reserve()` function of your strings.
 
+You can copy your data directly on a `simdjson::padded_string` as follows:
+
+```c++
+const char * data = "my data"; // 7 bytes
+simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer
+```
+
+Or as follows...
+
+```c++
+std::string data = "my data";
+simdjson::padded_string my_padded_data(data); // copies to a padded buffer
+```
+
+
 We recommend against creating many `std::string` or many `std::padding_string` instances in your application to store your JSON data.
 Consider reusing the same buffers and limiting memory allocations.
 
 By default, the simdjson library throws exceptions (`simdjson_error`) on errors. We omit `try`-`catch` clauses from our illustrating examples: if you omit `try`-`catch` in your code, an uncaught exception will halt your program. It is also possible to use simdjson without generating exceptions, and you may even build the library without exception support at all. See [Error Handling](#error-handling) for details.
 
+
+Some users may want to browse code along with the compiled assembly. You want to check out the following lists of examples:
+
+* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/98Kx9Kqjn)
+* [simdjson examples with errors without exceptions](https://godbolt.org/z/PKG7GdbPo)
+
+
 Documents are Iterators
 -----------------------
 
@@ -206,7 +238,9 @@ For best performance, a `parser` instance should be reused over several files: o
 needlessly reallocate memory, an expensive process. It is also possible to avoid entirely memory
 allocations during parsing when using simdjson. [See our performance notes for details](performance.md).
 
-C++11 Support and string_view
+If you need to have several documents active at once, you should have several parser instances.
+
+string_view
 -------------
 
 The simdjson library builds on compilers supporting the [C++11 standard](https://en.wikipedia.org/wiki/C%2B%2B11).
@@ -285,19 +319,35 @@ We invite you to keep the following rules in mind:
 2. Because On Demand is really just an iterator, you must fully consume the current object or array before accessing a sibling object or array.
 3. Values can only be consumed once, you should get the values and store them if you plan to need them multiple times. You are expected to access the keys of an object just once. You are expected to go through the values of an array just once.
 
+The simdjson library makes generous use of `std::string_view` instances. If you are unfamiliar
+with `std::string_view` in C++, make sure to [read the section on std::string_view](#string_view).
+They behave much like an immutable `std::string` but they require no memory allocation. You can
+create a `std::string` instance from an `std::string_view` when you need it.
+
 The following specific instructions indicate how to use the JSON when exceptions are enabled, but simdjson has full, idiomatic
 support for users who avoid exceptions. See [the simdjson error handling documentation](basics.md#error-handling) for more.
 
 * **Validate What You Use:** When calling `iterate`, the document is quickly indexed. If it is
   not a valid Unicode (UTF-8) string or if there is an unclosed string, an error may be reported right away.
   However, it is not fully validated. On Demand only fully validates the values you use and the
-  structure leading to it.
+  structure leading to it. It means that at every step as you traverse the document, you may encounter an error. You can handle errors either with exceptions or with error codes.
 * **Extracting Values:** You can cast a JSON element to a native type:
   `double(element)`. This works for `std::string_view`, double, uint64_t, int64_t, bool,
   ondemand::object and ondemand::array. We also have explicit methods such as `get_string()`, `get_double()`,
   `get_uint64()`, `get_int64()`, `get_bool()`, `get_object()` and `get_array()`. After a cast or an explicit method,
-  the number, string or boolean will be parsed, or the initial `[` or `{` will be verified. An exception is thrown if
-  the cast is not possible.
+  the number, string or boolean will be parsed, or the initial `{` or `[` will be verified for ondemand::object and ondemand::array. An exception is thrown if
+  the cast is not possible. Importantly, when getting an ondemand::object or ondemand::array instance, its content is
+  not validated: you are only guaranteed that the corresponding initial character (`{` or `[`) is present. Thus,
+  for example, you could have an ondemand::object instance pointing at the invalid JSON `{ "this is not a valid object" }`: the validation occurs as you access the content.
+  The `get_string()` returns a valid UTF-8 string, after
+  unescaping characters as needed: unmatched surrogate pairs are treated as an error unless you
+  pass `true` (`get_string(true)`) as a parameter to get replacement characters where errors
+  occur. If you somehow need to access non-UTF-8 strings in a lossless manner
+  (e.g., if you strings contain unpaired surrogates), you may use the `get_wobbly_string()` function to get a string in the [WTF-8 format](https://simonsapin.github.io/wtf-8).
+  When calling `get_uint64()` and `get_int64()`, if the number does not fit in a corresponding
+  64-bit integer type, it is also considered an error. When parsing numbers or other scalar values, the library checks
+  that the value is followed by an expected character, thus you *may* get a number parsing error when accessing the digits
+  as an integer in the following strings: `{"number":12332a`, `{"number":12332\0`, `{"number":12332` (the digits appear at the end). We always abide by the [RFC 8259](https://www.tbray.org/ongoing/When/201x/2017/12/14/rfc8259.html) JSON specification so that, for example, numbers prefixed by the `+` sign are in error.
 
   > IMPORTANT NOTE: values can only be parsed once. Since documents are *iterators*, once you have
   > parsed a value (such as by casting to double), you cannot get at it again. It is an error to call
@@ -305,8 +355,8 @@ support for users who avoid exceptions. See [the simdjson error handling documen
 * **Field Access:** To get the value of the "foo" field in an object, use `object["foo"]`. This will
   scan through the object looking for the field with the matching string, doing a character-by-character
   comparison. For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do
-  not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. If you consume an
-  object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]`, your code
+  not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. Keep in mind that On Demand does not buffer or save the result of the parsing: if you repeatedly access `object["foo"]`, then it must repeatedly seek the key and parse the content. The library does not provide a distinct function to check if a key is present, instead we recommend you attempt to access the key: e.g., by doing `ondemand::value val{}; if(!object["foo"].get(val)) {...}`, you have that `val` contains the requested value inside the if clause.  It is your responsability as a user to temporarily keep a reference to the value (`auto v = object["foo"]`), or to consume the content and store it in your own data structures. If you consume an
+  object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]` then your code
   is in error. Furthermore, you can only consume one field at a time, on the same object. The
   value instance you get from  `content["bids"]` becomes invalid when you call `content["asks"]`.
   If you have retrieved `content["bids"].get_array()` and you later call
@@ -321,8 +371,9 @@ support for users who avoid exceptions. See [the simdjson error handling documen
   > to support escaped keys, the method `unescaped_key()` provides the desired unescaped keys by
   > parsing and writing out the unescaped keys to a string buffer and returning a `std::string_view`
   > instance. You should expect a performance penalty when using `unescaped_key()`.
+  >
   > ```c++
-  > auto json = R"({"k\u0065y": 1})"_padded;
+  > auto json = R"({"k\u0065y": 1})"_padded;
   > ondemand::parser parser;
   > auto doc = parser.iterate(json);
   > ondemand::object object = doc.get_object();
@@ -429,7 +480,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
   ```
   This examples also show how we can string several operations and only check for the error once, a strategy we call  *error chaining*.
   Though error chaining makes the code very compact, it also makes error reporting less precise: in this instance, you may get the
-  same error whether the field "str", "123" or "abc" is missing. If you need to break down error handling per operation, avoid error chaining.
+  same error whether the field "str", "123" or "abc" is missing. If you need to break down error handling per operation, avoid error chaining. Furthermore, you should be mindful that chaining that harm performance by encouraging redundancies: writing both `doc["str"]["123"]["abc"].get(value)` and `doc["str"]["123"]["zyw"].get(value)` in the same program may force multiple accesses to the same keys (`"str"` and `"123"`).
 * **Counting elements in arrays:** Sometimes it is useful to scan an array to determine its length prior to parsing it.
   For this purpose, `array` instances have a `count_elements` method. Users should be
   aware that the `count_elements` method can be costly since it requires scanning the
@@ -482,8 +533,20 @@ support for users who avoid exceptions. See [the simdjson error handling documen
   std::cout << "Number of fields: " <<  count << std::endl; // Prints "Number of fields: 2"
   ```
 * **Tree Walking and JSON Element Types:** Sometimes you don't necessarily have a document
-  with a known type, and are trying to generically inspect or walk over JSON elements. To do that, you can use iterators and the `type()` method. You can also represent arbitrary JSON values with
-  `ondemand::value` instances: it can represent anything except a scalar document (lone number, string, null or Boolean). You can check for scalar documents with the method `scalar()`. You may also access [raw strings](#raw-strings).
+  with a known type, and are trying to generically inspect or walk over JSON elements.
+  You can also represent arbitrary JSON values with
+  `ondemand::value` instances: it can represent anything except a scalar document (lone number, string, null or Boolean). You can check for scalar documents with the method `scalar()`.
+  You can query the type of a document or a value with the `type()` method.
+  The `type()` method does not consume or validate documents and values, but it tells you whether they are
+  - arrays (`json_type::array`),
+  - objects (`json_type::object`)
+  - numbers (`json_type::number`),
+  - strings (`json_type::string`),
+  - Booleans (`json_type::boolean`),
+  - null (`json_type::null`).
+
+  You must still validate and consume the values (e.g., call `is_null()`) after calling `type()`.
+   You may also access [raw strings](#raw-strings).
   For example, the following is a quick and dirty recursive function that verbosely prints the JSON document as JSON. This example also illustrates lifecycle requirements: the `document` instance holds the iterator. The document must remain in scope while you are accessing instances of `value`, `object` and `array`.
   ```c++
   void recursive_print_json(ondemand::value element) {
@@ -532,7 +595,11 @@ support for users who avoid exceptions. See [the simdjson error handling documen
       cout << element.get_bool();
       break;
     case ondemand::json_type::null:
-      cout << "null";
+      // We check that the value is indeed null
+      // otherwise: an error is thrown.
+      if(element.is_null()) {
+        cout << "null";
+      }
       break;
     }
   }
@@ -597,6 +664,41 @@ for (ondemand::object car : parser.iterate(cars_json)) {
 }
 ```
 
+
+The previous example had an array of objects, but we can use essentially the same
+approach with an object of objects.
+
+```c++
+ondemand::parser parser;
+auto cars_json = R"( {
+  "identifier1":{ "make": "Toyota", "model": "Camry",  "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
+  "identifier2":{ "make": "Kia",    "model": "Soul",   "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
+  "identifier3":{ "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }
+} )"_padded;
+// Iterating through an array of objects
+ondemand::document doc = parser.iterate(cars_json);
+for (ondemand::field key_car : doc.get_object()) {
+  // If I need a string_view and/or, I can use key_car.unescaped_key() instead, but
+  // key_car.key() will be more performant otherwise.
+  cout << "identifier : " << key_car.key() << std::endl;
+  // I can now access the subobject:
+  ondemand::object car = key_car.value();
+  // Accessing a field by name
+  cout << "Make/Model: " << std::string_view(car["make"]) << "/" << std::string_view(car["model"]) << endl;
+
+  // Casting a JSON element to an integer
+  uint64_t year = car["year"];
+  cout << "- This car is " << 2020 - year << "years old." << endl;
+
+  // Iterating through an array of floats
+  double total_tire_pressure = 0;
+  for (double tire_pressure : car["tire_pressure"]) {
+    total_tire_pressure += tire_pressure;
+  }
+  cout << "- Average tire pressure: " << (total_tire_pressure / 4) << endl;
+}
+```
+
 The following example illustrates how you may also iterate through object values, effectively visiting all key-value pairs in the object.
 
 ```C++
@@ -931,11 +1033,11 @@ bool parse() {
   // Iterating through an array of objects
   auto error = parser.iterate(cars_json).get(doc);
   if(error) { std::cerr << error << std::endl; return false; }
-  ondemand::array cars;
+  ondemand::array cars; // invalid until the get() succeeds
   error = doc.get_array().get(cars);
 
   for (auto car_value : cars) {
-    ondemand::object car;
+    ondemand::object car; // invalid until the get() succeeds
     error = car_value.get_object().get(car);
     if(error) { std::cerr << error << std::endl; return false; }
 
@@ -950,7 +1052,7 @@ bool parse() {
     cout << "Make/Model: " << make << "/" << model << endl;
 
     // Casting a JSON element to an integer
-    uint64_t year;
+    uint64_t year{};
     error = car["year"].get(year);
     if(error) { std::cerr << error << std::endl; return false; }
     cout << "- This car is " << 2020 - year << " years old." << endl;
@@ -976,7 +1078,7 @@ For safety, you should only use our ondemand instances (e.g., `ondemand::object`
 after you have initialized them and checked that there is no error:
 
 ```c++
-    ondemand::object car;
+    ondemand::object car; // invalid until the get() succeeds
     // the `car` instance should not use used before it is initialized
     error = car_value.get_object().get(car);
     if(error) {
@@ -994,10 +1096,12 @@ having to handle exceptions.
   ondemand::document doc;
   auto error = parser.iterate(json).get(doc);
   if(error) { return false; }
-  ondemand::object object;
+  ondemand::object object; // invalid until the get() succeeds
   error = doc.get_object().get(object);
   if(error) { return false; }
   for(auto field : object) {
+    // We could replace 'field.key() with field.unescaped_key(),
+    // and ondemand::raw_json_string by std::string_view.
     ondemand::raw_json_string keyv;
     error = field.key().get(keyv);
     if(error) { return false; }
@@ -1046,10 +1150,32 @@ int main(void) {
 }
 ```
 
+
+You can do handle errors gracefully as well...
+
+```C++
+#include <iostream>
+#include "simdjson.h"
+int main(void) {
+  simdjson::ondemand::parser parser;
+  simdjson::padded_string json_string;
+  simdjson::ondemand::document doc;
+  try {
+    json_string = padded_string::load("twitter.json");
+    doc = parser.iterate(json_string);
+    uint64_t identifier = doc["statuses"].at(0)["id"];
+    std::cout << identifier << std::endl;
+  } catch (simdjson::simdjson_error &error) {
+    std::cerr << "JSON error: " << error.what() << " near "
+              << doc.current_location() << " in " << json_string << std::endl;
+  }
+}
+```
+
 ### Current location in document
 
 Sometimes, it might be helpful to know the current location in the document during iteration. This is especially useful when encountering errors. The `current_location()` method on a
-`document` instances makes it easy to identify common JSON errors. Users can call the `current_location()` method on a validdocument instance to retrieve a `const char *` pointer to the current location in the document. This method also works even after an error has invalidated the document and the parser (e.g. `TAPE_ERROR`, `INCOMPLETE_ARRAY_OR_OBJECT`).
+`document` instances makes it easy to identify common JSON errors. Users can call the `current_location()` method on a valid document instance to retrieve a `const char *` pointer to the current location in the document. This method also works even after an error has invalidated the document and the parser (e.g. `TAPE_ERROR`, `INCOMPLETE_ARRAY_OR_OBJECT`).
 When the input was a `padding_string` or another null-terminated source, then you may
 use the `const char *` pointer as a C string. As an example, consider the following
 example where we used the exception-free simdjson interface:
@@ -1062,6 +1188,12 @@ int64_t i;
 auto error = doc["integer"].get_int64().get(i);    // Expect to get integer from "integer" key, but get TAPE_ERROR
 if (error) {
   std::cout << error << std::endl;    // Prints TAPE_ERROR error message
+  // Recover a pointer to the location of the first error:
+  const char * ptr;
+  doc.current_location().get(ptr);
+  // ptr points at 'false, "integer": -343} " which is the location of the error
+  //
+  // Because we pad simdjson::padded_string instances with null characters, you may also do the following:
   std::cout<< doc.current_location() << std::endl;  // Prints "false, "integer": -343} " (location of TAPE_ERROR)
 }
 ```
@@ -1110,6 +1242,9 @@ for (auto val : doc) {
 std::cout << doc.current_location() << std::endl;   // Throws OUT_OF_BOUNDS
 ```
 
+Conversely, if `doc.current_location().error() == simdjson::SUCCESS`,
+then the document has more content.
+
 Finally, the `current_location()` method may also be used even when no exceptions/errors
 are thrown. This can be helpful for users that want to know the current state of iteration during parsing. For example:
 
@@ -1118,7 +1253,7 @@ auto json = R"( [[1,2,3], -23.4, {"key": "value"}, true] )"_padded;
 ondemand::parser parser;
 auto doc = parser.iterate(json);
 for (auto val : doc) {
-  ondemand::object obj;
+  ondemand::object obj; // invalid until the get() succeeds
   auto error = val.get_object().get(obj);     // Only get objects
   if (!error) {
     std::cout << doc.current_location() << std::endl;   // Prints ""key": "value"}, true] "
@@ -1135,6 +1270,32 @@ contains control characters that must be escaped and UNCLOSED_STRING if there
 is an unclosed string in the document. We do not provide location information for these
 errors.
 
+### Checking for trailing content
+
+The parser validates all parsed content, but your code may exhaust the content while
+not having processed the entire document. Thus, as a final optional step, you may
+call `at_end()` on the document instance. If it returns `false`, then you may
+conclude that you have trailing content and that your document is not valid JSON.
+You may then use `doc.current_location()` to obtain a pointer to the start of the trailing
+content.
+
+```C++
+  auto json = R"([1, 2] foo ])"_padded;
+  ondemand::parser parser;
+  ondemand::document doc = parser.iterate(json);
+  ondemand::array array = doc.get_array();
+  for (uint64_t values : array) {
+    std::cout << values << std::endl;
+  }
+  if(!doc.at_end()) {
+    // In this instance, we will be left pointing at 'foo' since we have consumed the array [1,2].
+    std::cerr << "trailing content at byte index " << doc.current_location() - json.data() << std::endl;
+  }
+```
+
+The `at_end()` method is equivalent to `doc.current_location().error() == simdjson::SUCCESS` but
+more convenient.
+
 Rewinding
 ----------
 
@@ -1173,41 +1334,6 @@ internal string buffer. Thus you should consume values only once
 even if you can iterate through the array or object more than once.
 If you unescape a string within an array more than once, you have unsafe code.
 
-Direct Access to the Raw String
---------------------------------
-
-The simdjson library makes explicit assumptions about types. For examples, numbers
-must be integers (up to 64-bit integers) or binary64 floating-point numbers. Some users
-have different needs. For example, some users might want to support big integers.
-The library makes this possible by providing a `raw_json_token` method which returns
-a `std::string_view` instance containing the value as a string which you may then
-parse as you see fit.
-
-```C++
-simdjson::ondemand::parser parser;
-simdjson::padded_string docdata =  R"({"value":12321323213213213213213213213211223})"_padded;
-simdjson::ondemand::document doc = parser.iterate(docdata);
-simdjson::ondemand::object obj = doc.get_object();
-std::string_view token = obj["value"].raw_json_token();
-// token has value 12321323213213213213213213213211223, it points inside the input string
-```
-
-The `raw_json_token` method even works when the JSON value is a string. In such cases, it
-will return the complete string with the quotes and with eventual escaped sequences as in the
-source document.
-
-```C++
-simdjson::ondemand::parser parser;
-simdjson::padded_string docdata =  R"({"value":"12321323213213213213213213213211223"})"_padded;
-simdjson::ondemand::document doc = parser.iterate(docdata);
-simdjson::ondemand::object obj = doc.get_object();
-string_view token = obj["value"].raw_json_token();
-// token has value "12321323213213213213213213213211223", it points inside the input string
-```
-
-The `raw_json_token()` should be fast and free of allocation.
-
-
 
 Newline-Delimited JSON (ndjson) and JSON lines
 ----------------------------------------------
@@ -1396,6 +1522,8 @@ if (error) {
 It is also important to note that when dealing an invalid number inside a string, simdjson will report a `NUMBER_ERROR` error if the string begins with a number whereas simdjson
 will report a `INCORRECT_TYPE` error otherwise.
 
+The `*_in_string` methods can also be called on a single document instance:
+e.g., when your document consist solely of a quoted number.
 
 Dynamic Number Types
 ------------------------------
@@ -1510,6 +1638,78 @@ JSON string to a user-provided buffer:
     }
 ```
 
+
+General Direct Access to the Raw JSON String
+--------------------------------
+If your value is a string, the `raw_json_string` gives you direct access to the unprocess
+string. The simdjson library allows you to have access to the raw underlying JSON
+more generally.
+
+The simdjson library makes explicit assumptions about types. For examples, numbers
+must be integers (up to 64-bit integers) or binary64 floating-point numbers. Some users
+have different needs. For example, some users might want to support big integers.
+The library makes this possible by providing a `raw_json_token` method which returns
+a `std::string_view` instance containing the value as a string which you may then
+parse as you see fit.
+
+```C++
+simdjson::ondemand::parser parser;
+simdjson::padded_string docdata =  R"({"value":12321323213213213213213213213211223})"_padded;
+simdjson::ondemand::document doc = parser.iterate(docdata);
+simdjson::ondemand::object obj = doc.get_object();
+std::string_view token = obj["value"].raw_json_token();
+// token has value 12321323213213213213213213213211223, it points inside the input string
+```
+
+The `raw_json_token` method even works when the JSON value is a string. In such cases, it
+will return the complete string with the quotes and with eventual escaped sequences as in the
+source document.
+
+```C++
+simdjson::ondemand::parser parser;
+simdjson::padded_string docdata =  R"({"value":"12321323213213213213213213213211223"})"_padded;
+simdjson::ondemand::document doc = parser.iterate(docdata);
+simdjson::ondemand::object obj = doc.get_object();
+string_view token = obj["value"].raw_json_token();
+// token has value "12321323213213213213213213213211223", it points inside the input string
+```
+
+The `raw_json_token()` should be fast and free of allocation.
+
+If you value is an array or an object, `raw_json_token()` returns effectively a single
+character (`[`) or (`}`) which is not very useful. For arrays and objects, we have another
+method called `raw_json()` which consumes (traverse) the array or the object.
+
+```C++
+simdjson::ondemand::parser parser;
+simdjson::padded_string docdata =  R"({"value":123})"_padded;
+simdjson::ondemand::document doc = parser.iterate(docdata);
+simdjson::ondemand::object obj = doc.get_object();
+string_view token = obj.raw_json(); // gives you `{"value":123}`
+```
+
+
+```C++
+simdjson::ondemand::parser parser;
+simdjson::padded_string docdata =  R"([1,2,3])"_padded;
+simdjson::ondemand::document doc = parser.iterate(docdata);
+simdjson::ondemand::array arr = doc.get_array();
+string_view token = arr.raw_json(); // gives you `[1,2,3]`
+```
+
+Because `raw_json()` consumes to object or the array, if you want to both have
+access to the raw string, and also use the array or object, you should call `reset()`.
+
+```C++
+simdjson::ondemand::parser parser;
+simdjson::padded_string docdata =  R"({"value":123})"_padded;
+simdjson::ondemand::document doc = parser.iterate(docdata);
+simdjson::ondemand::object obj = doc.get_object();
+string_view token = obj.raw_json(); // gives you `{"value":123}`
+obj.reset(); // revise the object
+uint64_t x = obj["value"]; // gives me 123
+```
+
 Thread Safety
 -------------
 
@@ -1550,3 +1750,215 @@ Backwards Compatibility
 The only header file supported by simdjson is `simdjson.h`. Older versions of simdjson published a
 number of other include files such as `document.h` or `ParsedJson.h` alongside `simdjson.h`; these headers
 may be moved or removed in future versions.
+
+Examples
+--------
+
+Some users like to have example. The following code samples illustrate how to process specific JSON inputs.
+For simplicity, we do not include full error support: this code would throw exceptions on error.
+
+
+* Example 1: ZuluBBox
+
+```C++
+struct ZuluBBox {
+  double xmin;
+  double ymin;
+  double width;
+  double height;
+
+  void print() {
+    std::cout << xmin << ", " << ymin << ", " << width << ", " << height
+              << std::endl;
+  }
+};
+
+bool example() {
+
+  auto json = R"+( {
+  "ZuluROI": {
+    "ZuluBBox": {
+      "xmin": 0,
+      "ymin": 0,
+      "width": 1,
+      "height": 1
+    },
+    "SubObjects": [
+      {
+        "ZuluDetection": {
+          "label": "car",
+          "class_id": 3,
+          "confidence": 0.7587034106254578,
+          "ZuluBBox": {
+            "xmin": 0.3843536376953125,
+            "ymin": 0.4532909393310547,
+            "width": 0.09115534275770187,
+            "height": 0.04127710685133934
+          },
+          "SubObjects": []
+        }
+      },
+      {
+        "ZuluDetection": {
+          "label": "car",
+          "class_id": 3,
+          "confidence": 0.6718865633010864,
+          "ZuluBBox": {
+            "xmin": 0.7500002980232239,
+            "ymin": 0.5212296843528748,
+            "width": 0.07592231780290604,
+            "height": 0.038947589695453644
+          },
+          "SubObjects": []
+        }
+      },
+      {
+        "ZuluDetection": {
+          "label": "car",
+          "class_id": 3,
+          "confidence": 0.5806200504302979,
+          "ZuluBBox": {
+            "xmin": 0.9025363922119141,
+            "ymin": 0.5925348401069641,
+            "width": 0.05478987470269203,
+            "height": 0.046337299048900604
+          },
+          "SubObjects": []
+        }
+      }
+    ]
+  },
+  "timestamp (ms)": 1677085594421,
+  "buffer_offset": 35673
+} )+"_padded;
+  ondemand::parser parser;
+  ondemand::document doc = parser.iterate(json);
+  ondemand::object root_object = doc.get_object();
+  ondemand::object roi_object = root_object["ZuluROI"];
+
+  ondemand::object box_roi_object = roi_object["ZuluBBox"];
+  ZuluBBox box = {
+      double(box_roi_object["xmin"]), double(box_roi_object["ymin"]),
+      double(box_roi_object["width"]), double(box_roi_object["height"])};
+  box.print();
+
+  for (ondemand::object value : roi_object["SubObjects"]) {
+    ondemand::object detect = value["ZuluDetection"];
+    std::cout << detect["label"].get_string() << std::endl;
+    std::cout << detect["class_id"].get_uint64() << std::endl;
+    std::cout << detect["confidence"].get_double() << std::endl;
+
+    ondemand::object vbox_roi_object = detect["ZuluBBox"];
+    ZuluBBox vbox = {
+        double(vbox_roi_object["xmin"]), double(vbox_roi_object["ymin"]),
+        double(vbox_roi_object["width"]), double(vbox_roi_object["height"])};
+    vbox.print();
+  }
+
+  std::cout << root_object["timestamp (ms)"].get_uint64() << std::endl;
+  std::cout << root_object["buffer_offset"].get_uint64() << std::endl;
+  return true;
+}
+```
+
+
+* Example 2: Demos
+
+```C++
+bool example() {
+  auto json = R"+( {
+    "5f08a730b280e54fd1e75a7046b93fdc": {
+        "file": "/DEMOS/0-9/10_Orbyte.sid",
+        "len": [
+            "1:17"
+        ],
+        "loud": [
+            "-22.8"
+        ],
+        "name": "10 Orbyte",
+        "author": "Michael Becker (Premium)",
+        "release": "2014 Tristar & Red Sector Inc.",
+        "bits": 20
+    },
+    "2727236ead44a62f0c6e01f6dd4dc484": {
+        "file": "/DEMOS/0-9/12345.sid",
+        "len": [
+            "0:56"
+        ],
+        "loud": [
+            "-33.3"
+        ],
+        "name": "12345",
+        "author": "Beal",
+        "release": "1988 Beal",
+        "bits": 20
+    },
+    "7ea765fce6c0f92570b18adc7bf52f54": {
+        "file": "/DEMOS/0-9/128_Byte_Blues_BASIC.sid",
+        "len": [
+            "0:18"
+        ],
+        "loud": [
+            "-27.1"
+        ],
+        "name": "128 Byte Blues",
+        "author": "Leonard J. Paul (Freaky DNA)",
+        "release": "2005 Freaky DNA",
+        "bits": 62
+    }
+} )+"_padded;
+  ondemand::parser parser;
+  ondemand::document doc = parser.iterate(json);
+  ondemand::object root_object = doc.get_object();
+  for(auto key_value : root_object) {
+    // could get std::string_view with 'unescaped_key()':
+    std::cout << "key: " << key_value.key() << std::endl;
+    ondemand::object obj = key_value.value();
+
+    std::cout << "file: " << std::string_view(obj["file"]) << std::endl;
+
+    std::cout << "len: ";
+    for(std::string_view values : obj["len"]) {
+      std::cout << values << std::endl;
+    }
+    std::cout << std::endl;
+
+    std::cout << "loud: ";
+    for(std::string_view values : obj["loud"]) {
+      std::cout << values << std::endl;
+    }
+    std::cout << std::endl;
+
+    std::cout << "name: " << std::string_view(obj["name"]) << std::endl;
+    std::cout << "author: " << std::string_view(obj["author"]) << std::endl;
+    std::cout << "release: " << std::string_view(obj["release"]) << std::endl;
+    std::cout << "bits: " << uint64_t(obj["bits"]) << std::endl;
+  }
+  return true;
+}
+```
+
+
+Performance Tips
+--------
+
+
+- The On Demand front-end works best when doing a single pass over the input: avoid calling `count_elements`, `rewind` and similar methods.
+- If you are familiar with assembly language, you may use the online tool godbolt to explore the compiled code. The following example may work: [https://godbolt.org/z/xE4GWs573](https://godbolt.org/z/xE4GWs573).
+- Given a field `field` in an object, calling `field.key()` is often faster than `field.unescaped_key()` so if you do not need an unescaped `std::string_view` instance, prefer `field.key()`.
+- For release builds, we recommend setting `NDEBUG` pre-processor directive when compiling the `simdjson` library. Importantly, using the optimization flags `-O2` or `-O3` under GCC and LLVM clang does not set the `NDEBUG` directive, you must set it manually (e.g., `-DNDEBUG`).
+- For long streams of JSON documents, consider [`iterate_many`](iterate_many.md) and [`parse_many`](parse_many.md) for better performance.
+- If possible, refer to each object and array in your code once. For example, the following code repeatedly refers to the `"data"` key to create an object...
+	```C++
+	std::string_view make = o["data"]["make"];
+	std::string_view model = o["data"]["model"];
+	std::string_view year = o["data"]["year"];
+  ```
+  We expect that it is more efficient to access the `"data"` key once:
+  ```C++
+	simdjson::ondemand::object data = o["data"];
+	std::string_view model = data["model"];
+	std::string_view year = data["year"];
+	std::string_view rating = data["rating"];
+  ```
+- To better understand the operation of your On Demand parser, and whether it is performing as well as you think it should be, there is a  logger feature built in to simdjson! To use it, define the pre-processor directive `SIMDJSON_VERBOSE_LOGGING` prior to including the `simdjson.h` header, which enables logging in simdjson. Run your code. It may generate a lot of logging output; adding printouts from your application that show each section may be helpful. The log’s output will show step-by-step information on state, buffer pointer position, depth, and key retrieval status. The log level is set to `INFO` by default, but can be modified with the environment variable `SIMDJSON_LOG_LEVEL`. Setting the environment variable `SIMDJSON_LOG_LEVEL` to `ERROR` will log only errors.

doc/basics_doxygen.md

@@ -1,5 +1,5 @@
 We take our documentation seriously. Please start reading the documentation before you attempt to use simdjson. We hope you will enjoy reading us.
 
 * Basics: https://github.com/simdjson/simdjson/blob/master/doc/basics.md is an overview of how to use simdjson and its APIs.
-* parse_many: https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md describes an interface providing features to work with files or streams containing multiple small JSON documents. As fast and convenient as possible.
+* iterate_many: https://github.com/simdjson/simdjson/blob/master/doc/iterate_many.md describes an interface providing features to work with files or streams containing multiple small JSON documents. As fast and convenient as possible.
 * Performance: https://github.com/simdjson/simdjson/blob/master/doc/performance.md shows some more advanced scenarios and how to tune for them.

doc/dom.md

@@ -16,6 +16,7 @@ An overview of what you need to know to use simdjson, with examples.
 * [Server Loops: Long-Running Processes and Memory Capacity](#server-loops-long-running-processes-and-memory-capacity)
 * [Best Use of the DOM API](#best-use-of-the-dom-api)
 * [Padding and Temporary Copies](#padding-and-temporary-copies)
+* [Performance Tips](#performance-tips)
 
 DOM vs On Demand
 ----------------------------------------------
@@ -44,10 +45,24 @@ dom::parser parser;
 dom::element doc = parser.parse("[1,2,3]"_padded); // parse a string, the _padded suffix creates a simdjson::padded_string instance
 ```
 
+You can copy your data directly on a `simdjson::padded_string` as follows:
+
+```c++
+const char * data = "my data"; // 7 bytes
+simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer
+```
+
+Or as follows...
+
+```c++
+std::string data = "my data";
+simdjson::padded_string my_padded_data(data); // copies to a padded buffer
+```
+
 The parsed document resulting from the `parser.load` and `parser.parse` calls depends on the `parser` instance. Thus the `parser` instance must remain in scope. Furthermore, you must have at most one parsed document in play per `parser` instance.
 You cannot copy a `parser` instance, you may only move it.
 
-If you need to keep a document around long term, you can keep or move the parser instance. Note that moving a parser instance, or keeping one in a movable data structure like vector or map, can cause any outstanding `element`, `object` or `array` instances to be invalidated. If you need to store a parser in a movable data structure, you should use a `std::unique_ptr` to avoid this invalidation(e.g., `std::unique_ptr<dom::parser> parser(new dom::parser{})`).
+If you need to keep a document around long term, you can keep or move the parser instance. Note that moving a parser instance, or keeping one in a movable data structure like vector or map, can cause any outstanding `element`, `object` or `array` instances to be invalidated. The `element`, `object` or `array` instances are mere thin wrappers akin to an `std::vector<int>::iterator`: they are invalid when default constructed, they must be tied to a valid document instance. If you need to store a parser in a movable data structure, you should use a `std::unique_ptr` to avoid this invalidation(e.g., `std::unique_ptr<dom::parser> parser(new dom::parser{})`).
 
 During the`load` or `parse` calls, neither the input file nor the input string are ever modified. After calling `load` or `parse`, the source (either a file or a string) can be safely discarded. All of the JSON data is stored in the `parser` instance.  The parsed document is also immutable in simdjson: you do not modify it by accessing it.
 
@@ -65,7 +80,7 @@ Once you have an element, you can navigate it with idiomatic C++ iterators, oper
 * **Extracting Values (with exceptions):** You can cast a JSON element to a native type: `double(element)` or
   `double x = json_element`. This works for double, uint64_t, int64_t, bool,
   dom::object and dom::array. An exception (`simdjson::simdjson_error`) is thrown if the cast is not possible.
-* **Extracting Values (without exceptions):** You can use a variant usage of `get()` with error codes to avoid exceptions. You first declare the variable of the appropriate type (`double`, `uint64_t`, `int64_t`, `bool`,
+* **Extracting Values (without exceptions):** You can use a variant usage of `get()` with error codes to avoid exceptions. You first declare the variable of the appropriate type (`double`, `uint64_t`, `int64_t`, `bool`, `std::string_view`,
   `dom::object` and `dom::array`) and pass it by reference to `get()` which gives you back an error code: e.g.,
   ```c++
   simdjson::error_code error;
@@ -76,6 +91,7 @@ Once you have an element, you can navigate it with idiomatic C++ iterators, oper
   if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
   std::cout << "I parsed " << value << " from " << numberstring.data() << std::endl;
   ```
+  The strings contain unescaped valid UTF-8 strings: no unmatched surrogate is allowed.
 * **Field Access:** To get the value of the "foo" field in an object, use `object["foo"]`.
 * **Array Iteration:** To iterate through an array, use `for (auto value : array) { ... }`. If you
   know the type of the value, you can cast it right there, too! `for (double value : array) { ... }`
@@ -88,7 +104,7 @@ Once you have an element, you can navigate it with idiomatic C++ iterators, oper
 * **Array and Object size** Given an array or an object, you can get its size (number of elements or keys)
   with the `size()` method.
 * **Checking an Element Type:** You can check an element's type with `element.type()`. It
-  returns an `element_type` with values such as `simdjson::dom::element_type::ARRAY`, `simdjson::dom::element_type::OBJECT`, `simdjson::dom::element_type::INT64`,  `simdjson::dom::element_type::UINT64`,`simdjson::dom::element_type::DOUBLE`, `simdjson::dom::element_type::BOOL` or, `simdjson::dom::element_type::NULL_VALUE`.
+  returns an `element_type` with values such as `simdjson::dom::element_type::ARRAY`, `simdjson::dom::element_type::OBJECT`, `simdjson::dom::element_type::INT64`,  `simdjson::dom::element_type::UINT64`,`simdjson::dom::element_type::DOUBLE`, `simdjson::dom::element_type::STRING`, `simdjson::dom::element_type::BOOL` or, `simdjson::dom::element_type::NULL_VALUE`.
 * **Output to streams and strings:** Given a document or an element (or node) out of a JSON document, you can output a minified string version using the C++ stream idiom (`out << element`). You can also request the construction of a minified string version (`simdjson::minify(element)`). Numbers are serialized as 64-bit floating-point numbers (`double`).
 
 ### Examples
@@ -167,7 +183,7 @@ While the simdjson library can be used in any project using C++ 11 and above, fi
 ```c++
 padded_string json = R"(  { "foo": 1, "bar": 2 }  )"_padded;
 dom::parser parser;
-dom::object object;
+dom::object object; // invalid until the get() succeeds
 auto error = parser.parse(json).get(object);
 if (error) { cerr << error << endl; return; }
 for (auto [key, value] : object) {
@@ -181,7 +197,7 @@ For comparison, here is the C++ 11 version of the same code:
 // C++ 11 version for comparison
 padded_string json = R"(  { "foo": 1, "bar": 2 }  )"_padded;
 dom::parser parser;
-dom::object object;
+dom::object object; // invalid until the get() succeeds
 auto error = parser.parse(json).get(object);
 if (error) { cerr << error << endl; return; }
 for (dom::key_value_pair field : object) {
@@ -227,7 +243,7 @@ dom::parser parser;
 dom::element cars = parser.parse(cars_json);
 cout << cars.at_pointer("/0/tire_pressure/1") << endl; // Prints 39.9
 for (dom::element car_element : cars) {
-    dom::object car;
+    dom::object car; // invalid until the get() succeeds
     simdjson::error_code error;
     if ((error = car_element.get(car))) { std::cerr << error << std::endl; return; }
     double x = car.at_pointer("/tire_pressure/1");
@@ -283,11 +299,11 @@ it selects the key "count" within that object.
 
 int main(void) {
   simdjson::dom::parser parser;
-  simdjson::dom::element tweets;
+  simdjson::dom::element tweets; // invalid until the get() succeeds
   auto error = parser.load("twitter.json").get(tweets);
   if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
 
-  simdjson::dom::element res;
+  simdjson::dom::element res; // invalid until the get() succeeds
   if ((error = tweets["search_metadata"]["count"].get(res))) {
     std::cerr << "could not access keys" << std::endl;
     return EXIT_FAILURE;
@@ -311,7 +327,7 @@ Observe how we use the `at` method when querying an index into an array, and not
 
 int main(void) {
   simdjson::dom::parser parser;
-  simdjson::dom::element tweets;
+  simdjson::dom::element tweets; // invalid until the get() succeeds
   auto error = parser.load("twitter.json").get(tweets);
   if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }
   uint64_t identifier;
@@ -336,7 +352,7 @@ auto cars_json = R"( [
   { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }
 ] )"_padded;
 dom::parser parser;
-dom::array cars;
+dom::array cars; // invalid until the get() succeeds
 auto error = parser.parse(cars_json).get(cars);
 if (error) { cerr << error << endl; exit(1); }
 
@@ -382,7 +398,7 @@ auto abstract_json = R"( [
     {  "12545" : {"a":11.44, "b":12.78, "c": 11111111}  }
   ] )"_padded;
 dom::parser parser;
-dom::array array;
+dom::array array; // invalid until after the next line
 auto error = parser.parse(abstract_json).get(array);
 if (error) { cerr << error << endl; exit(1); }
 // Iterate through an array of objects
@@ -436,7 +452,7 @@ bool parse_double(const char *j, double &d) {
 
 bool parse_string(const char *j, std::string &s) {
   std::string_view answer;
-  auto error = parser.parse(j,strlen(j))
+  auto error = parser.parse(j, strlen(j))
         .at(0)
         .get(answer, error);
   if (error) { return false; }
@@ -643,3 +659,9 @@ simdjson::dom::element element = parser.parse(padded_json_copy.get(), json_len,
 ````
 
 Setting the `realloc_if_needed` parameter `false` in this manner may lead to better performance since copies are avoided, but it requires that the user takes more responsibilities: the simdjson library cannot verify that the input buffer was padded with SIMDJSON_PADDING extra bytes.
+
+Performance Tips
+---------------------
+
+- For release builds, we recommend setting `NDEBUG` pre-processor directive when compiling the `simdjson` library. Importantly, using the optimization flags `-O2` or `-O3` under GCC and LLVM clang does not set the `NDEBUG` directrive, you must set it manually (e.g., `-DNDEBUG`).
+- For long streams of JSON documents, consider [`iterate_many`](iterate_many.md) and [`parse_many`](parse_many.md) for better performance.

doc/implementation-selection.md

@@ -21,12 +21,13 @@ The current implementations are:
 * haswell: AVX2 (2013 Intel Haswell or later)
 * westmere: SSE4.2 (2010 Westmere or later).
 * arm64: 64-bit ARMv8-A NEON
-* ppc64: 64-bit POWER8 and POWER9 with VSX and ALTIVEC extensions. Both big endian and little endian are implemented, depends on the compiler you are using.
+* ppc64: 64-bit POWER8 and POWER9 with VSX and ALTIVEC extensions. Both big endian and little endian are implemented, depends on the compiler you are using. The library is tested on recent, little-endian, POWER systems.
 * fallback: A generic implementation that runs on any 64-bit processor.
 
 In many cases, you don't know where your compiled binary is going to run, so simdjson automatically
-compiles *all* the implementations into the executable. On Intel, it will include 3 implementations
-(haswell, westmere and fallback), on ARM it will include 2 (arm64 and fallback), and on PPC it will include 2 (ppc64 and fallback).
+compiles *all* the implementations into the executable. On Intel, it will include 4 implementations
+(icelake, haswell, westmere and fallback), on ARM it will include 2 (arm64 and fallback), and on PPC
+it will include 2 (ppc64 and fallback).
 
 If you know more about where you're going to run and want to save the space, you can disable any of
 these implementations at compile time with `-DSIMDJSON_IMPLEMENTATION_X=0` (where X is ICELAKE, HASWELL,
@@ -51,7 +52,7 @@ Inspecting the Detected Implementation
 You can check what implementation is running with `active_implementation`:
 
 ```c++
-cout << "simdjson v" << SIMDJSON_STRINGIFY(SIMDJSON_VERSION) << endl;
+cout << "simdjson v" << SIMDJSON_VERSION << endl;
 cout << "Detected the best implementation for your machine: " << simdjson::get_active_implementation()->name();
 cout << "(" << simdjson::get_active_implementation()->description() << ")" << endl;
 ```

doc/iterate_many.md

@@ -237,3 +237,8 @@ This will print:
 ```
 
 Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
+
+Comma separated documents
+-----------
+
+`iterate_many` also takes in an option to allow parsing of comma separated documents. In this mode, the entire buffer is processed in 1 batch and batch size will be increased to be as large as the JSON passed. Therefore, the capacity of the parser has to be sufficient to support the batch size set.

doc/ondemand_design.md

@@ -752,7 +752,7 @@ On relevant systems, the On Demand API provides some support for runtime dispatc
 
 Some users wish to run at the best possible speed. Under recent Intel and AMD processors, these users should take additional steps to verify that their code is well optimized.
 
-Given that the On Demand API offer limited runtime dispatching, it matters that your code is compiled against a specific CPU target. You should verify that the code is compiled against the target you expect. Thankfully, the simdjson library will tell you exactly what it detects as an implementation: `haswell` (AVX2 x64 processors), `westmere` (SSE4 x64 processors), `arm64` (64-bit ARM), `ppc64` (64-bit POWER), `fallback` (others). Under x64 processors, many programmers will want to target `haswell` whereas under ARM, most programmers will want to target `arm64` (and it should do so automatically). The `fallback` is probably only good for testing purposes, not for deployment.
+Given that the On Demand API offer limited runtime dispatching, it matters that your code is compiled against a specific CPU target. You should verify that the code is compiled against the target you expect. Thankfully, the simdjson library will tell you exactly what it detects as an implementation: `icelake` (AVX512 x64 processors), `haswell` (AVX2 x64 processors), `westmere` (SSE4 x64 processors), `arm64` (64-bit ARM), `ppc64` (64-bit POWER), `fallback` (others). Under x64 processors, many programmers will want to target `haswell` whereas under ARM, most programmers will want to target `arm64` (and it should do so automatically). The `fallback` is probably only good for testing purposes, not for deployment.
 
 ```C++
   std::cout << simdjson::builtin_implementation()->name() << std::endl;

doc/performance.md

@@ -3,7 +3,10 @@ Performance Notes
 
 simdjson strives to be at its fastest *without tuning*, and generally achieves this. However, there
 are still some scenarios where tuning can enhance performance.
-
+Once your code is tested, we
+further encourage you to define `NDEBUG` in your Release builds to disable additional runtime
+testing and get the best performance.
+* [NDEBUG directive](#ndebug-directive)
 * [Reusing the parser for maximum efficiency](#reusing-the-parser-for-maximum-efficiency)
 * [Reusing string buffers](#reusing-string-buffers)
 * [Server Loops: Long-Running Processes and Memory Capacity](#server-loops-long-running-processes-and-memory-capacity)
@@ -12,6 +15,18 @@ are still some scenarios where tuning can enhance performance.
 * [Visual Studio](#visual-studio)
 * [Power Usage and Downclocking](#power-usage-and-downclocking)
 
+
+NDEBUG directive
+-------------
+
+In C/C++, the `NDEBUG` pre-processor directive is not set by default. When it is not set, the simdjson library does
+many additional checks that may impact negatively the performance. We recommend that, once your code
+is well tested, you define `NDEBUG` directive in your Release builds. The `NDEBUG` directive should be defined
+prior to including the `simdjson.h` header.
+
+The `NDEBUG` directive is generally independent from optimization flags. For example, setting `-O3` under
+GCC does not set the `NDEBUG` directive.
+
 Reusing the parser for maximum efficiency
 -----------------------------------------
 
@@ -138,7 +153,7 @@ few hundred megabytes per second if your JSON documents are densely packed with
 Visual Studio
 --------------
 
-On Intel and AMD Windows platforms, Microsoft Visual Studio enables programmers to build either 32-bit (x86) or 64-bit (x64) binaries. We urge you to always use 64-bit mode. Visual Studio 2019 should default on 64-bit builds when you have a 64-bit version of Windows, which we recommend.
+On Intel and AMD Windows platforms, Microsoft Visual Studio enables programmers to build either 32-bit (x86) or 64-bit (x64 or ARM64) binaries. We urge you to always use 64-bit mode. Visual Studio 2019 should default on 64-bit builds when you have a 64-bit version of Windows, which we recommend.
 
 When compiling with Visual Studio, we recommend the flags `/Ob2 /O2` or better. We do not recommend that you compile simdjson with architecture-specific flags such as  `arch:AVX2`. The simdjson library automatically selects the best execution kernel at runtime.
 
@@ -160,8 +175,7 @@ You should not expect the simdjson library to cause *downclocking* of your recen
 - [Whenever 512-bit AVX-512 instructions are used](https://lemire.me/blog/2018/09/07/avx-512-when-and-how-to-use-these-new-instructions/).
 - Whenever heavy 256-bit or wider instructions are used. Heavy instructions are those involving floating point operations or integer multiplications (since these execute on the floating point unit).
 
-The simdjson library does not make use of heavy 256-bit instructions. We do use vectorized multiplications, but only using 128-bit registers. Thus there should be no downclocking due to simdjson on recent processors, except when AVX-512 is allowed and
-detected. However, we only allow AVX-512 on recent processors (Ice Lake/Tiger Lake or better) where [little to no frequency throttling is expected](https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html). If you can still concerned, you can easily disable AVX-512 with the CMake option `SIMDJSON_AVX512_ALLOWED` set to `OFF` (e.g., `cmake -D SIMDJSON_AVX512_ALLOWED=OFF -B build && cmake --build build`) or by setting
+The simdjson library does not generally make use of heavy 256-bit instructions. On AVX2 kernels, we use vectorized multiplications, but only using 128-bit registers. On recent processors (Ice Lake/Tiger Lake or better, AMD Zen 4 or better) [no frequency throttling is found](https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html) due to SIMD instructions: we are thus more aggressive with SIMD on these machines. If you can still concerned, you can easily disable AVX-512 with the CMake option `SIMDJSON_AVX512_ALLOWED` set to `OFF` (e.g., `cmake -D SIMDJSON_AVX512_ALLOWED=OFF -B build && cmake --build build`) or by setting
 the macro `SIMDJSON_AVX512_ALLOWED` to `0` in C++ prior to importing the headers.
 
 You may still be worried about which SIMD instruction set is used by simdjson.  Thankfully,  [you can always determine and change which architecture-specific implementation is used](implementation-selection.md) by simdjson. Thus even if your CPU supports AVX2, you do not need to use AVX2. You are in control.

include/simdjson/arm64/bitmanipulation.h

@@ -9,6 +9,10 @@ namespace {
 // but the algorithms do not end up using the returned value.
 // Sadly, sanitizers are not smart enough to figure it out.
 SIMDJSON_NO_SANITIZE_UNDEFINED
+// This function can be used safely even if not all bytes have been
+// initialized.
+// See issue https://github.com/simdjson/simdjson/issues/1965
+SIMDJSON_NO_SANITIZE_MEMORY
 simdjson_inline int trailing_zeroes(uint64_t input_num) {
 #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
   unsigned long ret;

include/simdjson/arm64/implementation.h

@@ -12,6 +12,9 @@ using namespace simdjson;
 using namespace simdjson::dom;
 }
 
+/**
+ * @private
+ */
 class implementation final : public simdjson::implementation {
 public:
   simdjson_inline implementation() : simdjson::implementation("arm64", "ARM NEON", internal::instruction_set::NEON) {}

include/simdjson/common_defs.h

@@ -70,7 +70,7 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
 
 #define SIMDJSON_ISALIGNED_N(ptr, n) (((uintptr_t)(ptr) & ((n)-1)) == 0)
 
-#if defined(SIMDJSON_REGULAR_VISUAL_STUDIO)
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
 
   #define simdjson_really_inline __forceinline
   #define simdjson_never_inline __declspec(noinline)
@@ -155,7 +155,7 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
 
   #define SIMDJSON_PRAGMA(P) _Pragma(#P)
   #define SIMDJSON_DISABLE_GCC_WARNING(WARNING) SIMDJSON_PRAGMA(GCC diagnostic ignored #WARNING)
-  #if defined(SIMDJSON_CLANG_VISUAL_STUDIO)
+  #if SIMDJSON_CLANG_VISUAL_STUDIO
   #define SIMDJSON_DISABLE_UNDESIRED_WARNINGS SIMDJSON_DISABLE_GCC_WARNING(-Wmicrosoft-include)
   #else
   #define SIMDJSON_DISABLE_UNDESIRED_WARNINGS
@@ -180,7 +180,7 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
   #define simdjson_inline simdjson_really_inline
 #endif
 
-#if defined(SIMDJSON_VISUAL_STUDIO)
+#if SIMDJSON_VISUAL_STUDIO
     /**
      * Windows users need to do some extra work when building
      * or using a dynamic library (DLL). When building, we need
@@ -322,4 +322,15 @@ namespace std {
 # define simdjson_fallthrough do {} while (0)  /* fallthrough */
 #endif // simdjson_fallthrough
 
+
+#if SIMDJSON_DEVELOPMENT_CHECKS
+#define SIMDJSON_DEVELOPMENT_ASSERT(expr) do { assert ((expr)); } while (0)
+#else
+#define SIMDJSON_DEVELOPMENT_ASSERT(expr) do { } while (0)
+#endif
+
+#ifndef SIMDJSON_UTF8VALIDATION
+#define SIMDJSON_UTF8VALIDATION 1
+#endif
+
 #endif // SIMDJSON_COMMON_DEFS_H

include/simdjson/dom/array-inl.h

@@ -53,18 +53,23 @@ namespace dom {
 simdjson_inline array::array() noexcept : tape{} {}
 simdjson_inline array::array(const internal::tape_ref &_tape) noexcept : tape{_tape} {}
 inline array::iterator array::begin() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   return internal::tape_ref(tape.doc, tape.json_index + 1);
 }
 inline array::iterator array::end() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   return internal::tape_ref(tape.doc, tape.after_element() - 1);
 }
 inline size_t array::size() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   return tape.scope_count();
 }
 inline size_t array::number_of_slots() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   return tape.matching_brace_index() - tape.json_index;
 }
 inline simdjson_result<element> array::at_pointer(std::string_view json_pointer) const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   if(json_pointer.empty()) { // an empty string means that we return the current node
       return element(this->tape); // copy the current node
   } else if(json_pointer[0] != '/') { // otherwise there is an error
@@ -105,6 +110,7 @@ inline simdjson_result<element> array::at_pointer(std::string_view json_pointer)
 }
 
 inline simdjson_result<element> array::at(size_t index) const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   size_t i=0;
   for (auto element : *this) {
     if (i == index) { return element; }

include/simdjson/dom/element-inl.h

@@ -186,11 +186,13 @@ simdjson_inline element::element() noexcept : tape{} {}
 simdjson_inline element::element(const internal::tape_ref &_tape) noexcept : tape{_tape} { }
 
 inline element_type element::type() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   auto tape_type = tape.tape_ref_type();
   return tape_type == internal::tape_type::FALSE_VALUE ? element_type::BOOL : static_cast<element_type>(tape_type);
 }
 
 inline simdjson_result<bool> element::get_bool() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   if(tape.is_true()) {
     return true;
   } else if(tape.is_false()) {
@@ -199,6 +201,7 @@ inline simdjson_result<bool> element::get_bool() const noexcept {
   return INCORRECT_TYPE;
 }
 inline simdjson_result<const char *> element::get_c_str() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   switch (tape.tape_ref_type()) {
     case internal::tape_type::STRING: {
       return tape.get_c_str();
@@ -208,6 +211,7 @@ inline simdjson_result<const char *> element::get_c_str() const noexcept {
   }
 }
 inline simdjson_result<size_t> element::get_string_length() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   switch (tape.tape_ref_type()) {
     case internal::tape_type::STRING: {
       return tape.get_string_length();
@@ -217,6 +221,7 @@ inline simdjson_result<size_t> element::get_string_length() const noexcept {
   }
 }
 inline simdjson_result<std::string_view> element::get_string() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   switch (tape.tape_ref_type()) {
     case internal::tape_type::STRING:
       return tape.get_string_view();
@@ -225,6 +230,7 @@ inline simdjson_result<std::string_view> element::get_string() const noexcept {
   }
 }
 inline simdjson_result<uint64_t> element::get_uint64() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   if(simdjson_unlikely(!tape.is_uint64())) { // branch rarely taken
     if(tape.is_int64()) {
       int64_t result = tape.next_tape_value<int64_t>();
@@ -238,6 +244,7 @@ inline simdjson_result<uint64_t> element::get_uint64() const noexcept {
   return tape.next_tape_value<int64_t>();
 }
 inline simdjson_result<int64_t> element::get_int64() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   if(simdjson_unlikely(!tape.is_int64())) { // branch rarely taken
     if(tape.is_uint64()) {
       uint64_t result = tape.next_tape_value<uint64_t>();
@@ -252,6 +259,7 @@ inline simdjson_result<int64_t> element::get_int64() const noexcept {
   return tape.next_tape_value<int64_t>();
 }
 inline simdjson_result<double> element::get_double() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   // Performance considerations:
   // 1. Querying tape_ref_type() implies doing a shift, it is fast to just do a straight
   //   comparison.
@@ -273,6 +281,7 @@ inline simdjson_result<double> element::get_double() const noexcept {
   return tape.next_tape_value<double>();
 }
 inline simdjson_result<array> element::get_array() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   switch (tape.tape_ref_type()) {
     case internal::tape_type::START_ARRAY:
       return array(tape);
@@ -281,6 +290,7 @@ inline simdjson_result<array> element::get_array() const noexcept {
   }
 }
 inline simdjson_result<object> element::get_object() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   switch (tape.tape_ref_type()) {
     case internal::tape_type::START_OBJECT:
       return object(tape);
@@ -360,6 +370,7 @@ inline simdjson_result<element> element::operator[](const char *key) const noexc
 }
 
 inline simdjson_result<element> element::at_pointer(std::string_view json_pointer) const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   switch (tape.tape_ref_type()) {
     case internal::tape_type::START_OBJECT:
       return object(tape).at_pointer(json_pointer);
@@ -395,6 +406,7 @@ inline simdjson_result<element> element::at_key_case_insensitive(std::string_vie
 }
 
 inline bool element::dump_raw_tape(std::ostream &out) const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   return tape.doc->dump_raw_tape(out);
 }
 

include/simdjson/dom/object-inl.h

@@ -65,12 +65,15 @@ namespace dom {
 simdjson_inline object::object() noexcept : tape{} {}
 simdjson_inline object::object(const internal::tape_ref &_tape) noexcept : tape{_tape} { }
 inline object::iterator object::begin() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   return internal::tape_ref(tape.doc, tape.json_index + 1);
 }
 inline object::iterator object::end() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   return internal::tape_ref(tape.doc, tape.after_element() - 1);
 }
 inline size_t object::size() const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   return tape.scope_count();
 }
 
@@ -81,6 +84,7 @@ inline simdjson_result<element> object::operator[](const char *key) const noexce
   return at_key(key);
 }
 inline simdjson_result<element> object::at_pointer(std::string_view json_pointer) const noexcept {
+  SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   if(json_pointer.empty()) { // an empty string means that we return the current node
       return element(this->tape); // copy the current node
   } else if(json_pointer[0] != '/') { // otherwise there is an error

include/simdjson/dom/parser-inl.h

@@ -43,7 +43,7 @@ inline simdjson_result<size_t> parser::read_file(const std::string &path) noexce
 
   // Get the file size
   int ret;
-#if defined(SIMDJSON_VISUAL_STUDIO) && !SIMDJSON_IS_32BITS
+#if SIMDJSON_VISUAL_STUDIO && !SIMDJSON_IS_32BITS
   ret = _fseeki64(fp, 0, SEEK_END);
 #else
   ret = std::fseek(fp, 0, SEEK_END);
@@ -52,7 +52,7 @@ inline simdjson_result<size_t> parser::read_file(const std::string &path) noexce
     std::fclose(fp);
     return IO_ERROR;
   }
-#if defined(SIMDJSON_VISUAL_STUDIO) && !SIMDJSON_IS_32BITS
+#if SIMDJSON_VISUAL_STUDIO && !SIMDJSON_IS_32BITS
   __int64 len = _ftelli64(fp);
   if(len == -1L) {
     std::fclose(fp);
@@ -148,6 +148,9 @@ simdjson_inline simdjson_result<element> parser::parse(const std::string &s) & n
 simdjson_inline simdjson_result<element> parser::parse(const padded_string &s) & noexcept {
   return parse(s.data(), s.length(), false);
 }
+simdjson_inline simdjson_result<element> parser::parse(const padded_string_view &v) & noexcept {
+  return parse(v.data(), v.length(), false);
+}
 
 inline simdjson_result<document_stream> parser::parse_many(const uint8_t *buf, size_t len, size_t batch_size) noexcept {
   if(batch_size < MINIMAL_BATCH_SIZE) { batch_size = MINIMAL_BATCH_SIZE; }
@@ -223,7 +226,7 @@ inline error_code parser::ensure_capacity(document& target_document, size_t desi
 }
 
 simdjson_inline void parser::set_max_capacity(size_t max_capacity) noexcept {
-  if(max_capacity < MINIMAL_DOCUMENT_CAPACITY) {
+  if(max_capacity > MINIMAL_DOCUMENT_CAPACITY) {
     _max_capacity = max_capacity;
   } else {
     _max_capacity = MINIMAL_DOCUMENT_CAPACITY;

include/simdjson/dom/parser.h

@@ -145,7 +145,9 @@ public:
    * ### REQUIRED: Buffer Padding
    *
    * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
-   * those bytes are initialized to, as long as they are allocated.
+   * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
+   * using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
+   * SIMDJSON_PADDING bytes to avoid runtime warnings.
    *
    * If realloc_if_needed is true (the default), it is assumed that the buffer does *not* have enough padding,
    * and it is copied into an enlarged temporary buffer before parsing. Thus the following is safe:
@@ -199,6 +201,9 @@ public:
   /** @overload parse(const uint8_t *buf, size_t len, bool realloc_if_needed) */
   simdjson_inline simdjson_result<element> parse(const padded_string &s) & noexcept;
   simdjson_inline simdjson_result<element> parse(const padded_string &s) && =delete;
+  /** @overload parse(const uint8_t *buf, size_t len, bool realloc_if_needed) */
+  simdjson_inline simdjson_result<element> parse(const padded_string_view &v) & noexcept;
+  simdjson_inline simdjson_result<element> parse(const padded_string_view &v) && =delete;
 
   /** @private We do not want to allow implicit conversion from C string to std::string. */
   simdjson_inline simdjson_result<element> parse(const char *buf) noexcept = delete;
@@ -405,7 +410,9 @@ public:
    * ### REQUIRED: Buffer Padding
    *
    * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
-   * those bytes are initialized to, as long as they are allocated.
+   * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
+   * using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
+   * SIMDJSON_PADDING bytes to avoid runtime warnings.
    *
    * ### Threads
    *

include/simdjson/fallback/implementation.h

@@ -11,6 +11,9 @@ using namespace simdjson;
 using namespace simdjson::dom;
 }
 
+/**
+ * @private
+ */
 class implementation final : public simdjson::implementation {
 public:
   simdjson_inline implementation() : simdjson::implementation(

include/simdjson/generic/dom_parser_implementation.h

@@ -35,7 +35,8 @@ public:
   simdjson_warn_unused error_code stage1(const uint8_t *buf, size_t len, stage1_mode partial) noexcept final;
   simdjson_warn_unused error_code stage2(dom::document &doc) noexcept final;
   simdjson_warn_unused error_code stage2_next(dom::document &doc) noexcept final;
-  simdjson_warn_unused uint8_t *parse_string(const uint8_t *src, uint8_t *dst) const noexcept final;
+  simdjson_warn_unused uint8_t *parse_string(const uint8_t *src, uint8_t *dst, bool allow_replacement) const noexcept final;
+  simdjson_warn_unused uint8_t *parse_wobbly_string(const uint8_t *src, uint8_t *dst) const noexcept final;
   inline simdjson_warn_unused error_code set_capacity(size_t capacity) noexcept final;
   inline simdjson_warn_unused error_code set_max_depth(size_t max_depth) noexcept final;
 private:

include/simdjson/generic/jsoncharutils.h

@@ -72,7 +72,7 @@ simdjson_inline size_t codepoint_to_utf8(uint32_t cp, uint8_t *c) {
   return 0; // bad r
 }
 
-#ifdef SIMDJSON_IS_32BITS // _umul128 for x86, arm
+#if SIMDJSON_IS_32BITS // _umul128 for x86, arm
 // this is a slow emulation routine for 32-bit
 //
 static simdjson_inline uint64_t __emulu(uint32_t x, uint32_t y) {
@@ -94,7 +94,7 @@ using internal::value128;
 
 simdjson_inline value128 full_multiplication(uint64_t value1, uint64_t value2) {
   value128 answer;
-#if defined(SIMDJSON_REGULAR_VISUAL_STUDIO) || defined(SIMDJSON_IS_32BITS)
+#if SIMDJSON_REGULAR_VISUAL_STUDIO || SIMDJSON_IS_32BITS
 #ifdef _M_ARM64
   // ARM64 has native support for 64-bit multiplications, no need to emultate
   answer.high = __umulh(value1, value2);
@@ -102,7 +102,7 @@ simdjson_inline value128 full_multiplication(uint64_t value1, uint64_t value2) {
 #else
   answer.low = _umul128(value1, value2, &answer.high); // _umul128 not available on ARM64
 #endif // _M_ARM64
-#else // defined(SIMDJSON_REGULAR_VISUAL_STUDIO) || defined(SIMDJSON_IS_32BITS)
+#else // SIMDJSON_REGULAR_VISUAL_STUDIO || SIMDJSON_IS_32BITS
   __uint128_t r = (static_cast<__uint128_t>(value1)) * value2;
   answer.low = uint64_t(r);
   answer.high = uint64_t(r >> 64);

include/simdjson/generic/ondemand/array-inl.h

@@ -205,4 +205,8 @@ simdjson_inline  simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> simdj
   if (error()) { return error(); }
   return first.at_pointer(json_pointer);
 }
+simdjson_inline  simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::array>::raw_json() noexcept {
+  if (error()) { return error(); }
+  return first.raw_json();
+}
 } // namespace simdjson

include/simdjson/generic/ondemand/array.h

@@ -186,6 +186,8 @@ public:
   inline simdjson_result<bool> reset() & noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> at(size_t index) noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> at_pointer(std::string_view json_pointer) noexcept;
+  simdjson_inline simdjson_result<std::string_view> raw_json() noexcept;
+
 };
 
 } // namespace simdjson

include/simdjson/generic/ondemand/document-inl.h

@@ -20,7 +20,7 @@ inline std::string document::to_debug_string() noexcept {
   return iter.to_string();
 }
 
-inline simdjson_result<const char *> document::current_location() noexcept {
+inline simdjson_result<const char *> document::current_location() const noexcept {
   return iter.current_location();
 }
 
@@ -28,6 +28,11 @@ inline int32_t document::current_depth() const noexcept {
   return iter.depth();
 }
 
+inline bool document::at_end() const noexcept {
+  return iter.at_end();
+}
+
+
 inline bool document::is_alive() noexcept {
   return iter.is_alive();
 }
@@ -49,14 +54,24 @@ simdjson_inline simdjson_result<value> document::get_value() noexcept {
   // gets called.
   iter.assert_at_document_depth();
   switch (*iter.peek()) {
-    case '[':
-    case '{':
+    case '[': {
+      // The following lines check that the document ends with ].
+      auto value_iterator = get_root_value_iterator();
+      auto error = value_iterator.check_root_array();
+      if(error) { return error; }
       return value(get_root_value_iterator());
+    }
+    case '{': {
+      // The following lines would check that the document ends with }.
+      auto value_iterator = get_root_value_iterator();
+      auto error = value_iterator.check_root_object();
+      if(error) { return error; }
+      return value(get_root_value_iterator());
+    }
     default:
       // Unfortunately, scalar documents are a special case in simdjson and they cannot
       // be safely converted to value instances.
       return SCALAR_DOCUMENT_AS_VALUE;
-      // return value(get_root_value_iterator());
   }
 }
 simdjson_inline simdjson_result<array> document::get_array() & noexcept {
@@ -67,41 +82,53 @@ simdjson_inline simdjson_result<object> document::get_object() & noexcept {
   auto value = get_root_value_iterator();
   return object::start_root(value);
 }
+
+/**
+ * We decided that calling 'get_double()' on the JSON document '1.233 blabla' should
+ * give an error, so we check for trailing content. We want to disallow trailing
+ * content.
+ * Thus, in several implementations below, we pass a 'true' parameter value to
+ * a get_root_value_iterator() method: this indicates that we disallow trailing content.
+ */
+
 simdjson_inline simdjson_result<uint64_t> document::get_uint64() noexcept {
-  return get_root_value_iterator().get_root_uint64();
+  return get_root_value_iterator().get_root_uint64(true);
 }
 simdjson_inline simdjson_result<uint64_t> document::get_uint64_in_string() noexcept {
-  return get_root_value_iterator().get_root_uint64_in_string();
+  return get_root_value_iterator().get_root_uint64_in_string(true);
 }
 simdjson_inline simdjson_result<int64_t> document::get_int64() noexcept {
-  return get_root_value_iterator().get_root_int64();
+  return get_root_value_iterator().get_root_int64(true);
 }
 simdjson_inline simdjson_result<int64_t> document::get_int64_in_string() noexcept {
-  return get_root_value_iterator().get_root_int64_in_string();
+  return get_root_value_iterator().get_root_int64_in_string(true);
 }
 simdjson_inline simdjson_result<double> document::get_double() noexcept {
-  return get_root_value_iterator().get_root_double();
+  return get_root_value_iterator().get_root_double(true);
 }
 simdjson_inline simdjson_result<double> document::get_double_in_string() noexcept {
-  return get_root_value_iterator().get_root_double_in_string();
+  return get_root_value_iterator().get_root_double_in_string(true);
 }
-simdjson_inline simdjson_result<std::string_view> document::get_string() noexcept {
-  return get_root_value_iterator().get_root_string();
+simdjson_inline simdjson_result<std::string_view> document::get_string(bool allow_replacement) noexcept {
+  return get_root_value_iterator().get_root_string(true, allow_replacement);
+}
+simdjson_inline simdjson_result<std::string_view> document::get_wobbly_string() noexcept {
+  return get_root_value_iterator().get_root_wobbly_string(true);
 }
 simdjson_inline simdjson_result<raw_json_string> document::get_raw_json_string() noexcept {
-  return get_root_value_iterator().get_root_raw_json_string();
+  return get_root_value_iterator().get_root_raw_json_string(true);
 }
 simdjson_inline simdjson_result<bool> document::get_bool() noexcept {
-  return get_root_value_iterator().get_root_bool();
+  return get_root_value_iterator().get_root_bool(true);
 }
-simdjson_inline bool document::is_null() noexcept {
-  return get_root_value_iterator().is_root_null();
+simdjson_inline simdjson_result<bool> document::is_null() noexcept {
+  return get_root_value_iterator().is_root_null(true);
 }
 
 template<> simdjson_inline simdjson_result<array> document::get() & noexcept { return get_array(); }
 template<> simdjson_inline simdjson_result<object> document::get() & noexcept { return get_object(); }
 template<> simdjson_inline simdjson_result<raw_json_string> document::get() & noexcept { return get_raw_json_string(); }
-template<> simdjson_inline simdjson_result<std::string_view> document::get() & noexcept { return get_string(); }
+template<> simdjson_inline simdjson_result<std::string_view> document::get() & noexcept { return get_string(false); }
 template<> simdjson_inline simdjson_result<double> document::get() & noexcept { return get_double(); }
 template<> simdjson_inline simdjson_result<uint64_t> document::get() & noexcept { return get_uint64(); }
 template<> simdjson_inline simdjson_result<int64_t> document::get() & noexcept { return get_int64(); }
@@ -109,7 +136,7 @@ template<> simdjson_inline simdjson_result<bool> document::get() & noexcept { re
 template<> simdjson_inline simdjson_result<value> document::get() & noexcept { return get_value(); }
 
 template<> simdjson_inline simdjson_result<raw_json_string> document::get() && noexcept { return get_raw_json_string(); }
-template<> simdjson_inline simdjson_result<std::string_view> document::get() && noexcept { return get_string(); }
+template<> simdjson_inline simdjson_result<std::string_view> document::get() && noexcept { return get_string(false); }
 template<> simdjson_inline simdjson_result<double> document::get() && noexcept { return std::forward<document>(*this).get_double(); }
 template<> simdjson_inline simdjson_result<uint64_t> document::get() && noexcept { return std::forward<document>(*this).get_uint64(); }
 template<> simdjson_inline simdjson_result<int64_t> document::get() && noexcept { return std::forward<document>(*this).get_int64(); }
@@ -129,7 +156,7 @@ simdjson_inline document::operator object() & noexcept(false) { return get_objec
 simdjson_inline document::operator uint64_t() noexcept(false) { return get_uint64(); }
 simdjson_inline document::operator int64_t() noexcept(false) { return get_int64(); }
 simdjson_inline document::operator double() noexcept(false) { return get_double(); }
-simdjson_inline document::operator std::string_view() noexcept(false) { return get_string(); }
+simdjson_inline document::operator std::string_view() noexcept(false) { return get_string(false); }
 simdjson_inline document::operator raw_json_string() noexcept(false) { return get_raw_json_string(); }
 simdjson_inline document::operator bool() noexcept(false) { return get_bool(); }
 simdjson_inline document::operator value() noexcept(false) { return get_value(); }
@@ -213,15 +240,15 @@ simdjson_inline bool document::is_negative() noexcept {
 }
 
 simdjson_inline simdjson_result<bool> document::is_integer() noexcept {
-  return get_root_value_iterator().is_root_integer();
+  return get_root_value_iterator().is_root_integer(true);
 }
 
 simdjson_inline simdjson_result<number_type> document::get_number_type() noexcept {
-  return get_root_value_iterator().get_root_number_type();
+  return get_root_value_iterator().get_root_number_type(true);
 }
 
 simdjson_inline simdjson_result<number> document::get_number() noexcept {
-  return get_root_value_iterator().get_root_number();
+  return get_root_value_iterator().get_root_number(true);
 }
 
 
@@ -330,17 +357,33 @@ simdjson_inline simdjson_result<uint64_t> simdjson_result<SIMDJSON_IMPLEMENTATIO
   if (error()) { return error(); }
   return first.get_uint64();
 }
+simdjson_inline simdjson_result<uint64_t> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_uint64_in_string() noexcept {
+  if (error()) { return error(); }
+  return first.get_uint64_in_string();
+}
 simdjson_inline simdjson_result<int64_t> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_int64() noexcept {
   if (error()) { return error(); }
   return first.get_int64();
 }
+simdjson_inline simdjson_result<int64_t> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_int64_in_string() noexcept {
+  if (error()) { return error(); }
+  return first.get_int64_in_string();
+}
 simdjson_inline simdjson_result<double> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_double() noexcept {
   if (error()) { return error(); }
   return first.get_double();
 }
-simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_string() noexcept {
+simdjson_inline simdjson_result<double> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_double_in_string() noexcept {
   if (error()) { return error(); }
-  return first.get_string();
+  return first.get_double_in_string();
+}
+simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_string(bool allow_replacement) noexcept {
+  if (error()) { return error(); }
+  return first.get_string(allow_replacement);
+}
+simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_wobbly_string() noexcept {
+  if (error()) { return error(); }
+  return first.get_wobbly_string();
 }
 simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get_raw_json_string() noexcept {
   if (error()) { return error(); }
@@ -354,7 +397,7 @@ simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> simdjs
   if (error()) { return error(); }
   return first.get_value();
 }
-simdjson_inline bool simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::is_null() noexcept {
+simdjson_inline simdjson_result<bool> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::is_null() noexcept {
   if (error()) { return error(); }
   return first.is_null();
 }
@@ -469,6 +512,12 @@ simdjson_inline simdjson_result<const char *> simdjson_result<SIMDJSON_IMPLEMENT
   return first.current_location();
 }
 
+simdjson_inline bool simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::at_end() const noexcept {
+  if (error()) { return error(); }
+  return first.at_end();
+}
+
+
 simdjson_inline int32_t simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::current_depth() const noexcept {
   if (error()) { return error(); }
   return first.current_depth();
@@ -497,24 +546,42 @@ simdjson_inline document_reference::document_reference(document &d) noexcept : d
 simdjson_inline void document_reference::rewind() noexcept { doc->rewind(); }
 simdjson_inline simdjson_result<array> document_reference::get_array() & noexcept { return doc->get_array(); }
 simdjson_inline simdjson_result<object> document_reference::get_object() & noexcept { return doc->get_object(); }
-simdjson_inline simdjson_result<uint64_t> document_reference::get_uint64() noexcept { return doc->get_uint64(); }
-simdjson_inline simdjson_result<int64_t> document_reference::get_int64() noexcept { return doc->get_int64(); }
-simdjson_inline simdjson_result<double> document_reference::get_double() noexcept { return doc->get_double(); }
-simdjson_inline simdjson_result<std::string_view> document_reference::get_string() noexcept { return doc->get_string(); }
-simdjson_inline simdjson_result<raw_json_string> document_reference::get_raw_json_string() noexcept { return doc->get_raw_json_string(); }
-simdjson_inline simdjson_result<bool> document_reference::get_bool() noexcept { return doc->get_bool(); }
+/**
+ * The document_reference instances are used primarily/solely for streams of JSON
+ * documents.
+ * We decided that calling 'get_double()' on the JSON document '1.233 blabla' should
+ * give an error, so we check for trailing content.
+ *
+ * However, for streams of JSON documents, we want to be able to start from
+ * "321" "321" "321"
+ * and parse it successfully as a stream of JSON documents, calling get_uint64_in_string()
+ * successfully each time.
+ *
+ * To achieve this result, we pass a 'false' to a get_root_value_iterator() method:
+ * this indicates that we allow trailing content.
+ */
+simdjson_inline simdjson_result<uint64_t> document_reference::get_uint64() noexcept { return doc->get_root_value_iterator().get_root_uint64(false); }
+simdjson_inline simdjson_result<uint64_t> document_reference::get_uint64_in_string() noexcept { return doc->get_root_value_iterator().get_root_uint64_in_string(false); }
+simdjson_inline simdjson_result<int64_t> document_reference::get_int64() noexcept { return doc->get_root_value_iterator().get_root_int64(false); }
+simdjson_inline simdjson_result<int64_t> document_reference::get_int64_in_string() noexcept { return doc->get_root_value_iterator().get_root_int64_in_string(false); }
+simdjson_inline simdjson_result<double> document_reference::get_double() noexcept { return doc->get_root_value_iterator().get_root_double(false); }
+simdjson_inline simdjson_result<double> document_reference::get_double_in_string() noexcept { return doc->get_root_value_iterator().get_root_double(false); }
+simdjson_inline simdjson_result<std::string_view> document_reference::get_string(bool allow_replacement) noexcept { return doc->get_root_value_iterator().get_root_string(false, allow_replacement); }
+simdjson_inline simdjson_result<std::string_view> document_reference::get_wobbly_string() noexcept { return doc->get_root_value_iterator().get_root_wobbly_string(false); }
+simdjson_inline simdjson_result<raw_json_string> document_reference::get_raw_json_string() noexcept { return doc->get_root_value_iterator().get_root_raw_json_string(false); }
+simdjson_inline simdjson_result<bool> document_reference::get_bool() noexcept { return doc->get_root_value_iterator().get_root_bool(false); }
 simdjson_inline simdjson_result<value> document_reference::get_value() noexcept { return doc->get_value(); }
-simdjson_inline bool document_reference::is_null() noexcept { return doc->is_null(); }
+simdjson_inline simdjson_result<bool> document_reference::is_null() noexcept { return doc->get_root_value_iterator().is_root_null(false); }
 
 #if SIMDJSON_EXCEPTIONS
 simdjson_inline document_reference::operator array() & noexcept(false) { return array(*doc); }
 simdjson_inline document_reference::operator object() & noexcept(false) { return object(*doc); }
-simdjson_inline document_reference::operator uint64_t() noexcept(false) { return uint64_t(*doc); }
-simdjson_inline document_reference::operator int64_t() noexcept(false) { return int64_t(*doc); }
-simdjson_inline document_reference::operator double() noexcept(false) { return double(*doc); }
+simdjson_inline document_reference::operator uint64_t() noexcept(false) { return get_uint64(); }
+simdjson_inline document_reference::operator int64_t() noexcept(false) { return get_int64(); }
+simdjson_inline document_reference::operator double() noexcept(false) { return get_double(); }
 simdjson_inline document_reference::operator std::string_view() noexcept(false) { return std::string_view(*doc); }
 simdjson_inline document_reference::operator raw_json_string() noexcept(false) { return raw_json_string(*doc); }
-simdjson_inline document_reference::operator bool() noexcept(false) { return bool(*doc); }
+simdjson_inline document_reference::operator bool() noexcept(false) { return get_bool(); }
 simdjson_inline document_reference::operator value() noexcept(false) { return value(*doc); }
 #endif
 simdjson_inline simdjson_result<size_t> document_reference::count_elements() & noexcept { return doc->count_elements(); }
@@ -533,9 +600,9 @@ simdjson_inline simdjson_result<bool> document_reference::is_scalar() noexcept {
 simdjson_inline simdjson_result<const char *> document_reference::current_location() noexcept { return doc->current_location(); }
 simdjson_inline int32_t document_reference::current_depth() const noexcept { return doc->current_depth(); }
 simdjson_inline bool document_reference::is_negative() noexcept { return doc->is_negative(); }
-simdjson_inline simdjson_result<bool> document_reference::is_integer() noexcept { return doc->is_integer(); }
-simdjson_inline simdjson_result<number_type> document_reference::get_number_type() noexcept { return doc->get_number_type(); }
-simdjson_inline simdjson_result<number> document_reference::get_number() noexcept { return doc->get_number(); }
+simdjson_inline simdjson_result<bool> document_reference::is_integer() noexcept { return doc->get_root_value_iterator().is_root_integer(false); }
+simdjson_inline simdjson_result<number_type> document_reference::get_number_type() noexcept { return doc->get_root_value_iterator().get_root_number_type(false); }
+simdjson_inline simdjson_result<number> document_reference::get_number() noexcept { return doc->get_root_value_iterator().get_root_number(false); }
 simdjson_inline simdjson_result<std::string_view> document_reference::raw_json_token() noexcept { return doc->raw_json_token(); }
 simdjson_inline simdjson_result<value> document_reference::at_pointer(std::string_view json_pointer) noexcept { return doc->at_pointer(json_pointer); }
 simdjson_inline simdjson_result<std::string_view> document_reference::raw_json() noexcept { return doc->raw_json();}
@@ -612,17 +679,33 @@ simdjson_inline simdjson_result<uint64_t> simdjson_result<SIMDJSON_IMPLEMENTATIO
   if (error()) { return error(); }
   return first.get_uint64();
 }
+simdjson_inline simdjson_result<uint64_t> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_uint64_in_string() noexcept {
+  if (error()) { return error(); }
+  return first.get_uint64_in_string();
+}
 simdjson_inline simdjson_result<int64_t> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_int64() noexcept {
   if (error()) { return error(); }
   return first.get_int64();
 }
+simdjson_inline simdjson_result<int64_t> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_int64_in_string() noexcept {
+  if (error()) { return error(); }
+  return first.get_int64_in_string();
+}
 simdjson_inline simdjson_result<double> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_double() noexcept {
   if (error()) { return error(); }
   return first.get_double();
 }
-simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_string() noexcept {
+simdjson_inline simdjson_result<double> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_double_in_string() noexcept {
   if (error()) { return error(); }
-  return first.get_string();
+  return first.get_double_in_string();
+}
+simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_string(bool allow_replacement) noexcept {
+  if (error()) { return error(); }
+  return first.get_string(allow_replacement);
+}
+simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_wobbly_string() noexcept {
+  if (error()) { return error(); }
+  return first.get_wobbly_string();
 }
 simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::get_raw_json_string() noexcept {
   if (error()) { return error(); }
@@ -636,7 +719,7 @@ simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> simdjs
   if (error()) { return error(); }
   return first.get_value();
 }
-simdjson_inline bool simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::is_null() noexcept {
+simdjson_inline simdjson_result<bool> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::is_null() noexcept {
   if (error()) { return error(); }
   return first.is_null();
 }
@@ -648,7 +731,7 @@ simdjson_inline simdjson_result<bool> simdjson_result<SIMDJSON_IMPLEMENTATION::o
   if (error()) { return error(); }
   return first.is_scalar();
 }
-simdjson_inline bool simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::is_negative() noexcept {
+simdjson_inline simdjson_result<bool> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document_reference>::is_negative() noexcept {
   if (error()) { return error(); }
   return first.is_negative();
 }

include/simdjson/generic/ondemand/document.h

@@ -96,11 +96,24 @@ public:
    *
    * Important: Calling get_string() twice on the same document is an error.
    *
+   * @param Whether to allow a replacement character for unmatched surrogate pairs.
    * @returns An UTF-8 string. The string is stored in the parser and will be invalidated the next
    *          time it parses a document or when it is destroyed.
    * @returns INCORRECT_TYPE if the JSON value is not a string.
    */
-  simdjson_inline simdjson_result<std::string_view> get_string() noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_string(bool allow_replacement = false) noexcept;
+  /**
+   * Cast this JSON value to a string.
+   *
+   * The string is not guaranteed to be valid UTF-8. See https://simonsapin.github.io/wtf-8/
+   *
+   * Important: Calling get_wobbly_string() twice on the same document is an error.
+   *
+   * @returns An UTF-8 string. The string is stored in the parser and will be invalidated the next
+   *          time it parses a document or when it is destroyed.
+   * @returns INCORRECT_TYPE if the JSON value is not a string.
+   */
+  simdjson_inline simdjson_result<std::string_view> get_wobbly_string() noexcept;
   /**
    * Cast this JSON value to a raw_json_string.
    *
@@ -126,11 +139,14 @@ public:
   simdjson_inline simdjson_result<value> get_value() noexcept;
 
   /**
-   * Checks if this JSON value is null.
+   * Checks if this JSON value is null.  If and only if the value is
+   * null, then it is consumed (we advance). If we find a token that
+   * begins with 'n' but is not 'null', then an error is returned.
    *
    * @returns Whether the value is null.
+   * @returns INCORRECT_TYPE If the JSON value begins with 'n' and is not 'null'.
    */
-  simdjson_inline bool is_null() noexcept;
+  simdjson_inline simdjson_result<bool> is_null() noexcept;
 
   /**
    * Get this value as the given type.
@@ -363,7 +379,9 @@ public:
   simdjson_inline simdjson_result<value> operator[](const char *key) & noexcept;
 
   /**
-   * Get the type of this JSON value.
+   * Get the type of this JSON value. It does not validate or consume the value.
+   * E.g., you must still call "is_null()" to check that a value is null even if
+   * "type()" returns json_type::null.
    *
    * NOTE: If you're only expecting a value to be one type (a typical case), it's generally
    * better to just call .get_double, .get_string, etc. and check for INCORRECT_TYPE (or just
@@ -493,7 +511,14 @@ public:
   /**
    * Returns the current location in the document if in bounds.
    */
-  inline simdjson_result<const char *> current_location() noexcept;
+  inline simdjson_result<const char *> current_location() const noexcept;
+
+  /**
+   * Returns true if this document has been fully parsed.
+   * If you have consumed the whole document and at_end() returns
+   * false, then there may be trailing content.
+   */
+  inline bool at_end() const noexcept;
 
   /**
    * Returns the current depth in the document if in bounds.
@@ -572,6 +597,7 @@ protected:
   friend class field;
   friend class token;
   friend class document_stream;
+  friend class document_reference;
 };
 
 
@@ -588,14 +614,18 @@ public:
   simdjson_inline simdjson_result<array> get_array() & noexcept;
   simdjson_inline simdjson_result<object> get_object() & noexcept;
   simdjson_inline simdjson_result<uint64_t> get_uint64() noexcept;
+  simdjson_inline simdjson_result<uint64_t> get_uint64_in_string() noexcept;
   simdjson_inline simdjson_result<int64_t> get_int64() noexcept;
+  simdjson_inline simdjson_result<int64_t> get_int64_in_string() noexcept;
   simdjson_inline simdjson_result<double> get_double() noexcept;
-  simdjson_inline simdjson_result<std::string_view> get_string() noexcept;
+  simdjson_inline simdjson_result<double> get_double_in_string() noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_string(bool allow_replacement = false) noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_wobbly_string() noexcept;
   simdjson_inline simdjson_result<raw_json_string> get_raw_json_string() noexcept;
   simdjson_inline simdjson_result<bool> get_bool() noexcept;
   simdjson_inline simdjson_result<value> get_value() noexcept;
 
-  simdjson_inline bool is_null() noexcept;
+  simdjson_inline simdjson_result<bool> is_null() noexcept;
   simdjson_inline simdjson_result<std::string_view> raw_json() noexcept;
   simdjson_inline operator document&() const noexcept;
 
@@ -653,14 +683,17 @@ public:
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::array> get_array() & noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::object> get_object() & noexcept;
   simdjson_inline simdjson_result<uint64_t> get_uint64() noexcept;
+  simdjson_inline simdjson_result<uint64_t> get_uint64_in_string() noexcept;
   simdjson_inline simdjson_result<int64_t> get_int64() noexcept;
+  simdjson_inline simdjson_result<int64_t> get_int64_in_string() noexcept;
   simdjson_inline simdjson_result<double> get_double() noexcept;
-  simdjson_inline simdjson_result<double> get_double_from_string() noexcept;
-  simdjson_inline simdjson_result<std::string_view> get_string() noexcept;
+  simdjson_inline simdjson_result<double> get_double_in_string() noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_string(bool allow_replacement = false) noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_wobbly_string() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string> get_raw_json_string() noexcept;
   simdjson_inline simdjson_result<bool> get_bool() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> get_value() noexcept;
-  simdjson_inline bool is_null() noexcept;
+  simdjson_inline simdjson_result<bool> is_null() noexcept;
 
   template<typename T> simdjson_inline simdjson_result<T> get() & noexcept;
   template<typename T> simdjson_inline simdjson_result<T> get() && noexcept;
@@ -694,6 +727,7 @@ public:
   simdjson_inline simdjson_result<bool> is_scalar() noexcept;
   simdjson_inline simdjson_result<const char *> current_location() noexcept;
   simdjson_inline int32_t current_depth() const noexcept;
+  simdjson_inline bool at_end() const noexcept;
   simdjson_inline bool is_negative() noexcept;
   simdjson_inline simdjson_result<bool> is_integer() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::number_type> get_number_type() noexcept;
@@ -721,13 +755,17 @@ public:
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::array> get_array() & noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::object> get_object() & noexcept;
   simdjson_inline simdjson_result<uint64_t> get_uint64() noexcept;
+  simdjson_inline simdjson_result<uint64_t> get_uint64_in_string() noexcept;
   simdjson_inline simdjson_result<int64_t> get_int64() noexcept;
+  simdjson_inline simdjson_result<int64_t> get_int64_in_string() noexcept;
   simdjson_inline simdjson_result<double> get_double() noexcept;
-  simdjson_inline simdjson_result<std::string_view> get_string() noexcept;
+  simdjson_inline simdjson_result<double> get_double_in_string() noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_string(bool allow_replacement = false) noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_wobbly_string() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string> get_raw_json_string() noexcept;
   simdjson_inline simdjson_result<bool> get_bool() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> get_value() noexcept;
-  simdjson_inline bool is_null() noexcept;
+  simdjson_inline simdjson_result<bool> is_null() noexcept;
 
 #if SIMDJSON_EXCEPTIONS
   simdjson_inline operator SIMDJSON_IMPLEMENTATION::ondemand::array() & noexcept(false);
@@ -754,8 +792,8 @@ public:
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::json_type> type() noexcept;
   simdjson_inline simdjson_result<bool> is_scalar() noexcept;
   simdjson_inline simdjson_result<const char *> current_location() noexcept;
-  simdjson_inline int32_t current_depth() const noexcept;
-  simdjson_inline bool is_negative() noexcept;
+  simdjson_inline simdjson_result<int32_t> current_depth() const noexcept;
+  simdjson_inline simdjson_result<bool> is_negative() noexcept;
   simdjson_inline simdjson_result<bool> is_integer() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::number_type> get_number_type() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::number> get_number() noexcept;

include/simdjson/generic/ondemand/document_stream-inl.h

@@ -84,12 +84,14 @@ simdjson_inline document_stream::document_stream(
   ondemand::parser &_parser,
   const uint8_t *_buf,
   size_t _len,
-  size_t _batch_size
+  size_t _batch_size,
+  bool _allow_comma_separated
 ) noexcept
   : parser{&_parser},
     buf{_buf},
     len{_len},
     batch_size{_batch_size <= MINIMAL_BATCH_SIZE ? MINIMAL_BATCH_SIZE : _batch_size},
+    allow_comma_separated{_allow_comma_separated},
     error{SUCCESS}
     #ifdef SIMDJSON_THREADS_ENABLED
     , use_thread(_parser.threaded) // we need to make a copy because _parser.threaded can change
@@ -107,6 +109,7 @@ simdjson_inline document_stream::document_stream() noexcept
     buf{nullptr},
     len{0},
     batch_size{0},
+    allow_comma_separated{false},
     error{UNINITIALIZED}
     #ifdef SIMDJSON_THREADS_ENABLED
     , use_thread(false)
@@ -290,6 +293,8 @@ inline void document_stream::next_document() noexcept {
   if (error) { return; }
   // Always set depth=1 at the start of document
   doc.iter._depth = 1;
+  // consume comma if comma separated is allowed
+  if (allow_comma_separated) { doc.iter.consume_character(','); }
   // Resets the string buffer at the beginning, thus invalidating the strings.
   doc.iter._string_buf_loc = parser->string_buf.get();
   doc.iter._root = doc.iter.position();

include/simdjson/generic/ondemand/document_stream.h

@@ -222,7 +222,8 @@ private:
     ondemand::parser &parser,
     const uint8_t *buf,
     size_t len,
-    size_t batch_size
+    size_t batch_size,
+    bool allow_comma_separated
   ) noexcept;
 
   /**
@@ -271,6 +272,7 @@ private:
   const uint8_t *buf;
   size_t len;
   size_t batch_size;
+  bool allow_comma_separated;
   /**
    * We are going to use just one document instance. The document owns
    * the json_iterator. It implies that we only ever pass a reference

include/simdjson/generic/ondemand/field-inl.h

@@ -21,9 +21,9 @@ simdjson_inline simdjson_result<field> field::start(const value_iterator &parent
     return field(key, parent_iter.child());
 }
 
-simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> field::unescaped_key() noexcept {
+simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> field::unescaped_key(bool allow_replacement) noexcept {
   SIMDJSON_ASSUME(first.buf != nullptr); // We would like to call .alive() but Visual Studio won't let us.
-  simdjson_result<std::string_view> answer = first.unescape(second.iter.json_iter());
+  simdjson_result<std::string_view> answer = first.unescape(second.iter.json_iter(), allow_replacement);
   first.consume();
   return answer;
 }
@@ -66,9 +66,9 @@ simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_stri
   if (error()) { return error(); }
   return first.key();
 }
-simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::field>::unescaped_key() noexcept {
+simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::field>::unescaped_key(bool allow_replacement) noexcept {
   if (error()) { return error(); }
-  return first.unescaped_key();
+  return first.unescaped_key(allow_replacement);
 }
 simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::field>::value() noexcept {
   if (error()) { return error(); }

include/simdjson/generic/ondemand/field.h

@@ -28,7 +28,7 @@ public:
    * This consumes the key: once you have called unescaped_key(), you cannot
    * call it again nor can you call key().
    */
-  simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> unescaped_key() noexcept;
+  simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> unescaped_key(bool allow_replacement) noexcept;
   /**
    * Get the key as a raw_json_string. Can be used for direct comparison with
    * an unescaped C string: e.g., key() == "test".
@@ -64,7 +64,7 @@ public:
   simdjson_inline simdjson_result(error_code error) noexcept; ///< @private
   simdjson_inline simdjson_result() noexcept = default;
 
-  simdjson_inline simdjson_result<std::string_view> unescaped_key() noexcept;
+  simdjson_inline simdjson_result<std::string_view> unescaped_key(bool allow_replacement = false) noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string> key() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> value() noexcept;
 };

include/simdjson/generic/ondemand/json_iterator-inl.h

@@ -208,7 +208,7 @@ inline std::string json_iterator::to_string() const noexcept {
           + std::string(" ]");
 }
 
-inline simdjson_result<const char *> json_iterator::current_location() noexcept {
+inline simdjson_result<const char *> json_iterator::current_location() const noexcept {
   if (!is_alive()) {    // Unrecoverable error
     if (!at_root()) {
       return reinterpret_cast<const char *>(token.peek(-1));
@@ -316,8 +316,12 @@ simdjson_inline token_position json_iterator::position() const noexcept {
   return token.position();
 }
 
-simdjson_inline simdjson_result<std::string_view> json_iterator::unescape(raw_json_string in) noexcept {
-  return parser->unescape(in, _string_buf_loc);
+simdjson_inline simdjson_result<std::string_view> json_iterator::unescape(raw_json_string in, bool allow_replacement) noexcept {
+  return parser->unescape(in, _string_buf_loc, allow_replacement);
+}
+
+simdjson_inline simdjson_result<std::string_view> json_iterator::unescape_wobbly(raw_json_string in) noexcept {
+  return parser->unescape_wobbly(in, _string_buf_loc);
 }
 
 simdjson_inline void json_iterator::reenter_child(token_position position, depth_t child_depth) noexcept {
@@ -325,6 +329,7 @@ simdjson_inline void json_iterator::reenter_child(token_position position, depth
   SIMDJSON_ASSUME(_depth == child_depth - 1);
 #if SIMDJSON_DEVELOPMENT_CHECKS
 #ifndef SIMDJSON_CLANG_VISUAL_STUDIO
+  SIMDJSON_ASSUME(size_t(child_depth) < parser->max_depth());
   SIMDJSON_ASSUME(position >= parser->start_positions[child_depth]);
 #endif
 #endif
@@ -332,14 +337,24 @@ simdjson_inline void json_iterator::reenter_child(token_position position, depth
   _depth = child_depth;
 }
 
+simdjson_inline error_code json_iterator::consume_character(char c) noexcept {
+  if (*peek() == c) {
+    return_current_and_advance();
+    return SUCCESS;
+  }
+  return TAPE_ERROR;
+}
+
 #if SIMDJSON_DEVELOPMENT_CHECKS
 
 simdjson_inline token_position json_iterator::start_position(depth_t depth) const noexcept {
-  return parser->start_positions[depth];
+  SIMDJSON_ASSUME(size_t(depth) < parser->max_depth());
+  return size_t(depth) < parser->max_depth() ? parser->start_positions[depth] : 0;
 }
 
 simdjson_inline void json_iterator::set_start_position(depth_t depth, token_position position) noexcept {
-  parser->start_positions[depth] = position;
+  SIMDJSON_ASSUME(size_t(depth) < parser->max_depth());
+  if(size_t(depth) < parser->max_depth()) { parser->start_positions[depth] = position; }
 }
 
 #endif
@@ -351,19 +366,16 @@ simdjson_inline error_code json_iterator::optional_error(error_code _error, cons
   return _error;
 }
 
-template<int N>
-simdjson_warn_unused simdjson_inline bool json_iterator::copy_to_buffer(const uint8_t *json, uint32_t max_len, uint8_t (&tmpbuf)[N]) noexcept {
+
+simdjson_warn_unused simdjson_inline bool json_iterator::copy_to_buffer(const uint8_t *json, uint32_t max_len, uint8_t *tmpbuf, size_t N) noexcept {
+  // This function is not expected to be called in performance-sensitive settings.
   // Let us guard against silly cases:
   if((N < max_len) || (N == 0)) { return false; }
-  // Truncate whitespace to fit the buffer.
-  if (max_len > N-1) {
-    // if (jsoncharutils::is_not_structural_or_whitespace(json[N-1])) { return false; }
-    max_len = N-1;
-  }
-
   // Copy to the buffer.
   std::memcpy(tmpbuf, json, max_len);
-  tmpbuf[max_len] = ' ';
+  if(N > max_len) { // We pad whatever remains with ' '.
+    std::memset(tmpbuf + max_len, ' ', N - max_len);
+  }
   return true;
 }
 

include/simdjson/generic/ondemand/json_iterator.h

@@ -238,7 +238,12 @@ public:
    */
   simdjson_inline error_code optional_error(error_code error, const char *message) noexcept;
 
-  template<int N> simdjson_warn_unused simdjson_inline bool copy_to_buffer(const uint8_t *json, uint32_t max_len, uint8_t (&tmpbuf)[N]) noexcept;
+  /**
+   * Take an input in json containing max_len characters and attempt to copy it over to tmpbuf, a buffer with
+   * N bytes of capacity. It will return false if N is too small (smaller than max_len) of if it is zero.
+   * The buffer (tmpbuf) is padded with space characters.
+   */
+  simdjson_warn_unused simdjson_inline bool copy_to_buffer(const uint8_t *json, uint32_t max_len, uint8_t *tmpbuf, size_t N) noexcept;
 
   simdjson_inline token_position position() const noexcept;
   /**
@@ -246,9 +251,11 @@ public:
    * Each raw_json_string should be unescaped once, or else the string buffer might
    * overflow.
    */
-  simdjson_inline simdjson_result<std::string_view> unescape(raw_json_string in) noexcept;
+  simdjson_inline simdjson_result<std::string_view> unescape(raw_json_string in, bool allow_replacement) noexcept;
+  simdjson_inline simdjson_result<std::string_view> unescape_wobbly(raw_json_string in) noexcept;
   simdjson_inline void reenter_child(token_position position, depth_t child_depth) noexcept;
 
+  simdjson_inline error_code consume_character(char c) noexcept;
 #if SIMDJSON_DEVELOPMENT_CHECKS
   simdjson_inline token_position start_position(depth_t depth) const noexcept;
   simdjson_inline void set_start_position(depth_t depth, token_position position) noexcept;
@@ -260,7 +267,7 @@ public:
   /**
    * Returns the current location in the document if in bounds.
    */
-  inline simdjson_result<const char *> current_location() noexcept;
+  inline simdjson_result<const char *> current_location() const noexcept;
 
   /**
    * Updates this json iterator so that it is back at the beginning of the document,
@@ -291,8 +298,8 @@ protected:
   friend class raw_json_string;
   friend class parser;
   friend class value_iterator;
-  friend simdjson_inline void logger::log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta) noexcept;
-  friend simdjson_inline void logger::log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail) noexcept;
+  friend simdjson_inline void logger::log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta, logger::log_level level) noexcept;
+  friend simdjson_inline void logger::log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail, logger::log_level level) noexcept;
 }; // json_iterator
 
 } // namespace ondemand

include/simdjson/generic/ondemand/logger-inl.h

@@ -1,3 +1,4 @@
+#include <memory>
 namespace simdjson {
 namespace SIMDJSON_IMPLEMENTATION {
 namespace ondemand {
@@ -18,36 +19,68 @@ static inline char printable_char(char c) {
   }
 }
 
+static inline log_level get_log_level_from_env()
+{
+  SIMDJSON_PUSH_DISABLE_WARNINGS
+  SIMDJSON_DISABLE_DEPRECATED_WARNING // Disable CRT_SECURE warning on MSVC: manually verified this is safe
+  char *lvl = getenv("SIMDJSON_LOG_LEVEL");
+  SIMDJSON_POP_DISABLE_WARNINGS
+  if (lvl && simdjson_strcasecmp(lvl, "ERROR") == 0) { return log_level::LOG_ERROR; }
+  return log_level::LOG_INFO;
+}
+
+static inline log_level log_threshold()
+{
+  static log_level threshold = get_log_level_from_env();
+  return threshold;
+}
+
+static inline bool should_log(log_level level)
+{
+  return level >= log_threshold();
+}
+
+template<typename... Args>
+inline std::string string_format(const std::string& format, const Args&... args)
+{
+  int size_s = std::snprintf(nullptr, 0, format.c_str(), args...) + 1;
+  auto size = static_cast<size_t>(size_s);
+  if (size <= 0) return std::string();
+  std::unique_ptr<char[]> buf(new char[size]);
+  std::snprintf(buf.get(), size, format.c_str(), args...);
+  return std::string(buf.get(), buf.get() + size - 1);
+}
+
 inline void log_event(const json_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept {
-  log_line(iter, "", type, detail, delta, depth_delta);
+  log_line(iter, "", type, detail, delta, depth_delta, log_level::LOG_INFO);
 }
 
 inline void log_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail) noexcept {
-  log_line(iter, index, depth, "", type, detail);
+  log_line(iter, index, depth, "", type, detail, log_level::LOG_INFO);
 }
 inline void log_value(const json_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept {
-  log_line(iter, "", type, detail, delta, depth_delta);
+  log_line(iter, "", type, detail, delta, depth_delta, log_level::LOG_INFO);
 }
 
 inline void log_start_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail) noexcept {
-  log_line(iter, index, depth, "+", type, detail);
+  log_line(iter, index, depth, "+", type, detail, log_level::LOG_INFO);
   if (LOG_ENABLED) { log_depth++; }
 }
 inline void log_start_value(const json_iterator &iter, const char *type, int delta, int depth_delta) noexcept {
-  log_line(iter, "+", type, "", delta, depth_delta);
+  log_line(iter, "+", type, "", delta, depth_delta, log_level::LOG_INFO);
   if (LOG_ENABLED) { log_depth++; }
 }
 
 inline void log_end_value(const json_iterator &iter, const char *type, int delta, int depth_delta) noexcept {
   if (LOG_ENABLED) { log_depth--; }
-  log_line(iter, "-", type, "", delta, depth_delta);
+  log_line(iter, "-", type, "", delta, depth_delta, log_level::LOG_INFO);
 }
 
 inline void log_error(const json_iterator &iter, const char *error, const char *detail, int delta, int depth_delta) noexcept {
-  log_line(iter, "ERROR: ", error, detail, delta, depth_delta);
+  log_line(iter, "ERROR: ", error, detail, delta, depth_delta, log_level::LOG_ERROR);
 }
 inline void log_error(const json_iterator &iter, token_position index, depth_t depth, const char *error, const char *detail) noexcept {
-  log_line(iter, index, depth, "ERROR: ", error, detail);
+  log_line(iter, index, depth, "ERROR: ", error, detail, log_level::LOG_ERROR);
 }
 
 inline void log_event(const value_iterator &iter, const char *type, std::string_view detail, int delta, int depth_delta) noexcept {
@@ -71,7 +104,7 @@ inline void log_error(const value_iterator &iter, const char *error, const char
 }
 
 inline void log_headers() noexcept {
-  if (LOG_ENABLED) {
+  if (LOG_ENABLED && simdjson_unlikely(should_log(log_level::LOG_INFO))) {
     // Technically a static variable is not thread-safe, but if you are using threads
     // and logging... well...
     static bool displayed_hint{false};
@@ -121,11 +154,11 @@ inline void log_headers() noexcept {
   }
 }
 
-inline void log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta) noexcept {
-  log_line(iter, iter.position()+delta, depth_t(iter.depth()+depth_delta), title_prefix, title, detail);
+inline void log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta, log_level level) noexcept {
+  log_line(iter, iter.position()+delta, depth_t(iter.depth()+depth_delta), title_prefix, title, detail, level);
 }
-inline void log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail) noexcept {
-  if (LOG_ENABLED) {
+inline void log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail, log_level level) noexcept {
+  if (LOG_ENABLED && simdjson_unlikely(should_log(level))) {
     const int indent = depth*2;
     const auto buf = iter.token.buf;
     printf("| %*s%s%-*s ",
@@ -136,9 +169,15 @@ inline void log_line(const json_iterator &iter, token_position index, depth_t de
     {
       // Print the current structural.
       printf("| ");
-      auto current_structural = &buf[*index];
-      for (int i=0;i<LOG_BUFFER_LEN;i++) {
-        printf("%c", printable_char(current_structural[i]));
+      // Before we begin, the index might point right before the document.
+      // This could be unsafe, see https://github.com/simdjson/simdjson/discussions/1938
+      if(index < iter._root) {
+        printf("%*s", LOG_BUFFER_LEN, "");
+      } else {
+        auto current_structural = &buf[*index];
+        for (int i=0;i<LOG_BUFFER_LEN;i++) {
+            printf("%c", printable_char(current_structural[i]));
+        }
       }
       printf(" ");
     }
@@ -153,7 +192,7 @@ inline void log_line(const json_iterator &iter, token_position index, depth_t de
     }
     // printf("| %5u ", *(index+1));
     printf("| %5i ", depth);
-    printf("| %.*s ", int(detail.size()), detail.data());
+    printf("| %6.*s ", int(detail.size()) , detail.data());
     printf("|\n");
     fflush(stdout);
   }

include/simdjson/generic/ondemand/logger.h

@@ -7,6 +7,11 @@ class value_iterator;
 
 namespace logger {
 
+enum class log_level : int32_t {
+  LOG_INFO = 0,
+  LOG_ERROR = 1
+};
+
 #if SIMDJSON_VERBOSE_LOGGING
   static constexpr const bool LOG_ENABLED = true;
 #else
@@ -16,9 +21,11 @@ namespace logger {
 // We do not want these functions to be 'really inlined' since real inlining is
 // for performance purposes and if you are using the loggers, you do not care about
 // performance (or should not).
+template<typename... Args>
+static inline std::string string_format(const std::string& format, const Args&... args);
 static inline void log_headers() noexcept;
-static inline void log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail) noexcept;
-static inline void log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta) noexcept;
+static inline void log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail, log_level level) noexcept;
+static inline void log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta, log_level level) noexcept;
 static inline void log_event(const json_iterator &iter, const char *type, std::string_view detail="", int delta=0, int depth_delta=0) noexcept;
 static inline void log_value(const json_iterator &iter, token_position index, depth_t depth, const char *type, std::string_view detail="") noexcept;
 static inline void log_value(const json_iterator &iter, const char *type, std::string_view detail="", int delta=-1, int depth_delta=0) noexcept;

include/simdjson/generic/ondemand/object-inl.h

@@ -5,13 +5,23 @@ namespace ondemand {
 simdjson_inline simdjson_result<value> object::find_field_unordered(const std::string_view key) & noexcept {
   bool has_value;
   SIMDJSON_TRY( iter.find_field_unordered_raw(key).get(has_value) );
-  if (!has_value) { return NO_SUCH_FIELD; }
+  if (!has_value) {
+    auto key_str = std::string(key.data(), key.size());
+    auto msg = logger::string_format("Cannot find key: %s", key_str.c_str());
+    logger::log_error(iter, msg.c_str());
+    return NO_SUCH_FIELD;
+  }
   return value(iter.child());
 }
 simdjson_inline simdjson_result<value> object::find_field_unordered(const std::string_view key) && noexcept {
   bool has_value;
   SIMDJSON_TRY( iter.find_field_unordered_raw(key).get(has_value) );
-  if (!has_value) { return NO_SUCH_FIELD; }
+  if (!has_value) {
+    auto key_str = std::string(key.data(), key.size());
+    auto msg = logger::string_format("Cannot find key: %s", key_str.c_str());
+    logger::log_error(iter, msg.c_str());
+    return NO_SUCH_FIELD;
+  }
   return value(iter.child());
 }
 simdjson_inline simdjson_result<value> object::operator[](const std::string_view key) & noexcept {
@@ -23,13 +33,23 @@ simdjson_inline simdjson_result<value> object::operator[](const std::string_view
 simdjson_inline simdjson_result<value> object::find_field(const std::string_view key) & noexcept {
   bool has_value;
   SIMDJSON_TRY( iter.find_field_raw(key).get(has_value) );
-  if (!has_value) { return NO_SUCH_FIELD; }
+  if (!has_value) {
+    auto key_str = std::string(key.data(), key.size());
+    auto msg = logger::string_format("Cannot find key: %s", key_str.c_str());
+    logger::log_error(iter, msg.c_str());
+    return NO_SUCH_FIELD;
+  }
   return value(iter.child());
 }
 simdjson_inline simdjson_result<value> object::find_field(const std::string_view key) && noexcept {
   bool has_value;
   SIMDJSON_TRY( iter.find_field_raw(key).get(has_value) );
-  if (!has_value) { return NO_SUCH_FIELD; }
+  if (!has_value) {
+    auto key_str = std::string(key.data(), key.size());
+    auto msg = logger::string_format("Cannot find key: %s", key_str.c_str());
+    logger::log_error(iter, msg.c_str());
+    return NO_SUCH_FIELD;
+  }
   return value(iter.child());
 }
 
@@ -226,4 +246,8 @@ simdjson_inline  simdjson_result<size_t> simdjson_result<SIMDJSON_IMPLEMENTATION
   return first.count_fields();
 }
 
+simdjson_inline  simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::object>::raw_json() noexcept {
+  if (error()) { return error(); }
+  return first.raw_json();
+}
 } // namespace simdjson

include/simdjson/generic/ondemand/object.h

@@ -223,6 +223,7 @@ public:
   inline simdjson_result<bool> reset() noexcept;
   inline simdjson_result<bool> is_empty() noexcept;
   inline simdjson_result<size_t> count_fields() & noexcept;
+  inline simdjson_result<std::string_view> raw_json() noexcept;
 
 };
 

include/simdjson/generic/ondemand/parser-inl.h

@@ -8,12 +8,15 @@ simdjson_inline parser::parser(size_t max_capacity) noexcept
 
 simdjson_warn_unused simdjson_inline error_code parser::allocate(size_t new_capacity, size_t new_max_depth) noexcept {
   if (new_capacity > max_capacity()) { return CAPACITY; }
-  if (string_buf && new_capacity == capacity()) { return SUCCESS; }
+  if (string_buf && new_capacity == capacity() && new_max_depth == max_depth()) { return SUCCESS; }
 
   // string_capacity copied from document::allocate
   _capacity = 0;
   size_t string_capacity = SIMDJSON_ROUNDUP_N(5 * new_capacity / 3 + SIMDJSON_PADDING, 64);
   string_buf.reset(new (std::nothrow) uint8_t[string_capacity]);
+#if SIMDJSON_DEVELOPMENT_CHECKS
+  start_positions.reset(new (std::nothrow) token_position[new_max_depth]);
+#endif
   if (implementation) {
     SIMDJSON_TRY( implementation->set_capacity(new_capacity) );
     SIMDJSON_TRY( implementation->set_max_depth(new_max_depth) );
@@ -21,6 +24,7 @@ simdjson_warn_unused simdjson_inline error_code parser::allocate(size_t new_capa
     SIMDJSON_TRY( simdjson::get_active_implementation()->create_dom_parser_implementation(new_capacity, new_max_depth, implementation) );
   }
   _capacity = new_capacity;
+  _max_depth = new_max_depth;
   return SUCCESS;
 }
 
@@ -29,7 +33,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<document> parser::iterate(p
 
   // Allocate if needed
   if (capacity() < json.length() || !string_buf) {
-    SIMDJSON_TRY( allocate(json.length()) );
+    SIMDJSON_TRY( allocate(json.length(), max_depth()) );
   }
 
   // Run stage 1.
@@ -72,7 +76,7 @@ simdjson_warn_unused simdjson_inline simdjson_result<json_iterator> parser::iter
 
   // Allocate if needed
   if (capacity() < json.length()) {
-    SIMDJSON_TRY( allocate(json.length()) );
+    SIMDJSON_TRY( allocate(json.length(), max_depth()) );
   }
 
   // Run stage 1.
@@ -80,18 +84,19 @@ simdjson_warn_unused simdjson_inline simdjson_result<json_iterator> parser::iter
   return json_iterator(reinterpret_cast<const uint8_t *>(json.data()), this);
 }
 
-inline simdjson_result<document_stream> parser::iterate_many(const uint8_t *buf, size_t len, size_t batch_size) noexcept {
+inline simdjson_result<document_stream> parser::iterate_many(const uint8_t *buf, size_t len, size_t batch_size, bool allow_comma_separated) noexcept {
   if(batch_size < MINIMAL_BATCH_SIZE) { batch_size = MINIMAL_BATCH_SIZE; }
-  return document_stream(*this, buf, len, batch_size);
+  if(allow_comma_separated && batch_size < len) { batch_size = len; }
+  return document_stream(*this, buf, len, batch_size, allow_comma_separated);
 }
-inline simdjson_result<document_stream> parser::iterate_many(const char *buf, size_t len, size_t batch_size) noexcept {
-  return iterate_many(reinterpret_cast<const uint8_t *>(buf), len, batch_size);
+inline simdjson_result<document_stream> parser::iterate_many(const char *buf, size_t len, size_t batch_size, bool allow_comma_separated) noexcept {
+  return iterate_many(reinterpret_cast<const uint8_t *>(buf), len, batch_size, allow_comma_separated);
 }
-inline simdjson_result<document_stream> parser::iterate_many(const std::string &s, size_t batch_size) noexcept {
-  return iterate_many(s.data(), s.length(), batch_size);
+inline simdjson_result<document_stream> parser::iterate_many(const std::string &s, size_t batch_size, bool allow_comma_separated) noexcept {
+  return iterate_many(s.data(), s.length(), batch_size, allow_comma_separated);
 }
-inline simdjson_result<document_stream> parser::iterate_many(const padded_string &s, size_t batch_size) noexcept {
-  return iterate_many(s.data(), s.length(), batch_size);
+inline simdjson_result<document_stream> parser::iterate_many(const padded_string &s, size_t batch_size, bool allow_comma_separated) noexcept {
+  return iterate_many(s.data(), s.length(), batch_size, allow_comma_separated);
 }
 
 simdjson_inline size_t parser::capacity() const noexcept {
@@ -100,10 +105,10 @@ simdjson_inline size_t parser::capacity() const noexcept {
 simdjson_inline size_t parser::max_capacity() const noexcept {
   return _max_capacity;
 }
-// deprecated method
 simdjson_inline size_t parser::max_depth() const noexcept {
-  return DEFAULT_MAX_DEPTH;
+  return _max_depth;
 }
+
 simdjson_inline void parser::set_max_capacity(size_t max_capacity) noexcept {
   if(max_capacity < dom::MINIMAL_DOCUMENT_CAPACITY) {
     _max_capacity = max_capacity;
@@ -112,8 +117,16 @@ simdjson_inline void parser::set_max_capacity(size_t max_capacity) noexcept {
   }
 }
 
-simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> parser::unescape(raw_json_string in, uint8_t *&dst) const noexcept {
-  uint8_t *end = implementation->parse_string(in.buf, dst);
+simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> parser::unescape(raw_json_string in, uint8_t *&dst, bool allow_replacement) const noexcept {
+  uint8_t *end = implementation->parse_string(in.buf, dst, allow_replacement);
+  if (!end) { return STRING_ERROR; }
+  std::string_view result(reinterpret_cast<const char *>(dst), end-dst);
+  dst = end;
+  return result;
+}
+
+simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> parser::unescape_wobbly(raw_json_string in, uint8_t *&dst) const noexcept {
+  uint8_t *end = implementation->parse_wobbly_string(in.buf, dst);
   if (!end) { return STRING_ERROR; }
   std::string_view result(reinterpret_cast<const char *>(dst), end-dst);
   dst = end;

include/simdjson/generic/ondemand/parser.h

@@ -78,7 +78,9 @@ public:
    * ### REQUIRED: Buffer Padding
    *
    * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
-   * those bytes are initialized to, as long as they are allocated.
+   * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
+   * using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
+   * SIMDJSON_PADDING bytes to avoid runtime warnings.
    *
    * @param json The JSON to parse.
    * @param len The length of the JSON.
@@ -134,7 +136,9 @@ public:
    * ### REQUIRED: Buffer Padding
    *
    * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
-   * those bytes are initialized to, as long as they are allocated.
+   * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
+   * using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
+   * SIMDJSON_PADDING bytes to avoid runtime warnings.
    *
    * @param json The JSON to parse.
    *
@@ -188,7 +192,9 @@ public:
    * ### REQUIRED: Buffer Padding
    *
    * The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what
-   * those bytes are initialized to, as long as they are allocated.
+   * those bytes are initialized to, as long as they are allocated. These bytes will be read: if you
+   * using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the
+   * SIMDJSON_PADDING bytes to avoid runtime warnings.
    *
    * ### Threads
    *
@@ -212,15 +218,15 @@ public:
    *         - other json errors if parsing fails. You should not rely on these errors to always the same for the
    *           same document: they may vary under runtime dispatch (so they may vary depending on your system and hardware).
    */
-  inline simdjson_result<document_stream> iterate_many(const uint8_t *buf, size_t len, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept;
+  inline simdjson_result<document_stream> iterate_many(const uint8_t *buf, size_t len, size_t batch_size = DEFAULT_BATCH_SIZE, bool allow_comma_separated = false) noexcept;
   /** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */
-  inline simdjson_result<document_stream> iterate_many(const char *buf, size_t len, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept;
+  inline simdjson_result<document_stream> iterate_many(const char *buf, size_t len, size_t batch_size = DEFAULT_BATCH_SIZE, bool allow_comma_separated = false) noexcept;
   /** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */
-  inline simdjson_result<document_stream> iterate_many(const std::string &s, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept;
-  inline simdjson_result<document_stream> iterate_many(const std::string &&s, size_t batch_size) = delete;// unsafe
+  inline simdjson_result<document_stream> iterate_many(const std::string &s, size_t batch_size = DEFAULT_BATCH_SIZE, bool allow_comma_separated = false) noexcept;
+  inline simdjson_result<document_stream> iterate_many(const std::string &&s, size_t batch_size, bool allow_comma_separated = false) = delete;// unsafe
   /** @overload parse_many(const uint8_t *buf, size_t len, size_t batch_size) */
-  inline simdjson_result<document_stream> iterate_many(const padded_string &s, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept;
-  inline simdjson_result<document_stream> iterate_many(const padded_string &&s, size_t batch_size) = delete;// unsafe
+  inline simdjson_result<document_stream> iterate_many(const padded_string &s, size_t batch_size = DEFAULT_BATCH_SIZE, bool allow_comma_separated = false) noexcept;
+  inline simdjson_result<document_stream> iterate_many(const padded_string &&s, size_t batch_size, bool allow_comma_separated = false) = delete;// unsafe
 
   /** @private We do not want to allow implicit conversion from C string to std::string. */
   simdjson_result<document_stream> iterate_many(const char *buf, size_t batch_size = DEFAULT_BATCH_SIZE) noexcept = delete;
@@ -230,14 +236,24 @@ public:
   /** The maximum capacity of this parser (the largest document it is allowed to process). */
   simdjson_inline size_t max_capacity() const noexcept;
   simdjson_inline void set_max_capacity(size_t max_capacity) noexcept;
-  /** Deprecated method. */
+  /**
+   * The maximum depth of this parser (the most deeply nested objects and arrays it can process).
+   * This parameter is only relevant when the macro SIMDJSON_DEVELOPMENT_CHECKS is set to true.
+   * The document's instance current_depth() method should be used to monitor the parsing
+   * depth and limit it if desired.
+   */
   simdjson_inline size_t max_depth() const noexcept;
+
   /**
    * Ensure this parser has enough memory to process JSON documents up to `capacity` bytes in length
    * and `max_depth` depth.
    *
+   * The max_depth parameter is only relevant when the macro SIMDJSON_DEVELOPMENT_CHECKS is set to true.
+   * The document's instance current_depth() method should be used to monitor the parsing
+   * depth and limit it if desired.
+   *
    * @param capacity The new capacity.
-   * @param max_depth The new max_depth. Defaults to DEFAULT_MAX_DEPTH. (This parameter has no effect.)
+   * @param max_depth The new max_depth. Defaults to DEFAULT_MAX_DEPTH.
    * @return The error, if there is one.
    */
   simdjson_warn_unused error_code allocate(size_t capacity, size_t max_depth=DEFAULT_MAX_DEPTH) noexcept;
@@ -253,6 +269,31 @@ public:
 
   /**
    * Unescape this JSON string, replacing \\ with \, \n with newline, etc. to a user-provided buffer.
+   * The result must be valid UTF-8.
+   * The provided pointer is advanced to the end of the string by reference, and a string_view instance
+   * is returned. You can ensure that your buffer is large enough by allocating a block of memory at least
+   * as large as the input JSON plus SIMDJSON_PADDING and then unescape all strings to this one buffer.
+   *
+   * This unescape function is a low-level function. If you want a more user-friendly approach, you should
+   * avoid raw_json_string instances (e.g., by calling unescaped_key() instead of key() or get_string()
+   * instead of get_raw_json_string()).
+   *
+   * ## IMPORTANT: string_view lifetime
+   *
+   * The string_view is only valid as long as the bytes in dst.
+   *
+   * @param raw_json_string input
+   * @param dst A pointer to a buffer at least large enough to write this string as well as
+   *            an additional SIMDJSON_PADDING bytes.
+   * @param allow_replacement Whether we allow a replacement if the input string contains unmatched surrogate pairs.
+   * @return A string_view pointing at the unescaped string in dst
+   * @error STRING_ERROR if escapes are incorrect.
+   */
+  simdjson_inline simdjson_result<std::string_view> unescape(raw_json_string in, uint8_t *&dst, bool allow_replacement = false) const noexcept;
+
+  /**
+   * Unescape this JSON string, replacing \\ with \, \n with newline, etc. to a user-provided buffer.
+   * The result may not be valid UTF-8. See https://simonsapin.github.io/wtf-8/
    * The provided pointer is advanced to the end of the string by reference, and a string_view instance
    * is returned. You can ensure that your buffer is large enough by allocating a block of memory at least
    * as large as the input JSON plus SIMDJSON_PADDING and then unescape all strings to this one buffer.
@@ -271,13 +312,18 @@ public:
    * @return A string_view pointing at the unescaped string in dst
    * @error STRING_ERROR if escapes are incorrect.
    */
-  simdjson_inline simdjson_result<std::string_view> unescape(raw_json_string in, uint8_t *&dst) const noexcept;
+  simdjson_inline simdjson_result<std::string_view> unescape_wobbly(raw_json_string in, uint8_t *&dst) const noexcept;
+
 private:
   /** @private [for benchmarking access] The implementation to use */
   std::unique_ptr<internal::dom_parser_implementation> implementation{};
   size_t _capacity{0};
   size_t _max_capacity;
+  size_t _max_depth{DEFAULT_MAX_DEPTH};
   std::unique_ptr<uint8_t[]> string_buf{};
+#if SIMDJSON_DEVELOPMENT_CHECKS
+  std::unique_ptr<token_position[]> start_positions{};
+#endif
 
   friend class json_iterator;
   friend class document_stream;

include/simdjson/generic/ondemand/raw_json_string-inl.h

@@ -143,10 +143,13 @@ simdjson_unused simdjson_inline bool operator!=(std::string_view c, const raw_js
 }
 
 
-simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> raw_json_string::unescape(json_iterator &iter) const noexcept {
-  return iter.unescape(*this);
+simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> raw_json_string::unescape(json_iterator &iter, bool allow_replacement) const noexcept {
+  return iter.unescape(*this, allow_replacement);
 }
 
+simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> raw_json_string::unescape_wobbly(json_iterator &iter) const noexcept {
+  return iter.unescape_wobbly(*this);
+}
 
 simdjson_unused simdjson_inline std::ostream &operator<<(std::ostream &out, const raw_json_string &str) noexcept {
   bool in_escape = false;
@@ -177,9 +180,12 @@ simdjson_inline simdjson_result<const char *> simdjson_result<SIMDJSON_IMPLEMENT
   if (error()) { return error(); }
   return first.raw();
 }
-simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string>::unescape(SIMDJSON_IMPLEMENTATION::ondemand::json_iterator &iter) const noexcept {
+simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string>::unescape(SIMDJSON_IMPLEMENTATION::ondemand::json_iterator &iter, bool allow_replacement) const noexcept {
   if (error()) { return error(); }
-  return first.unescape(iter);
+  return first.unescape(iter, allow_replacement);
+}
+simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string>::unescape_wobbly(SIMDJSON_IMPLEMENTATION::ondemand::json_iterator &iter) const noexcept {
+  if (error()) { return error(); }
+  return first.unescape_wobbly(iter);
 }
-
 } // namespace simdjson

include/simdjson/generic/ondemand/raw_json_string.h

@@ -139,6 +139,20 @@ private:
 
   /**
    * Unescape this JSON string, replacing \\ with \, \n with newline, etc.
+   * The result will be a valid UTF-8.
+   *
+   * ## IMPORTANT: string_view lifetime
+   *
+   * The string_view is only valid until the next parse() call on the parser.
+   *
+   * @param iter A json_iterator, which contains a buffer where the string will be written.
+   * @param allow_replacement Whether we allow replacement of invalid surrogate pairs.
+   */
+  simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> unescape(json_iterator &iter, bool allow_replacement) const noexcept;
+
+  /**
+   * Unescape this JSON string, replacing \\ with \, \n with newline, etc.
+   * The result may not be a valid UTF-8. https://simonsapin.github.io/wtf-8/
    *
    * ## IMPORTANT: string_view lifetime
    *
@@ -146,8 +160,7 @@ private:
    *
    * @param iter A json_iterator, which contains a buffer where the string will be written.
    */
-  simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> unescape(json_iterator &iter) const noexcept;
-
+  simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> unescape_wobbly(json_iterator &iter) const noexcept;
   const uint8_t * buf{};
   friend class object;
   friend class field;
@@ -182,7 +195,8 @@ public:
   simdjson_inline ~simdjson_result() noexcept = default; ///< @private
 
   simdjson_inline simdjson_result<const char *> raw() const noexcept;
-  simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> unescape(SIMDJSON_IMPLEMENTATION::ondemand::json_iterator &iter) const noexcept;
+  simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> unescape(SIMDJSON_IMPLEMENTATION::ondemand::json_iterator &iter, bool allow_replacement) const noexcept;
+  simdjson_inline simdjson_warn_unused simdjson_result<std::string_view> unescape_wobbly(SIMDJSON_IMPLEMENTATION::ondemand::json_iterator &iter) const noexcept;
 };
 
 } // namespace simdjson

include/simdjson/generic/ondemand/serialization.h

@@ -4,23 +4,27 @@
 namespace simdjson {
 /**
  * Create a string-view instance out of a document instance. The string-view instance
- * contains JSON text that is suitable to be parsed as JSON again.
+ * contains JSON text that is suitable to be parsed as JSON again. It does not
+ * validate the content.
  */
 inline simdjson_result<std::string_view> to_json_string(SIMDJSON_IMPLEMENTATION::ondemand::document& x) noexcept;
 /**
  * Create a string-view instance out of a value instance. The string-view instance
  * contains JSON text that is suitable to be parsed as JSON again. The value must
- * not have been accessed previously.
+ * not have been accessed previously. It does not
+ * validate the content.
  */
 inline simdjson_result<std::string_view> to_json_string(SIMDJSON_IMPLEMENTATION::ondemand::value& x) noexcept;
 /**
  * Create a string-view instance out of an object instance. The string-view instance
- * contains JSON text that is suitable to be parsed as JSON again.
+ * contains JSON text that is suitable to be parsed as JSON again. It does not
+ * validate the content.
  */
 inline simdjson_result<std::string_view> to_json_string(SIMDJSON_IMPLEMENTATION::ondemand::object& x) noexcept;
 /**
  * Create a string-view instance out of an array instance. The string-view instance
- * contains JSON text that is suitable to be parsed as JSON again.
+ * contains JSON text that is suitable to be parsed as JSON again. It does not
+ * validate the content.
  */
 inline simdjson_result<std::string_view> to_json_string(SIMDJSON_IMPLEMENTATION::ondemand::array& x) noexcept;
 inline simdjson_result<std::string_view> to_json_string(simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document> x);
@@ -39,7 +43,8 @@ inline simdjson_result<std::string_view> to_json_string(simdjson_result<SIMDJSON
 namespace simdjson { namespace SIMDJSON_IMPLEMENTATION { namespace ondemand {
 
 /**
- * Print JSON to an output stream.
+ * Print JSON to an output stream.  It does not
+ * validate the content.
  *
  * @param out The output stream.
  * @param value The element.
@@ -50,7 +55,8 @@ inline std::ostream& operator<<(std::ostream& out, simdjson::SIMDJSON_IMPLEMENTA
 inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::SIMDJSON_IMPLEMENTATION::ondemand::value> x);
 #endif
 /**
- * Print JSON to an output stream.
+ * Print JSON to an output stream. It does not
+ * validate the content.
  *
  * @param out The output stream.
  * @param value The array.
@@ -61,7 +67,8 @@ inline std::ostream& operator<<(std::ostream& out, simdjson::SIMDJSON_IMPLEMENTA
 inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::SIMDJSON_IMPLEMENTATION::ondemand::array> x);
 #endif
 /**
- * Print JSON to an output stream.
+ * Print JSON to an output stream. It does not
+ * validate the content.
  *
  * @param out The output stream.
  * @param value The array.
@@ -76,7 +83,8 @@ inline std::ostream& operator<<(std::ostream& out, simdjson::SIMDJSON_IMPLEMENTA
 inline std::ostream& operator<<(std::ostream& out, simdjson::simdjson_result<simdjson::SIMDJSON_IMPLEMENTATION::ondemand::document_reference>&& x);
 #endif
 /**
- * Print JSON to an output stream.
+ * Print JSON to an output stream. It does not
+ * validate the content.
  *
  * @param out The output stream.
  * @param value The object.

include/simdjson/generic/ondemand/token_iterator.h

@@ -116,8 +116,8 @@ protected:
   friend class json_iterator;
   friend class value_iterator;
   friend class object;
-  friend simdjson_inline void logger::log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta) noexcept;
-  friend simdjson_inline void logger::log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail) noexcept;
+  friend simdjson_inline void logger::log_line(const json_iterator &iter, const char *title_prefix, const char *title, std::string_view detail, int delta, int depth_delta, logger::log_level lvl) noexcept;
+  friend simdjson_inline void logger::log_line(const json_iterator &iter, token_position index, depth_t depth, const char *title_prefix, const char *title, std::string_view detail, logger::log_level lvl) noexcept;
 };
 
 } // namespace ondemand

include/simdjson/generic/ondemand/value-inl.h

@@ -30,8 +30,11 @@ simdjson_inline simdjson_result<object> value::start_or_resume_object() noexcept
 simdjson_inline simdjson_result<raw_json_string> value::get_raw_json_string() noexcept {
   return iter.get_raw_json_string();
 }
-simdjson_inline simdjson_result<std::string_view> value::get_string() noexcept {
-  return iter.get_string();
+simdjson_inline simdjson_result<std::string_view> value::get_string(bool allow_replacement) noexcept {
+  return iter.get_string(allow_replacement);
+}
+simdjson_inline simdjson_result<std::string_view> value::get_wobbly_string() noexcept {
+  return iter.get_wobbly_string();
 }
 simdjson_inline simdjson_result<double> value::get_double() noexcept {
   return iter.get_double();
@@ -54,14 +57,13 @@ simdjson_inline simdjson_result<int64_t> value::get_int64_in_string() noexcept {
 simdjson_inline simdjson_result<bool> value::get_bool() noexcept {
   return iter.get_bool();
 }
-simdjson_inline bool value::is_null() noexcept {
+simdjson_inline simdjson_result<bool> value::is_null() noexcept {
   return iter.is_null();
 }
-
 template<> simdjson_inline simdjson_result<array> value::get() noexcept { return get_array(); }
 template<> simdjson_inline simdjson_result<object> value::get() noexcept { return get_object(); }
 template<> simdjson_inline simdjson_result<raw_json_string> value::get() noexcept { return get_raw_json_string(); }
-template<> simdjson_inline simdjson_result<std::string_view> value::get() noexcept { return get_string(); }
+template<> simdjson_inline simdjson_result<std::string_view> value::get() noexcept { return get_string(false); }
 template<> simdjson_inline simdjson_result<number> value::get() noexcept { return get_number(); }
 template<> simdjson_inline simdjson_result<double> value::get() noexcept { return get_double(); }
 template<> simdjson_inline simdjson_result<uint64_t> value::get() noexcept { return get_uint64(); }
@@ -89,7 +91,7 @@ simdjson_inline value::operator double() noexcept(false) {
   return get_double();
 }
 simdjson_inline value::operator std::string_view() noexcept(false) {
-  return get_string();
+  return get_string(false);
 }
 simdjson_inline value::operator raw_json_string() noexcept(false) {
   return get_raw_json_string();
@@ -299,9 +301,13 @@ simdjson_inline simdjson_result<double> simdjson_result<SIMDJSON_IMPLEMENTATION:
   if (error()) { return error(); }
   return first.get_double_in_string();
 }
-simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value>::get_string() noexcept {
+simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value>::get_string(bool allow_replacement) noexcept {
   if (error()) { return error(); }
-  return first.get_string();
+  return first.get_string(allow_replacement);
+}
+simdjson_inline simdjson_result<std::string_view> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value>::get_wobbly_string() noexcept {
+  if (error()) { return error(); }
+  return first.get_wobbly_string();
 }
 simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value>::get_raw_json_string() noexcept {
   if (error()) { return error(); }
@@ -311,8 +317,8 @@ simdjson_inline simdjson_result<bool> simdjson_result<SIMDJSON_IMPLEMENTATION::o
   if (error()) { return error(); }
   return first.get_bool();
 }
-simdjson_inline bool simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value>::is_null() noexcept {
-  if (error()) { return false; }
+simdjson_inline simdjson_result<bool> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value>::is_null() noexcept {
+  if (error()) { return error(); }
   return first.is_null();
 }
 
@@ -404,7 +410,7 @@ simdjson_inline simdjson_result<const char *> simdjson_result<SIMDJSON_IMPLEMENT
   return first.current_location();
 }
 
-simdjson_inline int32_t simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value>::current_depth() const noexcept {
+simdjson_inline simdjson_result<int32_t> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value>::current_depth() const noexcept {
   if (error()) { return error(); }
   return first.current_depth();
 }

include/simdjson/generic/ondemand/value.h

@@ -129,8 +129,23 @@ public:
    *          time it parses a document or when it is destroyed.
    * @returns INCORRECT_TYPE if the JSON value is not a string.
    */
-  simdjson_inline simdjson_result<std::string_view> get_string() noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_string(bool allow_replacement = false) noexcept;
 
+
+  /**
+   * Cast this JSON value to a "wobbly" string.
+   *
+   * The string is may not be a valid UTF-8 string.
+   * See https://simonsapin.github.io/wtf-8/
+   *
+   * Important: a value should be consumed once. Calling get_wobbly_string() twice on the same value
+   * is an error.
+   *
+   * @returns An UTF-8 string. The string is stored in the parser and will be invalidated the next
+   *          time it parses a document or when it is destroyed.
+   * @returns INCORRECT_TYPE if the JSON value is not a string.
+   */
+  simdjson_inline simdjson_result<std::string_view> get_wobbly_string() noexcept;
   /**
    * Cast this JSON value to a raw_json_string.
    *
@@ -150,11 +165,14 @@ public:
   simdjson_inline simdjson_result<bool> get_bool() noexcept;
 
   /**
-   * Checks if this JSON value is null.
+   * Checks if this JSON value is null. If and only if the value is
+   * null, then it is consumed (we advance). If we find a token that
+   * begins with 'n' but is not 'null', then an error is returned.
    *
    * @returns Whether the value is null.
+   * @returns INCORRECT_TYPE If the JSON value begins with 'n' and is not 'null'.
    */
-  simdjson_inline bool is_null() noexcept;
+  simdjson_inline simdjson_result<bool> is_null() noexcept;
 
 #if SIMDJSON_EXCEPTIONS
   /**
@@ -334,7 +352,9 @@ public:
   simdjson_inline simdjson_result<value> operator[](const char *key) noexcept;
 
   /**
-   * Get the type of this JSON value.
+   * Get the type of this JSON value. It does not validate or consume the value.
+   * E.g., you must still call "is_null()" to check that a value is null even if
+   * "type()" returns json_type::null.
    *
    * NOTE: If you're only expecting a value to be one type (a typical case), it's generally
    * better to just call .get_double, .get_string, etc. and check for INCORRECT_TYPE (or just
@@ -580,10 +600,11 @@ public:
   simdjson_inline simdjson_result<int64_t> get_int64_in_string() noexcept;
   simdjson_inline simdjson_result<double> get_double() noexcept;
   simdjson_inline simdjson_result<double> get_double_in_string() noexcept;
-  simdjson_inline simdjson_result<std::string_view> get_string() noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_string(bool allow_replacement = false) noexcept;
+  simdjson_inline simdjson_result<std::string_view> get_wobbly_string() noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::raw_json_string> get_raw_json_string() noexcept;
   simdjson_inline simdjson_result<bool> get_bool() noexcept;
-  simdjson_inline bool is_null() noexcept;
+  simdjson_inline simdjson_result<bool> is_null() noexcept;
 
   template<typename T> simdjson_inline simdjson_result<T> get() noexcept;
 
@@ -676,7 +697,7 @@ public:
   /** @copydoc simdjson_inline simdjson_result<const char *> current_location() noexcept */
   simdjson_inline simdjson_result<const char *> current_location() noexcept;
   /** @copydoc simdjson_inline int32_t current_depth() const noexcept */
-  simdjson_inline int32_t current_depth() const noexcept;
+  simdjson_inline simdjson_result<int32_t> current_depth() const noexcept;
   simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::value> at_pointer(std::string_view json_pointer) noexcept;
 };
 

include/simdjson/generic/ondemand/value_iterator-inl.h

@@ -34,12 +34,17 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::start
   return true;
 }
 
-simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::started_root_object() noexcept {
+simdjson_warn_unused simdjson_inline error_code value_iterator::check_root_object() noexcept {
   // When in streaming mode, we cannot expect peek_last() to be the last structural element of the
   // current document. It only works in the normal mode where we have indexed a single document.
   // Note that adding a check for 'streaming' is not expensive since we only have at most
   // one root element.
   if ( ! _json_iter->streaming() ) {
+    // The following lines do not fully protect against garbage content within the
+    // object: e.g., `{"a":2} foo }`. Users concerned with garbage content should
+    // call `at_end()` on the document instance at the end of the processing to
+    // ensure that the processing has finished at the end.
+    //
     if (*_json_iter->peek_last() != '}') {
       _json_iter->abandon();
       return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "missing } at end");
@@ -56,6 +61,12 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::start
       return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "the document is unbalanced");
     }
   }
+  return SUCCESS;
+}
+
+simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::started_root_object() noexcept {
+  auto error = check_root_object();
+  if(error) { return error; }
   return started_object();
 }
 
@@ -419,12 +430,17 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::start
   return true;
 }
 
-simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::started_root_array() noexcept {
+simdjson_warn_unused simdjson_inline error_code value_iterator::check_root_array() noexcept {
   // When in streaming mode, we cannot expect peek_last() to be the last structural element of the
   // current document. It only works in the normal mode where we have indexed a single document.
   // Note that adding a check for 'streaming' is not expensive since we only have at most
   // one root element.
   if ( ! _json_iter->streaming() ) {
+    // The following lines do not fully protect against garbage content within the
+    // array: e.g., `[1, 2] foo]`. Users concerned with garbage content should
+    // also call `at_end()` on the document instance at the end of the processing to
+    // ensure that the processing has finished at the end.
+    //
     if (*_json_iter->peek_last() != ']') {
       _json_iter->abandon();
       return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "missing ] at end");
@@ -441,6 +457,12 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::start
       return report_error(INCOMPLETE_ARRAY_OR_OBJECT, "the document is unbalanced");
     }
   }
+  return SUCCESS;
+}
+
+simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::started_root_array() noexcept {
+  auto error = check_root_array();
+  if (error) { return error; }
   return started_array();
 }
 
@@ -468,12 +490,18 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::parse
   if (error) { return incorrect_type_error("Not a boolean"); }
   return simdjson_result<bool>(!not_true);
 }
-simdjson_inline bool value_iterator::parse_null(const uint8_t *json) const noexcept {
-  return !atomparsing::str4ncmp(json, "null") && jsoncharutils::is_structural_or_whitespace(json[4]);
+simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::parse_null(const uint8_t *json) const noexcept {
+  bool is_null_string = !atomparsing::str4ncmp(json, "null") && jsoncharutils::is_structural_or_whitespace(json[4]);
+  // if we start with 'n', we must be a null
+  if(!is_null_string && json[0]=='n') { return incorrect_type_error("Not a null but starts with n"); }
+  return is_null_string;
 }
 
-simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> value_iterator::get_string() noexcept {
-  return get_raw_json_string().unescape(json_iter());
+simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> value_iterator::get_string(bool allow_replacement) noexcept {
+  return get_raw_json_string().unescape(json_iter(), allow_replacement);
+}
+simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> value_iterator::get_wobbly_string() noexcept {
+  return get_raw_json_string().unescape_wobbly(json_iter());
 }
 simdjson_warn_unused simdjson_inline simdjson_result<raw_json_string> value_iterator::get_raw_json_string() noexcept {
   auto json = peek_scalar("string");
@@ -516,10 +544,11 @@ simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::get_b
   if(result.error() == SUCCESS) { advance_non_root_scalar("bool"); }
   return result;
 }
-simdjson_inline bool value_iterator::is_null() noexcept {
-  auto result = parse_null(peek_non_root_scalar("null"));
-  if(result) { advance_non_root_scalar("null"); }
-  return result;
+simdjson_inline simdjson_result<bool> value_iterator::is_null() noexcept {
+  bool is_null_value;
+  SIMDJSON_TRY(parse_null(peek_non_root_scalar("null")).get(is_null_value));
+  if(is_null_value) { advance_non_root_scalar("null"); }
+  return is_null_value;
 }
 simdjson_inline bool value_iterator::is_negative() noexcept {
   return numberparsing::is_negative(peek_non_root_scalar("numbersign"));
@@ -540,184 +569,187 @@ simdjson_inline simdjson_result<number> value_iterator::get_number() noexcept {
   return num;
 }
 
-simdjson_inline simdjson_result<bool> value_iterator::is_root_integer() noexcept {
+simdjson_inline simdjson_result<bool> value_iterator::is_root_integer(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("is_root_integer");
   uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
     return false; // if there are more than 20 characters, it cannot be represented as an integer.
   }
   auto answer = numberparsing::is_integer(tmpbuf);
   // If the parsing was a success, we must still check that it is
   // a single scalar. Note that we parse first because of cases like '[]' where
   // getting TRAILING_CONTENT is wrong.
-  if((answer.error() == SUCCESS) && (!_json_iter->is_single_token())) { return TRAILING_CONTENT; }
+  if(check_trailing && (answer.error() == SUCCESS) && (!_json_iter->is_single_token())) { return TRAILING_CONTENT; }
   return answer;
 }
 
-simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::number_type> value_iterator::get_root_number_type() noexcept {
-  if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::number_type> value_iterator::get_root_number_type(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("number");
   // Per https://www.exploringbinary.com/maximum-number-of-decimal-digits-in-binary-floating-point-numbers/,
   // 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest
   // number: -0.<fraction>e-308.
   uint8_t tmpbuf[1074+8+1];
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 1074+8+1)) {
     logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters");
     return NUMBER_ERROR;
   }
-  // If the parsing was a success, we must still check that it is
-  // a single scalar. Note that we parse first because of cases like '[]' where
-  // getting TRAILING_CONTENT is wrong.
   auto answer = numberparsing::get_number_type(tmpbuf);
-  if((answer.error() == SUCCESS) && (!_json_iter->is_single_token())) { return TRAILING_CONTENT; }
+  if (check_trailing && (answer.error() == SUCCESS)  && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
   return answer;
 }
-simdjson_inline simdjson_result<number> value_iterator::get_root_number() noexcept {
+simdjson_inline simdjson_result<number> value_iterator::get_root_number(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("number");
   // Per https://www.exploringbinary.com/maximum-number-of-decimal-digits-in-binary-floating-point-numbers/,
   // 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest
   // number: -0.<fraction>e-308.
   uint8_t tmpbuf[1074+8+1];
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 1074+8+1)) {
     logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters");
     return NUMBER_ERROR;
   }
   number num;
   error_code error =  numberparsing::parse_number(tmpbuf, num);
   if(error) { return error; }
-  if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+  if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
   advance_root_scalar("number");
   return num;
 }
-
-simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> value_iterator::get_root_string() noexcept {
-  return get_string();
+simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> value_iterator::get_root_string(bool check_trailing, bool allow_replacement) noexcept {
+  return get_root_raw_json_string(check_trailing).unescape(json_iter(), allow_replacement);
 }
-simdjson_warn_unused simdjson_inline simdjson_result<raw_json_string> value_iterator::get_root_raw_json_string() noexcept {
-  return get_raw_json_string();
+simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> value_iterator::get_root_wobbly_string(bool check_trailing) noexcept {
+  return get_root_raw_json_string(check_trailing).unescape_wobbly(json_iter());
 }
-simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> value_iterator::get_root_uint64() noexcept {
+simdjson_warn_unused simdjson_inline simdjson_result<raw_json_string> value_iterator::get_root_raw_json_string(bool check_trailing) noexcept {
+  auto json = peek_scalar("string");
+  if (*json != '"') { return incorrect_type_error("Not a string"); }
+  if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+  advance_scalar("string");
+  return raw_json_string(json+1);
+}
+simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> value_iterator::get_root_uint64(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("uint64");
   uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
     logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters");
     return NUMBER_ERROR;
   }
   auto result = numberparsing::parse_unsigned(tmpbuf);
   if(result.error() == SUCCESS) {
-    if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+    if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
     advance_root_scalar("uint64");
   }
   return result;
 }
-simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> value_iterator::get_root_uint64_in_string() noexcept {
+simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> value_iterator::get_root_uint64_in_string(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("uint64");
   uint8_t tmpbuf[20+1]; // <20 digits> is the longest possible unsigned integer
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
     logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters");
     return NUMBER_ERROR;
   }
   auto result = numberparsing::parse_unsigned_in_string(tmpbuf);
   if(result.error() == SUCCESS) {
-    if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+    if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
     advance_root_scalar("uint64");
   }
   return result;
 }
-simdjson_warn_unused simdjson_inline simdjson_result<int64_t> value_iterator::get_root_int64() noexcept {
+simdjson_warn_unused simdjson_inline simdjson_result<int64_t> value_iterator::get_root_int64(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("int64");
   uint8_t tmpbuf[20+1]; // -<19 digits> is the longest possible integer
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
     logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters");
     return NUMBER_ERROR;
   }
 
   auto result = numberparsing::parse_integer(tmpbuf);
   if(result.error() == SUCCESS) {
-    if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+    if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
     advance_root_scalar("int64");
   }
   return result;
 }
-simdjson_warn_unused simdjson_inline simdjson_result<int64_t> value_iterator::get_root_int64_in_string() noexcept {
+simdjson_warn_unused simdjson_inline simdjson_result<int64_t> value_iterator::get_root_int64_in_string(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("int64");
   uint8_t tmpbuf[20+1]; // -<19 digits> is the longest possible integer
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 20+1)) {
     logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 20 characters");
     return NUMBER_ERROR;
   }
 
   auto result = numberparsing::parse_integer_in_string(tmpbuf);
   if(result.error() == SUCCESS) {
-    if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+    if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
     advance_root_scalar("int64");
   }
   return result;
 }
-simdjson_warn_unused simdjson_inline simdjson_result<double> value_iterator::get_root_double() noexcept {
+simdjson_warn_unused simdjson_inline simdjson_result<double> value_iterator::get_root_double(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("double");
   // Per https://www.exploringbinary.com/maximum-number-of-decimal-digits-in-binary-floating-point-numbers/,
   // 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest
   // number: -0.<fraction>e-308.
   uint8_t tmpbuf[1074+8+1];
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 1074+8+1)) {
     logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters");
     return NUMBER_ERROR;
   }
   auto result = numberparsing::parse_double(tmpbuf);
   if(result.error() == SUCCESS) {
-    if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+    if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
     advance_root_scalar("double");
   }
   return result;
 }
 
-simdjson_warn_unused simdjson_inline simdjson_result<double> value_iterator::get_root_double_in_string() noexcept {
+simdjson_warn_unused simdjson_inline simdjson_result<double> value_iterator::get_root_double_in_string(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("double");
   // Per https://www.exploringbinary.com/maximum-number-of-decimal-digits-in-binary-floating-point-numbers/,
   // 1074 is the maximum number of significant fractional digits. Add 8 more digits for the biggest
   // number: -0.<fraction>e-308.
   uint8_t tmpbuf[1074+8+1];
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) {
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 1074+8+1)) {
     logger::log_error(*_json_iter, start_position(), depth(), "Root number more than 1082 characters");
     return NUMBER_ERROR;
   }
   auto result = numberparsing::parse_double_in_string(tmpbuf);
   if(result.error() == SUCCESS) {
-    if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+    if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
     advance_root_scalar("double");
   }
   return result;
 }
-simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::get_root_bool() noexcept {
+simdjson_warn_unused simdjson_inline simdjson_result<bool> value_iterator::get_root_bool(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("bool");
   uint8_t tmpbuf[5+1];
-  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf)) { return incorrect_type_error("Not a boolean"); }
+  if (!_json_iter->copy_to_buffer(json, max_len, tmpbuf, 5+1)) { return incorrect_type_error("Not a boolean"); }
   auto result = parse_bool(tmpbuf);
   if(result.error() == SUCCESS) {
-    if (!_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+    if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
     advance_root_scalar("bool");
   }
   return result;
 }
-simdjson_inline bool value_iterator::is_root_null() noexcept {
-  // If there is trailing content, then the document is not null.
-  if (!_json_iter->is_single_token()) { return false; }
+simdjson_inline simdjson_result<bool> value_iterator::is_root_null(bool check_trailing) noexcept {
   auto max_len = peek_start_length();
   auto json = peek_root_scalar("null");
   bool result = (max_len >= 4 && !atomparsing::str4ncmp(json, "null") &&
-         (max_len == 4 || jsoncharutils::is_structural_or_whitespace(json[5])));
-  if(result) { advance_root_scalar("null"); }
+         (max_len == 4 || jsoncharutils::is_structural_or_whitespace(json[4])));
+  if(result) { // we have something that looks like a null.
+    if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
+    advance_root_scalar("null");
+  }
   return result;
 }
 
@@ -914,11 +946,13 @@ simdjson_inline void value_iterator::move_at_container_start() noexcept {
 }
 
 simdjson_inline simdjson_result<bool> value_iterator::reset_array() noexcept {
+  if(error()) { return error(); }
   move_at_container_start();
   return started_array();
 }
 
 simdjson_inline simdjson_result<bool> value_iterator::reset_object() noexcept {
+  if(error()) { return error(); }
   move_at_container_start();
   return started_object();
 }

include/simdjson/generic/ondemand/value_iterator.h

@@ -109,7 +109,14 @@ public:
    * @error TAPE_ERROR if there is no matching } at end of document
    */
   simdjson_warn_unused simdjson_inline simdjson_result<bool> start_root_object() noexcept;
-
+  /**
+   * Checks whether an object could be started from the root. May be called by start_root_object.
+   *
+   * @returns SUCCESS if it is possible to safely start an object from the root (document level).
+   * @error INCORRECT_TYPE if there is no opening {
+   * @error TAPE_ERROR if there is no matching } at end of document
+   */
+  simdjson_warn_unused simdjson_inline error_code check_root_object() noexcept;
   /**
    * Start an object iteration after the user has already checked and moved past the {.
    *
@@ -234,7 +241,14 @@ public:
    * @error TAPE_ERROR if there is no matching ] at end of document
    */
   simdjson_warn_unused simdjson_inline simdjson_result<bool> start_root_array() noexcept;
-
+  /**
+   * Checks whether an array could be started from the root. May be called by start_root_array.
+   *
+   * @returns SUCCESS if it is possible to safely start an array from the root (document level).
+   * @error INCORRECT_TYPE If there is no [.
+   * @error TAPE_ERROR if there is no matching ] at end of document
+   */
+  simdjson_warn_unused simdjson_inline error_code check_root_array() noexcept;
   /**
    * Start an array iteration, after the user has already checked and moved past the [.
    *
@@ -280,7 +294,8 @@ public:
    * @{
    */
 
-  simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> get_string() noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> get_string(bool allow_replacement) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> get_wobbly_string() noexcept;
   simdjson_warn_unused simdjson_inline simdjson_result<raw_json_string> get_raw_json_string() noexcept;
   simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> get_uint64() noexcept;
   simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> get_uint64_in_string() noexcept;
@@ -289,26 +304,27 @@ public:
   simdjson_warn_unused simdjson_inline simdjson_result<double> get_double() noexcept;
   simdjson_warn_unused simdjson_inline simdjson_result<double> get_double_in_string() noexcept;
   simdjson_warn_unused simdjson_inline simdjson_result<bool> get_bool() noexcept;
-  simdjson_inline bool is_null() noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<bool> is_null() noexcept;
   simdjson_warn_unused simdjson_inline bool is_negative() noexcept;
   simdjson_warn_unused simdjson_inline simdjson_result<bool> is_integer() noexcept;
   simdjson_warn_unused simdjson_inline simdjson_result<number_type> get_number_type() noexcept;
   simdjson_warn_unused simdjson_inline simdjson_result<number> get_number() noexcept;
 
-  simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> get_root_string() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<raw_json_string> get_root_raw_json_string() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> get_root_uint64() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> get_root_uint64_in_string() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<int64_t> get_root_int64() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<int64_t> get_root_int64_in_string() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<double> get_root_double() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<double> get_root_double_in_string() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<bool> get_root_bool() noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> get_root_string(bool check_trailing, bool allow_replacement) noexcept;
+    simdjson_warn_unused simdjson_inline simdjson_result<std::string_view> get_root_wobbly_string(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<raw_json_string> get_root_raw_json_string(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> get_root_uint64(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<uint64_t> get_root_uint64_in_string(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<int64_t> get_root_int64(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<int64_t> get_root_int64_in_string(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<double> get_root_double(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<double> get_root_double_in_string(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<bool> get_root_bool(bool check_trailing) noexcept;
   simdjson_warn_unused simdjson_inline bool is_root_negative() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<bool> is_root_integer() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<number_type> get_root_number_type() noexcept;
-  simdjson_warn_unused simdjson_inline simdjson_result<number> get_root_number() noexcept;
-  simdjson_inline bool is_root_null() noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<bool> is_root_integer(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<number_type> get_root_number_type(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<number> get_root_number(bool check_trailing) noexcept;
+  simdjson_warn_unused simdjson_inline simdjson_result<bool> is_root_null(bool check_trailing) noexcept;
 
   simdjson_inline error_code error() const noexcept;
   simdjson_inline uint8_t *&string_buf_loc() noexcept;
@@ -350,7 +366,7 @@ protected:
   inline std::string to_string() const noexcept;
   simdjson_inline value_iterator(json_iterator *json_iter, depth_t depth, token_position start_index) noexcept;
 
-  simdjson_inline bool parse_null(const uint8_t *json) const noexcept;
+  simdjson_inline simdjson_result<bool> parse_null(const uint8_t *json) const noexcept;
   simdjson_inline simdjson_result<bool> parse_bool(const uint8_t *json) const noexcept;
   simdjson_inline const uint8_t *peek_start() const noexcept;
   simdjson_inline uint32_t peek_start_length() const noexcept;

include/simdjson/haswell.h

@@ -9,7 +9,7 @@
 #define SIMDJSON_TARGET_HASWELL
 #define SIMDJSON_UNTARGET_HASWELL
 #else
-#define SIMDJSON_TARGET_HASWELL SIMDJSON_TARGET_REGION("avx2,bmi,pclmul,lzcnt")
+#define SIMDJSON_TARGET_HASWELL SIMDJSON_TARGET_REGION("avx2,bmi,pclmul,lzcnt,popcnt")
 #define SIMDJSON_UNTARGET_HASWELL SIMDJSON_UNTARGET_REGION
 #endif
 

include/simdjson/haswell/bitmanipulation.h

@@ -9,8 +9,12 @@ namespace {
 // but the algorithms do not end up using the returned value.
 // Sadly, sanitizers are not smart enough to figure it out.
 SIMDJSON_NO_SANITIZE_UNDEFINED
+// This function can be used safely even if not all bytes have been
+// initialized.
+// See issue https://github.com/simdjson/simdjson/issues/1965
+SIMDJSON_NO_SANITIZE_MEMORY
 simdjson_inline int trailing_zeroes(uint64_t input_num) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   return (int)_tzcnt_u64(input_num);
 #else // SIMDJSON_REGULAR_VISUAL_STUDIO
   ////////
@@ -32,7 +36,7 @@ simdjson_inline int leading_zeroes(uint64_t input_num) {
   return int(_lzcnt_u64(input_num));
 }
 
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
 simdjson_inline unsigned __int64 count_ones(uint64_t input_num) {
   // note: we do not support legacy 32-bit Windows
   return __popcnt64(input_num);// Visual Studio wants two underscores
@@ -45,7 +49,7 @@ simdjson_inline long long int count_ones(uint64_t input_num) {
 
 simdjson_inline bool add_overflow(uint64_t value1, uint64_t value2,
                                 uint64_t *result) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   return _addcarry_u64(0, value1, value2,
                        reinterpret_cast<unsigned __int64 *>(result));
 #else

include/simdjson/haswell/implementation.h

@@ -9,6 +9,9 @@ namespace haswell {
 
 using namespace simdjson;
 
+/**
+ * @private
+ */
 class implementation final : public simdjson::implementation {
 public:
   simdjson_inline implementation() : simdjson::implementation(

include/simdjson/haswell/intrinsics.h

@@ -3,14 +3,14 @@
 
 #include "simdjson/base.h"
 
-#ifdef SIMDJSON_VISUAL_STUDIO
+#if SIMDJSON_VISUAL_STUDIO
 // under clang within visual studio, this will include <x86intrin.h>
 #include <intrin.h>  // visual studio or clang
 #else
 #include <x86intrin.h> // elsewhere
 #endif // SIMDJSON_VISUAL_STUDIO
 
-#ifdef SIMDJSON_CLANG_VISUAL_STUDIO
+#if SIMDJSON_CLANG_VISUAL_STUDIO
 /**
  * You are not supposed, normally, to include these
  * headers directly. Instead you should either include intrin.h

include/simdjson/icelake.h

@@ -9,7 +9,7 @@
 #define SIMDJSON_TARGET_ICELAKE
 #define SIMDJSON_UNTARGET_ICELAKE
 #else
-#define SIMDJSON_TARGET_ICELAKE SIMDJSON_TARGET_REGION("avx512f,avx512dq,avx512cd,avx512bw,avx512vbmi,avx512vbmi2,avx512vl,avx2,bmi,pclmul,lzcnt")
+#define SIMDJSON_TARGET_ICELAKE SIMDJSON_TARGET_REGION("avx512f,avx512dq,avx512cd,avx512bw,avx512vbmi,avx512vbmi2,avx512vl,avx2,bmi,pclmul,lzcnt,popcnt")
 #define SIMDJSON_UNTARGET_ICELAKE SIMDJSON_UNTARGET_REGION
 #endif
 

include/simdjson/icelake/bitmanipulation.h

@@ -9,8 +9,12 @@ namespace {
 // but the algorithms do not end up using the returned value.
 // Sadly, sanitizers are not smart enough to figure it out.
 SIMDJSON_NO_SANITIZE_UNDEFINED
+// This function can be used safely even if not all bytes have been
+// initialized.
+// See issue https://github.com/simdjson/simdjson/issues/1965
+SIMDJSON_NO_SANITIZE_MEMORY
 simdjson_inline int trailing_zeroes(uint64_t input_num) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   return (int)_tzcnt_u64(input_num);
 #else // SIMDJSON_REGULAR_VISUAL_STUDIO
   ////////
@@ -32,7 +36,7 @@ simdjson_inline int leading_zeroes(uint64_t input_num) {
   return int(_lzcnt_u64(input_num));
 }
 
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
 simdjson_inline unsigned __int64 count_ones(uint64_t input_num) {
   // note: we do not support legacy 32-bit Windows
   return __popcnt64(input_num);// Visual Studio wants two underscores
@@ -45,7 +49,7 @@ simdjson_inline long long int count_ones(uint64_t input_num) {
 
 simdjson_inline bool add_overflow(uint64_t value1, uint64_t value2,
                                 uint64_t *result) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   return _addcarry_u64(0, value1, value2,
                        reinterpret_cast<unsigned __int64 *>(result));
 #else

include/simdjson/icelake/implementation.h

@@ -9,6 +9,9 @@ namespace icelake {
 
 using namespace simdjson;
 
+/**
+ * @private
+ */
 class implementation final : public simdjson::implementation {
 public:
   simdjson_inline implementation() : simdjson::implementation(

include/simdjson/icelake/intrinsics.h

@@ -3,14 +3,14 @@
 
 #include "simdjson/base.h"
 
-#ifdef SIMDJSON_VISUAL_STUDIO
+#if SIMDJSON_VISUAL_STUDIO
 // under clang within visual studio, this will include <x86intrin.h>
 #include <intrin.h>  // visual studio or clang
 #else
 #include <x86intrin.h> // elsewhere
 #endif // SIMDJSON_VISUAL_STUDIO
 
-#ifdef SIMDJSON_CLANG_VISUAL_STUDIO
+#if SIMDJSON_CLANG_VISUAL_STUDIO
 /**
  * You are not supposed, normally, to include these
  * headers directly. Instead you should either include intrin.h

include/simdjson/icelake/simd.h

@@ -79,13 +79,9 @@ namespace simd {
 
     template<int N=1>
     simdjson_inline simd8<T> prev(const simd8<T> prev_chunk) const {
-#if SIMDJSON_GCC8
      // workaround for compilers unable to figure out that 16 - N is a constant (GCC 8)
       constexpr int shift = 16 - N;
       return _mm512_alignr_epi8(*this, _mm512_permutex2var_epi64(prev_chunk, _mm512_set_epi64(13, 12, 11, 10, 9, 8, 7, 6), *this), shift);
-#else
-      return _mm512_alignr_epi8(*this, _mm512_permutex2var_epi64(prev_chunk, _mm512_set_epi64(13, 12, 11, 10, 9, 8, 7, 6), *this), 16 - N);
-#endif
     }
   };
 

include/simdjson/implementation.h

@@ -87,7 +87,7 @@ public:
    *
    * @return a mask of all required `internal::instruction_set::` values.
    */
-  virtual uint32_t required_instruction_sets() const { return _required_instruction_sets; };
+  virtual uint32_t required_instruction_sets() const { return _required_instruction_sets; }
 
   /**
    * @private For internal implementation use

include/simdjson/implementations.h

@@ -51,8 +51,13 @@
 // Default Haswell to on if this is x86-64. Even if we're not compiled for it, it could be selected
 // at runtime.
 #ifndef SIMDJSON_IMPLEMENTATION_HASWELL
+#if SIMDJSON_CAN_ALWAYS_RUN_ICELAKE
+// if icelake is always available, never enable haswell.
+#define SIMDJSON_IMPLEMENTATION_HASWELL 0
+#else
 #define SIMDJSON_IMPLEMENTATION_HASWELL SIMDJSON_IS_X86_64
 #endif
+#endif
 #ifdef _MSC_VER
 // To see why  (__BMI__) && (__PCLMUL__) && (__LZCNT__) are not part of this next line, see
 // https://github.com/simdjson/simdjson/issues/1247
@@ -61,20 +66,30 @@
 #define SIMDJSON_CAN_ALWAYS_RUN_HASWELL ((SIMDJSON_IMPLEMENTATION_HASWELL) && (SIMDJSON_IS_X86_64) && (__AVX2__) && (__BMI__) && (__PCLMUL__) && (__LZCNT__))
 #endif
 
-// Default Westmere to on if this is x86-64. Note that the macro SIMDJSON_REQUIRES_HASWELL appears unused.
+// Default Westmere to on if this is x86-64.
 #ifndef SIMDJSON_IMPLEMENTATION_WESTMERE
-#define SIMDJSON_IMPLEMENTATION_WESTMERE (SIMDJSON_IS_X86_64 && !SIMDJSON_REQUIRES_HASWELL)
+#if SIMDJSON_CAN_ALWAYS_RUN_ICELAKE || SIMDJSON_CAN_ALWAYS_RUN_HASWELL
+// if icelake or haswell are always available, never enable westmere.
+#define SIMDJSON_IMPLEMENTATION_WESTMERE 0
+#else
+#define SIMDJSON_IMPLEMENTATION_WESTMERE SIMDJSON_IS_X86_64
+#endif
 #endif
 #define SIMDJSON_CAN_ALWAYS_RUN_WESTMERE (SIMDJSON_IMPLEMENTATION_WESTMERE && SIMDJSON_IS_X86_64 && __SSE4_2__ && __PCLMUL__)
 
 #ifndef SIMDJSON_IMPLEMENTATION_PPC64
-#define SIMDJSON_IMPLEMENTATION_PPC64 (SIMDJSON_IS_PPC64)
+#define SIMDJSON_IMPLEMENTATION_PPC64 (SIMDJSON_IS_PPC64 && SIMDJSON_IS_PPC64_VMX)
 #endif
-#define SIMDJSON_CAN_ALWAYS_RUN_PPC64 SIMDJSON_IMPLEMENTATION_PPC64 && SIMDJSON_IS_PPC64
+#define SIMDJSON_CAN_ALWAYS_RUN_PPC64 SIMDJSON_IMPLEMENTATION_PPC64 && SIMDJSON_IS_PPC64 && SIMDJSON_IS_PPC64_VMX
 
 // Default Fallback to on unless a builtin implementation has already been selected.
 #ifndef SIMDJSON_IMPLEMENTATION_FALLBACK
-#define SIMDJSON_IMPLEMENTATION_FALLBACK 1 // (!SIMDJSON_CAN_ALWAYS_RUN_ARM64 && !SIMDJSON_CAN_ALWAYS_RUN_HASWELL && !SIMDJSON_CAN_ALWAYS_RUN_WESTMERE && !SIMDJSON_CAN_ALWAYS_RUN_PPC64)
+#if SIMDJSON_CAN_ALWAYS_RUN_ARM64 || SIMDJSON_CAN_ALWAYS_RUN_ICELAKE || SIMDJSON_CAN_ALWAYS_RUN_HASWELL || SIMDJSON_CAN_ALWAYS_RUN_WESTMERE || SIMDJSON_CAN_ALWAYS_RUN_PPC64
+// if anything at all except fallback can always run, then disable fallback.
+#define SIMDJSON_IMPLEMENTATION_FALLBACK 0
+#else
+#define SIMDJSON_IMPLEMENTATION_FALLBACK 1
+#endif
 #endif
 #define SIMDJSON_CAN_ALWAYS_RUN_FALLBACK SIMDJSON_IMPLEMENTATION_FALLBACK
 

include/simdjson/internal/dom_parser_implementation.h

@@ -114,9 +114,26 @@ public:
    *
    * @param str pointer to the beginning of a valid UTF-8 JSON string, must end with an unescaped quote.
    * @param dst pointer to a destination buffer, it must point a region in memory of sufficient size.
+   * @param allow_replacement whether we allow a replacement character when the UTF-8 contains unmatched surrogate pairs.
    * @return end of the of the written region (exclusive) or nullptr in case of error.
    */
-  simdjson_warn_unused virtual uint8_t *parse_string(const uint8_t *src, uint8_t *dst) const noexcept = 0;
+  simdjson_warn_unused virtual uint8_t *parse_string(const uint8_t *src, uint8_t *dst, bool allow_replacement) const noexcept = 0;
+
+  /**
+   * Unescape a NON-valid UTF-8 string from src to dst, stopping at a final unescaped quote. There
+   * must be an unescaped quote terminating the string. It returns the final output
+   * position as pointer. In case of error (e.g., the string has bad escaped codes),
+   * then null_nullptrptr is returned. It is assumed that the output buffer is large
+   * enough. E.g., if src points at 'joe"', then dst needs to have four free bytes +
+   * SIMDJSON_PADDING bytes.
+   *
+   * Overridden by each implementation.
+   *
+   * @param str pointer to the beginning of a possibly invalid UTF-8 JSON string, must end with an unescaped quote.
+   * @param dst pointer to a destination buffer, it must point a region in memory of sufficient size.
+   * @return end of the of the written region (exclusive) or nullptr in case of error.
+   */
+  simdjson_warn_unused virtual uint8_t *parse_wobbly_string(const uint8_t *src, uint8_t *dst) const noexcept = 0;
 
   /**
    * Change the capacity of this parser.

include/simdjson/internal/isadetection.h

@@ -57,7 +57,6 @@ POSSIBILITY OF SUCH DAMAGE.
 namespace simdjson {
 namespace internal {
 
-
 enum instruction_set {
   DEFAULT = 0x0,
   NEON = 0x1,
@@ -84,22 +83,12 @@ static inline uint32_t detect_supported_architectures() {
   return instruction_set::ALTIVEC;
 }
 
-#elif defined(__arm__) || defined(__aarch64__) // incl. armel, armhf, arm64
-
-#if defined(__ARM_NEON)
+#elif defined(__aarch64__) || defined(_M_ARM64)
 
 static inline uint32_t detect_supported_architectures() {
   return instruction_set::NEON;
 }
 
-#else // ARM without NEON
-
-static inline uint32_t detect_supported_architectures() {
-  return instruction_set::DEFAULT;
-}
-
-#endif
-
 #elif defined(__x86_64__) || defined(_M_AMD64) // x64
 
 
@@ -117,7 +106,10 @@ constexpr uint32_t cpuid_avx512cd_bit = 1 << 28;    ///< @private bit 28 of EBX
 constexpr uint32_t cpuid_avx512bw_bit = 1 << 30;    ///< @private bit 30 of EBX for EAX=0x7
 constexpr uint32_t cpuid_avx512vl_bit = 1U << 31;    ///< @private bit 31 of EBX for EAX=0x7
 constexpr uint32_t cpuid_avx512vbmi2_bit = 1 << 6;  ///< @private bit 6 of ECX for EAX=0x7
+constexpr uint64_t cpuid_avx256_saved = uint64_t(1) << 2; ///< @private bit 2 = AVX
+constexpr uint64_t cpuid_avx512_saved = uint64_t(7) << 5; ///< @private bits 5,6,7 = opmask, ZMM_hi256, hi16_ZMM
 constexpr uint32_t cpuid_sse42_bit = 1 << 20;       ///< @private bit 20 of ECX for EAX=0x1
+constexpr uint32_t cpuid_osxsave = (uint32_t(1) << 26) | (uint32_t(1) << 27); ///< @private bits 26+27 of ECX for EAX=0x1
 constexpr uint32_t cpuid_pclmulqdq_bit = 1 << 1;    ///< @private bit  1 of ECX for EAX=0x1
 }
 
@@ -127,7 +119,7 @@ static inline void cpuid(uint32_t *eax, uint32_t *ebx, uint32_t *ecx,
                          uint32_t *edx) {
 #if defined(_MSC_VER)
   int cpu_info[4];
-  __cpuid(cpu_info, *eax);
+  __cpuidex(cpu_info, *eax, *ecx);
   *eax = cpu_info[0];
   *ebx = cpu_info[1];
   *ecx = cpu_info[2];
@@ -145,10 +137,48 @@ static inline void cpuid(uint32_t *eax, uint32_t *ebx, uint32_t *ecx,
 #endif
 }
 
+
+static inline uint64_t xgetbv() {
+#if defined(_MSC_VER)
+  return _xgetbv(0);
+#else
+  uint32_t xcr0_lo, xcr0_hi;
+  asm volatile("xgetbv\n\t" : "=a" (xcr0_lo), "=d" (xcr0_hi) : "c" (0));
+  return xcr0_lo | (uint64_t(xcr0_hi) << 32);
+#endif
+}
+
 static inline uint32_t detect_supported_architectures() {
   uint32_t eax, ebx, ecx, edx;
   uint32_t host_isa = 0x0;
 
+  // EBX for EAX=0x1
+  eax = 0x1;
+  ecx = 0x0;
+  cpuid(&eax, &ebx, &ecx, &edx);
+
+  if (ecx & cpuid_sse42_bit) {
+    host_isa |= instruction_set::SSE42;
+  } else {
+    return host_isa; // everything after is redundant
+  }
+
+  if (ecx & cpuid_pclmulqdq_bit) {
+    host_isa |= instruction_set::PCLMULQDQ;
+  }
+
+
+  if ((ecx & cpuid_osxsave) != cpuid_osxsave) {
+    return host_isa;
+  }
+
+  // xgetbv for checking if the OS saves registers
+  uint64_t xcr0 = xgetbv();
+
+  if ((xcr0 & cpuid_avx256_saved) == 0) {
+    return host_isa;
+  }
+
   // ECX for EAX=0x7
   eax = 0x7;
   ecx = 0x0;
@@ -164,6 +194,10 @@ static inline uint32_t detect_supported_architectures() {
     host_isa |= instruction_set::BMI2;
   }
 
+  if (!((xcr0 & cpuid_avx512_saved) == cpuid_avx512_saved)) {
+     return host_isa;
+  }
+
   if (ebx & cpuid_avx512f_bit) {
     host_isa |= instruction_set::AVX512F;
   }
@@ -200,18 +234,6 @@ static inline uint32_t detect_supported_architectures() {
     host_isa |= instruction_set::AVX512VBMI2;
   }
 
-  // EBX for EAX=0x1
-  eax = 0x1;
-  cpuid(&eax, &ebx, &ecx, &edx);
-
-  if (ecx & cpuid_sse42_bit) {
-    host_isa |= instruction_set::SSE42;
-  }
-
-  if (ecx & cpuid_pclmulqdq_bit) {
-    host_isa |= instruction_set::PCLMULQDQ;
-  }
-
   return host_isa;
 }
 #else // fallback

include/simdjson/internal/tape_ref-inl.h

@@ -17,7 +17,9 @@ simdjson_inline tape_ref::tape_ref(const dom::document *_doc, size_t _json_index
 simdjson_inline bool tape_ref::is_document_root() const noexcept {
   return json_index == 1; // should we ever change the structure of the tape, this should get updated.
 }
-
+simdjson_inline bool tape_ref::usable() const noexcept {
+  return doc != nullptr; // when the document pointer is null, this tape_ref is uninitialized (should not be accessed).
+}
 // Some value types have a specific on-tape word value. It can be faster
 // to check the type by doing a word-to-word comparison instead of extracting the
 // most significant 8 bits.

include/simdjson/internal/tape_ref.h

@@ -38,6 +38,7 @@ public:
   simdjson_inline const char * get_c_str() const noexcept;
   inline std::string_view get_string_view() const noexcept;
   simdjson_inline bool is_document_root() const noexcept;
+  simdjson_inline bool usable() const noexcept;
 
   /** The document this element references. */
   const dom::document *doc;

include/simdjson/nonstd/string_view.hpp

@@ -12,7 +12,7 @@
 #define NONSTD_SV_LITE_H_INCLUDED
 
 #define string_view_lite_MAJOR  1
-#define string_view_lite_MINOR  6
+#define string_view_lite_MINOR  7
 #define string_view_lite_PATCH  0
 
 #define string_view_lite_VERSION  nssv_STRINGIFY(string_view_lite_MAJOR) "." nssv_STRINGIFY(string_view_lite_MINOR) "." nssv_STRINGIFY(string_view_lite_PATCH)
@@ -72,7 +72,7 @@
 // Control presence of exception handling (try and auto discover):
 
 #ifndef nssv_CONFIG_NO_EXCEPTIONS
-# if _MSC_VER
+# if defined(_MSC_VER)
 #  include <cstddef>    // for _HAS_EXCEPTIONS
 # endif
 # if defined(__cpp_exceptions) || defined(__EXCEPTIONS) || (_HAS_EXCEPTIONS)
@@ -82,7 +82,7 @@
 # endif
 #endif
 
-// C++ language version detection (C++20 is speculative):
+// C++ language version detection (C++23 is speculative):
 // Note: VC14.0/1900 (VS2015) lacks too much from C++14.
 
 #ifndef   nssv_CPLUSPLUS
@@ -98,7 +98,8 @@
 #define nssv_CPP11_OR_GREATER_ ( nssv_CPLUSPLUS >= 201103L )
 #define nssv_CPP14_OR_GREATER  ( nssv_CPLUSPLUS >= 201402L )
 #define nssv_CPP17_OR_GREATER  ( nssv_CPLUSPLUS >= 201703L )
-#define nssv_CPP20_OR_GREATER  ( nssv_CPLUSPLUS >= 202000L )
+#define nssv_CPP20_OR_GREATER  ( nssv_CPLUSPLUS >= 202002L )
+#define nssv_CPP23_OR_GREATER  ( nssv_CPLUSPLUS >= 202300L )
 
 // use C++17 std::string_view if available and requested:
 
@@ -288,6 +289,8 @@ using std::operator<<;
 #define nssv_HAVE_CONSTEXPR_11          nssv_CPP11_140
 #define nssv_HAVE_EXPLICIT_CONVERSION   nssv_CPP11_140
 #define nssv_HAVE_INLINE_NAMESPACE      nssv_CPP11_140
+#define nssv_HAVE_IS_DEFAULT            nssv_CPP11_140
+#define nssv_HAVE_IS_DELETE             nssv_CPP11_140
 #define nssv_HAVE_NOEXCEPT              nssv_CPP11_140
 #define nssv_HAVE_NULLPTR               nssv_CPP11_100
 #define nssv_HAVE_REF_QUALIFIER         nssv_CPP11_140
@@ -469,6 +472,17 @@ nssv_DISABLE_MSVC_WARNINGS( 4455 26481 26472 )
 
 namespace nonstd { namespace sv_lite {
 
+//
+// basic_string_view declaration:
+//
+
+template
+<
+    class CharT,
+    class Traits = std::char_traits<CharT>
+>
+class basic_string_view;
+
 namespace detail {
 
 // support constexpr comparison in C++14;
@@ -536,14 +550,33 @@ inline nssv_constexpr14 std::size_t length( CharT * s )
 
 #endif // OPTIMIZE
 
-} // namespace detail
+#if nssv_CPP11_OR_GREATER && ! nssv_CPP17_OR_GREATER
+#if defined(__OPTIMIZE__)
 
-template
-<
-    class CharT,
-    class Traits = std::char_traits<CharT>
->
-class basic_string_view;
+// gcc, clang provide __OPTIMIZE__
+// Expect tail call optimization to make search() non-recursive:
+
+template< class CharT, class Traits = std::char_traits<CharT> >
+constexpr const CharT* search( basic_string_view<CharT, Traits> haystack, basic_string_view<CharT, Traits> needle )
+{
+    return haystack.starts_with( needle ) ? haystack.begin() :
+        haystack.empty() ? haystack.end() : search( haystack.substr(1), needle );
+}
+
+#else // OPTIMIZE
+
+// non-recursive:
+
+template< class CharT, class Traits = std::char_traits<CharT> >
+constexpr const CharT* search( basic_string_view<CharT, Traits> haystack, basic_string_view<CharT, Traits> needle )
+{
+    return std::search( haystack.begin(), haystack.end(), needle.begin(), needle.end() );
+}
+
+#endif // OPTIMIZE
+#endif // nssv_CPP11_OR_GREATER && ! nssv_CPP17_OR_GREATER
+
+} // namespace detail
 
 //
 // basic_string_view:
@@ -570,7 +603,7 @@ public:
     typedef const_pointer iterator;
     typedef const_pointer const_iterator;
     typedef std::reverse_iterator< const_iterator > reverse_iterator;
-    typedef	std::reverse_iterator< const_iterator > const_reverse_iterator;
+    typedef std::reverse_iterator< const_iterator > const_reverse_iterator;
 
     typedef std::size_t     size_type;
     typedef std::ptrdiff_t  difference_type;
@@ -607,6 +640,14 @@ public:
 #endif
     {}
 
+#if  nssv_HAVE_NULLPTR
+# if nssv_HAVE_IS_DELETE
+    nssv_constexpr basic_string_view( std::nullptr_t ) nssv_noexcept = delete;
+# else
+    private: nssv_constexpr basic_string_view( std::nullptr_t ) nssv_noexcept; public:
+# endif
+#endif
+
     // Assignment:
 
 #if nssv_CPP11_OR_GREATER
@@ -804,25 +845,30 @@ public:
 
     // find(), 4x:
 
-    nssv_constexpr14 size_type find( basic_string_view v, size_type pos = 0 ) const nssv_noexcept  // (1)
+    nssv_constexpr size_type find( basic_string_view v, size_type pos = 0 ) const nssv_noexcept  // (1)
     {
         return assert( v.size() == 0 || v.data() != nssv_nullptr )
             , pos >= size()
-            ? npos
-            : to_pos( std::search( cbegin() + pos, cend(), v.cbegin(), v.cend(), Traits::eq ) );
+            ? npos : to_pos(
+#if nssv_CPP11_OR_GREATER && ! nssv_CPP17_OR_GREATER
+                detail::search( substr(pos), v )
+#else
+                std::search( cbegin() + pos, cend(), v.cbegin(), v.cend(), Traits::eq )
+#endif
+            );
     }
 
-    nssv_constexpr14 size_type find( CharT c, size_type pos = 0 ) const nssv_noexcept  // (2)
+    nssv_constexpr size_type find( CharT c, size_type pos = 0 ) const nssv_noexcept  // (2)
     {
         return find( basic_string_view( &c, 1 ), pos );
     }
 
-    nssv_constexpr14 size_type find( CharT const * s, size_type pos, size_type n ) const  // (3)
+    nssv_constexpr size_type find( CharT const * s, size_type pos, size_type n ) const  // (3)
     {
         return find( basic_string_view( s, n ), pos );
     }
 
-    nssv_constexpr14 size_type find( CharT const * s, size_type pos = 0 ) const  // (4)
+    nssv_constexpr size_type find( CharT const * s, size_type pos = 0 ) const  // (4)
     {
         return find( basic_string_view( s ), pos );
     }
@@ -1381,7 +1427,7 @@ Stream & write_to_stream( Stream & os, View const & sv )
 {
     typename Stream::sentry sentry( os );
 
-    if ( !os )
+    if ( !sentry )
         return os;
 
     const std::streamsize length = static_cast<std::streamsize>( sv.length() );

include/simdjson/padded_string-inl.h

@@ -35,9 +35,8 @@ inline char *allocate_padded_buffer(size_t length) noexcept {
   if (padded_buffer == nullptr) {
     return nullptr;
   }
-  // We write zeroes in the padded region to avoid having uninitized
-  // garbage. If nothing else, garbage getting read might trigger a
-  // warning in a memory checking.
+  // We write nulls in the padded region to avoid having uninitialized
+  // content which may trigger warning for some sanitizers
   std::memset(padded_buffer + length, 0, totalpaddedlength - length);
   return padded_buffer;
 } // allocate_padded_buffer()
@@ -67,7 +66,7 @@ inline padded_string::padded_string(std::string_view sv_) noexcept
     : viable_size(sv_.size()), data_ptr(internal::allocate_padded_buffer(sv_.size())) {
   if(simdjson_unlikely(!data_ptr)) {
     //allocation failed or zero size
-    viable_size=0;
+    viable_size = 0;
     return;
   }
   if (sv_.size()) {
@@ -128,7 +127,7 @@ inline simdjson_result<padded_string> padded_string::load(std::string_view filen
 
   // Get the file size
   int ret;
-#if defined(SIMDJSON_VISUAL_STUDIO) && !SIMDJSON_IS_32BITS
+#if SIMDJSON_VISUAL_STUDIO && !SIMDJSON_IS_32BITS
   ret = _fseeki64(fp, 0, SEEK_END);
 #else
   ret = std::fseek(fp, 0, SEEK_END);
@@ -137,7 +136,7 @@ inline simdjson_result<padded_string> padded_string::load(std::string_view filen
     std::fclose(fp);
     return IO_ERROR;
   }
-#if defined(SIMDJSON_VISUAL_STUDIO) && !SIMDJSON_IS_32BITS
+#if SIMDJSON_VISUAL_STUDIO && !SIMDJSON_IS_32BITS
   __int64 llen = _ftelli64(fp);
   if(llen == -1L) {
     std::fclose(fp);

include/simdjson/portability.h

@@ -32,19 +32,14 @@
 #endif // __clang__
 #endif // _MSC_VER
 
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
-// https://en.wikipedia.org/wiki/C_alternative_tokens
-// This header should have no effect, except maybe
-// under Visual Studio.
-#include <iso646.h>
-#endif
-
 #if defined(__x86_64__) || defined(_M_AMD64)
 #define SIMDJSON_IS_X86_64 1
 #elif defined(__aarch64__) || defined(_M_ARM64)
 #define SIMDJSON_IS_ARM64 1
 #elif defined(__PPC64__) || defined(_M_PPC64)
-#define SIMDJSON_IS_PPC64 1
+#if defined(__ALTIVEC__)
+#define SIMDJSON_IS_PPC64_VMX 1
+#endif // defined(__ALTIVEC__)
 #else
 #define SIMDJSON_IS_32BITS 1
 
@@ -59,8 +54,11 @@
 #endif
 
 #endif // defined(__x86_64__) || defined(_M_AMD64)
+#ifndef SIMDJSON_IS_32BITS
+#define SIMDJSON_IS_32BITS 0
+#endif
 
-#ifdef SIMDJSON_IS_32BITS
+#if SIMDJSON_IS_32BITS
 #ifndef SIMDJSON_NO_PORTABILITY_WARNING
 #pragma message("The simdjson library is designed \
 for 64-bit processors and it seems that you are not \
@@ -91,7 +89,7 @@ use a 64-bit target such as x64, 64-bit ARM or 64-bit PPC.")
 //
 
 // We are going to use runtime dispatch.
-#ifdef SIMDJSON_IS_X86_64
+#if SIMDJSON_IS_X86_64
 #ifdef __clang__
 // clang does not have GCC push pop
 // warning: clang attribute push can't be used within a namespace in clang up
@@ -145,7 +143,20 @@ use a 64-bit target such as x64, 64-bit ARM or 64-bit PPC.")
 #define SIMDJSON_NO_SANITIZE_UNDEFINED
 #endif
 
-#ifdef SIMDJSON_VISUAL_STUDIO
+
+#if defined(__clang__) || defined(__GNUC__)
+#if defined(__has_feature)
+#  if __has_feature(memory_sanitizer)
+#define SIMDJSON_NO_SANITIZE_MEMORY __attribute__((no_sanitize("memory")))
+#  endif // if __has_feature(memory_sanitizer)
+#endif // defined(__has_feature)
+#endif
+// make sure it is defined as 'nothing' if it is unapplicable.
+#ifndef SIMDJSON_NO_SANITIZE_MEMORY
+#define SIMDJSON_NO_SANITIZE_MEMORY
+#endif
+
+#if SIMDJSON_VISUAL_STUDIO
 // This is one case where we do not distinguish between
 // regular visual studio and clang under visual studio.
 // clang under Windows has _stricmp (like visual studio) but not strcasecmp (as clang normally has)
@@ -159,9 +170,10 @@ use a 64-bit target such as x64, 64-bit ARM or 64-bit PPC.")
 #define simdjson_strncasecmp strncasecmp
 #endif
 
-#ifdef NDEBUG
-
-#ifdef SIMDJSON_VISUAL_STUDIO
+#if defined(NDEBUG) || defined(__OPTIMIZE__) || (defined(_MSC_VER) && !defined(_DEBUG))
+// If NDEBUG is set, or __OPTIMIZE__ is set, or we are under MSVC in release mode,
+// then do away with asserts and use __assume.
+#if SIMDJSON_VISUAL_STUDIO
 #define SIMDJSON_UNREACHABLE() __assume(0)
 #define SIMDJSON_ASSUME(COND) __assume(COND)
 #else
@@ -169,8 +181,8 @@ use a 64-bit target such as x64, 64-bit ARM or 64-bit PPC.")
 #define SIMDJSON_ASSUME(COND) do { if (!(COND)) __builtin_unreachable(); } while (0)
 #endif
 
-#else // NDEBUG
-
+#else // defined(NDEBUG) || defined(__OPTIMIZE__) || (defined(_MSC_VER) && !defined(_DEBUG))
+// This should only ever be enabled in debug mode.
 #define SIMDJSON_UNREACHABLE() assert(0);
 #define SIMDJSON_ASSUME(COND) assert(COND)
 

include/simdjson/ppc64/bitmanipulation.h

@@ -9,8 +9,12 @@ namespace {
 // but the algorithms do not end up using the returned value.
 // Sadly, sanitizers are not smart enough to figure it out.
 SIMDJSON_NO_SANITIZE_UNDEFINED
+// This function can be used safely even if not all bytes have been
+// initialized.
+// See issue https://github.com/simdjson/simdjson/issues/1965
+SIMDJSON_NO_SANITIZE_MEMORY
 simdjson_inline int trailing_zeroes(uint64_t input_num) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   unsigned long ret;
   // Search the mask data from least significant bit (LSB)
   // to the most significant bit (MSB) for a set bit (1).
@@ -28,7 +32,7 @@ simdjson_inline uint64_t clear_lowest_bit(uint64_t input_num) {
 
 /* result might be undefined when input_num is zero */
 simdjson_inline int leading_zeroes(uint64_t input_num) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   unsigned long leading_zero = 0;
   // Search the mask data from most significant bit (MSB)
   // to least significant bit (LSB) for a set bit (1).
@@ -41,7 +45,7 @@ simdjson_inline int leading_zeroes(uint64_t input_num) {
 #endif // SIMDJSON_REGULAR_VISUAL_STUDIO
 }
 
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
 simdjson_inline int count_ones(uint64_t input_num) {
   // note: we do not support legacy 32-bit Windows
   return __popcnt64(input_num); // Visual Studio wants two underscores
@@ -54,7 +58,7 @@ simdjson_inline int count_ones(uint64_t input_num) {
 
 simdjson_inline bool add_overflow(uint64_t value1, uint64_t value2,
                                          uint64_t *result) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   *result = value1 + value2;
   return *result < value1;
 #else

include/simdjson/ppc64/implementation.h

@@ -12,6 +12,9 @@ using namespace simdjson;
 using namespace simdjson::dom;
 } // namespace
 
+/**
+ * @private
+ */
 class implementation final : public simdjson::implementation {
 public:
   simdjson_inline implementation()

include/simdjson/simdjson_version.h

@@ -4,14 +4,14 @@
 #define SIMDJSON_SIMDJSON_VERSION_H
 
 /** The version of simdjson being used (major.minor.revision) */
-#define SIMDJSON_VERSION 2.2.3
+#define SIMDJSON_VERSION "3.2.0"
 
 namespace simdjson {
 enum {
   /**
    * The major version (MAJOR.minor.revision) of simdjson being used.
    */
-  SIMDJSON_VERSION_MAJOR = 2,
+  SIMDJSON_VERSION_MAJOR = 3,
   /**
    * The minor version (major.MINOR.revision) of simdjson being used.
    */
@@ -19,7 +19,7 @@ enum {
   /**
    * The revision (major.minor.REVISION) of simdjson being used.
    */
-  SIMDJSON_VERSION_REVISION = 3
+  SIMDJSON_VERSION_REVISION = 0
 };
 } // namespace simdjson
 

include/simdjson/westmere.h

@@ -9,7 +9,7 @@
 #define SIMDJSON_TARGET_WESTMERE
 #define SIMDJSON_UNTARGET_WESTMERE
 #else
-#define SIMDJSON_TARGET_WESTMERE SIMDJSON_TARGET_REGION("sse4.2,pclmul")
+#define SIMDJSON_TARGET_WESTMERE SIMDJSON_TARGET_REGION("sse4.2,pclmul,popcnt")
 #define SIMDJSON_UNTARGET_WESTMERE SIMDJSON_UNTARGET_REGION
 #endif
 

include/simdjson/westmere/bitmanipulation.h

@@ -9,8 +9,12 @@ namespace {
 // but the algorithms do not end up using the returned value.
 // Sadly, sanitizers are not smart enough to figure it out.
 SIMDJSON_NO_SANITIZE_UNDEFINED
+// This function can be used safely even if not all bytes have been
+// initialized.
+// See issue https://github.com/simdjson/simdjson/issues/1965
+SIMDJSON_NO_SANITIZE_MEMORY
 simdjson_inline int trailing_zeroes(uint64_t input_num) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   unsigned long ret;
   // Search the mask data from least significant bit (LSB)
   // to the most significant bit (MSB) for a set bit (1).
@@ -28,7 +32,7 @@ simdjson_inline uint64_t clear_lowest_bit(uint64_t input_num) {
 
 /* result might be undefined when input_num is zero */
 simdjson_inline int leading_zeroes(uint64_t input_num) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   unsigned long leading_zero = 0;
   // Search the mask data from most significant bit (MSB)
   // to least significant bit (LSB) for a set bit (1).
@@ -41,7 +45,7 @@ simdjson_inline int leading_zeroes(uint64_t input_num) {
 #endif// SIMDJSON_REGULAR_VISUAL_STUDIO
 }
 
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
 simdjson_inline unsigned __int64 count_ones(uint64_t input_num) {
   // note: we do not support legacy 32-bit Windows
   return __popcnt64(input_num);// Visual Studio wants two underscores
@@ -54,7 +58,7 @@ simdjson_inline long long int count_ones(uint64_t input_num) {
 
 simdjson_inline bool add_overflow(uint64_t value1, uint64_t value2,
                                 uint64_t *result) {
-#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
+#if SIMDJSON_REGULAR_VISUAL_STUDIO
   return _addcarry_u64(0, value1, value2,
                        reinterpret_cast<unsigned __int64 *>(result));
 #else

include/simdjson/westmere/implementation.h

@@ -12,6 +12,9 @@ using namespace simdjson;
 using namespace simdjson::dom;
 }
 
+/**
+ * @private
+ */
 class implementation final : public simdjson::implementation {
 public:
   simdjson_inline implementation() : simdjson::implementation("westmere", "Intel/AMD SSE4.2", internal::instruction_set::SSE42 | internal::instruction_set::PCLMULQDQ) {}

include/simdjson/westmere/intrinsics.h

@@ -1,7 +1,7 @@
 #ifndef SIMDJSON_WESTMERE_INTRINSICS_H
 #define SIMDJSON_WESTMERE_INTRINSICS_H
 
-#ifdef SIMDJSON_VISUAL_STUDIO
+#if SIMDJSON_VISUAL_STUDIO
 // under clang within visual studio, this will include <x86intrin.h>
 #include <intrin.h> // visual studio or clang
 #else
@@ -9,7 +9,7 @@
 #endif // SIMDJSON_VISUAL_STUDIO
 
 
-#ifdef SIMDJSON_CLANG_VISUAL_STUDIO
+#if SIMDJSON_CLANG_VISUAL_STUDIO
 /**
  * You are not supposed, normally, to include these
  * headers directly. Instead you should either include intrin.h

singleheader/amalgamate.py

@@ -112,7 +112,8 @@ def dofile(fid, prepath, filename):
                     print(f"// redefining SIMDJSON_IMPLEMENTATION to \"{current_implementation}\"\n// {line}", file=fid)
                 elif undefines_simdjson_implementation.search(line):
                     # Don't include #undef SIMDJSON_IMPLEMENTATION since we're handling it ourselves
-                    print(f"// {line}")
+                    # print(f"// {line}")
+                    pass
                 else:
                     # copy the line, with SIMDJSON_IMPLEMENTATION replace to what it is currently defined to
                     print(uses_simdjson_implementation.sub(current_implementation+"\\1",line), file=fid)

src/arm64/dom_parser_implementation.cpp

@@ -152,8 +152,12 @@ simdjson_warn_unused error_code dom_parser_implementation::stage2_next(dom::docu
   return stage2::tape_builder::parse_document<true>(*this, _doc);
 }
 
-simdjson_warn_unused uint8_t *dom_parser_implementation::parse_string(const uint8_t *src, uint8_t *dst) const noexcept {
-  return arm64::stringparsing::parse_string(src, dst);
+simdjson_warn_unused uint8_t *dom_parser_implementation::parse_string(const uint8_t *src, uint8_t *dst, bool allow_replacement) const noexcept {
+  return arm64::stringparsing::parse_string(src, dst, allow_replacement);
+}
+
+simdjson_warn_unused uint8_t *dom_parser_implementation::parse_wobbly_string(const uint8_t *src, uint8_t *dst) const noexcept {
+  return arm64::stringparsing::parse_wobbly_string(src, dst);
 }
 
 simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *_buf, size_t _len, dom::document &_doc) noexcept {

src/fallback/dom_parser_implementation.cpp

@@ -360,8 +360,12 @@ simdjson_warn_unused error_code dom_parser_implementation::stage2_next(dom::docu
   return stage2::tape_builder::parse_document<true>(*this, _doc);
 }
 
-simdjson_warn_unused uint8_t *dom_parser_implementation::parse_string(const uint8_t *src, uint8_t *dst) const noexcept {
-  return fallback::stringparsing::parse_string(src, dst);
+simdjson_warn_unused uint8_t *dom_parser_implementation::parse_string(const uint8_t *src, uint8_t *dst, bool replacement_char) const noexcept {
+  return fallback::stringparsing::parse_string(src, dst, replacement_char);
+}
+
+simdjson_warn_unused uint8_t *dom_parser_implementation::parse_wobbly_string(const uint8_t *src, uint8_t *dst) const noexcept {
+  return fallback::stringparsing::parse_wobbly_string(src, dst);
 }
 
 simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *_buf, size_t _len, dom::document &_doc) noexcept {

src/fallback/implementation.cpp

@@ -1,5 +1,4 @@
 #include "simdjson/fallback/begin.h"
-
 namespace simdjson {
 namespace SIMDJSON_IMPLEMENTATION {