Fixing typo

* tag_invoke based custom types (#2219)

* tag_invoke based custom types

Now you can use tag_invoke to add a custom type or a group of custom types.

* Fixing macro usage + Fixing noexcept

* Fixing the usage of #include

We don't need <concepts> at all seems like it

* Fixing tag_invoke impl for MSVC

* Making `tag_invoke` to support `ondemand::document` as well + docs (#2228)

* Making `tag_invoke` to support `ondemand::document` as well + docs

* Fix typos and doc update by @lemire

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Better docs by @lemire

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* Preserving the old, disallowing in the new

I'm disabling `document::get() &&` if the user has provided a `tag_invoke`d version; otherwise, we retain the compatibility.

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>

* fix: correct small issues with deserialize (#2232)

* Extending the deserialization code with more defaults + docs (#2233)

* Make custom types easier with some predefined cases + docs

* missing include

* adding Ubuntu 24 CXX 20

* using concepts all the way

* minor tweak

* tiny tweak

* tweaks

* more tweaking

* saving

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>

* Making `tag_invoke` a "put" as opposed to a "get" (#2256)

* fix: add tests related to issue 2227 (#2229)

* fix: add tests related to issue 2227

* avoiding name clash

* pedantic fix

* deprecate rvalue get on document

* selectively deprecating

* Fix ndjson spec link (#2234)

* fix ndjson spec link

The link in the readme of parse_many links to a casino spam site

* fix link

* [no-ci] Update README.md

* Make simdjson compile again

* Enable SIMDJSON_SINGLEHEADER=OFF in VS Code

With singleheader on, clangd can't find the right
include files.

* Add missing include directives to static build targets of simdjson. (#2240)

* adding a warning

* adding warning regarding SIMDJSON_BUILD_STATIC_LIB

* release candidate

* pedantic viable size

* Making tag_invoke a feeder instead of a producer

* adding missing undef silencer (#2253)

* Ignore pragma once when amalgamating source files (#2248)

With gcc it causes an error in `simdjson.cpp`:
```
simdjson.cpp:548:9: warning: #pragma once in main file
  548 | #pragma once
      |         ^~~~
```

It had previously been commented out in:
https://github.com/simdjson/simdjson/commit/6ef555e6fb79363fae057a9a46b52cd208d9e305

However, this was lost in an upgrade:
https://github.com/simdjson/simdjson/commit/2a4ff7346813b120f2b5b40e95d69352b593cc9c

* Update CI (#2254)

* adding missing undef silencer

* Updating CI

* more fixes

* fix

* big endian fix

* Moving to the new tag_invoke signature

* Fix nlohmann ambiguity on C++23-enabled clang

* Revert "Merge branch 'master' of https://github.com/simdjson/simdjson into builder_development_branch_extra"

This reverts commit 3eeecbab34, reversing
changes made to 6858b208b4.

---------

Co-authored-by: Daniel Lemire <daniel@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>

* update CI on the builder_development_branch (no code change) (#2262)

* typo

* General madness simpler, no simpler!!! (#2267)

* Minimal tag_invokes for STL types

* simpler madness

* adding a comment

* missing file

* minor tweaks to style

* fixing incorrect max/min usage

* updating single

* simplify

* validating the idea

* putting back the concept

* moving the include

* guarding

* Cheap General Madness (#2268)

* Some General Concepts and their deserializations

* Resolving ambiguity

* Add missing #include

* C++20 custom deserializer: better documentation (#2269)

* mostly a documentation update.

* missing cpp

* [no-ci] fix comment

* various minor fixes

---------

Co-authored-by: Daniel Lemire <dlemire@lemire.me>

---------

Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: M. Bahoosh <moisrex@gmail.com>

* minor update

* More documentation regarding builder (#2270)

* minor update

* more improvment to our documentation (builder branch)

* putting back missing functions

* merge candidate

---------

Co-authored-by: M. Bahoosh <moisrex@gmail.com>
Co-authored-by: Daniel Lemire <dlemire@lemire.me>
Co-authored-by: Sasha Lopoukhine <superlopuh@gmail.com>
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Tan Li Boon <undisputed-seraphim@users.noreply.github.com>
Co-authored-by: tobil4sk <tobil4sk@outlook.com>
Co-authored-by: M. Bahoosh <12122474+the-moisrex@users.noreply.github.com>

.clangd

@@ -25,6 +25,7 @@ CompileFlags:
 Diagnostics:
     Suppress:
         - pp_including_mainfile_in_preamble
+        - unused-includes
 ---
 # Amalgamated files that require or partly define an implementation
 If:

.github/workflows/vs17-arm-ci.yml

@@ -10,7 +10,6 @@ jobs:
        fail-fast: false
        matrix:
          include:
-           - {arch: ARM}
            - {arch: ARM64}
            - {arch: ARM64EC}
      steps:

CMakeLists.txt

@@ -3,7 +3,7 @@ cmake_minimum_required(VERSION 3.14)
 project(
     simdjson
     # The version number is modified by tools/release.py
-    VERSION 3.10.0
+    VERSION 3.10.1
     DESCRIPTION "Parsing gigabytes of JSON per second"
     HOMEPAGE_URL "https://simdjson.org/"
     LANGUAGES CXX C
@@ -23,7 +23,11 @@ string(
 set(SIMDJSON_LIB_VERSION "23.0.0" CACHE STRING "simdjson library version")
 set(SIMDJSON_LIB_SOVERSION "23" CACHE STRING "simdjson library soversion")
 
-option(SIMDJSON_BUILD_STATIC_LIB "Build simdjson_static library along with simdjson" OFF)
+option(SIMDJSON_BUILD_STATIC_LIB "Build simdjson_static library along with simdjson (only makes sense if BUILD_SHARED_LIBS=ON)" OFF)
+if(SIMDJSON_BUILD_STATIC_LIB AND NOT BUILD_SHARED_LIBS)
+  message(WARNING "SIMDJSON_BUILD_STATIC_LIB only makes sense if BUILD_SHARED_LIBS is set to ON")
+  message(WARNING "You might be building and installing a two identical static libraries.")
+endif()
 
 option(SIMDJSON_ENABLE_THREADS "Link with thread support" ON)
 
@@ -51,6 +55,7 @@ endif()
 if(is_top_project)
   option(SIMDJSON_DEVELOPER_MODE "Enable targets for developing simdjson" OFF)
   option(BUILD_SHARED_LIBS "Build simdjson as a shared library" OFF)
+  option(SIMDJSON_SINGLEHEADER "Disable singleheader generation" ON)
 endif()
 
 include(cmake/handle-deprecations.cmake)
@@ -155,11 +160,13 @@ endif()
 include(CMakePackageConfigHelpers)
 include(GNUInstallDirs)
 
-install(
+if(SIMDJSON_SINGLEHEADER)
-    FILES singleheader/simdjson.h
+  install(
-    DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
+      FILES singleheader/simdjson.h
-    COMPONENT simdjson_Development
+      DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
-)
+      COMPONENT simdjson_Development
+  )
+endif()
 
 install(
     TARGETS simdjson
@@ -203,6 +210,7 @@ if(SIMDJSON_BUILD_STATIC_LIB)
       TARGETS simdjson_static
       EXPORT simdjson_staticTargets
       ARCHIVE COMPONENT simdjson_Development
+      INCLUDES DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
   )
   install(
       EXPORT simdjson_staticTargets
@@ -279,6 +287,7 @@ enable_testing()
 add_custom_target(all_tests)
 
 add_subdirectory(windows)
+include(cmake/CPM.cmake)
 add_subdirectory(dependencies) ## This needs to be before tools because of cxxopts
 add_subdirectory(tools)  ## This needs to be before tests because of cxxopts
 
@@ -286,8 +295,9 @@ add_subdirectory(tools)  ## This needs to be before tests because of cxxopts
 # most of the data has been moved to https://github.com/simdjson/simdjson-data
 add_subdirectory(jsonexamples)
 
-
+if(SIMDJSON_SINGLEHEADER)
 add_subdirectory(singleheader)
+endif()
 
 
 

Doxyfile

@@ -38,7 +38,7 @@ PROJECT_NAME           = simdjson
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER         = "3.10.0"
+PROJECT_NUMBER         = "3.10.1"
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a

README.md

@@ -201,8 +201,8 @@ For the video inclined, <br />
 Funding
 -------
 
-The work is supported by the Natural Sciences and Engineering Research Council of Canada under grant
+The work is supported by the Natural Sciences and Engineering Research Council of Canada under grants
-number RGPIN-2017-03910.
+RGPIN-2017-03910 and RGPIN-2024-03787.
 
 [license]: LICENSE
 [license img]: https://img.shields.io/badge/License-Apache%202-blue.svg

cmake/CPM.cmake

@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: MIT
+#
+# SPDX-FileCopyrightText: Copyright (c) 2019-2023 Lars Melchior and contributors
+
+set(CPM_DOWNLOAD_VERSION 0.40.2)
+set(CPM_HASH_SUM "c8cdc32c03816538ce22781ed72964dc864b2a34a310d3b7104812a5ca2d835d")
+
+if(CPM_SOURCE_CACHE)
+  set(CPM_DOWNLOAD_LOCATION "${CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
+elseif(DEFINED ENV{CPM_SOURCE_CACHE})
+  set(CPM_DOWNLOAD_LOCATION "$ENV{CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
+else()
+  set(CPM_DOWNLOAD_LOCATION "${CMAKE_BINARY_DIR}/cmake/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
+endif()
+
+# Expand relative path. This is important if the provided path contains a tilde (~)
+get_filename_component(CPM_DOWNLOAD_LOCATION ${CPM_DOWNLOAD_LOCATION} ABSOLUTE)
+
+file(DOWNLOAD
+     https://github.com/cpm-cmake/CPM.cmake/releases/download/v${CPM_DOWNLOAD_VERSION}/CPM.cmake
+     ${CPM_DOWNLOAD_LOCATION} EXPECTED_HASH SHA256=${CPM_HASH_SUM}
+)
+
+include(${CPM_DOWNLOAD_LOCATION})

dependencies/CMakeLists.txt

@@ -1,5 +1,4 @@
 include(CMakeDependentOption)
-include(import.cmake)
 
 option(SIMDJSON_ALLOW_DOWNLOADS
         "Allow dependencies to be downloaded during configure time"
@@ -11,17 +10,21 @@ cmake_dependent_option(SIMDJSON_GOOGLE_BENCHMARKS "compile the Google Benchmark
         SIMDJSON_ALLOW_DOWNLOADS OFF)
 
 if(SIMDJSON_GOOGLE_BENCHMARKS)
-  set_off(BENCHMARK_ENABLE_TESTING)
+  CPMAddPackage(
-  set_off(BENCHMARK_ENABLE_INSTALL)
+        NAME google_benchmarks
-  set_off(BENCHMARK_ENABLE_WERROR)
+        URL https://github.com/google/benchmark/archive/refs/tags/v1.7.1.zip
-
+        OPTIONS
-  import_dependency(google_benchmarks google/benchmark v1.7.1)
+        "BENCHMARK_ENABLE_TESTING OFF"
-  add_dependency(google_benchmarks)
+        "BENCHMARK_ENABLE_INSTALL OFF"
+        "BENCHMARK_ENABLE_WERROR OFF"
+  )
 endif()
 
-# The bulk of our benchmarking and testing data has been moved simdjson/simdjson-data
+CPMAddPackage(
-import_dependency(simdjson-data simdjson/simdjson-data a5b13babe65c1bba7186b41b43d4cbdc20a5c470)
+        NAME simdjson-data
-add_dependency(simdjson-data)
+        URL https://github.com/simdjson/simdjson-data/archive/a5b13babe65c1bba7186b41b43d4cbdc20a5c470.zip
+)
+
 option(SIMDJSON_USE_BOOST_JSON "Try to include BOOST_JSON, this may break your binaries under some systems." OFF)
 # This prevents variables declared with set() from unnecessarily escaping and
 # should not be called more than once
@@ -38,20 +41,30 @@ function(competition_scope_)
 int main() {}
 ]] SIMDJSON_FOUND_STRING_VIEW)
   if(SIMDJSON_FOUND_STRING_VIEW AND SIMDJSON_USE_BOOST_JSON)
-    import_dependency(boostjson boostorg/json ee8d72d)
+    CPMAddPackage(
+        NAME boostjson
+        URL https://github.com/boostorg/json/archive/ee8d72d8502b409b5561200299cad30ccdb91415.zip
+    )
     add_library(boostjson STATIC "${boostjson_SOURCE_DIR}/src/src.cpp")
     target_compile_definitions(boostjson PUBLIC BOOST_JSON_STANDALONE)
     target_include_directories(boostjson SYSTEM PUBLIC
             "${boostjson_SOURCE_DIR}/include")
     target_compile_definitions(boostjson INTERFACE SIMDJSON_COMPETITION_BOOSTJSON)
   endif()
-
+  CPMAddPackage(
-  import_dependency(cjson DaveGamble/cJSON c69134d)
+      NAME cjson
+      URL https://github.com/DaveGamble/cJSON/archive/c69134d01746dcf551dd7724b4edb12f922eb0d1.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(cjson STATIC "${cjson_SOURCE_DIR}/cJSON.c")
   target_include_directories(cjson SYSTEM PUBLIC "${cjson_SOURCE_DIR}")
   target_compile_definitions(cjson INTERFACE SIMDJSON_COMPETITION_CJSON)
 
-  import_dependency(fastjson mikeando/fastjson 485f994)
+  CPMAddPackage(
+      NAME fastjson
+      URL https://github.com/mikeando/fastjson/archive/485f994a61a64ac73fa6a40d4d639b99b463563b.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(fastjson STATIC
           "${fastjson_SOURCE_DIR}/src/fastjson.cpp"
           "${fastjson_SOURCE_DIR}/src/fastjson2.cpp"
@@ -60,28 +73,36 @@ int main() {}
           "${fastjson_SOURCE_DIR}/include")
   target_compile_definitions(fastjson INTERFACE SIMDJSON_COMPETITION_FASTJSON)
 
-  import_dependency(gason vivkin/gason 7aee524)
+  CPMAddPackage(
+      NAME gason
+      URL https://github.com/vivkin/gason/archive/7aee524189da1c1ecd19f67981e3d903dae25470.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(gason STATIC "${gason_SOURCE_DIR}/src/gason.cpp")
   target_include_directories(gason SYSTEM PUBLIC "${gason_SOURCE_DIR}/src")
   target_compile_definitions(gason INTERFACE SIMDJSON_COMPETITION_GASON)
 
-  import_dependency(jsmn zserge/jsmn 18e9fe4)
+  CPMAddPackage(
+      NAME jsmn
+      URL https://github.com/zserge/jsmn/archive/18e9fe42cbfe21d65076f5c77ae2be379ad1270f.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(jsmn STATIC "${jsmn_SOURCE_DIR}/jsmn.c")
   target_include_directories(jsmn SYSTEM PUBLIC "${jsmn_SOURCE_DIR}")
   target_compile_definitions(jsmn INTERFACE SIMDJSON_COMPETITION_JSMN)
 
-  message(STATUS "Importing json (nlohmann/json@v3.10.5)")
+  CPMAddPackage(
-  set(nlohmann_json_SOURCE_DIR "${dep_root}/json")
+      NAME nlohmann_json
-  if(NOT EXISTS "${nlohmann_json_SOURCE_DIR}")
+      URL https://github.com/nlohmann/json/archive/refs/tags/v3.10.5.zip
-    file(DOWNLOAD
+  )
-            "https://github.com/nlohmann/json/releases/download/v3.10.5/json.hpp"
-            "${nlohmann_json_SOURCE_DIR}/nlohmann/json.hpp")
-  endif()
-  add_library(nlohmann_json INTERFACE)
-  target_include_directories(nlohmann_json SYSTEM INTERFACE "${nlohmann_json_SOURCE_DIR}")
-  target_compile_definitions(nlohmann_json INTERFACE SIMDJSON_COMPETITION_NLOHMANN_JSON)
 
-  import_dependency(json11 dropbox/json11 ec4e452)
+  set_property(TARGET nlohmann_json APPEND PROPERTY INTERFACE_COMPILE_DEFINITIONS SIMDJSON_COMPETITION_NLOHMANN_JSON)
+
+  CPMAddPackage(
+      NAME json11
+      URL https://github.com/dropbox/json11/archive/ec4e45219af1d7cde3d58b49ed762376fccf1ace.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(json11 STATIC "${json11_SOURCE_DIR}/json11.cpp")
   target_include_directories(json11 SYSTEM PUBLIC "${json11_SOURCE_DIR}")
   target_compile_definitions(json11 INTERFACE SIMDJSON_COMPETITION_JSON11)
@@ -91,7 +112,11 @@ int main() {}
   target_include_directories(jsoncpp SYSTEM PUBLIC "${jsoncpp_SOURCE_DIR}")
   target_compile_definitions(jsoncpp INTERFACE SIMDJSON_COMPETITION_JSONCPP)
 
-  import_dependency(rapidjson Tencent/rapidjson f54b0e4)
+  CPMAddPackage(
+      NAME rapidjson
+      URL https://github.com/Tencent/rapidjson/archive/f54b0e47a08782a6131cc3d60f94d038fa6e0a51.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(rapidjson INTERFACE)
   target_compile_definitions(rapidjson INTERFACE RAPIDJSON_HAS_STDSTRING)
   include (TestBigEndian)
@@ -110,14 +135,22 @@ int main() {}
     target_compile_definitions(rapidjson INTERFACE SIMDJSON_COMPETITION_RAPIDJSON)
   endif()
 
-  import_dependency(sajson chadaustin/sajson 2dcfd35)
+  CPMAddPackage(
+      NAME sajson
+      URL https://github.com/chadaustin/sajson/archive/2dcfd350586375f9910f74821d4f07d67ae455ba.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(sajson INTERFACE)
   target_compile_definitions(sajson INTERFACE SAJSON_UNSORTED_OBJECT_KEYS)
   target_include_directories(sajson SYSTEM INTERFACE
           "${sajson_SOURCE_DIR}/include")
   target_compile_definitions(sajson INTERFACE SIMDJSON_COMPETITION_SAJSON)
 
-  import_dependency(ujson4c esnme/ujson4c e14f3fd)
+  CPMAddPackage(
+      NAME ujson4c
+      URL https://github.com/esnme/ujson4c/archive/e14f3fd5207fe30d1bdea723f260609e69d1abfa.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(ujson4c STATIC
           "${ujson4c_SOURCE_DIR}/src/ujdecode.c"
           "${ujson4c_SOURCE_DIR}/3rdparty/ultrajsondec.c")
@@ -126,7 +159,11 @@ int main() {}
           "${ujson4c_SOURCE_DIR}/3rdparty")
   target_compile_definitions(ujson4c INTERFACE SIMDJSON_COMPETITION_UJSON4C)
 
-  import_dependency(yyjson ibireme/yyjson c385651)
+  CPMAddPackage(
+      NAME yyjson
+      URL https://github.com/ibireme/yyjson/archive/c3856514de0a67d7b66939bf3ed491a2d6e61277.zip
+      DOWNLOAD_ONLY YES
+  )
   add_library(yyjson STATIC "${yyjson_SOURCE_DIR}/src/yyjson.c")
   target_include_directories(yyjson SYSTEM PUBLIC "${yyjson_SOURCE_DIR}/src")
   target_compile_definitions(yyjson INTERFACE SIMDJSON_COMPETITION_YYJSON)
@@ -151,10 +188,12 @@ cmake_dependent_option(SIMDJSON_CXXOPTS "Download cxxopts (necessary for tools)"
         SIMDJSON_ALLOW_DOWNLOADS OFF)
 
 if(SIMDJSON_CXXOPTS)
-  set_off(CXXOPTS_BUILD_EXAMPLES)
+  CPMAddPackage(
-  set_off(CXXOPTS_BUILD_TESTS)
+        NAME cxxopts
-  set_off(CXXOPTS_ENABLE_INSTALL)
+        URL https://github.com/jarro2783/cxxopts/archive/59656709c0c58fcd0ed18b38e02938dbe05284c5.zip
-
+        OPTIONS
-  import_dependency(cxxopts jarro2783/cxxopts 5965670)
+        "CXXOPTS_BUILD_EXAMPLES OFF"
-  add_dependency(cxxopts)
+        "CXXOPTS_BUILD_TESTS OFF"
+        "CXXOPTS_ENABLE_INSTALL OFF"
+  )
 endif()

dependencies/import.cmake

@@ -1,48 +0,0 @@
-set(dep_root "${simdjson_SOURCE_DIR}/dependencies/.cache")
-if(DEFINED ENV{simdjson_DEPENDENCY_CACHE_DIR})
-  set(dep_root "$ENV{simdjson_DEPENDENCY_CACHE_DIR}")
-endif()
-
-function(import_dependency NAME GITHUB_REPO COMMIT)
-  message(STATUS "Importing ${NAME} (${GITHUB_REPO}@${COMMIT})")
-  set(target "${dep_root}/${NAME}")
-
-  # If the folder exists in the cache, then we assume that everything is as
-  # should be and do nothing
-  if(EXISTS "${target}")
-    set("${NAME}_SOURCE_DIR" "${target}" PARENT_SCOPE)
-    return()
-  endif()
-
-  set(zip_url "https://github.com/${GITHUB_REPO}/archive/${COMMIT}.zip")
-  set(archive "${dep_root}/archive.zip")
-  set(dest "${dep_root}/_extract")
-
-  file(DOWNLOAD "${zip_url}" "${archive}")
-  file(MAKE_DIRECTORY "${dest}")
-  execute_process(
-          WORKING_DIRECTORY "${dest}"
-          COMMAND "${CMAKE_COMMAND}" -E tar xf "${archive}")
-  file(REMOVE "${archive}")
-
-  # GitHub archives only ever have one folder component at the root, so this
-  # will always match that single folder
-  file(GLOB dir LIST_DIRECTORIES YES "${dest}/*")
-
-  file(RENAME "${dir}" "${target}")
-
-  set("${NAME}_SOURCE_DIR" "${target}" PARENT_SCOPE)
-endfunction()
-
-# Delegates to the dependency
-macro(add_dependency NAME)
-  if(NOT DEFINED "${NAME}_SOURCE_DIR")
-    message(FATAL_ERROR "Missing ${NAME}_SOURCE_DIR variable")
-  endif()
-
-  add_subdirectory("${${NAME}_SOURCE_DIR}" "${PROJECT_BINARY_DIR}/_deps/${NAME}" EXCLUDE_FROM_ALL)
-endmacro()
-
-function(set_off NAME)
-  set("${NAME}" OFF CACHE INTERNAL "")
-endfunction()

doc/basics.md

@@ -249,7 +249,7 @@ documents.
 For code safety, you should keep (1) the `parser` instance, (2) the input string and (3) the document instance alive throughout your parsing. Additionally, you should follow the following rules:
 
 - A `parser` may have at most one document open at a time, since it holds allocated memory used for the parsing.
-- By design, you should only have one `document` instance per JSON document. Thus, if you must pass a document instance to a function, you should avoid passing it by value: choose to pass it by reference instance to avoid the copy. (We also provide a `document_reference` class if you need to pass by value.)
+- By design, you should only have one `document` instance per JSON document. Thus, if you must pass a document instance to a function, you should avoid passing it by value: choose to pass it by reference instance to avoid the copy. In any case, the `document` class does not have a copy constructor.
 
 During the `iterate` call, the original JSON text is never modified--only read. After you are done
 with the document, the source (whether file or string) can be safely discarded.
@@ -1428,9 +1428,11 @@ be represented as `value` instances. You can check that a document is a scalar w
 JSONPath
 ------------
 
-The simdjson library now supports a subset of [JSONPath](https://datatracker.ietf.org/doc/html/draft-normington-jsonpath-00) through the `at_path()` method, allowing you to reach further into the document in a single call. The subset of JSONPath that is implemented is the subset that is trivially convertible into the JSON Pointer format, using `.` to access a field and `[]` to access a specific index.
+The simdjson library supports a subset of [JSONPath](https://datatracker.ietf.org/doc/html/draft-normington-jsonpath-00) through the `at_path()` method, allowing you to reach further into the document in a single call. The subset of JSONPath that is implemented is the subset that is trivially convertible into the JSON Pointer format, using `.` to access a field and `[]` to access a specific index.
 
-This implementation relies on `at_path()` converting its argument to JSON Pointer and then calling `at_pointer`, which makes use of [`rewind`](#rewind) to reset the parser at the beginning of the document. Hence, it invalidates all previously parsed values, objects and arrays: make sure to consume the values between each call to `at_path`.
+This implementation relies on `at_path()` converting its argument to JSON Pointer and then calling `at_pointer`, which makes use of
+[`rewind`](#rewind) to reset the parser at the beginning of the document. Hence, it invalidates all previously parsed values, objects
+ and arrays: make sure to consume the values between each call to `at_path`.
 
 Consider the following example:
 
@@ -1471,6 +1473,18 @@ doc.at_path(".\\u00E9") == 123; // true
 doc.at_path((const char*)u8".\u00E9") // returns an error (NO_SUCH_FIELD)
 ```
 
+
+We also support the `$` prefix. When you start a JSONPath expression with $, you are indicating that the path starts from the root of the JSON document. E.g.,
+
+```c++
+auto json = R"( { "c" :{ "foo": { "a": [ 10, 20, 30 ] }}, "d": { "foo2": { "a": [ 10, 20, 30 ] }} , "e": 120 })"_padded;
+ondemand::parser parser;
+ondemand::document doc = parser.iterate(json);
+ondemand::object obj = doc.get_object();
+int64_t x = obj.at_path("$.c.foo.a[1]"); // 20
+x = obj.at_path("$.d.foo2.a.2"); // 30
+```
+
 Error handling
 --------------
 
@@ -2876,6 +2890,7 @@ Performance tips
 	std::string_view year = data["year"];
 	std::string_view rating = data["rating"];
   ```
+- You will get better performance if you seek the keys in the order in which they appear in the document. So if processing `{"a":1, "b":2, "c":3}`, do `value1 = data["a"]; value2 = data["b"]; value3 data["c"];` and not `value2 = data["b"]; value1 = data["a"]; value3 data["c"];`. Of course, it is not always possible to know for sure in which order the keys appear.
 
 
 

doc/dom.md

@@ -8,6 +8,7 @@ An overview of what you need to know to use simdjson, with examples.
 * [Using the Parsed JSON](#using-the-parsed-json)
 * [C++17 Support](#c17-support)
 * [JSON Pointer](#json-pointer)
+* [JSONPath](#jsonpath)
 * [Error Handling](#error-handling)
   * [Error Handling Example](#error-handling-example)
   * [Exceptions](#exceptions)
@@ -257,6 +258,49 @@ for (dom::element car_element : cars) {
 }
 ```
 
+JSONPath
+------------
+
+
+The simdjson library supports a subset of [JSONPath](https://datatracker.ietf.org/doc/html/draft-normington-jsonpath-00) through the `at_path()` method, allowing you to reach further into the document in a single call. The subset of JSONPath that is implemented is the subset that is trivially convertible into the JSON Pointer format, using `.` to access a field and `[]` to access a specific index.
+
+Consider the following example:
+
+```c++
+auto cars_json = R"( [
+  { "make": "Toyota", "model": "Camry",  "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
+  { "make": "Kia",    "model": "Soul",   "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
+  { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }
+] )"_padded;
+dom::parser parser;
+dom::element doc;
+auto error = parser.parse(cars_json).get(doc);
+if(error) { /*won't happen*/ }
+double p;
+error = doc.at_path("[0].tire_pressure[1]").get(p);
+if(error) { /*won't happen*/ }
+cout << p << endl; // Prints 39.9
+```
+
+
+We also support the `$` prefix. When you start a JSONPath expression with $, you are indicating that the path starts from the root of the JSON document. E.g.,
+
+```c++
+auto json = R"( { "c" :{ "foo": { "a": [ 10, 20, 30 ] }}, "d": { "foo2": { "a": [ 10, 20, 30 ] }} , "e": 120 })"_padded;
+dom::parser parser;
+dom::element doc;
+auto error = parser.parse(json).get(doc);
+if(error) { /*won't happen*/ }
+dom::object obj;
+error = doc.get_object().get(obj);
+if(error) { /*won't happen*/ }
+int64_t x;
+error = obj.at_path("$[3].foo.a[1]").get(x);
+if(error) { /*won't happen*/ }
+if(x != 20) { /*won't happen*/ }
+x = obj.at_path("$.d.foo2.a.2");
+if(error) { /*won't happen*/ }
+```
 
 
 Error Handling

doc/iterate_many.md

@@ -103,7 +103,12 @@ remove almost entirely its cost and replaces it by the overhead of a thread, whi
 cheaper. Ain't that awesome!
 
 Thread support is only active if thread supported is detected in which case the macro
-SIMDJSON_THREADS_ENABLED is set. Otherwise the library runs in  single-thread mode.
+SIMDJSON_THREADS_ENABLED is set.  You can also manually pass `SIMDJSON_THREADS_ENABLED=1` flag
+to the library. Otherwise the library runs in single-thread mode.
+
+You should be consistent. If you link against the simdjson library built for multithreading
+(i.e., with `SIMDJSON_THREADS_ENABLED`), then you should build your application with multithreading
+system (setting `SIMDJSON_THREADS_ENABLED=1` and linking against a thread library).
 
 A `document_stream` instance uses at most two threads: there is a main thread and a worker thread.
 

doc/parse_many.md

@@ -103,7 +103,12 @@ cases, remove almost entirely its cost and replaces it by the overhead of a thre
 of magnitude cheaper. Ain't that awesome!
 
 Thread support is only active if thread supported is detected in which case the macro
-SIMDJSON_THREADS_ENABLED is set. Otherwise the library runs in  single-thread mode.
+SIMDJSON_THREADS_ENABLED is set.  You can also manually pass `SIMDJSON_THREADS_ENABLED=1` flag
+to the library. Otherwise the library runs in single-thread mode.
+
+You should be consistent. If you link against the simdjson library built for multithreading
+(i.e., with `SIMDJSON_THREADS_ENABLED`), then you should build your application with multithreading
+system (setting `SIMDJSON_THREADS_ENABLED=1` and linking against a thread library).
 
 A `document_stream` instance uses at most two threads: there is a main thread and a worker thread.
 You should expect the main thread to be fully occupied while the worker thread is partially busy
@@ -125,7 +130,7 @@ Whitespace Characters:
 - **Nothing**
 
 Some official formats **(non-exhaustive list)**:
-- [Newline-Delimited JSON (NDJSON)](http://ndjson.org/)
+- [Newline-Delimited JSON (NDJSON)](https://github.com/ndjson/ndjson-spec)
 - [JSON lines (JSONL)](http://jsonlines.org/)
 - [Record separator-delimited JSON (RFC 7464)](https://tools.ietf.org/html/rfc7464) <- Not supported by JsonStream!
 - [More on Wikipedia...](https://en.wikipedia.org/wiki/JSON_streaming)

doc/performance.md

@@ -241,8 +241,8 @@ long page_size() {
 // Returns true if the buffer + len + simdjson::SIMDJSON_PADDING crosses the
 // page boundary.
 bool need_allocation(const char *buf, size_t len) {
-  return ((reinterpret_cast<uintptr_t>(buf + len - 1) % page_size()) <
+  return ((reinterpret_cast<uintptr_t>(buf + len - 1) % page_size())
-          simdjson::SIMDJSON_PADDING);
+    + simdjson::SIMDJSON_PADDING > static_cast<uintptr_t>(page_size()));
 }
 
 simdjson::padded_string_view

include/simdjson/arm64/bitmanipulation.h

@@ -35,6 +35,12 @@ simdjson_inline uint64_t clear_lowest_bit(uint64_t input_num) {
   return input_num & (input_num-1);
 }
 
+// We sometimes call leading_zeroes on inputs that are zero,
+// but the algorithms do not end up using the returned value.
+// Sadly, sanitizers are not smart enough to figure it out.
+// Applies only when SIMDJSON_PREFER_REVERSE_BITS is defined and true.
+// (See below.)
+SIMDJSON_NO_SANITIZE_UNDEFINED
 /* result might be undefined when input_num is zero */
 simdjson_inline int leading_zeroes(uint64_t input_num) {
 #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO

include/simdjson/arm64/implementation.h

@@ -23,6 +23,7 @@ public:
   ) const noexcept final;
   simdjson_warn_unused error_code minify(const uint8_t *buf, size_t len, uint8_t *dst, size_t &dst_len) const noexcept final;
   simdjson_warn_unused bool validate_utf8(const char *buf, size_t len) const noexcept final;
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final;
 };
 
 } // namespace arm64

include/simdjson/arm64/numberparsing_defs.h

@@ -51,6 +51,12 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace arm64
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_ARM64_NUMBERPARSING_DEFS_H

include/simdjson/arm64/simd.h

@@ -134,6 +134,12 @@ namespace {
       tmp = vpaddq_u8(tmp, tmp);
       return vgetq_lane_u16(vreinterpretq_u16_u8(tmp), 0);
     }
+    // Returns 4-bit out of each byte, alternating between the high 4 bits and low
+    // bits result it is 64 bit.
+    simdjson_inline uint64_t to_bitmask64() const {
+      return vget_lane_u64(
+          vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(*this), 4)), 0);
+    }
     simdjson_inline bool any() const { return vmaxvq_u32(vreinterpretq_u32_u8(*this)) != 0; }
   };
 
@@ -210,7 +216,7 @@ namespace {
 
     // Bit-specific operations
     simdjson_inline simd8<bool> any_bits_set(simd8<uint8_t> bits) const { return vtstq_u8(*this, bits); }
-    simdjson_inline bool any_bits_set_anywhere() const { return this->max_val() != 0; }
+    simdjson_inline bool any_bits_set_anywhere() const { return vmaxvq_u32(vreinterpretq_u32_u8(*this)) != 0; }
     simdjson_inline bool any_bits_set_anywhere(simd8<uint8_t> bits) const { return (*this & bits).any_bits_set_anywhere(); }
     template<int N>
     simdjson_inline simd8<uint8_t> shr() const { return vshrq_n_u8(*this, N); }
@@ -223,7 +229,12 @@ namespace {
       return lookup_table.apply_lookup_16_to(*this);
     }
 
-
+    // Returns 4-bit out of each byte, alternating between the high 4 bits and low
+    // bits result it is 64 bit.
+    simdjson_inline uint64_t to_bitmask64() const {
+      return vget_lane_u64(
+          vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(*this), 4)), 0);
+    }
     // Copies to 'output" all bytes corresponding to a 0 in the mask (interpreted as a bitset).
     // Passing a 0 value for mask would be equivalent to writing out every byte to output.
     // Only the first 16 - count_ones(mask) bytes of the result are significant but 16 bytes

include/simdjson/arm64/stringparsing_defs.h

@@ -46,6 +46,32 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
   };
 }
 
+struct escaping {
+  static constexpr uint32_t BYTES_PROCESSED = 16;
+  simdjson_inline static escaping copy_and_find(const uint8_t *src, uint8_t *dst);
+
+  simdjson_inline bool has_escape() { return escape_bits != 0; }
+  simdjson_inline int escape_index() { return trailing_zeroes(escape_bits) / 4; }
+
+  uint64_t escape_bits;
+}; // struct escaping
+
+
+
+simdjson_inline escaping escaping::copy_and_find(const uint8_t *src, uint8_t *dst) {
+  static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "escaping finder must process fewer than SIMDJSON_PADDING bytes");
+  simd8<uint8_t> v(src);
+  v.store(dst);
+  simd8<bool> is_quote = (v == '"');
+  simd8<bool> is_backslash = (v == '\\');
+  simd8<bool> is_control = (v < 32);
+  return {
+    (is_backslash | is_quote | is_control).to_bitmask64()
+  };
+}
+
+
+
 } // unnamed namespace
 } // namespace arm64
 } // namespace simdjson

include/simdjson/common_defs.h

@@ -50,6 +50,8 @@ double from_chars(const char *first, const char* end) noexcept;
 #define SIMDJSON_ISALIGNED_N(ptr, n) (((uintptr_t)(ptr) & ((n)-1)) == 0)
 
 #if SIMDJSON_REGULAR_VISUAL_STUDIO
+  // We could use [[deprecated]] but it requires C++14
+  #define simdjson_deprecated __declspec(deprecated)
 
   #define simdjson_really_inline __forceinline
   #define simdjson_never_inline __declspec(noinline)
@@ -88,6 +90,8 @@ double from_chars(const char *first, const char* end) noexcept;
   #define SIMDJSON_POP_DISABLE_UNUSED_WARNINGS
 
 #else // SIMDJSON_REGULAR_VISUAL_STUDIO
+  // We could use [[deprecated]] but it requires C++14
+  #define simdjson_deprecated __attribute__((deprecated))
 
   #define simdjson_really_inline inline __attribute__((always_inline))
   #define simdjson_never_inline inline __attribute__((noinline))

include/simdjson/dom/array-inl.h

@@ -7,6 +7,7 @@
 #include "simdjson/dom/array.h"
 #include "simdjson/dom/element.h"
 #include "simdjson/error-inl.h"
+#include "simdjson/jsonpathutil.h"
 #include "simdjson/internal/tape_ref-inl.h"
 
 #include <limits>
@@ -44,6 +45,13 @@ inline simdjson_result<dom::element> simdjson_result<dom::array>::at_pointer(std
   if (error()) { return error(); }
   return first.at_pointer(json_pointer);
 }
+
+ inline simdjson_result<dom::element> simdjson_result<dom::array>::at_path(std::string_view json_path) const noexcept {
+  auto json_pointer = json_path_to_pointer_conversion(json_path);
+  if (json_pointer == "-1") { return INVALID_JSON_POINTER; }
+  return at_pointer(json_pointer);
+ }
+
 inline simdjson_result<dom::element> simdjson_result<dom::array>::at(size_t index) const noexcept {
   if (error()) { return error(); }
   return first.at(index);
@@ -113,6 +121,12 @@ inline simdjson_result<element> array::at_pointer(std::string_view json_pointer)
   return child;
 }
 
+inline simdjson_result<element> array::at_path(std::string_view json_path) const noexcept {
+  auto json_pointer = json_path_to_pointer_conversion(json_path);
+  if (json_pointer == "-1") { return INVALID_JSON_POINTER; }
+  return at_pointer(json_pointer);
+}
+
 inline simdjson_result<element> array::at(size_t index) const noexcept {
   SIMDJSON_DEVELOPMENT_ASSERT(tape.usable()); // https://github.com/simdjson/simdjson/issues/1914
   size_t i=0;

include/simdjson/dom/array.h

@@ -108,6 +108,21 @@ public:
    */
   inline simdjson_result<element> at_pointer(std::string_view json_pointer) const noexcept;
 
+  /**
+   * Get the value associated with the given JSONPath expression. We only support
+   * JSONPath queries that trivially convertible to JSON Pointer queries: key
+   * names and array indices.
+   *
+   * https://datatracker.ietf.org/doc/html/draft-normington-jsonpath-00
+   *
+   * @return The value associated with the given JSONPath expression, or:
+   *         - INVALID_JSON_POINTER if the JSONPath to JSON Pointer conversion fails
+   *         - NO_SUCH_FIELD if a field does not exist in an object
+   *         - INDEX_OUT_OF_BOUNDS if an array index is larger than an array length
+   *         - INCORRECT_TYPE if a non-integer is used to access an array
+  */
+  inline simdjson_result<element> at_path(std::string_view json_path) const noexcept;
+
   /**
    * Get the value at the given index. This function has linear-time complexity and
    * is equivalent to the following:
@@ -152,6 +167,7 @@ public:
   simdjson_inline simdjson_result(error_code error) noexcept; ///< @private
 
   inline simdjson_result<dom::element> at_pointer(std::string_view json_pointer) const noexcept;
+  inline simdjson_result<dom::element> at_path(std::string_view json_path) const noexcept;
   inline simdjson_result<dom::element> at(size_t index) const noexcept;
 
 #if SIMDJSON_EXCEPTIONS

include/simdjson/dom/element-inl.h

@@ -9,6 +9,7 @@
 
 #include "simdjson/dom/object-inl.h"
 #include "simdjson/error-inl.h"
+#include "simdjson/jsonpathutil.h"
 
 #include <ostream>
 #include <limits>
@@ -122,6 +123,11 @@ simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::at_
   if (error()) { return error(); }
   return first.at_pointer(json_pointer);
 }
+simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::at_path(const std::string_view json_path) const noexcept {
+  auto json_pointer = json_path_to_pointer_conversion(json_path);
+  if (json_pointer == "-1") { return INVALID_JSON_POINTER; }
+  return at_pointer(json_pointer);
+}
 #ifndef SIMDJSON_DISABLE_DEPRECATED_API
 [[deprecated("For standard compliance, use at_pointer instead, and prefix your pointers with a slash '/', see RFC6901 ")]]
 simdjson_inline simdjson_result<dom::element> simdjson_result<dom::element>::at(const std::string_view json_pointer) const noexcept {
@@ -412,6 +418,11 @@ inline simdjson_result<element> element::at_pointer(std::string_view json_pointe
     }
   }
 }
+inline simdjson_result<element> element::at_path(std::string_view json_path) const noexcept {
+  auto json_pointer = json_path_to_pointer_conversion(json_path);
+  if (json_pointer == "-1") { return INVALID_JSON_POINTER; }
+  return at_pointer(json_pointer);
+}
 #ifndef SIMDJSON_DISABLE_DEPRECATED_API
 [[deprecated("For standard compliance, use at_pointer instead, and prefix your pointers with a slash '/', see RFC6901 ")]]
 inline simdjson_result<element> element::at(std::string_view json_pointer) const noexcept {

include/simdjson/dom/element.h

@@ -397,6 +397,21 @@ public:
    */
   inline simdjson_result<element> at_pointer(const std::string_view json_pointer) const noexcept;
 
+  /**
+   * Get the value associated with the given JSONPath expression. We only support
+   * JSONPath queries that trivially convertible to JSON Pointer queries: key
+   * names and array indices.
+   *
+   * https://datatracker.ietf.org/doc/html/draft-normington-jsonpath-00
+   *
+   * @return The value associated with the given JSONPath expression, or:
+   *         - INVALID_JSON_POINTER if the JSONPath to JSON Pointer conversion fails
+   *         - NO_SUCH_FIELD if a field does not exist in an object
+   *         - INDEX_OUT_OF_BOUNDS if an array index is larger than an array length
+   *         - INCORRECT_TYPE if a non-integer is used to access an array
+  */
+  inline simdjson_result<element> at_path(std::string_view json_path) const noexcept;
+
 #ifndef SIMDJSON_DISABLE_DEPRECATED_API
   /**
    *
@@ -526,6 +541,7 @@ public:
   simdjson_inline simdjson_result<dom::element> operator[](std::string_view key) const noexcept;
   simdjson_inline simdjson_result<dom::element> operator[](const char *key) const noexcept;
   simdjson_inline simdjson_result<dom::element> at_pointer(const std::string_view json_pointer) const noexcept;
+  simdjson_inline simdjson_result<dom::element> at_path(const std::string_view json_path) const noexcept;
   [[deprecated("For standard compliance, use at_pointer instead, and prefix your pointers with a slash '/', see RFC6901 ")]]
   simdjson_inline simdjson_result<dom::element> at(const std::string_view json_pointer) const noexcept;
   simdjson_inline simdjson_result<dom::element> at(size_t index) const noexcept;

include/simdjson/dom/object-inl.h

@@ -7,6 +7,7 @@
 
 #include "simdjson/dom/element-inl.h"
 #include "simdjson/error-inl.h"
+#include "simdjson/jsonpathutil.h"
 
 #include <cstring>
 
@@ -34,6 +35,11 @@ inline simdjson_result<dom::element> simdjson_result<dom::object>::at_pointer(st
   if (error()) { return error(); }
   return first.at_pointer(json_pointer);
 }
+inline simdjson_result<dom::element> simdjson_result<dom::object>::at_path(std::string_view json_path) const noexcept {
+  auto json_pointer = json_path_to_pointer_conversion(json_path);
+  if (json_pointer == "-1") { return INVALID_JSON_POINTER; }
+  return at_pointer(json_pointer);
+}
 inline simdjson_result<dom::element> simdjson_result<dom::object>::at_key(std::string_view key) const noexcept {
   if (error()) { return error(); }
   return first.at_key(key);
@@ -131,6 +137,12 @@ inline simdjson_result<element> object::at_pointer(std::string_view json_pointer
   return child;
 }
 
+inline simdjson_result<element> object::at_path(std::string_view json_path) const noexcept {
+  auto json_pointer = json_path_to_pointer_conversion(json_path);
+  if (json_pointer == "-1") { return INVALID_JSON_POINTER; }
+  return at_pointer(json_pointer);
+}
+
 inline simdjson_result<element> object::at_key(std::string_view key) const noexcept {
   iterator end_field = end();
   for (iterator field = begin(); field != end_field; ++field) {

include/simdjson/dom/object.h

@@ -171,6 +171,21 @@ public:
    */
   inline simdjson_result<element> at_pointer(std::string_view json_pointer) const noexcept;
 
+  /**
+   * Get the value associated with the given JSONPath expression. We only support
+   * JSONPath queries that trivially convertible to JSON Pointer queries: key
+   * names and array indices.
+   *
+   * https://datatracker.ietf.org/doc/html/draft-normington-jsonpath-00
+   *
+   * @return The value associated with the given JSONPath expression, or:
+   *         - INVALID_JSON_POINTER if the JSONPath to JSON Pointer conversion fails
+   *         - NO_SUCH_FIELD if a field does not exist in an object
+   *         - INDEX_OUT_OF_BOUNDS if an array index is larger than an array length
+   *         - INCORRECT_TYPE if a non-integer is used to access an array
+  */
+  inline simdjson_result<element> at_path(std::string_view json_path) const noexcept;
+
   /**
    * Get the value associated with the given key.
    *
@@ -244,6 +259,7 @@ public:
   inline simdjson_result<dom::element> operator[](std::string_view key) const noexcept;
   inline simdjson_result<dom::element> operator[](const char *key) const noexcept;
   inline simdjson_result<dom::element> at_pointer(std::string_view json_pointer) const noexcept;
+  inline simdjson_result<dom::element> at_path(std::string_view json_path) const noexcept;
   inline simdjson_result<dom::element> at_key(std::string_view key) const noexcept;
   inline simdjson_result<dom::element> at_key_case_insensitive(std::string_view key) const noexcept;
 

include/simdjson/dom/parser.h

@@ -549,9 +549,14 @@ public:
   /**
    * The parser instance can use threads when they are available to speed up some
    * operations. It is enabled by default. Changing this attribute will change the
-   * behavior of the parser for future operations.
+   * behavior of the parser for future operations. Set to true by default.
    */
   bool threaded{true};
+#else
+  /**
+   * When SIMDJSON_THREADS_ENABLED is not defined, the parser instance cannot use threads.
+   */
+  bool threaded{false};
 #endif
   /** @private Use the new DOM API instead */
   class Iterator;

include/simdjson/error.h

@@ -49,7 +49,8 @@ enum error_code {
   SCALAR_DOCUMENT_AS_VALUE,   ///< A scalar document is treated as a value.
   OUT_OF_BOUNDS,              ///< Attempted to access location outside of document.
   TRAILING_CONTENT,           ///< Unexpected trailing content in the JSON input
-  NUM_ERROR_CODES
+  OUT_OF_CAPACITY,            ///< The capacity was exceeded, we cannot allocate enough memory.
+  NUM_ERROR_CODES             ///< Placeholder for end of error code list.
 };
 
 /**

include/simdjson/fallback/implementation.h

@@ -26,6 +26,7 @@ public:
   ) const noexcept final;
   simdjson_warn_unused error_code minify(const uint8_t *buf, size_t len, uint8_t *dst, size_t &dst_len) const noexcept final;
   simdjson_warn_unused bool validate_utf8(const char *buf, size_t len) const noexcept final;
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final;
 };
 
 } // namespace fallback

include/simdjson/fallback/numberparsing_defs.h

@@ -75,6 +75,12 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace fallback
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_FALLBACK_NUMBERPARSING_DEFS_H

include/simdjson/fallback/stringparsing_defs.h

@@ -29,6 +29,24 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
   return { src[0] };
 }
 
+
+struct escaping {
+  static constexpr uint32_t BYTES_PROCESSED = 1;
+  simdjson_inline static escaping copy_and_find(const uint8_t *src, uint8_t *dst);
+
+  simdjson_inline bool has_escape() { return escape_bits; }
+  simdjson_inline int escape_index() { return 0; }
+
+  bool escape_bits;
+}; // struct escaping
+
+
+
+simdjson_inline escaping escaping::copy_and_find(const uint8_t *src, uint8_t *dst) {
+  dst[0] = src[0];
+  return { (src[0] == '\\') || (src[0] == '"') || (src[0] < 32) };
+}
+
 } // unnamed namespace
 } // namespace fallback
 } // namespace simdjson

include/simdjson/generic/numberparsing.h

@@ -574,7 +574,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //

include/simdjson/generic/ondemand/amalgamated.h

@@ -45,4 +45,8 @@
 #include "simdjson/generic/ondemand/token_iterator-inl.h"
 #include "simdjson/generic/ondemand/value_iterator-inl.h"
 
+// JSON builder, ideally they should not be part of the ondemand directory
+// but it is convenient for now to have them here.
+#include "simdjson/generic/ondemand/json_string_builder.h"
+#include "simdjson/generic/ondemand/json_string_builder-inl.h"
 

include/simdjson/generic/ondemand/array-inl.h

@@ -2,6 +2,7 @@
 
 #ifndef SIMDJSON_CONDITIONAL_INCLUDE
 #define SIMDJSON_GENERIC_ONDEMAND_ARRAY_INL_H
+#include "simdjson/jsonpathutil.h"
 #include "simdjson/generic/ondemand/base.h"
 #include "simdjson/generic/ondemand/array.h"
 #include "simdjson/generic/ondemand/array_iterator-inl.h"
@@ -163,53 +164,6 @@ inline simdjson_result<value> array::at_pointer(std::string_view json_pointer) n
   return child;
 }
 
-inline std::string json_path_to_pointer_conversion(std::string_view json_path) {
-  if (json_path.empty() || (json_path.front() != '.' &&
-      json_path.front() != '[')) {
-    return "-1"; // This is just a sentinel value, the caller should check for this and return an error.
-  }
-
-  std::string result;
-  // Reserve space to reduce allocations, adjusting for potential increases due
-  // to escaping.
-  result.reserve(json_path.size() * 2);
-
-  size_t i = 0;
-
-  while (i < json_path.length()) {
-    if (json_path[i] == '.') {
-      result += '/';
-    } else if (json_path[i] == '[') {
-      result += '/';
-      ++i; // Move past the '['
-      while (i < json_path.length() && json_path[i] != ']') {
-          if (json_path[i] == '~') {
-            result += "~0";
-          } else if (json_path[i] == '/') {
-            result += "~1";
-          } else {
-            result += json_path[i];
-          }
-          ++i;
-      }
-      if (i == json_path.length() || json_path[i] != ']') {
-          return "-1"; // Using sentinel value that will be handled as an error by the caller.
-      }
-    } else {
-      if (json_path[i] == '~') {
-          result += "~0";
-      } else if (json_path[i] == '/') {
-          result += "~1";
-      } else {
-          result += json_path[i];
-      }
-    }
-    ++i;
-  }
-
-  return result;
-}
-
 inline simdjson_result<value> array::at_path(std::string_view json_path) noexcept {
   auto json_pointer = json_path_to_pointer_conversion(json_path);
   if (json_pointer == "-1") { return INVALID_JSON_POINTER; }

include/simdjson/generic/ondemand/dependencies.h

@@ -13,5 +13,6 @@
 #include "simdjson/padded_string.h"
 #include "simdjson/padded_string_view.h"
 #include "simdjson/internal/dom_parser_implementation.h"
+#include "simdjson/jsonpathutil.h"
 
 #endif // SIMDJSON_GENERIC_ONDEMAND_DEPENDENCIES_H

include/simdjson/generic/ondemand/document-inl.h

@@ -3,17 +3,15 @@
 #ifndef SIMDJSON_CONDITIONAL_INCLUDE
 #define SIMDJSON_GENERIC_ONDEMAND_DOCUMENT_INL_H
 #include "simdjson/generic/ondemand/base.h"
-#include "simdjson/generic/ondemand/array-inl.h"
 #include "simdjson/generic/ondemand/array_iterator.h"
 #include "simdjson/generic/ondemand/document.h"
-#include "simdjson/generic/ondemand/json_iterator-inl.h"
-#include "simdjson/generic/ondemand/json_path_to_pointer_conversion.h"
-#include "simdjson/generic/ondemand/json_path_to_pointer_conversion-inl.h"
 #include "simdjson/generic/ondemand/json_type.h"
-#include "simdjson/generic/ondemand/object-inl.h"
 #include "simdjson/generic/ondemand/raw_json_string.h"
 #include "simdjson/generic/ondemand/value.h"
 #include "simdjson/generic/ondemand/value-inl.h"
+#include "simdjson/generic/ondemand/array-inl.h"
+#include "simdjson/generic/ondemand/json_iterator-inl.h"
+#include "simdjson/generic/ondemand/object-inl.h"
 #include "simdjson/generic/ondemand/value_iterator-inl.h"
 #include "simdjson/generic/ondemand/deserialize.h"
 #endif // SIMDJSON_CONDITIONAL_INCLUDE
@@ -179,10 +177,19 @@ template<> simdjson_inline error_code document::get(int64_t& out) & noexcept { r
 template<> simdjson_inline error_code document::get(bool& out) & noexcept { return get_bool().get(out); }
 template<> simdjson_inline error_code document::get(value& out) & noexcept { return get_value().get(out); }
 
+template<> simdjson_deprecated simdjson_inline simdjson_result<raw_json_string> document::get() && noexcept { return get_raw_json_string(); }
+template<> simdjson_deprecated simdjson_inline simdjson_result<std::string_view> document::get() && noexcept { return get_string(false); }
+template<> simdjson_deprecated simdjson_inline simdjson_result<double> document::get() && noexcept { return std::forward<document>(*this).get_double(); }
+template<> simdjson_deprecated simdjson_inline simdjson_result<uint64_t> document::get() && noexcept { return std::forward<document>(*this).get_uint64(); }
+template<> simdjson_deprecated simdjson_inline simdjson_result<int64_t> document::get() && noexcept { return std::forward<document>(*this).get_int64(); }
+template<> simdjson_deprecated simdjson_inline simdjson_result<bool> document::get() && noexcept { return std::forward<document>(*this).get_bool(); }
+template<> simdjson_deprecated simdjson_inline simdjson_result<value> document::get() && noexcept { return get_value(); }
 
 #if SIMDJSON_EXCEPTIONS
 template <class T>
-simdjson_inline document::operator T() noexcept(false) { return get<T>(); }
+simdjson_deprecated simdjson_inline document::operator T() && noexcept(false) { return get<T>(); }
+template <class T>
+simdjson_inline document::operator T() & noexcept(false) { return get<T>(); }
 simdjson_inline document::operator array() & noexcept(false) { return get_array(); }
 simdjson_inline document::operator object() & noexcept(false) { return get_object(); }
 simdjson_inline document::operator uint64_t() noexcept(false) { return get_uint64(); }
@@ -469,7 +476,7 @@ simdjson_inline simdjson_result<T> simdjson_result<SIMDJSON_IMPLEMENTATION::onde
   return first.get<T>();
 }
 template<typename T>
-simdjson_inline simdjson_result<T> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get() && noexcept {
+simdjson_deprecated simdjson_inline simdjson_result<T> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get() && noexcept {
   if (error()) { return error(); }
   return std::forward<SIMDJSON_IMPLEMENTATION::ondemand::document>(first).get<T>();
 }
@@ -485,7 +492,7 @@ simdjson_inline error_code simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::do
 }
 
 template<> simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get<SIMDJSON_IMPLEMENTATION::ondemand::document>() & noexcept = delete;
-template<> simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get<SIMDJSON_IMPLEMENTATION::ondemand::document>() && noexcept {
+template<> simdjson_deprecated simdjson_inline simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document> simdjson_result<SIMDJSON_IMPLEMENTATION::ondemand::document>::get<SIMDJSON_IMPLEMENTATION::ondemand::document>() && noexcept {
   if (error()) { return error(); }
   return std::forward<SIMDJSON_IMPLEMENTATION::ondemand::document>(first);
 }

include/simdjson/generic/ondemand/document.h

@@ -254,7 +254,7 @@ public:
 #endif
   }
   /** @overload template<typename T> error_code get(T &out) & noexcept */
-  template<typename T> simdjson_inline error_code get(T &out) && noexcept;
+  template<typename T> simdjson_deprecated simdjson_inline error_code get(T &out) && noexcept;
 
 #if SIMDJSON_EXCEPTIONS
   /**
@@ -267,7 +267,10 @@ public:
    * @returns An instance of type T
    */
   template <class T>
-  explicit simdjson_inline operator T() noexcept(false);
+  explicit simdjson_inline operator T() & noexcept(false);
+  template <class T>
+  explicit simdjson_deprecated simdjson_inline operator T() && noexcept(false);
+
   /**
    * Cast this JSON value to an array.
    *
@@ -731,6 +734,11 @@ protected:
 
 /**
  * A document_reference is a thin wrapper around a document reference instance.
+ * The document_reference instances are used primarily/solely for streams of JSON
+ * documents. They differ from document instances when parsing a scalar value
+ * (a document that is not an array or an object). In the case of a document,
+ * we expect the document to be fully consumed. In the case of a document_reference,
+ * we allow trailing content.
  */
 class document_reference {
 public:
@@ -896,7 +904,7 @@ public:
   simdjson_inline simdjson_result<bool> is_null() noexcept;
 
   template<typename T> simdjson_inline simdjson_result<T> get() & noexcept;
-  template<typename T> simdjson_inline simdjson_result<T> get() && noexcept;
+  template<typename T> simdjson_deprecated simdjson_inline simdjson_result<T> get() && noexcept;
 
   template<typename T> simdjson_inline error_code get(T &out) & noexcept;
   template<typename T> simdjson_inline error_code get(T &out) && noexcept;

include/simdjson/generic/ondemand/document_stream.h

@@ -316,7 +316,7 @@ private:
   friend class document;
   friend class json_iterator;
   friend struct simdjson_result<ondemand::document_stream>;
-  friend struct internal::simdjson_result_base<ondemand::document_stream>;
+  friend struct simdjson::internal::simdjson_result_base<ondemand::document_stream>;
 };  // document_stream
 
 } // namespace ondemand

include/simdjson/generic/ondemand/json_path_to_pointer_conversion-inl.h

@@ -1,67 +0,0 @@
-#pragma once
-
-#ifndef SIMDJSON_ONDEMAND_GENERIC_JSON_PATH_TO_POINTER_CONVERSION_INL_H
-#define SIMDJSON_ONDEMAND_GENERIC_JSON_PATH_TO_POINTER_CONVERSION_INL_H
-
-#ifndef SIMDJSON_CONDITIONAL_INCLUDE
-#define SIMDJSON_GENERIC_ONDEMAND_JSON_PATH_TO_POINTER_CONVERSION_INL_H
-#include "simdjson/generic/ondemand/json_path_to_pointer_conversion.h"
-#endif // SIMDJSON_CONDITIONAL_INCLUDE
-
-namespace simdjson {
-namespace SIMDJSON_IMPLEMENTATION {}
-namespace ondemand {
-
-simdjson_inline std::string json_path_to_pointer_conversion(std::string_view json_path) {
-  if (json_path.empty() || (json_path.front() != '.' && json_path.front() != '[') {
-    return "-1"; // Sentinel value to be handled as an error by the caller.
-  }
-
-  std::string result;
-  // Reserve space to reduce allocations, adjusting for potential increases due
-  // to escaping.
-  result.reserve(json_path.size() * 2);
-
-  // Skip the initial '.' as it's assumed every path starts with it.
-  size_t i = 0;
-
-  while (i < json_path.length()) {
-    if (json_path[i] == '.') {
-      result += '/';
-    } else if (json_path[i] == '[') {
-      result += '/';
-      ++i; // Move past the '['
-      while (i < json_path.length() && json_path[i] != ']') {
-        if (json_path[i] == '~') {
-          result += "~0";
-        } else if (json_path[i] == '/') {
-          result += "~1";
-        } else {
-          result += json_path[i];
-        }
-        ++i;
-      }
-      if (i == json_path.length() || json_path[i] != ']') {
-        return "-1"; // Returning sentinel value that will be handled as an error by the caller
-      }
-    } else {
-      if (json_path[i] == '~') {
-        result += "~0";
-      } else if (json_path[i] == '/') {
-        result += "~1";
-      } else {
-        result += json_path[i];
-      }
-    }
-    ++i;
-  }
-
-  return simdjson_result<std::string>(result);
-}
-
-
-} // namespace ondemand
-} // namespace SIMDJSON_IMPLEMENTATION
-} // namespace simdjson
-
-#endif // SIMDJSON_ONDEMAND_GENERIC_JSON_PATH_TO_POINTER_CONVERSION_INL_H

include/simdjson/generic/ondemand/json_path_to_pointer_conversion.h

@@ -1,22 +0,0 @@
-#pragma once
-
-#ifndef SIMDJSON_ONDEMAND_GENERIC_JSON_PATH_TO_POINTER_CONVERSION_H
-#define SIMDJSON_ONDEMAND_GENERIC_JSON_PATH_TO_POINTER_CONVERSION_H
-
-namespace simdjson {
-namespace SIMDJSON_IMPLEMENTATION {
-namespace internal {
-
-/**
- * Converts JSONPath to JSON Pointer.
- * @param json_path The JSONPath string to be converted.
- * @return A string containing the equivalent JSON Pointer.
- * @throws simdjson_error If the conversion fails.
- */
-simdjson_inline std::string json_path_to_pointer_conversion(std::string_view json_path);
-
-} // namespace internal
-} // namespace SIMDJSON_IMPLEMENTATION
-} // namespace simdjson
-
-#endif // SIMDJSON_JSON_PATH_TO_POINTER_CONVERSION_H

include/simdjson/generic/ondemand/json_string_builder-inl.h

@@ -0,0 +1,281 @@
+/**
+ * This file is part of the builder API. It is temporarily in the ondemand directory
+ * but we will move it to a builder directory later.
+ */
+#include <type_traits>
+#ifndef SIMDJSON_GENERIC_BUILDER_INL_H
+
+#ifndef SIMDJSON_CONDITIONAL_INCLUDE
+#define SIMDJSON_GENERIC_BUILDER_INL_H
+#include "simdjson/generic/builder/json_string_builder.h"
+#endif // SIMDJSON_CONDITIONAL_INCLUDE
+
+namespace simdjson {
+namespace SIMDJSON_IMPLEMENTATION {
+namespace builder {
+
+simdjson_inline string_builder::string_builder(size_t initial_capacity) :
+  buffer(new (std::nothrow) char[initial_capacity]),
+  position(0), capacity(buffer.get() != nullptr ? initial_capacity : 0),
+  is_valid(buffer.get() != nullptr) {}
+
+simdjson_inline bool string_builder::capacity_check(size_t upcoming_bytes) {
+  // We use the convention that when is_valid is false, then the capacity and
+  // the position are 0.
+  // Most of the time, this function will return true.
+  if (simdjson_likely(upcoming_bytes <= capacity - position)) { return true; }
+  // check for overflow, most of the time there is no overflow
+  if (simdjson_likely(position + upcoming_bytes < position)) { return false; }
+  // We will rarely get here.
+  grow_buffer((std::max)(capacity * 2, position + upcoming_bytes));
+  // If the buffer allocation failed, we set is_valid to false.
+  return is_valid;
+}
+
+simdjson_inline void string_builder::grow_buffer(size_t desired_capacity) {
+  if (!is_valid) { return; }
+  std::unique_ptr<char[]> new_buffer(new (std::nothrow) char[desired_capacity]);
+  if (new_buffer.get() == nullptr) {
+    set_valid(false);
+    return;
+  }
+  std::memcpy(new_buffer.get(), buffer.get(), position);
+  buffer.swap(new_buffer);
+  capacity = desired_capacity;
+}
+
+simdjson_inline void string_builder::set_valid(bool valid) noexcept {
+  if(!valid) {
+    is_valid = false;
+    capacity = 0;
+    position = 0;
+    buffer.reset();
+  } else {
+    is_valid = true;
+  }
+}
+
+simdjson_inline size_t string_builder::size() const noexcept {
+  return position;
+}
+
+
+simdjson_inline void string_builder::append(char c) noexcept {
+    if(capacity_check(1)) {
+      buffer.get()[position++] = c;
+    }
+}
+
+
+simdjson_inline void string_builder::append_null() noexcept {
+  constexpr char null_literal[] = "null";
+  constexpr size_t null_len = sizeof(null_literal) - 1;
+  if(capacity_check(null_len)) {
+    std::memcpy(buffer.get() + position, null_literal, null_len);
+    position += null_len;
+  }
+}
+
+simdjson_inline void string_builder::clear()  noexcept {
+  position = 0;
+  // if it was invalid, we should try to repair it
+  if(!is_valid) {
+    capacity = 0;
+    buffer.reset();
+    is_valid = true;
+  }
+}
+
+namespace internal {
+
+// We could specialize further for 32-bit integers.
+int int_log2(uint32_t x) { return (63 - leading_zeroes(x | 1)); }
+
+int fast_digit_count_32(uint32_t x) {
+  static uint64_t table[] = {
+      4294967296,  8589934582,  8589934582,  8589934582,  12884901788,
+      12884901788, 12884901788, 17179868184, 17179868184, 17179868184,
+      21474826480, 21474826480, 21474826480, 21474826480, 25769703776,
+      25769703776, 25769703776, 30063771072, 30063771072, 30063771072,
+      34349738368, 34349738368, 34349738368, 34349738368, 38554705664,
+      38554705664, 38554705664, 41949672960, 41949672960, 41949672960,
+      42949672960, 42949672960};
+  return uint32_t((x + table[int_log2(x)]) >> 32);
+}
+
+int int_log2(uint64_t x) { return 63 - leading_zeroes(x | 1); }
+
+int digit_count_64(uint64_t x) {
+  static uint64_t table[] = {9,
+                             99,
+                             999,
+                             9999,
+                             99999,
+                             999999,
+                             9999999,
+                             99999999,
+                             999999999,
+                             9999999999,
+                             99999999999,
+                             999999999999,
+                             9999999999999,
+                             99999999999999,
+                             999999999999999ULL,
+                             9999999999999999ULL,
+                             99999999999999999ULL,
+                             999999999999999999ULL,
+                             9999999999999999999ULL};
+  int y = (19 * int_log2(x) >> 6);
+  y += x > table[y];
+  return y + 1;
+}
+
+template<typename number_type,
+         typename = typename std::enable_if<std::is_unsigned<number_type>::value>::type>
+simdjson_inline size_t digit_count(number_type v) noexcept {
+  static_assert(sizeof(number_type) == 8
+   || sizeof(number_type) == 4
+   || sizeof(number_type) == 2
+   || sizeof(number_type) == 1, "We only support 8-bit, 16-bit, 32-bit and 64-bit numbers");
+  if (sizeof(number_type) <= 4) {
+    return fast_digit_count_32(v);
+  } else {
+    return digit_count_64(v);
+  }
+}
+
+} // internal
+
+template<typename number_type, typename>
+simdjson_inline void string_builder::append(number_type v) noexcept {
+  static_assert(std::is_same<number_type, bool>::value
+    || std::is_integral<number_type>::value || std::is_floating_point<number_type>::value, "Unsupported number type");
+  // If C++17 is available, we can 'if constexpr' here.
+  if constexpr (std::is_same<number_type, bool>::value) {
+    if (v) {
+      constexpr char true_literal[] = "true";
+      constexpr size_t true_len = sizeof(true_literal) - 1;
+      if(capacity_check(true_len)) {
+        std::memcpy(buffer.get() + position, true_literal, true_len);
+        position += true_len;
+      }
+    } else {
+      constexpr char false_literal[] = "false";
+      constexpr size_t false_len = sizeof(false_literal) - 1;
+      if(capacity_check(false_len)) {
+        std::memcpy(buffer.get() + position, false_literal, false_len);
+        position += false_len;
+      }
+    }
+  } else if constexpr (std::is_unsigned<number_type>::value) {
+    constexpr size_t max_number_size = 20;
+    if(capacity_check(max_number_size)) {
+      using unsigned_type = typename std::make_unsigned<number_type>::type;
+      unsigned_type pv = static_cast<unsigned_type>(v);
+      size_t dc = internal::digit_count(pv);
+      char *write_pointer = buffer.get() + position + dc - 1;
+      // optimization opportunity: if v is large, we can do better.
+      while(pv >= 10) {
+        *write_pointer-- = char('0' + (pv % 10));
+        pv /= 10;
+      }
+      *write_pointer = char('0' + pv);
+      position += dc;
+    }
+  } else if constexpr (std::is_integral<number_type>::value) {
+    constexpr size_t max_number_size = 20;
+    if(capacity_check(max_number_size)) {
+      using unsigned_type = typename std::make_unsigned<number_type>::type;
+      bool negative = v < 0;
+      unsigned_type pv = static_cast<unsigned_type>(negative ? -v : v);
+      size_t dc = internal::digit_count(pv);
+      if(negative) {
+        buffer.get()[position++] = '-';
+      }
+      char *write_pointer = buffer.get() + position + dc - 1;
+      // optimization opportunity: if v is large, we can do better.
+      while(pv >= 10) {
+        *write_pointer-- = char('0' + (pv % 10));
+        pv /= 10;
+      }
+      *write_pointer = char('0' + pv);
+      position += dc;
+    }
+  } else if constexpr (std::is_floating_point<number_type>::value) {
+    constexpr size_t max_number_size = 24;
+    if(capacity_check(max_number_size)) {
+      // We could specialize for float.
+      char *end = simdjson::internal::to_chars(buffer.get() + position, nullptr, double(v));
+      position = end - buffer.get();
+    }
+  }
+}
+
+simdjson_inline void string_builder::escape_and_append(std::string_view input)  noexcept {
+  // escaping might turn a control character into \x00xx so 6 characters.
+  if(capacity_check(6 * input.size())) {
+    position += simdjson::write_string_escaped(input, buffer.get() + position);
+  }
+}
+
+simdjson_inline void string_builder::escape_and_append_with_quotes(std::string_view input)  noexcept {
+  // escaping might turn a control character into \x00xx so 6 characters.
+  if(capacity_check(2 + 6 * input.size())) {
+    buffer.get()[position++] = '"';
+    position += simdjson::write_string_escaped(input, buffer.get() + position);
+    buffer.get()[position++] = '"';
+  }
+}
+
+simdjson_inline void string_builder::append_raw(const char *c)  noexcept {
+  if(capacity_check(1)) {
+    buffer.get()[position++] = *c;
+  }
+}
+
+simdjson_inline void string_builder::append_raw(std::string_view input)  noexcept {
+  if(capacity_check(input.size())) {
+    std::memcpy(buffer.get() + position, input.data(), input.size());
+    position += input.size();
+  }
+}
+
+simdjson_inline void string_builder::append_raw(const char *str, size_t len)  noexcept {
+  if(capacity_check(len)) {
+    std::memcpy(buffer.get() + position, str, len);
+    position += len;
+  }
+}
+
+#if SIMDJSON_EXCEPTIONS
+simdjson_inline string_builder::operator std::string() const noexcept(false) {
+  return std::string(std::string_view());
+}
+
+simdjson_inline string_builder::operator std::string_view() const noexcept(false) {
+  return view();
+}
+#endif
+
+simdjson_inline simdjson_result<std::string_view> string_builder::view() const noexcept {
+  if (!is_valid) { return simdjson::OUT_OF_CAPACITY; }
+  return std::string_view(buffer.get(), position);
+}
+
+simdjson_inline simdjson_result<const char *> string_builder::c_str() noexcept {
+  if(capacity_check(1)) {
+    buffer.get()[position] = '\0';
+    return buffer.get();
+  }
+  return simdjson::OUT_OF_CAPACITY;
+}
+
+simdjson_inline bool string_builder::validate_unicode() const noexcept {
+  return simdjson::validate_utf8(buffer.get(), position);
+}
+
+} // namespace builder
+} // namespace SIMDJSON_IMPLEMENTATION
+} // namespace simdjson
+
+#endif // SIMDJSON_GENERIC_BUILDER_INL_H

include/simdjson/generic/ondemand/json_string_builder.h

@@ -0,0 +1,167 @@
+/**
+ * This file is part of the builder API. It is temporarily in the ondemand directory
+ * but we will move it to a builder directory later.
+ */
+#ifndef SIMDJSON_GENERIC_BUILDER_H
+
+#ifndef SIMDJSON_CONDITIONAL_INCLUDE
+#define SIMDJSON_GENERIC_BUILDER_H
+#include "simdjson/generic/implementation_simdjson_result_base.h"
+#endif // SIMDJSON_CONDITIONAL_INCLUDE
+
+namespace simdjson {
+namespace SIMDJSON_IMPLEMENTATION {
+namespace builder {
+
+/**
+ * A builder for JSON strings representing documents. This is a low-level
+ * builder that is not meant to be used directly by end-users. Though it
+ * supports atomic types (Booleans, strings), it does not support composed
+ * types (arrays and objects).
+ *
+ * Ultimately, this class should support kernel-specific optimizations. E.g.,
+ * it may make use of SIMD instructions to escape strings faster.
+ */
+class string_builder {
+public:
+  simdjson_inline string_builder(size_t initial_capacity = 1024);
+
+  /**
+   * Append number (includes Booleans). Booleans are mapped to the strings
+   * false and true. Numbers are converted to strings abiding by the JSON standard.
+   * Floating-point numbers are converted to the shortest string that 'correctly'
+   * represents the number.
+   */
+  template<typename number_type,
+         typename = typename std::enable_if<std::is_arithmetic<number_type>::value>::type>
+  simdjson_inline void append(number_type v) noexcept;
+
+  /**
+   * Append character c.
+   */
+  simdjson_inline void append(char c)  noexcept;
+
+  /**
+   * Append the string 'null'.
+   */
+  simdjson_inline void append_null()  noexcept;
+
+  /**
+   * Clear the content.
+   */
+  simdjson_inline void clear()  noexcept;
+
+  /**
+   * Append the std::string_view, after escaping it.
+   * There is no UTF-8 validation.
+   */
+  simdjson_inline void escape_and_append(std::string_view input)  noexcept;
+
+  /**
+   * Append the std::string_view surrounded by double quotes, after escaping it.
+   * There is no UTF-8 validation.
+   */
+  simdjson_inline void escape_and_append_with_quotes(std::string_view input)  noexcept;
+
+  /**
+   * Append the C string directly, without escaping.
+   * There is no UTF-8 validation.
+   */
+  simdjson_inline void append_raw(const char *c)  noexcept;
+
+  /**
+   * Append the std::string_view directly, without escaping.
+   * There is no UTF-8 validation.
+   */
+  simdjson_inline void append_raw(std::string_view input)  noexcept;
+
+  /**
+   * Append len characters from str.
+   * There is no UTF-8 validation.
+   */
+  simdjson_inline void append_raw(const char *str, size_t len) noexcept;
+#if SIMDJSON_EXCEPTIONS
+  /**
+   * Creates an std::string from the written JSON buffer.
+   * Throws if memory allocation failed
+   *
+   * The result may not be valid UTF-8 if some of your content was not valid UTF-8.
+   * Use validate_unicode() to check the content if needed.
+   */
+  simdjson_inline operator std::string() const noexcept(false);
+
+  /**
+   * Creates an std::string_view from the written JSON buffer.
+   * Throws if memory allocation failed.
+   *
+   * The result may not be valid UTF-8 if some of your content was not valid UTF-8.
+   * Use validate_unicode() to check the content if needed.
+   */
+  simdjson_inline operator std::string_view() const noexcept(false);
+#endif
+
+  /**
+   * Returns a view on the written JSON buffer. Returns an error
+   * if memory allocation failed.
+   *
+   * The result may not be valid UTF-8 if some of your content was not valid UTF-8.
+   * Use validate_unicode() to check the content.
+   */
+  simdjson_inline simdjson_result<std::string_view> view() const noexcept;
+
+  /**
+   * Appends the null character to the buffer and returns
+   * a pointer to the beginning of the written JSON buffer.
+   * Returns an error if memory allocation failed.
+   * The result is null-terminated.
+   *
+   * The result may not be valid UTF-8 if some of your content was not valid UTF-8.
+   * Use validate_unicode() to check the content.
+   */
+  simdjson_inline simdjson_result<const char *> c_str() noexcept;
+
+  /**
+   * Return true if the content is valid UTF-8.
+   */
+  simdjson_inline bool validate_unicode() const noexcept;
+
+  /**
+   * Returns the current size of the written JSON buffer.
+   * If an error occurred, returns 0.
+   */
+  simdjson_inline size_t size() const noexcept;
+
+private:
+  /**
+   * Returns true if we can write at least upcoming_bytes bytes.
+   * The underlying buffer is reallocated if needed. It is designed
+   * to be called before writing to the buffer. It should be fast.
+   */
+  simdjson_inline bool capacity_check(size_t upcoming_bytes);
+
+  /**
+   * Grow the buffer to at least desired_capacity bytes.
+   * If the allocation fails, is_valid is set to false. We expect
+   * that this function would not be repeatedly called.
+   */
+  simdjson_inline void grow_buffer(size_t desired_capacity);
+
+  /**
+   * We use this helper function to make sure that is_valid is kept consistent.
+   */
+  simdjson_inline void set_valid(bool valid) noexcept;
+
+
+  std::unique_ptr<char[]> buffer{};
+  size_t position{0};
+  size_t capacity{0};
+  bool is_valid{true};
+};
+
+
+
+}
+}
+} // namespace simdjson
+
+#endif // SIMDJSON_GENERIC_BUILDER_H

include/simdjson/generic/ondemand/parser.h

@@ -278,8 +278,12 @@ public:
    * behavior of the parser for future operations.
    */
   bool threaded{true};
+  #else
+  /**
+   * When SIMDJSON_THREADS_ENABLED is not defined, the parser instance cannot use threads.
+   */
+  bool threaded{false};
   #endif
-
   /**
    * Unescape this JSON string, replacing \\ with \, \n with newline, etc. to a user-provided buffer.
    * The result must be valid UTF-8.
@@ -340,7 +344,7 @@ public:
 
 private:
   /** @private [for benchmarking access] The implementation to use */
-  std::unique_ptr<internal::dom_parser_implementation> implementation{};
+  std::unique_ptr<simdjson::internal::dom_parser_implementation> implementation{};
   size_t _capacity{0};
   size_t _max_capacity;
   size_t _max_depth{DEFAULT_MAX_DEPTH};

include/simdjson/generic/ondemand/value-inl.h

@@ -6,8 +6,6 @@
 #include "simdjson/generic/ondemand/array.h"
 #include "simdjson/generic/ondemand/array_iterator.h"
 #include "simdjson/generic/ondemand/json_iterator.h"
-#include "simdjson/generic/ondemand/json_path_to_pointer_conversion.h"
-#include "simdjson/generic/ondemand/json_path_to_pointer_conversion-inl.h"
 #include "simdjson/generic/ondemand/json_type.h"
 #include "simdjson/generic/ondemand/object.h"
 #include "simdjson/generic/ondemand/raw_json_string.h"

include/simdjson/generic/ondemand/value_iterator-inl.h

@@ -6,9 +6,9 @@
 #include "simdjson/generic/atomparsing.h"
 #include "simdjson/generic/numberparsing.h"
 #include "simdjson/generic/ondemand/json_iterator.h"
+#include "simdjson/generic/ondemand/value_iterator.h"
 #include "simdjson/generic/ondemand/json_type-inl.h"
 #include "simdjson/generic/ondemand/raw_json_string-inl.h"
-#include "simdjson/generic/ondemand/value_iterator.h"
 #endif // SIMDJSON_CONDITIONAL_INCLUDE
 
 namespace simdjson {
@@ -799,6 +799,8 @@ simdjson_inline simdjson_result<bool> value_iterator::is_root_null(bool check_tr
   if(result) { // we have something that looks like a null.
     if (check_trailing && !_json_iter->is_single_token()) { return TRAILING_CONTENT; }
     advance_root_scalar("null");
+  } else if (json[0] == 'n') {
+    return incorrect_type_error("Not a null but starts with n");
   }
   return result;
 }

include/simdjson/haswell/begin.h

@@ -4,8 +4,14 @@
 #include "simdjson/haswell/intrinsics.h"
 
 #if !SIMDJSON_CAN_ALWAYS_RUN_HASWELL
+// We enable bmi2 only if LLVM/clang is used, because GCC may not
+// make good use of it. See https://github.com/simdjson/simdjson/pull/2243
+#if defined(__clang__)
+SIMDJSON_TARGET_REGION("avx2,bmi,bmi2,pclmul,lzcnt,popcnt")
+#else
 SIMDJSON_TARGET_REGION("avx2,bmi,pclmul,lzcnt,popcnt")
 #endif
+#endif
 
 #include "simdjson/haswell/bitmanipulation.h"
 #include "simdjson/haswell/bitmask.h"

include/simdjson/haswell/implementation.h

@@ -28,6 +28,7 @@ public:
   ) const noexcept final;
   simdjson_warn_unused error_code minify(const uint8_t *buf, size_t len, uint8_t *dst, size_t &dst_len) const noexcept final;
   simdjson_warn_unused bool validate_utf8(const char *buf, size_t len) const noexcept final;
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final;
 };
 
 } // namespace haswell

include/simdjson/haswell/stringparsing_defs.h

@@ -41,6 +41,31 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
   };
 }
 
+
+struct escaping {
+  static constexpr uint32_t BYTES_PROCESSED = 32;
+  simdjson_inline static escaping copy_and_find(const uint8_t *src, uint8_t *dst);
+
+  simdjson_inline bool has_escape() { return escape_bits != 0; }
+  simdjson_inline int escape_index() { return trailing_zeroes(escape_bits); }
+
+  uint64_t escape_bits;
+}; // struct escaping
+
+
+
+simdjson_inline escaping escaping::copy_and_find(const uint8_t *src, uint8_t *dst) {
+  static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "escaping finder must process fewer than SIMDJSON_PADDING bytes");
+  simd8<uint8_t> v(src);
+  v.store(dst);
+  simd8<bool> is_quote = (v == '"');
+  simd8<bool> is_backslash = (v == '\\');
+  simd8<bool> is_control = (v < 32);
+  return {
+    uint64_t((is_backslash | is_quote | is_control).to_bitmask())
+  };
+}
+
 } // unnamed namespace
 } // namespace haswell
 } // namespace simdjson

include/simdjson/icelake/implementation.h

@@ -28,6 +28,7 @@ public:
   ) const noexcept final;
   simdjson_warn_unused error_code minify(const uint8_t *buf, size_t len, uint8_t *dst, size_t &dst_len) const noexcept final;
   simdjson_warn_unused bool validate_utf8(const char *buf, size_t len) const noexcept final;
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final;
 };
 
 } // namespace icelake

include/simdjson/icelake/simd.h

@@ -77,7 +77,6 @@ namespace simd {
     friend simdjson_really_inline uint64_t operator==(const simd8<T> lhs, const simd8<T> rhs) {
       return _mm512_cmpeq_epi8_mask(lhs, rhs);
     }
-
     static const int SIZE = sizeof(base<T>::value);
 
     template<int N=1>

include/simdjson/icelake/stringparsing_defs.h

@@ -41,6 +41,35 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
   };
 }
 
+
+
+struct escaping {
+  static constexpr uint32_t BYTES_PROCESSED = 64;
+  simdjson_inline static escaping copy_and_find(const uint8_t *src, uint8_t *dst);
+
+  simdjson_inline bool has_escape() { return escape_bits != 0; }
+  simdjson_inline int escape_index() { return trailing_zeroes(uint64_t(escape_bits)); }
+
+  __mmask64 escape_bits;
+}; // struct escaping
+
+
+
+simdjson_inline escaping escaping::copy_and_find(const uint8_t *src, uint8_t *dst) {
+  static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "escaping finder must process fewer than SIMDJSON_PADDING bytes");
+  simd8<uint8_t> v(src);
+  v.store(dst);
+  __mmask64 is_quote = _mm512_cmpeq_epi8_mask(v, _mm512_set1_epi8('"'));
+  __mmask64 is_backslash = _mm512_cmpeq_epi8_mask(v, _mm512_set1_epi8('\\'));
+  __mmask64 is_control = _mm512_cmplt_epi8_mask(v, _mm512_set1_epi8(32));
+  return {
+    (is_backslash | is_quote | is_control)
+  };
+}
+
+
+
+
 } // unnamed namespace
 } // namespace icelake
 } // namespace simdjson

include/simdjson/implementation.h

@@ -26,6 +26,15 @@ simdjson_inline simdjson_warn_unused bool validate_utf8(const std::string_view s
   return validate_utf8(sv.data(), sv.size());
 }
 
+/**
+ * Write the string to the output buffer while escaping double-quote, backlash and ascii control characters.
+ *
+ * @param input the string_view to escape
+ * @param out output buffer (for escaped string): to be safe, it should have 6 * input.size() allocated bytes.
+ * @return number of bytes written
+ */
+simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) noexcept;
+
 /**
  * Validate the UTF-8 string.
  *
@@ -127,6 +136,14 @@ public:
    */
   simdjson_warn_unused virtual bool validate_utf8(const char *buf, size_t len) const noexcept = 0;
 
+  /**
+   * Write the string to the output buffer while escaping double-quote, backlash and ascii control characters.
+   *
+   * @param input the string_view to escape
+   * @param out output buffer (for escaped string): to be safe, it should have 6 * input.size() allocated bytes.
+   * @return number of bytes written
+   */
+  simdjson_warn_unused virtual size_t write_string_escaped(const std::string_view input, char *out) const noexcept = 0;
 protected:
   /** @private Construct an implementation with the given name and description. For subclasses. */
   simdjson_inline implementation(

include/simdjson/jsonpathutil.h

@@ -0,0 +1,64 @@
+#ifndef SIMDJSON_JSONPATHUTIL_H
+#define SIMDJSON_JSONPATHUTIL_H
+
+#include <string>
+#include <string_view>
+
+namespace simdjson {
+/**
+ * Converts JSONPath to JSON Pointer.
+ * @param json_path The JSONPath string to be converted.
+ * @return A string containing the equivalent JSON Pointer.
+ */
+inline std::string json_path_to_pointer_conversion(std::string_view json_path) {
+  size_t i = 0;
+
+  // if JSONPath starts with $, skip it
+  if (!json_path.empty() && json_path.front() == '$') {
+    i = 1;
+  }
+  if (json_path.empty() || (json_path[i] != '.' &&
+      json_path[i] != '[')) {
+    return "-1"; // This is just a sentinel value, the caller should check for this and return an error.
+  }
+
+  std::string result;
+  // Reserve space to reduce allocations, adjusting for potential increases due
+  // to escaping.
+  result.reserve(json_path.size() * 2);
+
+  while (i < json_path.length()) {
+    if (json_path[i] == '.') {
+      result += '/';
+    } else if (json_path[i] == '[') {
+      result += '/';
+      ++i; // Move past the '['
+      while (i < json_path.length() && json_path[i] != ']') {
+          if (json_path[i] == '~') {
+            result += "~0";
+          } else if (json_path[i] == '/') {
+            result += "~1";
+          } else {
+            result += json_path[i];
+          }
+          ++i;
+      }
+      if (i == json_path.length() || json_path[i] != ']') {
+          return "-1"; // Using sentinel value that will be handled as an error by the caller.
+      }
+    } else {
+      if (json_path[i] == '~') {
+          result += "~0";
+      } else if (json_path[i] == '/') {
+          result += "~1";
+      } else {
+          result += json_path[i];
+      }
+    }
+    ++i;
+  }
+
+  return result;
+}
+} // namespace simdjson
+#endif // SIMDJSON_JSONPATHUTIL_H

include/simdjson/lasx/implementation.h

@@ -23,6 +23,7 @@ public:
   ) const noexcept final;
   simdjson_warn_unused error_code minify(const uint8_t *buf, size_t len, uint8_t *dst, size_t &dst_len) const noexcept final;
   simdjson_warn_unused bool validate_utf8(const char *buf, size_t len) const noexcept final;
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final;
 };
 
 } // namespace lasx

include/simdjson/lasx/numberparsing_defs.h

@@ -36,6 +36,12 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace lasx
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_LASX_NUMBERPARSING_DEFS_H

include/simdjson/lasx/stringparsing_defs.h

@@ -40,6 +40,31 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
   };
 }
 
+
+struct escaping {
+  static constexpr uint32_t BYTES_PROCESSED = 16;
+  simdjson_inline static escaping copy_and_find(const uint8_t *src, uint8_t *dst);
+
+  simdjson_inline bool has_escape() { return escape_bits != 0; }
+  simdjson_inline int escape_index() { return trailing_zeroes(escape_bits); }
+
+  uint64_t escape_bits;
+}; // struct escaping
+
+
+
+simdjson_inline escaping escaping::copy_and_find(const uint8_t *src, uint8_t *dst) {
+  static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "escaping finder must process fewer than SIMDJSON_PADDING bytes");
+  simd8<uint8_t> v(src);
+  v.store(dst);
+  simd8<bool> is_quote = (v == '"');
+  simd8<bool> is_backslash = (v == '\\');
+  simd8<bool> is_control = (v < 32);
+  return {
+    (is_backslash | is_quote | is_control).to_bitmask()
+  };
+}
+
 } // unnamed namespace
 } // namespace lasx
 } // namespace simdjson

include/simdjson/lsx/implementation.h

@@ -23,6 +23,7 @@ public:
   ) const noexcept final;
   simdjson_warn_unused error_code minify(const uint8_t *buf, size_t len, uint8_t *dst, size_t &dst_len) const noexcept final;
   simdjson_warn_unused bool validate_utf8(const char *buf, size_t len) const noexcept final;
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final;
 };
 
 } // namespace lsx

include/simdjson/lsx/numberparsing_defs.h

@@ -36,6 +36,12 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace lsx
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_LSX_NUMBERPARSING_DEFS_H

include/simdjson/lsx/stringparsing_defs.h

@@ -46,6 +46,31 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
   };
 }
 
+
+struct escaping {
+  static constexpr uint32_t BYTES_PROCESSED = 16;
+  simdjson_inline static escaping copy_and_find(const uint8_t *src, uint8_t *dst);
+
+  simdjson_inline bool has_escape() { return escape_bits != 0; }
+  simdjson_inline int escape_index() { return trailing_zeroes(escape_bits); }
+
+  uint64_t escape_bits;
+}; // struct escaping
+
+
+
+simdjson_inline escaping escaping::copy_and_find(const uint8_t *src, uint8_t *dst) {
+  static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "escaping finder must process fewer than SIMDJSON_PADDING bytes");
+  simd8<uint8_t> v(src);
+  v.store(dst);
+  simd8<bool> is_quote = (v == '"');
+  simd8<bool> is_backslash = (v == '\\');
+  simd8<bool> is_control = (v < 32);
+  return {
+    (is_backslash | is_quote | is_control).to_bitmask()
+  };
+}
+
 } // unnamed namespace
 } // namespace lsx
 } // namespace simdjson

include/simdjson/ondemand.h

@@ -8,6 +8,10 @@ namespace simdjson {
    * @copydoc simdjson::builtin::ondemand
    */
   namespace ondemand = builtin::ondemand;
+  /**
+   * @copydoc simdjson::builtin::builder
+   */
+  namespace builder = builtin::builder;
 } // namespace simdjson
 
 #endif // SIMDJSON_ONDEMAND_H

include/simdjson/padded_string-inl.h

@@ -53,6 +53,9 @@ inline padded_string::padded_string(const char *data, size_t length) noexcept
   if ((data != nullptr) && (data_ptr != nullptr)) {
     std::memcpy(data_ptr, data, length);
   }
+  if (data_ptr == nullptr) {
+    viable_size = 0;
+  }
 }
 #ifdef __cpp_char8_t
 inline padded_string::padded_string(const char8_t *data, size_t length) noexcept
@@ -60,12 +63,17 @@ inline padded_string::padded_string(const char8_t *data, size_t length) noexcept
   if ((data != nullptr) && (data_ptr != nullptr)) {
     std::memcpy(data_ptr, reinterpret_cast<const char *>(data), length);
   }
+  if (data_ptr == nullptr) {
+    viable_size = 0;
+  }
 }
 #endif
 // note: do not pass std::string arguments by value
 inline padded_string::padded_string(const std::string & str_ ) noexcept
     : viable_size(str_.size()), data_ptr(internal::allocate_padded_buffer(str_.size())) {
-  if (data_ptr != nullptr) {
+  if (data_ptr == nullptr) {
+    viable_size = 0;
+  } else {
     std::memcpy(data_ptr, str_.data(), str_.size());
   }
 }

include/simdjson/portability.h

@@ -201,4 +201,43 @@ using std::size_t;
 
 #endif
 
+
+
+#if defined __BYTE_ORDER__ && defined __ORDER_BIG_ENDIAN__
+#define SIMDJSON_IS_BIG_ENDIAN (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
+#elif defined _WIN32
+#define SIMDJSON_IS_BIG_ENDIAN 0
+#else
+#if defined(__APPLE__) || defined(__FreeBSD__)
+#include <machine/endian.h>
+#elif defined(sun) || defined(__sun)
+#include <sys/byteorder.h>
+#elif defined(__MVS__)
+#include <sys/endian.h>
+#else
+#ifdef __has_include
+#if __has_include(<endian.h>)
+#include <endian.h>
+#endif //__has_include(<endian.h>)
+#endif //__has_include
+#endif
+#
+#ifndef __BYTE_ORDER__
+// safe choice
+#define SIMDJSON_IS_BIG_ENDIAN 0
+#endif
+#
+#ifndef __ORDER_LITTLE_ENDIAN__
+// safe choice
+#define SIMDJSON_IS_BIG_ENDIAN 0
+#endif
+#
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+#define SIMDJSON_IS_BIG_ENDIAN 0
+#else
+#define SIMDJSON_IS_BIG_ENDIAN 1
+#endif
+#endif
+
+
 #endif // SIMDJSON_PORTABILITY_H

include/simdjson/ppc64/implementation.h

@@ -32,6 +32,7 @@ public:
                                          size_t &dst_len) const noexcept final;
   simdjson_warn_unused bool validate_utf8(const char *buf,
                                           size_t len) const noexcept final;
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final;
 };
 
 } // namespace ppc64

include/simdjson/ppc64/numberparsing_defs.h

@@ -60,6 +60,12 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace ppc64
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_PPC64_NUMBERPARSING_DEFS_H

include/simdjson/ppc64/stringparsing_defs.h

@@ -58,6 +58,31 @@ backslash_and_quote::copy_and_find(const uint8_t *src, uint8_t *dst) {
   };
 }
 
+
+struct escaping {
+  static constexpr uint32_t BYTES_PROCESSED = 16;
+  simdjson_inline static escaping copy_and_find(const uint8_t *src, uint8_t *dst);
+
+  simdjson_inline bool has_escape() { return escape_bits != 0; }
+  simdjson_inline int escape_index() { return trailing_zeroes(escape_bits); }
+
+  uint64_t escape_bits;
+}; // struct escaping
+
+
+
+simdjson_inline escaping escaping::copy_and_find(const uint8_t *src, uint8_t *dst) {
+  static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "escaping finder must process fewer than SIMDJSON_PADDING bytes");
+  simd8<uint8_t> v(src);
+  v.store(dst);
+  simd8<bool> is_quote = (v == '"');
+  simd8<bool> is_backslash = (v == '\\');
+  simd8<bool> is_control = (v < 32);
+  return {
+    (is_backslash | is_quote | is_control).to_bitmask()
+  };
+}
+
 } // unnamed namespace
 } // namespace ppc64
 } // namespace simdjson

include/simdjson/simdjson_version.h

@@ -4,7 +4,7 @@
 #define SIMDJSON_SIMDJSON_VERSION_H
 
 /** The version of simdjson being used (major.minor.revision) */
-#define SIMDJSON_VERSION "3.10.0"
+#define SIMDJSON_VERSION "3.10.1"
 
 namespace simdjson {
 enum {
@@ -19,7 +19,7 @@ enum {
   /**
    * The revision (major.minor.REVISION) of simdjson being used.
    */
-  SIMDJSON_VERSION_REVISION = 0
+  SIMDJSON_VERSION_REVISION = 1
 };
 } // namespace simdjson
 

include/simdjson/westmere/implementation.h

@@ -24,6 +24,7 @@ public:
   ) const noexcept final;
   simdjson_warn_unused error_code minify(const uint8_t *buf, size_t len, uint8_t *dst, size_t &dst_len) const noexcept final;
   simdjson_warn_unused bool validate_utf8(const char *buf, size_t len) const noexcept final;
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final;
 };
 
 } // namespace westmere

include/simdjson/westmere/stringparsing_defs.h

@@ -40,6 +40,31 @@ simdjson_inline backslash_and_quote backslash_and_quote::copy_and_find(const uin
   };
 }
 
+
+struct escaping {
+  static constexpr uint32_t BYTES_PROCESSED = 16;
+  simdjson_inline static escaping copy_and_find(const uint8_t *src, uint8_t *dst);
+
+  simdjson_inline bool has_escape() { return escape_bits != 0; }
+  simdjson_inline int escape_index() { return trailing_zeroes(escape_bits); }
+
+  uint64_t escape_bits;
+}; // struct escaping
+
+
+
+simdjson_inline escaping escaping::copy_and_find(const uint8_t *src, uint8_t *dst) {
+  static_assert(SIMDJSON_PADDING >= (BYTES_PROCESSED - 1), "escaping finder must process fewer than SIMDJSON_PADDING bytes");
+  simd8<uint8_t> v(src);
+  v.store(dst);
+  simd8<bool> is_quote = (v == '"');
+  simd8<bool> is_backslash = (v == '\\');
+  simd8<bool> is_control = (v < 32);
+  return {
+    uint64_t((is_backslash | is_quote | is_control).to_bitmask())
+  };
+}
+
 } // unnamed namespace
 } // namespace westmere
 } // namespace simdjson

singleheader/amalgamate.py

@@ -10,12 +10,41 @@ import os
 import re
 import shutil
 import datetime
+import json
 from typing import Dict, List, Optional, Set, TextIO, Union, cast
 
+# Check for Python 3, this does not actually work.
 if sys.version_info < (3, 0):
     sys.stdout.write("Sorry, requires Python 3.x or better\n")
     sys.exit(1)
 
+rules = """
+
+We refer your to the HACKING.md file for more information on how the project is organized.
+
+To help understand the error, here are the rules for including files in simdjson:
+
+All implementation-specific files, including arm64.h, arm64/implementation.h and
+arm64/ondemand.h, must be within SIMDJSON_CONDITIONAL_INCLUDE blocks.
+
+Top-level headers must not be included in any SIMDJSON_CONDITIONAL_INCLUDE block.
+
+Generic files must be included only in amalgamator files (arm64.h,
+arm64/implementation.h, arm64/ondemand.h, generic/amalgamated.h).
+
+We fail if an implementation-specific file is included more than once in the same block.
+We fail if a generic file is included more than once per implementation in the same block.
+
+
+Tip: generally, "file" will search the including file's source directory first, then
+the search paths while <file> does it the other way around.
+We prefer to use <> in simdjson headers to avoid accidentally including a file from the
+wrong directory.
+
+The amalgamate.py script checks that all files are included.
+
+"""
+
 SCRIPTPATH = os.path.dirname(os.path.abspath(sys.argv[0]))
 PROJECTPATH = os.path.dirname(SCRIPTPATH)
 print(f"SCRIPTPATH={SCRIPTPATH} PROJECTPATH={PROJECTPATH}")
@@ -62,6 +91,22 @@ class SimdjsonFile:
     def __str__(self):
         return self.include_path
 
+    def dump(self):
+        return {
+            'root': self.root,
+            'include_path': self.include_path,
+            'includes': [include.include_path for include in self.includes],
+            'included_from': [included_from.include_path for included_from in self.included_from],
+            'editor_only_includes': [editor_only_include.include_path for editor_only_include in self.editor_only_includes],
+            'editor_only_included_from': [editor_only_included_from.include_path for editor_only_included_from in self.editor_only_included_from],
+            'processed': self.processed,
+            'dependency_file': self.dependency_file.include_path if self.dependency_file else None,
+            'is_amalgamator': self.is_amalgamator,
+            'implementation': self.implementation,
+        }
+    def json(self):
+        return json.dumps(self.dump(), indent=4, sort_keys=True, ensure_ascii=False)
+
     def __repr__(self):
         return self.include_path
 
@@ -162,20 +207,21 @@ class SimdjsonFile:
 
     def add_include(self, include: 'SimdjsonFile'):
         if self.is_conditional_include:
-            assert include.is_conditional_include, f"{self} cannot include {include} without #ifndef SIMDJSON_CONDITIONAL_INCLUDE."
+            # If I have a dependency file, I can only include something that has a dependency file.
+            assert include.is_conditional_include, f"{self} cannot include {include} without #ifndef SIMDJSON_CONDITIONAL_INCLUDE. {rules}"
             # TODO make sure we only include amalgamated files that are guaranteed to be included with us (or before us)
             # if include.amalgamator_file:
             #     assert include.amalgamator_file == self, f"{self} cannot include {include}: it should be included from {include.amalgamator_file} instead."
         else:
-            assert include.is_amalgamator or not include.is_conditional_include, f"{self} cannot include {include} because it is an amalgamated file."
+            assert include.is_amalgamator or not include.is_conditional_include, f"{self} cannot include {include} because it is an amalgamated file. {rules}"
 
         self.includes.append(include)
         include.included_from.add(self)
 
     def add_editor_only_include(self, include: 'SimdjsonFile'):
-        assert self.is_conditional_include, f"Cannot use #ifndef SIMDJSON_CONDITIONAL_INCLUDE in {self} because it is not an amalgamated file."
+        assert self.is_conditional_include, f"Cannot use #ifndef SIMDJSON_CONDITIONAL_INCLUDE in {self} because it is not an amalgamated file. {rules}"
         if not include.is_conditional_include:
-            assert self.dependency_file, f"{self} cannot include {include} without #ifndef SIMDJSON_CONDITIONAL_INCLUDE."
+            assert self.dependency_file, f"{self} cannot include {include} without #ifndef SIMDJSON_CONDITIONAL_INCLUDE. {rules}"
         # TODO make sure we only include amalgamated files that are guaranteed to be included with us (or before us)
         # elif include.amalgamator_file:
         #     assert self.is_amalgamated_before(self.amalgamator_file), f"{self} cannot include {include}: it should be included from {include.amalgamator_file} instead."
@@ -190,11 +236,11 @@ class SimdjsonFile:
                 if file.dependency_file == self:
                     for editor_only_include in file.editor_only_includes:
                         if not editor_only_include.is_conditional_include:
-                            assert editor_only_include in self.includes, f"{file} includes {editor_only_include}, but it is not included from {self}. It must be added to {self}."
+                            assert editor_only_include in self.includes, f"{file} includes {editor_only_include}, but it is not included from {self}. It must be added to {self}. {rules}"
                             if editor_only_include in extra_include_set:
                                 extra_include_set.remove(editor_only_include)
 
-            assert len(extra_include_set) == 0, f"{self} unnecessarily includes {extra_include_set}. They are not included in the corresponding amalgamated files."
+            assert len(extra_include_set) == 0, f"{self} unnecessarily includes {extra_include_set}. They are not included in the corresponding amalgamated files. {rules}"
 
 class SimdjsonRepository:
     def __init__(self, project_path: str, relative_roots: List[RelativeRoot]):
@@ -320,14 +366,19 @@ class Amalgamator:
 
         assert not self.editor_only_region
         with open(file.absolute_path, 'r') as fid2:
+            print(f"including: {file}")
             for line in fid2:
                 line = line.rstrip('\n')
 
+                # Ignore #pragma once, it causes warnings if it ends up in a .cpp file
+                if re.search(r'^#pragma once$', line):
+                    continue
+
                 # Ignore lines inside #ifndef SIMDJSON_CONDITIONAL_INCLUDE
                 if re.search(r'^#ifndef\s+SIMDJSON_CONDITIONAL_INCLUDE\s*$', line):
-                    assert file.is_conditional_include, f"{file} uses #ifndef SIMDJSON_CONDITIONAL_INCLUDE but is not an amalgamated file!"
+                    assert file.is_conditional_include, f"{file} uses #ifndef SIMDJSON_CONDITIONAL_INCLUDE but is not an amalgamated file! {rules}"
-                    assert self.in_conditional_include_block, f"{file} uses #ifndef SIMDJSON_CONDITIONAL_INCLUDE without a prior #define SIMDJSON_CONDITIONAL_INCLUDE: {self.include_stack}"
+                    assert self.in_conditional_include_block, f"{file} uses #ifndef SIMDJSON_CONDITIONAL_INCLUDE without a prior #define SIMDJSON_CONDITIONAL_INCLUDE: {self.include_stack} {rules}"
-                    assert not self.editor_only_region, f"{file} uses #ifndef SIMDJSON_CONDITIONAL_INCLUDE twice in a row"
+                    assert not self.editor_only_region, f"{file} uses #ifndef SIMDJSON_CONDITIONAL_INCLUDE twice in a row {rules}"
                     self.editor_only_region = True
 
                 # Handle ignored lines (and ending ignore blocks)
@@ -345,7 +396,7 @@ class Amalgamator:
                         self.editor_only_region = False
                     continue
 
-                assert not end_ignore, f"{file} has #endif // SIMDJSON_CONDITIONAL_INCLUDE without #ifndef SIMDJSON_CONDITIONAL_INCLUDE"
+                assert not end_ignore, f"{file} has #endif // SIMDJSON_CONDITIONAL_INCLUDE without #ifndef SIMDJSON_CONDITIONAL_INCLUDE {rules}"
 
                 # Handle #include lines
                 included = re.search(r'^#include\s+["<]([^">]*)[">]', line)
@@ -372,26 +423,26 @@ class Amalgamator:
                     self.implementation = None
                 elif re.search(r'\bSIMDJSON_IMPLEMENTATION\b', line) and file.include_path != IMPLEMENTATION_DETECTION_H:
                     # copy the line, with SIMDJSON_IMPLEMENTATION replace to what it is currently defined to
-                    assert self.implementation, f"Use of SIMDJSON_IMPLEMENTATION while not defined in {file}: {line}"
+                    assert self.implementation, f"Use of SIMDJSON_IMPLEMENTATION while not defined in {file}: {line}\n{rules}"
                     line = re.sub(r'\bSIMDJSON_IMPLEMENTATION\b',self.implementation,line)
 
                 # Handle defining and undefining SIMDJSON_CONDITIONAL_INCLUDE
                 defined = re.search(r'^#define\s+SIMDJSON_CONDITIONAL_INCLUDE\s*$', line)
                 if defined:
-                    assert not file.is_conditional_include, "SIMDJSON_CONDITIONAL_INCLUDE defined in amalgamated file {file}! Not allowed."
+                    assert not file.is_conditional_include, "SIMDJSON_CONDITIONAL_INCLUDE defined in amalgamated file {file}! Not allowed. {rules}"
-                    assert not self.in_conditional_include_block, f"{file} redefines SIMDJSON_CONDITIONAL_INCLUDE"
+                    assert not self.in_conditional_include_block, f"{file} redefines SIMDJSON_CONDITIONAL_INCLUDE {rules}"
                     self.in_conditional_include_block = True
                     self.found_includes_per_conditional_block.clear()
                     self.write(f'/* defining SIMDJSON_CONDITIONAL_INCLUDE */')
                 elif re.search(r'^#undef\s+SIMDJSON_CONDITIONAL_INCLUDE\s*$', line):
-                    assert not file.is_conditional_include, "SIMDJSON_CONDITIONAL_INCLUDE undefined in amalgamated file {file}! Not allowed."
+                    assert not file.is_conditional_include, "SIMDJSON_CONDITIONAL_INCLUDE undefined in amalgamated file {file}! Not allowed. {rules}"
-                    assert self.in_conditional_include_block, f"{file} undefines SIMDJSON_CONDITIONAL_INCLUDE without defining it"
+                    assert self.in_conditional_include_block, f"{file} undefines SIMDJSON_CONDITIONAL_INCLUDE without defining it {rules}"
                     self.write(f'/* undefining SIMDJSON_CONDITIONAL_INCLUDE */')
                     self.in_conditional_include_block = False
 
                 self.write(line)
 
-            assert not self.editor_only_region, f"{file} ended without #endif // SIMDJSON_CONDITIONAL_INCLUDE"
+            assert not self.editor_only_region, f"{file} ended without #endif // SIMDJSON_CONDITIONAL_INCLUDE {rules}"
 
         self.write(f"/* end file {self.file_to_str(file)} */")
 

singleheader/simdjson.cpp

@@ -1,4 +1,4 @@
-/* auto-generated on 2024-09-30 10:57:33 -0400. Do not edit! */
+/* auto-generated on 2024-10-28 20:27:26 -0400. Do not edit! */
 /* including simdjson.cpp:  */
 /* begin file simdjson.cpp */
 #define SIMDJSON_SRC_SIMDJSON_CPP
@@ -296,6 +296,45 @@ using std::size_t;
 
 #endif
 
+
+
+#if defined __BYTE_ORDER__ && defined __ORDER_BIG_ENDIAN__
+#define SIMDJSON_IS_BIG_ENDIAN (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
+#elif defined _WIN32
+#define SIMDJSON_IS_BIG_ENDIAN 0
+#else
+#if defined(__APPLE__) || defined(__FreeBSD__)
+#include <machine/endian.h>
+#elif defined(sun) || defined(__sun)
+#include <sys/byteorder.h>
+#elif defined(__MVS__)
+#include <sys/endian.h>
+#else
+#ifdef __has_include
+#if __has_include(<endian.h>)
+#include <endian.h>
+#endif //__has_include(<endian.h>)
+#endif //__has_include
+#endif
+#
+#ifndef __BYTE_ORDER__
+// safe choice
+#define SIMDJSON_IS_BIG_ENDIAN 0
+#endif
+#
+#ifndef __ORDER_LITTLE_ENDIAN__
+// safe choice
+#define SIMDJSON_IS_BIG_ENDIAN 0
+#endif
+#
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+#define SIMDJSON_IS_BIG_ENDIAN 0
+#else
+#define SIMDJSON_IS_BIG_ENDIAN 1
+#endif
+#endif
+
+
 #endif // SIMDJSON_PORTABILITY_H
 /* end file simdjson/portability.h */
 
@@ -344,6 +383,8 @@ double from_chars(const char *first, const char* end) noexcept;
 #define SIMDJSON_ISALIGNED_N(ptr, n) (((uintptr_t)(ptr) & ((n)-1)) == 0)
 
 #if SIMDJSON_REGULAR_VISUAL_STUDIO
+  // We could use [[deprecated]] but it requires C++14
+  #define simdjson_deprecated __declspec(deprecated)
 
   #define simdjson_really_inline __forceinline
   #define simdjson_never_inline __declspec(noinline)
@@ -382,6 +423,8 @@ double from_chars(const char *first, const char* end) noexcept;
   #define SIMDJSON_POP_DISABLE_UNUSED_WARNINGS
 
 #else // SIMDJSON_REGULAR_VISUAL_STUDIO
+  // We could use [[deprecated]] but it requires C++14
+  #define simdjson_deprecated __attribute__((deprecated))
 
   #define simdjson_really_inline inline __attribute__((always_inline))
   #define simdjson_never_inline inline __attribute__((noinline))
@@ -557,7 +600,6 @@ SIMDJSON_PUSH_DISABLE_ALL_WARNINGS
 // Distributed under the Boost Software License, Version 1.0.
 // (See accompanying file LICENSE.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 
-#pragma once
 
 #ifndef NONSTD_SV_LITE_H_INCLUDED
 #define NONSTD_SV_LITE_H_INCLUDED
@@ -2401,7 +2443,8 @@ enum error_code {
   SCALAR_DOCUMENT_AS_VALUE,   ///< A scalar document is treated as a value.
   OUT_OF_BOUNDS,              ///< Attempted to access location outside of document.
   TRAILING_CONTENT,           ///< Unexpected trailing content in the JSON input
-  NUM_ERROR_CODES
+  OUT_OF_CAPACITY,            ///< The capacity was exceeded, we cannot allocate enough memory.
+  NUM_ERROR_CODES             ///< Placeholder for end of error code list.
 };
 
 /**
@@ -7762,6 +7805,12 @@ simdjson_inline uint64_t clear_lowest_bit(uint64_t input_num) {
   return input_num & (input_num-1);
 }
 
+// We sometimes call leading_zeroes on inputs that are zero,
+// but the algorithms do not end up using the returned value.
+// Sadly, sanitizers are not smart enough to figure it out.
+// Applies only when SIMDJSON_PREFER_REVERSE_BITS is defined and true.
+// (See below.)
+SIMDJSON_NO_SANITIZE_UNDEFINED
 /* result might be undefined when input_num is zero */
 simdjson_inline int leading_zeroes(uint64_t input_num) {
 #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
@@ -7934,7 +7983,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace arm64
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_ARM64_NUMBERPARSING_DEFS_H
 /* end file simdjson/arm64/numberparsing_defs.h */
@@ -9551,7 +9606,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //
@@ -10518,6 +10572,12 @@ simdjson_inline uint64_t clear_lowest_bit(uint64_t input_num) {
   return input_num & (input_num-1);
 }
 
+// We sometimes call leading_zeroes on inputs that are zero,
+// but the algorithms do not end up using the returned value.
+// Sadly, sanitizers are not smart enough to figure it out.
+// Applies only when SIMDJSON_PREFER_REVERSE_BITS is defined and true.
+// (See below.)
+SIMDJSON_NO_SANITIZE_UNDEFINED
 /* result might be undefined when input_num is zero */
 simdjson_inline int leading_zeroes(uint64_t input_num) {
 #ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
@@ -10690,7 +10750,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace arm64
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_ARM64_NUMBERPARSING_DEFS_H
 /* end file simdjson/arm64/numberparsing_defs.h */
@@ -14238,8 +14304,14 @@ static_assert(sizeof(__m256i) <= simdjson::SIMDJSON_PADDING, "insufficient paddi
 /* end file simdjson/haswell/intrinsics.h */
 
 #if !SIMDJSON_CAN_ALWAYS_RUN_HASWELL
+// We enable bmi2 only if LLVM/clang is used, because GCC may not
+// make good use of it. See https://github.com/simdjson/simdjson/pull/2243
+#if defined(__clang__)
+SIMDJSON_TARGET_REGION("avx2,bmi,bmi2,pclmul,lzcnt,popcnt")
+#else
 SIMDJSON_TARGET_REGION("avx2,bmi,pclmul,lzcnt,popcnt")
 #endif
+#endif
 
 /* including simdjson/haswell/bitmanipulation.h: #include "simdjson/haswell/bitmanipulation.h" */
 /* begin file simdjson/haswell/bitmanipulation.h */
@@ -15893,7 +15965,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //
@@ -16871,8 +16942,14 @@ static_assert(sizeof(__m256i) <= simdjson::SIMDJSON_PADDING, "insufficient paddi
 /* end file simdjson/haswell/intrinsics.h */
 
 #if !SIMDJSON_CAN_ALWAYS_RUN_HASWELL
+// We enable bmi2 only if LLVM/clang is used, because GCC may not
+// make good use of it. See https://github.com/simdjson/simdjson/pull/2243
+#if defined(__clang__)
+SIMDJSON_TARGET_REGION("avx2,bmi,bmi2,pclmul,lzcnt,popcnt")
+#else
 SIMDJSON_TARGET_REGION("avx2,bmi,pclmul,lzcnt,popcnt")
 #endif
+#endif
 
 /* including simdjson/haswell/bitmanipulation.h: #include "simdjson/haswell/bitmanipulation.h" */
 /* begin file simdjson/haswell/bitmanipulation.h */
@@ -22107,7 +22184,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //
@@ -26873,7 +26949,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace ppc64
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_PPC64_NUMBERPARSING_DEFS_H
 /* end file simdjson/ppc64/numberparsing_defs.h */
@@ -28477,7 +28559,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //
@@ -29617,7 +29698,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace ppc64
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_PPC64_NUMBERPARSING_DEFS_H
 /* end file simdjson/ppc64/numberparsing_defs.h */
@@ -35213,7 +35300,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //
@@ -40299,7 +40385,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace lsx
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_LSX_NUMBERPARSING_DEFS_H
 /* end file simdjson/lsx/numberparsing_defs.h */
@@ -41773,7 +41865,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //
@@ -42828,7 +42919,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace lsx
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_LSX_NUMBERPARSING_DEFS_H
 /* end file simdjson/lsx/numberparsing_defs.h */
@@ -46282,7 +46379,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace lasx
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_LASX_NUMBERPARSING_DEFS_H
 /* end file simdjson/lasx/numberparsing_defs.h */
@@ -47772,7 +47875,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //
@@ -48827,7 +48929,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace lasx
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_LASX_NUMBERPARSING_DEFS_H
 /* end file simdjson/lasx/numberparsing_defs.h */
@@ -52311,7 +52419,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace fallback
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_FALLBACK_NUMBERPARSING_DEFS_H
 /* end file simdjson/fallback/numberparsing_defs.h */
@@ -53370,7 +53484,6 @@ simdjson_unused simdjson_inline simdjson_result<number_type> get_number_type(con
 // Our objective is accurate parsing (ULP of 0) at high speed.
 template<typename W>
 simdjson_inline error_code parse_number(const uint8_t *const src, W &writer) {
-
   //
   // Check for minus sign
   //
@@ -54445,7 +54558,13 @@ simdjson_inline internal::value128 full_multiplication(uint64_t value1, uint64_t
 } // namespace fallback
 } // namespace simdjson
 
+#ifndef SIMDJSON_SWAR_NUMBER_PARSING
+#if SIMDJSON_IS_BIG_ENDIAN
+#define SIMDJSON_SWAR_NUMBER_PARSING 0
+#else
 #define SIMDJSON_SWAR_NUMBER_PARSING 1
+#endif
+#endif
 
 #endif // SIMDJSON_FALLBACK_NUMBERPARSING_DEFS_H
 /* end file simdjson/fallback/numberparsing_defs.h */

src/arm64.cpp

@@ -164,6 +164,10 @@ simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *
   return stage2(_doc);
 }
 
+simdjson_warn_unused size_t implementation::write_string_escaped(const std::string_view input, char *out) const noexcept {
+  return arm64::stringparsing::write_string_escaped(input, out);
+}
+
 } // namespace arm64
 } // namespace simdjson
 

src/fallback.cpp

@@ -402,6 +402,10 @@ simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *
   return stage2(_doc);
 }
 
+simdjson_warn_unused size_t implementation::write_string_escaped(const std::string_view input, char *out) const noexcept {
+  return fallback::stringparsing::write_string_escaped(input, out);
+}
+
 } // namespace fallback
 } // namespace simdjson
 

src/generic/stage2/stringparsing.h

@@ -236,6 +236,101 @@ simdjson_warn_unused simdjson_inline uint8_t *parse_wobbly_string(const uint8_t
   }
 }
 
+simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) noexcept {
+  // We are making the following assumption: most strings will either be very short or they will not
+  // need escaping.
+  size_t i = 0;
+  size_t pos = 0;
+  if(input.size() >= escaping::BYTES_PROCESSED) {
+    auto vec_processing = [input,out]() -> size_t {
+      size_t i = 0;
+      size_t pos = 0;
+      for(;input.size() - i >= escaping::BYTES_PROCESSED; i += escaping::BYTES_PROCESSED) {
+        escaping vinput = escaping::copy_and_find(reinterpret_cast<const uint8_t *>(input.data()) + i, reinterpret_cast<uint8_t *>(out) + pos);
+        if(vinput.has_escape()) {
+          return i + vinput.escape_index(); // We have a character that needs escaping
+        }
+        pos += escaping::BYTES_PROCESSED;
+      }
+      if(i == input.size()) { return input.size(); }
+      // We virtually backtrack so we can load a full vector register
+      i = input.size() - escaping::BYTES_PROCESSED;
+      pos = i;
+      escaping vinput = escaping::copy_and_find(reinterpret_cast<const uint8_t *>(input.data()) + i, reinterpret_cast<uint8_t *>(out) + pos);
+      if(vinput.has_escape()) {
+        return i + vinput.escape_index(); // We have a character that needs escaping
+      }
+      return input.size();
+    };
+    i = vec_processing();
+    pos = i;
+    if(i == input.size()) { return pos; }
+    // Here we only continue if there was a character that needed escaping.
+  }
+  static std::string_view control_chars[] = {
+    "\\x0000", "\\x0001", "\\x0002", "\\x0003", "\\x0004", "\\x0005", "\\x0006",
+    "\\x0007", "\\x0008", "\\t",     "\\n",     "\\x000b", "\\f",     "\\r",
+    "\\x000e", "\\x000f", "\\x0010", "\\x0011", "\\x0012", "\\x0013", "\\x0014",
+    "\\x0015", "\\x0016", "\\x0017", "\\x0018", "\\x0019", "\\x001a", "\\x001b",
+    "\\x001c", "\\x001d", "\\x001e", "\\x001f"};
+  static std::array<uint8_t, 256> json_quotable_character =
+    []() constexpr {
+      std::array<uint8_t, 256> result{};
+      for (int i = 0; i < 32; i++) {
+        result[i] = 1;
+      }
+      for (int i : {'"', '\\'}) {
+        result[i] = 1;
+      }
+      return result;
+    }();
+  // The rest could possibly be vectorized, but consider that we expect most strings
+  // to be short or not to require escaping.
+  for (; i < input.size(); i++) {
+    uint8_t c = static_cast<uint8_t>(input[i]);
+    if(json_quotable_character[c]) {
+      switch (c) {
+      case '"':
+        out[pos++] = '\\';
+        out[pos++] = '"';
+        break;
+      case '\\':
+        out[pos++] = '\\';
+        out[pos++] = '\\';
+        break;
+      case '\b':
+        out[pos++] = '\\';
+        out[pos++] = 'b';
+        break;
+      case '\f':
+        out[pos++] = '\\';
+        out[pos++] = 'f';
+        break;
+      case '\n':
+        out[pos++] = '\\';
+        out[pos++] = 'n';
+        break;
+      case '\r':
+        out[pos++] = '\\';
+        out[pos++] = 'r';
+        break;
+      case '\t':
+        out[pos++] = '\\';
+        out[pos++] = 't';
+        break;
+      default:
+        control_chars[c].copy(out + pos, 6);
+        pos += 6;
+      }
+    } else {
+      out[pos++] = c;
+    }
+  }
+  return pos;
+}
+
+
+
 } // namespace stringparsing
 } // unnamed namespace
 } // namespace SIMDJSON_IMPLEMENTATION

src/haswell.cpp

@@ -161,6 +161,10 @@ simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *
   return stage2(_doc);
 }
 
+simdjson_warn_unused size_t implementation::write_string_escaped(const std::string_view input, char *out) const noexcept {
+  return haswell::stringparsing::write_string_escaped(input, out);
+}
+
 } // namespace SIMDJSON_IMPLEMENTATION
 } // namespace simdjson
 

src/icelake.cpp

@@ -207,6 +207,10 @@ simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *
   return stage2(_doc);
 }
 
+simdjson_warn_unused size_t implementation::write_string_escaped(const std::string_view input, char *out) const noexcept {
+  return icelake::stringparsing::write_string_escaped(input, out);
+}
+
 } // namespace icelake
 } // namespace simdjson
 

src/implementation.cpp

@@ -186,6 +186,9 @@ public:
   simdjson_warn_unused bool validate_utf8(const char * buf, size_t len) const noexcept final override {
     return set_best()->validate_utf8(buf, len);
   }
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final {
+    return set_best()->write_string_escaped(input, out);
+  }
   simdjson_inline detect_best_supported_implementation_on_first_use() noexcept : implementation("best_supported_detector", "Detects the best supported implementation and sets it", 0) {}
 private:
   const implementation *set_best() const noexcept;
@@ -236,6 +239,9 @@ public:
   simdjson_warn_unused error_code minify(const uint8_t *, size_t, uint8_t *, size_t &) const noexcept final override {
     return UNSUPPORTED_ARCHITECTURE;
   }
+  simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) const noexcept final {
+    return 0;
+  }
   simdjson_warn_unused bool validate_utf8(const char *, size_t) const noexcept final override {
     return false; // Just refuse to validate. Given that we have a fallback implementation
     // it seems unlikely that unsupported_implementation will ever be used. If it is used,
@@ -319,6 +325,9 @@ simdjson_warn_unused error_code minify(const char *buf, size_t len, char *dst, s
 simdjson_warn_unused bool validate_utf8(const char *buf, size_t len) noexcept {
   return get_active_implementation()->validate_utf8(buf, len);
 }
+simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) noexcept {
+  return get_active_implementation()->write_string_escaped(input, out);
+}
 const implementation * builtin_implementation() {
   static const implementation * builtin_impl = get_available_implementations()[SIMDJSON_STRINGIFY(SIMDJSON_BUILTIN_IMPLEMENTATION)];
   assert(builtin_impl);

src/lasx.cpp

@@ -124,6 +124,10 @@ simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *
   return stage2(_doc);
 }
 
+simdjson_warn_unused size_t implementation::write_string_escaped(const std::string_view input, char *out) const noexcept {
+  return lasx::stringparsing::write_string_escaped(input, out);
+}
+
 } // namespace lasx
 } // namespace simdjson
 

src/lsx.cpp

@@ -128,6 +128,10 @@ simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *
   return stage2(_doc);
 }
 
+simdjson_warn_unused size_t implementation::write_string_escaped(const std::string_view input, char *out) const noexcept {
+  return lsx::stringparsing::write_string_escaped(input, out);
+}
+
 } // namespace lsx
 } // namespace simdjson
 

src/ppc64.cpp

@@ -134,6 +134,10 @@ simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *
   return stage2(_doc);
 }
 
+simdjson_warn_unused size_t implementation::write_string_escaped(const std::string_view input, char *out) const noexcept {
+  return ppc64::stringparsing::write_string_escaped(input, out);
+}
+
 } // namespace ppc64
 } // namespace simdjson
 

src/westmere.cpp

@@ -166,6 +166,10 @@ simdjson_warn_unused error_code dom_parser_implementation::parse(const uint8_t *
   return stage2(_doc);
 }
 
+simdjson_warn_unused size_t implementation::write_string_escaped(const std::string_view input, char *out) const noexcept {
+  return westmere::stringparsing::write_string_escaped(input, out);
+}
+
 } // namespace westmere
 } // namespace simdjson
 

tests/CMakeLists.txt

@@ -26,3 +26,4 @@ endif()
 add_cpp_test(checkimplementation LABELS other per_implementation)
 
 add_subdirectory(compilation_failure_tests)
+add_subdirectory(builder)

tests/builder/CMakeLists.txt

@@ -0,0 +1,12 @@
+# All remaining tests link with simdjson proper
+link_libraries(simdjson)
+include_directories(..)
+add_cpp_test(builder_string_builder_tests              LABELS ondemand acceptance per_implementation)
+
+# Copy the simdjson dll into the tests directory
+if(MSVC AND BUILD_SHARED_LIBS)
+  add_custom_command(TARGET builder_string_builder_tests POST_BUILD        # Adds a post-build event
+    COMMAND ${CMAKE_COMMAND} -E copy_if_different  # which executes "cmake -E copy_if_different..."
+        "$<TARGET_FILE:simdjson>"      # <--this is in-file
+        "$<TARGET_FILE_DIR:builder_string_builder_tests>")                 # <--this is out-file path
+endif(MSVC AND BUILD_SHARED_LIBS)

tests/builder/builder_string_builder_tests.cpp

@@ -0,0 +1,178 @@
+#include "simdjson.h"
+#include "test_builder.h"
+#include <string_view>
+
+using namespace simdjson;
+
+namespace builder_tests {
+  using namespace std;
+
+    #if SIMDJSON_EXCEPTIONS
+    bool string_convertion_except() {
+        TEST_START();
+        simdjson::ondemand::parser p;
+        simdjson::builder::string_builder sb;
+        sb.append('a');
+        std::string r(sb);
+        ASSERT_EQUAL(r, "a");
+        TEST_SUCCEED();
+    }
+    #endif
+
+
+    bool append_char() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append('a');
+        ASSERT_EQUAL(sb.size(), 1);
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "a");
+        TEST_SUCCEED();
+    }
+
+
+    bool append_integer() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append(42);
+        ASSERT_EQUAL(sb.size(), 2);
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "42");
+        TEST_SUCCEED();
+    }
+
+    bool append_float() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append(1.1);
+        ASSERT_EQUAL(sb.size(), 3);
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "1.1");
+        TEST_SUCCEED();
+    }
+
+    bool append_null() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append_null();
+        ASSERT_EQUAL(sb.size(), 4);
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "null");
+        TEST_SUCCEED();
+    }
+
+    bool clear() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append('a');
+        sb.clear();
+        ASSERT_EQUAL(sb.size(), 0);
+        TEST_SUCCEED();
+    }
+
+    bool escape_and_append() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.escape_and_append("Hello, \"world\"!");
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "Hello, \"world\"!");
+        TEST_SUCCEED();
+    }
+
+    bool escape_and_append_with_quotes() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.escape_and_append_with_quotes("Hello, \"world\"!");
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "\"Hello, \\\"world\\\"!\"");
+        TEST_SUCCEED();
+    }
+
+    bool append_raw() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append_raw("Test");
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "Test");
+        TEST_SUCCEED();
+    }
+
+    bool raw_with_length() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append_raw("Test String", 4);
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "Test");
+        TEST_SUCCEED();
+    }
+
+    bool string_convertion() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append('a');
+        std::string_view p;
+        auto result = sb.view().get(p);
+        ASSERT_EQUAL(result, SUCCESS);
+        ASSERT_EQUAL(p, "a");
+        TEST_SUCCEED();
+    }
+
+    bool unicode_validation() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        sb.append('a');
+        ASSERT_TRUE(sb.validate_unicode());
+        TEST_SUCCEED();
+    }
+
+    bool buffer_growth() {
+        TEST_START();
+        simdjson::builder::string_builder sb;
+        for(int i = 0; i < 3; ++i) {
+            sb.append('a');
+        }
+        ASSERT_EQUAL(sb.size(), 3);
+        TEST_SUCCEED();
+    }
+
+    bool run() {
+        return
+    #if SIMDJSON_EXCEPTIONS
+            string_convertion_except() &&
+    #endif
+            append_char() &&
+            append_integer() &&
+            append_float() &&
+            append_null() &&
+            clear() &&
+            escape_and_append() &&
+            escape_and_append_with_quotes() &&
+            append_raw() &&
+            raw_with_length() &&
+            string_convertion() &&
+            buffer_growth() &&
+            unicode_validation() &&
+            true;
+    }
+
+} // namespace twitter_tests
+
+int main(int argc, char *argv[]) {
+  return test_main(argc, argv, builder_tests::run);
+}

tests/builder/test_builder.h

@@ -0,0 +1,51 @@
+#ifndef ONDEMAND_TEST_BUILDER_H
+#define ONDEMAND_TEST_BUILDER_H
+
+#include <iostream>
+#include <unistd.h>
+#include "simdjson.h"
+#include "cast_tester.h"
+#include "test_macros.h"
+
+template<typename F>
+int test_main(int argc, char *argv[], const F& test_function) {
+  std::cout << std::unitbuf;
+  int c;
+  while ((c = getopt(argc, argv, "a:")) != -1) {
+    switch (c) {
+    case 'a': {
+      const simdjson::implementation *impl = simdjson::get_available_implementations()[optarg];
+      if (!impl) {
+        std::fprintf(stderr, "Unsupported architecture value -a %s\n", optarg);
+        return EXIT_FAILURE;
+      }
+      simdjson::get_active_implementation() = impl;
+      break;
+    }
+    default:
+      std::fprintf(stderr, "Unexpected argument %c\n", c);
+      return EXIT_FAILURE;
+    }
+  }
+
+  // this is put here deliberately to check that the documentation is correct (README),
+  // should this fail to compile, you should update the documentation:
+  if (simdjson::get_active_implementation()->name() == "unsupported") {
+    std::printf("unsupported CPU\n");
+    std::abort();
+  }
+  // We want to know what we are testing.
+  std::cout << "builtin_implementation -- " << simdjson::builtin_implementation()->name() << std::endl;
+  std::cout << "------------------------------------------------------------" << std::endl;
+
+  std::cout << "Running tests." << std::endl;
+  if (test_function()) {
+    std::cout << "Success!" << std::endl;
+    return EXIT_SUCCESS;
+  } else {
+    std::cerr << "FAILED." << std::endl;
+    return EXIT_FAILURE;
+  }
+}
+
+#endif // ONDEMAND_TEST_BUILDER_H

tests/dom/CMakeLists.txt

@@ -12,6 +12,7 @@ add_cpp_test(errortests                 LABELS dom acceptance per_implementation
 add_cpp_test(extracting_values_example  LABELS dom acceptance per_implementation)
 add_cpp_test(integer_tests              LABELS dom acceptance per_implementation)
 add_cpp_test(jsoncheck                  LABELS dom acceptance per_implementation)
+add_cpp_test(json_path_tests            LABELS dom acceptance per_implementation)
 add_cpp_test(minefieldcheck             LABELS dom acceptance per_implementation)
 add_cpp_test(numberparsingcheck         LABELS dom acceptance per_implementation) # https://tools.ietf.org/html/rfc6901
 add_cpp_test(parse_many_test            LABELS dom acceptance per_implementation)

tests/dom/json_path_tests.cpp

@@ -0,0 +1,350 @@
+/**
+ * refer to pathcheck.cpp
+ */
+
+#include <iostream>
+#include <string>
+using namespace std::string_literals;
+
+#include "simdjson.h"
+#include "test_macros.h"
+
+// we define our own asserts to get around NDEBUG
+#ifndef ASSERT
+#define ASSERT(x)                                                              \
+  {                                                                            \
+    if (!(x)) {                                                                \
+      std::cerr << "Failed assertion " << #x << std::endl;                     \
+      return false;                                                            \
+    }                                                                          \
+  }
+#endif
+
+using namespace simdjson;
+
+bool demo() {
+#if SIMDJSON_EXCEPTIONS
+  std::cout << "demo test" << std::endl;
+  auto cars_json = R"( [
+  { "make": "Toyota", "model": "Camry",  "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
+  { "make": "Kia",    "model": "Soul",   "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
+  { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }
+] )"_padded;
+  dom::parser parser;
+  dom::element cars = parser.parse(cars_json);
+  double x = cars.at_path("$[0].tire_pressure[1]");
+  if (x != 39.9)
+    return false;
+  // Iterating through an array of objects
+  std::vector<double> measured;
+  for (dom::element car_element : cars) {
+    dom::object car;
+    simdjson::error_code error;
+    if ((error = car_element.get(car))) {
+      std::cerr << error << std::endl;
+      return false;
+    }
+    double x3 = car.at_path("$.tire_pressure[1]");
+    measured.push_back(x3);
+  }
+  std::vector<double> expected = {39.9, 31, 30};
+  if (measured != expected) {
+    return false;
+  }
+#endif
+  return true;
+}
+
+const padded_string TEST_JSON = R"(
+  {
+    "/~01abc": [
+      0,
+      {
+        "\\\" 0": [
+          "value0",
+          "value1"
+        ]
+      }
+    ],
+    "0": "0 ok",
+    "01": "01 ok",
+    "": "empty ok",
+    "arr": []
+  }
+)"_padded;
+
+const padded_string TEST_RFC_JSON = R"(
+   {
+      "foo": ["bar", "baz"],
+      "": 0,
+      "a/b": 1,
+      "c%d": 2,
+      "e^f": 3,
+      "g|h": 4,
+      "i\\j": 5,
+      "k\"l": 6,
+      " ": 7,
+      "m~n": 8
+   }
+)"_padded;
+
+bool run_success_test(const padded_string &source, const char *json_path,
+                      std::string_view expected_value) {
+  std::cout << "Running successful JSONPath test '" << json_path << "' ..."
+            << std::endl;
+  dom::parser parser;
+  dom::element doc;
+  auto error = parser.parse(source).get(doc);
+  if (error) {
+    std::cerr << "cannot parse: " << error << std::endl;
+    return false;
+  }
+  dom::element answer;
+  error = doc.at_path(json_path).get(answer);
+  if (error) {
+    std::cerr << "cannot access pointer: " << error << std::endl;
+    return false;
+  }
+  std::string str_answer = simdjson::minify(answer);
+  if (str_answer != expected_value) {
+    std::cerr << "They differ!!!" << std::endl;
+    std::cerr << "   found    '" << str_answer << "'" << std::endl;
+    std::cerr << "   expected '" << expected_value << "'" << std::endl;
+  }
+  ASSERT_EQUAL(str_answer, expected_value);
+  return true;
+}
+
+bool run_failure_test(const padded_string &source, const char *json_path,
+                      error_code expected_error) {
+  std::cout << "Running invalid JSONPath test '" << json_path << "' ..."
+            << std::endl;
+  dom::parser parser;
+  ASSERT_ERROR(parser.parse(source).at_path(json_path).error(), expected_error);
+  return true;
+}
+
+bool demo_relative_path() {
+  TEST_START();
+  auto cars_json = R"( [
+        { "make": "Toyota", "model": "Camry",  "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
+        { "make": "Kia",    "model": "Soul",   "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
+        { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }
+        ] )"_padded;
+
+  dom::parser parser;
+  dom::element cars;
+  std::vector<double> measured;
+  auto error = parser.parse(cars_json).get(cars);
+  if (error) {
+    std::cerr << "cannot parse: " << error << std::endl;
+    return false;
+  }
+  dom::array cars_array;
+  error = cars.get(cars_array);
+  if (error) {
+    std::cerr << "cannot get array: " << error << std::endl;
+    return false;
+  }
+  for (auto car_element : cars_array) {
+    double x;
+    ASSERT_SUCCESS(car_element.at_path(".tire_pressure[1]").get(x));
+    measured.push_back(x);
+  }
+
+  std::vector<double> expected = {39.9, 31, 30};
+  if (measured != expected) {
+    return false;
+  }
+  TEST_SUCCEED();
+}
+
+bool many_json_paths() {
+  TEST_START();
+  auto cars_json = R"( [
+        { "make": "Toyota", "model": "Camry",  "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
+        { "make": "Kia",    "model": "Soul",   "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
+        { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }
+        ] )"_padded;
+
+  dom::parser parser;
+  dom::element cars;
+  std::vector<double> measured;
+  ASSERT_SUCCESS(parser.parse(cars_json).get(cars));
+  for (int i = 0; i < 3; i++) {
+    double x;
+    std::string json_path = std::string("$[") + std::to_string(i) +
+                            std::string("].tire_pressure[1]");
+    ASSERT_SUCCESS(cars.at_path(json_path).get(x));
+    measured.push_back(x);
+  }
+
+  std::vector<double> expected = {39.9, 31, 30};
+  if (measured != expected) {
+    return false;
+  }
+  TEST_SUCCEED();
+}
+
+bool many_json_paths_object_array() {
+  TEST_START();
+  auto dogcatpotato =
+      R"( { "dog" : [1,2,3], "cat" : [5, 6, 7], "potato" : [1234]})"_padded;
+
+  dom::parser parser;
+  dom::element doc;
+  ASSERT_SUCCESS(parser.parse(dogcatpotato).get(doc));
+  dom::object obj;
+  ASSERT_SUCCESS(doc.get_object().get(obj));
+  int64_t x;
+  ASSERT_SUCCESS(obj.at_path("$.dog[1]").get(x));
+  ASSERT_EQUAL(x, 2);
+  ASSERT_SUCCESS(obj.at_path("$.potato[0]").get(x));
+  ASSERT_EQUAL(x, 1234);
+  TEST_SUCCEED();
+}
+bool many_json_paths_object() {
+  TEST_START();
+  auto cfoofoo2 =
+      R"( { "c" :{ "foo": { "a": [ 10, 20, 30 ] }}, "d": { "foo2": { "a": [ 10, 20, 30 ] }} , "e": 120 })"_padded;
+  dom::parser parser;
+  dom::element doc;
+  ASSERT_SUCCESS(parser.parse(cfoofoo2).get(doc));
+  dom::object obj;
+  ASSERT_SUCCESS(doc.get_object().get(obj));
+  int64_t x;
+  ASSERT_SUCCESS(obj.at_path("$.c.foo.a[1]").get(x));
+  ASSERT_EQUAL(x, 20);
+  ASSERT_SUCCESS(obj.at_path("$.d.foo2.a.2").get(x));
+  ASSERT_EQUAL(x, 30);
+  ASSERT_SUCCESS(obj.at_path("$.e").get(x));
+  ASSERT_EQUAL(x, 120);
+  TEST_SUCCEED();
+}
+bool many_json_paths_array() {
+  TEST_START();
+  auto cfoofoo2 =
+      R"( [ 111, 2, 3, { "foo": { "a": [ 10, 20, 33 ] }}, { "foo2": { "a": [ 10, 20, 30 ] }}, 1001 ])"_padded;
+  dom::parser parser;
+  dom::element doc;
+  ASSERT_SUCCESS(parser.parse(cfoofoo2).get(doc));
+  dom::array arr;
+  ASSERT_SUCCESS(doc.get_array().get(arr));
+  int64_t x;
+  ASSERT_SUCCESS(arr.at_path("$[3].foo.a[1]").get(x));
+  ASSERT_EQUAL(x, 20);
+  TEST_SUCCEED();
+}
+struct car_type {
+  std::string make;
+  std::string model;
+  uint64_t year;
+  std::vector<double> tire_pressure;
+  car_type(std::string_view _make, std::string_view _model, uint64_t _year,
+           std::vector<double> &&_tire_pressure)
+      : make{_make}, model{_model}, year(_year), tire_pressure(_tire_pressure) {
+  }
+};
+
+bool json_path_invalidation() {
+  TEST_START();
+  auto cars_json = R"( [
+        { "make": "Toyota", "model": "Camry",  "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
+        { "make": "Kia",    "model": "Soul",   "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
+        { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }
+        ] )"_padded;
+
+  dom::parser parser;
+  dom::element cars;
+  std::vector<double> measured;
+  ASSERT_SUCCESS(parser.parse(cars_json).get(cars));
+  std::vector<car_type> content;
+  for (int i = 0; i < 3; i++) {
+    dom::object obj;
+    std::string json_path =
+        std::string("$[") + std::to_string(i) + std::string("]");
+    // Each successive at_path call invalidates
+    // previously parsed values, strings, objects and array.
+    ASSERT_SUCCESS(cars.at_path(json_path).get(obj));
+    // We materialize the object.
+    std::string_view make;
+    ASSERT_SUCCESS(obj["make"].get(make));
+    std::string_view model;
+    ASSERT_SUCCESS(obj["model"].get(model));
+    uint64_t year;
+    ASSERT_SUCCESS(obj["year"].get(year));
+    // We materialize the array.
+    dom::array arr;
+    ASSERT_SUCCESS(obj["tire_pressure"].get(arr));
+    std::vector<double> values;
+    for (auto x : arr) {
+      double value_double;
+      ASSERT_SUCCESS(x.get(value_double));
+      values.push_back(value_double);
+    }
+    content.emplace_back(make, model, year, std::move(values));
+  }
+  std::string expected[] = {"Toyota", "Kia", "Toyota"};
+  int i = 0;
+  for (car_type c : content) {
+    std::cout << c.make << " " << c.model << " " << c.year << "\n";
+    ASSERT_EQUAL(expected[i++], c.make);
+  }
+  TEST_SUCCEED();
+}
+// for 0.5 version and following (standard compliant)
+bool modern_support() {
+#if SIMDJSON_EXCEPTIONS
+  std::cout << "modern test" << std::endl;
+  auto example_json = R"({"key": "value", "array": [0, 1, 2]})"_padded;
+  dom::parser parser;
+  dom::element example = parser.parse(example_json);
+  std::string_view value_str = example.at_path("$.key");
+  ASSERT_EQUAL(value_str, "value");
+  int64_t array0 = example.at_path("$.array[0]");
+  ASSERT_EQUAL(array0, 0);
+  array0 = example.at_path("$.array").at_path("$[0]");
+  ASSERT_EQUAL(array0, 0);
+  ASSERT_ERROR(example.at_path("$.no_such_key").error(), NO_SUCH_FIELD);
+  ASSERT_ERROR(example.at_path("$.array[9]").error(), INDEX_OUT_OF_BOUNDS);
+  ASSERT_ERROR(example.at_path("$.array.not_a_num").error(), INCORRECT_TYPE);
+  ASSERT_ERROR(example.at_path("$.array.").error(), INVALID_JSON_POINTER);
+#endif
+  return true;
+}
+
+int main() {
+  if (true && demo() && modern_support() &&
+      run_success_test(TEST_RFC_JSON, "$.foo", "[\"bar\",\"baz\"]") &&
+      run_success_test(TEST_RFC_JSON, "$.foo[0]", "\"bar\"") &&
+      run_success_test(TEST_RFC_JSON, "$.", "0") &&
+      run_success_test(TEST_RFC_JSON, "$.a/b", "1") &&
+      run_success_test(TEST_RFC_JSON, "$.c%d", "2") &&
+      run_success_test(TEST_RFC_JSON, "$.e^f", "3") &&
+      run_success_test(TEST_RFC_JSON, "$.g|h", "4") &&
+      run_success_test(TEST_RFC_JSON, "$.i\\j", "5") &&
+      run_success_test(TEST_RFC_JSON, "$.k\"l", "6") &&
+      run_success_test(TEST_RFC_JSON, "$. ", "7") &&
+      run_success_test(TEST_RFC_JSON, "$.m~n", "8") &&
+      run_success_test(TEST_JSON, "$./~01abc",
+                       R"([0,{"\\\" 0":["value0","value1"]}])") &&
+      run_success_test(TEST_JSON, "$./~01abc[1]",
+                       R"({"\\\" 0":["value0","value1"]})") &&
+      run_success_test(TEST_JSON, "$./~01abc[1].\\\" 0",
+                       R"(["value0","value1"])") &&
+      run_success_test(TEST_JSON, "$.arr", R"([])") // get array
+      &&
+      run_failure_test(TEST_JSON, R"($./~01abc[1].\\\" 0[2])", NO_SUCH_FIELD) &&
+      run_failure_test(TEST_JSON, "$.arr[0]", INDEX_OUT_OF_BOUNDS) &&
+      run_failure_test(TEST_JSON, "/~01abc", INVALID_JSON_POINTER) &&
+      run_failure_test(TEST_JSON, ".~1abc", NO_SUCH_FIELD) &&
+      run_failure_test(TEST_JSON, "./~01abc.01", INVALID_JSON_POINTER) &&
+      run_failure_test(TEST_JSON, "./~01abc.", INVALID_JSON_POINTER) &&
+      run_failure_test(TEST_JSON, "./~01abc.-", INDEX_OUT_OF_BOUNDS)) {
+    std::cout << "Success!" << std::endl;
+    return 0;
+  } else {
+    std::cerr << "Failed!" << std::endl;
+    return 1;
+  }
+}

tests/dom/readme_examples.cpp

@@ -461,7 +461,44 @@ void parse_documentation_lowlevel() {
   (void)element;
 }
 
+
+void jsondollar() {
+  dom::parser parser;
+  auto json = R"( { "c" :{ "foo": { "a": [ 10, 20, 30 ] }}, "d": { "foo2": { "a": [ 10, 20, 30 ] }} , "e": 120 })"_padded;
+  dom::element doc;
+  auto error = parser.parse(json).get(doc);
+  if(error) { exit(-1); }
+  dom::object obj;
+  error = doc.get_object().get(obj);
+  if(error) { exit(-1); }
+  int64_t x = 0; // initialization to silence unwarranted compiler warning
+  error = obj.at_path("$[3].foo.a[1]").get(x);
+  if(error) { exit(-1); }
+  if(x != 20) { exit(-1); }
+  x = obj.at_path("$.d.foo2.a.2");
+  if(x != 30) { exit(-1); }
+  if(error) { exit(-1); }
+}
+
+void jsonpath() {
+  auto cars_json = R"( [
+    { "make": "Toyota", "model": "Camry",  "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },
+    { "make": "Kia",    "model": "Soul",   "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },
+    { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }
+  ] )"_padded;
+  dom::parser parser;
+  dom::element doc;
+  auto error = parser.parse(cars_json).get(doc);
+  if(error) { exit(-1); }
+  double p = 0.0; // initialization to silence unwarranted compiler warning
+  error = doc.at_path("[0].tire_pressure[1]").get(p);
+  if(error) { exit(-1); }
+  if(p != 39.9) { exit(-1); }
+}
+
 int main() {
+  jsonpath();
+  jsondollar();
   basics_dom_1();
   basics_dom_2();
   basics_dom_3();

tests/ondemand/compilation_failure_tests/CMakeLists.txt

@@ -13,7 +13,8 @@ function(add_dual_compile_test TEST_NAME)
   target_compile_definitions(${TEST_NAME}_should_not_compile PRIVATE COMPILATION_TEST_USE_FAILING_CODE=1)
 endfunction(add_dual_compile_test)
 
-
+add_dual_compile_test(iterate_object)
+add_dual_compile_test(iterate_array)
 add_dual_compile_test(iterate_char_star)
 add_dual_compile_test(iterate_string_view)
 add_dual_compile_test(iterate_temporary_buffer)

tests/ondemand/compilation_failure_tests/iterate_array.cpp

@@ -0,0 +1,34 @@
+
+#include <iostream>
+#include "simdjson.h"
+
+using namespace simdjson;
+
+int main() {
+    auto json = "[1]"_padded;
+    ondemand::parser parser;
+    auto f = [](ondemand::parser& p, simdjson::padded_string& jsons) -> ondemand::document {
+        ondemand::document doc;
+        auto error = p.iterate(jsons).get(doc);
+        if(error) { std::abort(); }
+        return doc;
+    };
+    ondemand::array arrayv;
+#if COMPILATION_TEST_USE_FAILING_CODE
+    // Not allowed as this would be unsafe, the document must remain alive.
+    auto error = f(parser).get_array().get(arrayv);
+#else
+    ondemand::document doc = f(parser, json);
+    auto error = doc.get_array().get(arrayv);
+#endif
+    if(error) {
+        std::cout << "Failure" << std::endl;
+    }
+    int64_t a = 0;
+    error = arrayv.at(0).get_int64().get(a);
+    if(error) {
+        std::cout << "failure" << std::endl;
+    }
+    printf("a = %d\n", (int)a);
+    return EXIT_SUCCESS;
+}

tests/ondemand/compilation_failure_tests/iterate_object.cpp

@@ -0,0 +1,34 @@
+
+#include <iostream>
+#include "simdjson.h"
+
+using namespace simdjson;
+
+int main() {
+    auto json = "{\"a\":1}"_padded;
+    ondemand::parser parser;
+    auto f = [](ondemand::parser& p, simdjson::padded_string& jsons) -> ondemand::document {
+        ondemand::document doc;
+        auto error = p.iterate(jsons).get(doc);
+        if(error) { std::abort(); }
+        return doc;
+    };
+    ondemand::object objv;
+#if COMPILATION_TEST_USE_FAILING_CODE
+    // Not allowed as this would be unsafe, the document must remain alive.
+    auto error = f(parser).get_object().get(objv);
+#else
+    ondemand::document doc = f(parser, json);
+    auto error = doc.get_object().get(objv);
+#endif
+    if(error) {
+        std::cout << "Failure" << std::endl;
+    }
+    int64_t a = 0;
+    error = objv["a"].get_int64().get(a);
+    if(error) {
+        std::cout << "failure" << std::endl;
+    }
+    printf("a = %d\n", (int)a);
+    return EXIT_SUCCESS;
+}

tests/ondemand/ondemand_cacheline.cpp

@@ -21,8 +21,24 @@ long page_size() {
 // Returns true if the buffer + len + simdjson::SIMDJSON_PADDING crosses the
 // page boundary.
 bool need_allocation(const char *buf, size_t len) {
-  return ((reinterpret_cast<uintptr_t>(buf + len - 1) % page_size()) <
+  return ((reinterpret_cast<uintptr_t>(buf + len - 1) % page_size())
-          simdjson::SIMDJSON_PADDING);
+    + simdjson::SIMDJSON_PADDING > static_cast<uintptr_t>(page_size()));
+}
+
+bool check_need_allocation() {
+  long ps = page_size();
+  std::vector<std::tuple<uintptr_t,size_t,bool>> test_cases = {
+    {2*ps, 5, false}, {2*ps, ps, true}, {2*ps, 100, false},
+  };
+  for(auto t : test_cases) {
+    uintptr_t buf = std::get<0>(t);
+    size_t len = std::get<1>(t);
+    bool expected = std::get<2>(t);
+    if(need_allocation(reinterpret_cast<const char *>(buf), len) != expected) {
+      return false;
+    }
+  }
+  return true;
 }
 
 simdjson::padded_string_view
@@ -39,6 +55,10 @@ get_padded_string_view(const char *buf, size_t len,
 
 int main() {
   printf("page_size: %ld\n", page_size());
+  if(!check_need_allocation()) {
+    printf("Bug in the need_allocation function.\n");
+    return EXIT_FAILURE;
+  }
   const char *jsonpoiner = R"(
         {
             "key": "value"

tests/ondemand/ondemand_compilation_tests.cpp

@@ -59,6 +59,22 @@ void compilation_test_3() {
     }
   }
 }
+
+// Do not run this, it is only meant to compile
+void compilation_test_4() {
+  const padded_string bogus = ""_padded;
+  ondemand::parser parser;
+  int64_t x1 = (int64_t)parser.iterate(bogus);
+  double x2 = (double)parser.iterate(bogus);
+  std::string_view x = (std::string_view)parser.iterate(bogus);
+  uint64_t x3 = (uint64_t)parser.iterate(bogus);
+  bool x4 = (bool)parser.iterate(bogus);
+  (void) x1;
+  (void) x2;
+  (void) x3;
+  (void) x4;
+  (void) x;
+}
 #endif // SIMDJSON_EXCEPTIONS
 
 int main(void) {

tests/ondemand/ondemand_json_path_tests.cpp

@@ -346,6 +346,54 @@ namespace json_path_tests {
         TEST_SUCCEED();
     }
 
+    bool many_json_paths_with_prefix() {
+        TEST_START();
+        // object
+        {
+            auto cfoofoo2 = R"( { "c" :{ "foo": { "a": [ 10, 20, 30 ] }}, "d": { "foo2": { "a": [ 10, 20, 30 ] }} , "e": 120 })"_padded;
+            ondemand::parser parser;
+            ondemand::document doc;
+            ASSERT_SUCCESS(parser.iterate(cfoofoo2).get(doc));
+            ondemand::object obj;
+            ASSERT_SUCCESS(doc.get_object().get(obj));
+            int64_t x;
+            ASSERT_SUCCESS(obj.at_path("$.c.foo.a[1]").get(x));
+            ASSERT_EQUAL(x, 20);
+            ASSERT_SUCCESS(obj.at_path("$.d.foo2.a.2").get(x));
+            ASSERT_EQUAL(x, 30);
+            ASSERT_SUCCESS(obj.at_path("$.e").get(x));
+            ASSERT_EQUAL(x, 120);
+        }
+        // array
+        {
+            auto cfoofoo2 = R"( [ 111, 2, 3, { "foo": { "a": [ 10, 20, 33 ] }}, { "foo2": { "a": [ 10, 20, 30 ] }}, 1001 ])"_padded;
+            ondemand::parser parser;
+            ondemand::document doc;
+            ASSERT_SUCCESS(parser.iterate(cfoofoo2).get(doc));
+            ondemand::array arr;
+            ASSERT_SUCCESS(doc.get_array().get(arr));
+            int64_t x;
+            ASSERT_SUCCESS(arr.at_path("$[3].foo.a[1]").get(x));
+            ASSERT_EQUAL(x, 20);
+        }
+        // onject array
+        {
+            auto dogcatpotato = R"( { "dog" : [1,2,3], "cat" : [5, 6, 7], "potato" : [1234]})"_padded;
+
+            ondemand::parser parser;
+            ondemand::document doc;
+            ASSERT_SUCCESS(parser.iterate(dogcatpotato).get(doc));
+            ondemand::object obj;
+            ASSERT_SUCCESS(doc.get_object().get(obj));
+            int64_t x;
+            ASSERT_SUCCESS(obj.at_path("$.dog[1]").get(x));
+            ASSERT_EQUAL(x, 2);
+            ASSERT_SUCCESS(obj.at_path("$.potato[0]").get(x));
+            ASSERT_EQUAL(x, 1234);
+        }
+        TEST_SUCCEED();
+    }
+
 #if SIMDJSON_EXCEPTIONS
     bool json_path_invalidation_exceptions() {
         TEST_START();
@@ -392,6 +440,7 @@ namespace json_path_tests {
                 many_json_paths_array() &&
                 many_json_paths_object() &&
                 many_json_paths_object_array() &&
+                many_json_paths_with_prefix() &&
                 run_broken_tests() &&
                 json_path_invalidation() &&
                 demo_test() &&

tests/ondemand/ondemand_json_pointer_tests.cpp

@@ -164,8 +164,7 @@ namespace json_pointer_tests {
         ASSERT_TRUE(is_scalar);
         ASSERT_ERROR(doc.at_pointer("").get(val), simdjson::SCALAR_DOCUMENT_AS_VALUE);
         std::cout << "  checking true"<< std::endl;
-        ASSERT_SUCCESS(parser.iterate(true_json).get(doc));
+        ASSERT_SUCCESS(parser.iterate(true_json).is_scalar().get(is_scalar));
-        ASSERT_SUCCESS(doc.is_scalar().get(is_scalar));
         ASSERT_TRUE(is_scalar);
         ASSERT_ERROR(doc.at_pointer("").get(val), simdjson::SCALAR_DOCUMENT_AS_VALUE);
         std::cout << "  checking object"<< std::endl;

tests/ondemand/ondemand_readme_examples.cpp

@@ -221,6 +221,19 @@ bool gen_raw2() {
   TEST_SUCCEED();
 }
 
+bool jsondollar() {
+  TEST_START();
+  auto json = R"( { "c" :{ "foo": { "a": [ 10, 20, 30 ] }}, "d": { "foo2": { "a": [ 10, 20, 30 ] }} , "e": 120 })"_padded;
+  ondemand::parser parser;
+  ondemand::document doc = parser.iterate(json);
+  ondemand::object obj = doc.get_object();
+  int64_t x = obj.at_path("$.c.foo.a[1]"); // 20
+  ASSERT_EQUAL(x, 20);
+  x = obj.at_path("$.d.foo2.a.2"); // 30
+  ASSERT_EQUAL(x, 30);
+  TEST_SUCCEED();
+}
+
 bool gen_raw3() {
   TEST_START();
   simdjson::ondemand::parser parser;
@@ -1876,6 +1889,7 @@ bool run() {
 #if SIMDJSON_CPLUSPLUS17
     && big_int_array()
 #endif // SIMDJSON_CPLUSPLUS17
+    && jsondollar()
     && big_int_array_as_double()
     && key_raw_json_token()
     && to_optional()

tests/ondemand/ondemand_scalar_tests.cpp

@@ -369,15 +369,24 @@ namespace scalar_tests {
     ondemand::parser parser;
     ondemand::document doc;
     ASSERT_SUCCESS(parser.iterate(json).get(doc));
-    bool x;
+    ASSERT_ERROR(doc.is_null(), INCORRECT_TYPE);
-    ASSERT_SUCCESS(doc.is_null().get(x));
+    TEST_SUCCEED();
-    ASSERT_TRUE(!x);
+  }
+
+  bool nul() {
+    TEST_START();
+    auto json = R"( nul )"_padded;
+    ondemand::parser parser;
+    ondemand::document doc;
+    ASSERT_SUCCESS(parser.iterate(json).get(doc));
+    ASSERT_ERROR(doc.is_null(), INCORRECT_TYPE);
     TEST_SUCCEED();
   }
 
   bool run() {
     return
            nully() &&
+           nul() &&
            string_with_trailing() &&
            uint64_with_trailing() &&
            int64_with_trailing() &&