Compare commits

...

24 Commits

Author SHA1 Message Date
Daniel Lemire 44c69f64f6 bumping up cmake 2026-03-25 10:21:18 -04:00
Daniel Lemire 844e5eac23 bump google bench 2026-03-25 09:29:33 -04:00
Daniel Lemire 45caa861cd Update README with talk information
Added details about talks at QCon San Francisco 2019 and CppCon 2025.
2026-03-24 22:00:42 -04:00
Andrey Abramov 50bf372d3f Add SereneDB to the list of projects (#2641)
Add SereneDB to the list of projects
2026-03-22 12:37:23 -04:00
Daniel Lemire b4a0fd3ff4 4.4.2 2026-03-20 11:12:35 -04:00
Levi Zim a1ff05a5e8 Fix RISC-V compilation without RVV (#2637)
#2617 introduced an RVV check that only checks if the compiler supports RVV intrinsics without checking if RVV is enabled at compile time.

This has caused compilation failure of node.js on riscv64. Fix it by appending a check for __riscv_vector.
2026-03-20 11:11:38 -04:00
Daniel Lemire 5395aacb8b v4.4.1 2026-03-17 12:39:07 -04:00
Daniel Lemire 781ea4b495 fixing issue https://github.com/nodejs/node/pull/62257 (#2631) 2026-03-17 12:38:11 -04:00
Jaël Champagne Gareau 03e8d14eee fix failing CI and rename misleading workflow name (#2630) 2026-03-15 18:40:08 -04:00
Daniel Lemire be94694290 v4.4.0 2026-03-12 20:43:36 -04:00
Daniel Lemire 6065b3b424 disabling loongson LASX runtime dispatching everywhere but GCC15+ (#2626) 2026-03-12 20:40:43 -04:00
Daniel Lemire 019bf1dfe0 adding memory-file mapping (#2625)
* adding memory-file mapping

* tuning

* init _capacity

* adding missing header

* update doc
2026-03-12 14:36:01 -04:00
Daniel Lemire 2922822622 altivec version of find_next_json_quotable_character (#2622)
* altivec version of find_next_json_quotable_character

* simplify

* saving.
2026-03-08 14:06:01 -04:00
Daniel Lemire 262ddad037 Update bug report template with support policy details
Clarified support policy for obsolete compiler systems and vendors.
2026-03-02 14:08:04 -05:00
Hendrik ba007b0cb9 add mruby-fast-json binding (#2616)
Hi there,

i took simdjson as a learning opportunity to learn more about c++, as such i have covered most of the public API Surface simdjson got.

Have fun :)
2026-02-28 14:58:01 -05:00
Sudhanshu Shukla ec675d9c61 Add RISC-V RVV support for find_next_json_quotable_character (#2617) 2026-02-28 13:11:47 -05:00
Daniel Lemire 7dcc196f66 iterating over a simdjson_result<padded_string> (#2614) 2026-02-27 11:01:39 -05:00
Daniel Lemire 7ec4572d33 trimming whitespace 2026-02-26 16:09:34 -05:00
Daniel Lemire 5891d4b64b adding AI policy 2026-02-26 15:09:40 -05:00
Daniel Lemire 3cd9875f1f simplifying the find_next_json_quotable_character fnc for neon processors (#2613) 2026-02-25 01:06:08 -05:00
Daniel Lemire 6cf1bdba43 saving. 2026-02-24 18:01:16 -05:00
Sudhanshu Shukla 817f10dd7d Add LoongArch LSX support for find_next_json_quotable_character (#2608) 2026-02-23 15:00:26 -05:00
Daniel Lemire e829566589 4.3.1 2026-02-20 16:17:16 -05:00
Daniel Lemire dbe0c1543e fix bad hint (#2610) 2026-02-20 16:16:37 -05:00
52 changed files with 2502 additions and 335 deletions
+4
View File
@@ -29,6 +29,9 @@ We accept the identification of an issue by a sanitizer or some checker tool (e.
We recommend that you run your tests using different optimization levels. In particular, we recommend your run tests with the simdjson library and you code compiled in debug mode. The simdjson then sets the SIMDJSON_DEVELOPMENT_CHECKS macro to 1, and this triggers additional checks on your code and on the internals of the library. If possible, we recommend that you run tests with sanitizers (e.g., see [No more leaks with sanitize flags in gcc and clang](https://lemire.me/blog/2016/04/20/no-more-leaks-with-sanitize-flags-in-gcc-and-clang/)). You can compile the library with sanitizers for debugging purposes (e.g., set SIMDJSON_SANITIZE to ON using CMake), but you should also turn on sanitizers on your own code. You may also use tools like valgrind or the commercial equivalent.
Mixing debug and release simdjson code is unsafe: you either build all your code using simdjson in
release mode or all of it in debug mode.
Before reporting a bug, please ensure that you have read our documentation.
**To Reproduce**
@@ -55,6 +58,7 @@ We support up-to-date 64-bit ARM and x64 FreeBSD, macOS, Windows and Linux syste
* We do not support unreleased or experimental compilers. If you encounter an issue with a
pre-release version of a compiler, do not report it as a bug to simdjson. However, we always
invite contributions either in the form an analysis or of a code contribution.
* Vendors (e.g., Apple and Microsoft) stop supporting old systems. Once a compiler system is no longer supported by its vendor, we no longer support it. We will gladly accept code contributions, but we do not consider it a *bug* if you have issues with an obsolete compiler systems. This policy extends to obsolete standard libraries, linkers and other build tools. Please do not report it as an issue. If you cannot resolve the issue yourself, we encourage you to reach out to the vendor for legacy support.
Under Windows, we support Visual Studio (both with LLVM and without). We do not support MinGW and other alternate compiler systems. Windows users should be aware that there [is a long-running bug with GCC under Windows](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412).
+1 -1
View File
@@ -20,7 +20,7 @@ How to verify / test
Please read before contributing:
- CONTRIBUTING: https://github.com/simdjson/simdjson/blob/master/CONTRIBUTING.md
- HACKING: https://github.com/simdjson/simdjson/blob/master/HACKING.md
- AI Usage Policy: https://github.com/simdjson/simdjson/blob/master/AI_USAGE_POLICY.md
If you can, we recommend running our tests with the sanitizers turned on.
+1 -1
View File
@@ -24,6 +24,6 @@ jobs:
apt-get update -q -y
apt-get install -y cmake make g++
run: |
cmake -DSIMDJSON_SANITIZE_UNDEFINED=ON -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
cmake -DSIMDJSON_SANITIZE_UNDEFINED=ON -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
cmake --build build -j=2
ctest --output-on-failure --test-dir build
+3 -3
View File
@@ -18,13 +18,13 @@ jobs:
run: |
mkdir builddebug &&
cd builddebug &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cd .. &&
mkdir build &&
cd build &&
cmake -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cmake --install . &&
@@ -35,7 +35,7 @@ jobs:
run: |
mkdir buildshared &&
cd buildshared &&
cmake -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=destination -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cmake --install . &&
+1 -1
View File
@@ -43,6 +43,6 @@ jobs:
run: |
mkdir build
cd build
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_BUILD_TYPE=${{ matrix.type }} -DBUILD_SHARED_LIBS=OFF -DSIMDJSON_DO_NOT_USE_THREADS_NO_MATTER_WHAT=ON ..
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_BUILD_TYPE=${{ matrix.type }} -DBUILD_SHARED_LIBS=OFF -DSIMDJSON_DO_NOT_USE_THREADS_NO_MATTER_WHAT=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON ..
cmake --build . --verbose
ctest -j4 --output-on-failure -LE explicitonly
+1 -1
View File
@@ -24,6 +24,6 @@ jobs:
apt-get update -q -y
apt-get install -y cmake make g++
run: |
cmake -DCMAKE_BUILD_TYPE=Release -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
cmake -DCMAKE_BUILD_TYPE=Release -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
cmake --build build -j=2
ctest --output-on-failure --test-dir build
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
- name: Build
run: |
CC=clang-18 CXX=clang++-18 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv_zvbb" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
cmake --build build/ -j$(nproc) --config Release
- name: Test VLEN=1024
run: |
+2 -2
View File
@@ -20,7 +20,7 @@ jobs:
- name: Build
run: |
CC=clang-20 CXX=clang++-20 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
cmake --build build/ -j$(nproc) --config Release
- name: Test VLEN=128
run: |
@@ -30,7 +30,7 @@ jobs:
- name: Build VLS
run: |
CC=clang-20 CXX=clang++-20 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv_zvl128b_zba_zbb_zbc -mrvv-vector-bits=zvl" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build-vls
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build-vls
cmake --build build-vls/ -j$(nproc) --config Release
- name: Test VLEN=128 VLS
run: |
+2 -2
View File
@@ -20,7 +20,7 @@ jobs:
- name: Build
run: |
CC=riscv64-linux-gnu-gcc-14 CXX=riscv64-linux-gnu-g++-14 CFLAGS=-march=rv64gcv CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
cmake --build build/ -j$(nproc) --config Release
- name: Test VLEN=256
run: |
@@ -30,7 +30,7 @@ jobs:
- name: Build VLS
run: |
CC=riscv64-linux-gnu-gcc-14 CXX=riscv64-linux-gnu-g++-14 CFLAGS="-march=rv64gcv_zvl256b -mrvv-vector-bits=zvl" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build-vls
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build-vls
cmake --build build-vls/ -j$(nproc) --config Release
- name: Test VLEN=256 VLS
run: |
+2 -2
View File
@@ -20,7 +20,7 @@ jobs:
- name: Build
run: |
CC=clang-19 CXX=clang++-19 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
cmake --build build/ -j$(nproc) --config Release
- name: Test VLEN=512
run: |
@@ -30,7 +30,7 @@ jobs:
- name: Build VLS
run: |
CC=clang-19 CXX=clang++-19 CFLAGS="--target=riscv64-linux-gnu -march=rv64gcv_zvl512b_zba_zbb_zbc -mrvv-vector-bits=zvl" CXXFLAGS="${CFLAGS}" \
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -B build-vls
cmake --toolchain=cmake/toolchains-ci/riscv64-linux-gnu.cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build-vls
cmake --build build-vls/ -j$(nproc) --config Release
- name: Test VLEN=512 VLS
run: |
+1 -1
View File
@@ -24,6 +24,6 @@ jobs:
apt-get update -q -y
apt-get install -y cmake make g++
run: |
cmake -DCMAKE_BUILD_TYPE=Release -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
cmake -DCMAKE_BUILD_TYPE=Release -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
cmake --build build -j=2
ctest --output-on-failure --test-dir build
+1 -1
View File
@@ -18,6 +18,6 @@ jobs:
run: |
mkdir build &&
cd build &&
CXX=clang++-13 cmake -DSIMDJSON_DEVELOPER_MODE=ON .. &&
CXX=clang++-13 cmake -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j
@@ -19,6 +19,6 @@ jobs:
run: |
mkdir build &&
cd build &&
CXX=g++-12 cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GLIBCXX_ASSERTIONS=ON -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON .. &&
CXX=g++-12 cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GLIBCXX_ASSERTIONS=ON -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest . -E avoid_
+1 -1
View File
@@ -18,7 +18,7 @@ jobs:
run: |
mkdir build &&
cd build &&
CXX=g++-12 cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE_THREADS=ON .. &&
CXX=g++-12 cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE_THREADS=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . --target document_stream_tests --target ondemand_document_stream_tests --target parse_many_test &&
ctest --output-on-failure -R parse_many_test &&
ctest --output-on-failure -R document_stream_tests
+2 -2
View File
@@ -1,4 +1,4 @@
name: Performance check on Ubuntu 20.04 CI (GCC 9)
name: Performance check on Ubuntu 24.04 CI (GCC 13)
on:
push:
@@ -24,6 +24,6 @@ jobs:
run: |
mkdir build &&
cd build &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_ENABLE_DOM_CHECKPERF=ON -DCMAKE_CXX_FLAGS="-Werror=old-style-cast -pedantic -Wpedantic" -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_ENABLE_DOM_CHECKPERF=ON -DCMAKE_CXX_FLAGS="-Werror=old-style-cast -pedantic -Wpedantic" -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . --target checkperf &&
ctest --output-on-failure -R checkperf
@@ -13,7 +13,7 @@ jobs:
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- name: Prepare
run: cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_EXCEPTIONS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -B build
run: cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_EXCEPTIONS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
env:
CXX: ${{matrix.cxx}}
- name: Build
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- name: Prepare
run: cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_DEVELOPER_MODE=ON -B build
run: cmake -DSIMDJSON_CXX_STANDARD=20 -DSIMDJSON_DEVELOPER_MODE=ON -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
env:
CXX: ${{matrix.cxx}}
- name: Build
+3 -3
View File
@@ -1,4 +1,4 @@
name: Ubuntu 20.04 CI (GCC 9) without exceptions
name: Ubuntu 24.04 CI (GCC 13) without exceptions
on: [push, pull_request]
@@ -18,13 +18,13 @@ jobs:
run: |
mkdir builddebug &&
cd builddebug &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_EXCEPTIONS=OFF -DBUILD_SHARED_LIBS=OFF .. &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_EXCEPTIONS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cd .. &&
mkdir build &&
cd build &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_EXCEPTIONS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_EXCEPTIONS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
make install &&
+3 -3
View File
@@ -1,4 +1,4 @@
name: Ubuntu 20.04 CI (GCC 9) Without Threads
name: Ubuntu 24.04 CI (GCC 13) Without Threads
on: [push, pull_request]
@@ -18,13 +18,13 @@ jobs:
run: |
mkdir builddebug &&
cd builddebug &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_ENABLE_THREADS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_ENABLE_THREADS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
cd .. &&
mkdir build &&
cd build &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_ENABLE_THREADS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination .. &&
cmake -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_GOOGLE_BENCHMARKS=ON -DSIMDJSON_ENABLE_THREADS=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX:PATH=destination -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j &&
make install &&
+4 -4
View File
@@ -1,9 +1,9 @@
name: Ubuntu 20.04 CI (GCC 9) With Memory Sanitizer
name: Ubuntu 24.04 CI (GCC 13) With Memory Sanitizer
on: [push, pull_request]
jobs:
ubuntu-build-address-sanitizier:
ubuntu-build-address-sanitizer:
if: >-
! contains(toJSON(github.event.commits.*.message), '[skip ci]') &&
! contains(toJSON(github.event.commits.*.message), '[skip github]')
@@ -18,7 +18,7 @@ jobs:
run: |
mkdir builddebug &&
cd builddebug &&
cmake -DSIMDJSON_SANITIZE=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake -DSIMDJSON_SANITIZE=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j
ubuntu-build-undefined-sanitizer:
@@ -36,6 +36,6 @@ jobs:
run: |
mkdir builddebugundefsani &&
cd builddebugundefsani &&
cmake -DSIMDJSON_SANITIZE_UNDEFINED=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF .. &&
cmake -DSIMDJSON_SANITIZE_UNDEFINED=ON -DCMAKE_BUILD_TYPE=Debug -DSIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_DEVELOPER_MODE=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON .. &&
cmake --build . &&
ctest --output-on-failure -LE explicitonly -j
+1 -1
View File
@@ -16,7 +16,7 @@ jobs:
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- name: Prepare
run: cmake -DCMAKE_BUILD_TYPE=${{matrix.build_type}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE=${{matrix.sanitizer}} -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
run: cmake -DCMAKE_BUILD_TYPE=${{matrix.build_type}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_SANITIZE=${{matrix.sanitizer}} -DBUILD_SHARED_LIBS=${{matrix.shared}} -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
env:
CXX: ${{matrix.cxx}}
- name: Build
+1 -1
View File
@@ -17,5 +17,5 @@ jobs:
uses: actions/checkout@v4
- name: Use cmake
run: |
cmake -A ${{ matrix.arch }} -DCMAKE_SYSTEM_VERSION="10.0.22621.0" -DCMAKE_CROSSCOMPILING=1 -DSIMDJSON_DEVELOPER_MODE=ON -D SIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_EXCEPTIONS=OFF -B build &&
cmake -A ${{ matrix.arch }} -DCMAKE_SYSTEM_VERSION="10.0.22621.0" -DCMAKE_CROSSCOMPILING=1 -DSIMDJSON_DEVELOPER_MODE=ON -D SIMDJSON_GOOGLE_BENCHMARKS=OFF -DSIMDJSON_EXCEPTIONS=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build &&
cmake --build build --verbose
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
uses: actions/checkout@v4
- name: Configure
run: |
cmake -DSIMDJSON_CXX_STANDARD=20 -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -B build
cmake -DSIMDJSON_CXX_STANDARD=20 -G "${{matrix.gen}}" -A ${{matrix.arch}} -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DBUILD_SHARED_LIBS=${{matrix.shared}} -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
- name: Build Debug
run: cmake --build build --config Debug --verbose
- name: Build Release
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
uses: actions/checkout@v4
- name: Configure
run: |
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -T ClangCL -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -B build
cmake -G "${{matrix.gen}}" -A ${{matrix.arch}} -T ClangCL -DSIMDJSON_DEVELOPER_MODE=ON -DSIMDJSON_COMPETITION=OFF -DCMAKE_COMPILE_WARNING_AS_ERROR=ON -B build
- name: Build
run: cmake --build build --config ${{matrix.build_type}} --verbose
- name: Run tests
+56
View File
@@ -0,0 +1,56 @@
# AI Usage Policy
Contributors can use whatever tools they would like to
craft their contributions, but there must be a **human in the loop**.
**Contributors must read and review all LLM-generated code or text before they
ask other project members to review it.** The contributor is always the author
and is fully accountable for their contributions. Contributors should be
sufficiently confident that the contribution is high enough quality that asking
for a review is a good use of scarce maintainer time, and they should be **able
to answer questions about their work** during review.
We expect that new contributors will be less confident in their contributions,
and our guidance to them is to **start with small contributions** that they can
fully understand to build confidence. We aspire to be a welcoming community
that helps new contributors grow their expertise, but learning involves taking
small steps, getting feedback, and iterating. Passing maintainer feedback to an
LLM doesn't help anyone grow, and does not sustain our community.
This policy includes, but is not limited to, the following kinds of
contributions:
- Code, usually in the form of a pull request
- Issues or security vulnerabilities
- Comments and feedback on pull requests
## Extractive Contributions
The reason for our "human-in-the-loop" contribution policy is that processing
patches, PRs, RFCs, and comments is not free -- it takes a lot of
maintainer time and energy to review those contributions! Sending the
unreviewed output of an LLM to open source project maintainers *extracts* work
from them in the form of design and code review, so we call this kind of
contribution an "extractive contribution".
## Transparency
For contributions involving significant AI assistance, we encourage you to disclose
its use and explain your process. If a submission appears to rely heavily on AI
without disclosure, we may doubt that the **human-in-the-loop** requirement has
been met. Please show awareness of your use of AI.
## Copyright
Artificial intelligence systems raise many questions around copyright that have
yet to be answered. Our policy on AI tools is similar to our copyright policy:
Contributors are responsible for ensuring that they have the right to
contribute code under the terms of our license, typically meaning that either
they, their employer, or their collaborators hold the copyright. Using AI tools
to regenerate copyrighted material does not remove the copyright, and
contributors are responsible for ensuring that such material does not appear in
their contributions. Contributions found to violate this policy will be removed
just like any other offending contribution.
## Reference
- [LLVM AI Tool Use Policy](https://discourse.llvm.org/t/rfc-llvm-ai-tool-policy-human-in-the-loop/89159)
+4 -4
View File
@@ -1,4 +1,4 @@
cmake_minimum_required(VERSION 3.14)
cmake_minimum_required(VERSION 3.24)
# Build performance optimizations
set(CMAKE_EXPORT_COMPILE_COMMANDS ON CACHE BOOL "Export compile commands for faster IDE integration")
@@ -12,7 +12,7 @@ endif()
project(
simdjson
# The version number is modified by tools/release.py
VERSION 4.3.0
VERSION 4.4.2
DESCRIPTION "Parsing gigabytes of JSON per second"
HOMEPAGE_URL "https://simdjson.org/"
LANGUAGES CXX C
@@ -29,8 +29,8 @@ string(
# ---- Options, variables ----
# These version numbers are modified by tools/release.py
set(SIMDJSON_LIB_VERSION "30.0.0" CACHE STRING "simdjson library version")
set(SIMDJSON_LIB_SOVERSION "30" CACHE STRING "simdjson library soversion")
set(SIMDJSON_LIB_VERSION "31.0.0" CACHE STRING "simdjson library version")
set(SIMDJSON_LIB_SOVERSION "31" CACHE STRING "simdjson library soversion")
option(SIMDJSON_BUILD_STATIC_LIB "Build simdjson_static library along with simdjson (only makes sense if BUILD_SHARED_LIBS=ON)" OFF)
if(SIMDJSON_BUILD_STATIC_LIB AND NOT BUILD_SHARED_LIBS)
+7
View File
@@ -101,3 +101,10 @@ Getting Started Hacking
An overview of simdjson's directory structure, with pointers to architecture and design
considerations and other helpful notes, can be found at [HACKING.md](HACKING.md).
AI Usage Policy
---------------
Please also review our [AI Usage Policy](AI_USAGE_POLICY.md).
+1 -1
View File
@@ -38,7 +38,7 @@ PROJECT_NAME = simdjson
# could be handy for archiving the generated documentation or if some version
# control system is used.
PROJECT_NUMBER = "4.3.0"
PROJECT_NUMBER = "4.4.2"
# Using the PROJECT_BRIEF tag one can provide an optional one line description
# for a project that appears at the top of each page and should give viewer a
+9 -1
View File
@@ -65,6 +65,7 @@ Real-world usage
- [GreptimeDB](https://github.com/GreptimeTeam/greptimedb)
- [mamba](https://github.com/mamba-org/mamba)
- [Ladybird Browser](https://ladybird.org)
- [SereneDB](https://github.com/serenedb/serenedb)
If you are planning to use simdjson in a product, please work from one of our releases.
@@ -188,6 +189,7 @@ We distinguish between "bindings" (which just wrap the C++ code) and a port to a
- [JSON::SIMD](https://metacpan.org/pod/JSON::SIMD): Perl bindings; fully-featured JSON module that uses simdjson for decoding.
- [gemmaJSON](https://github.com/sainttttt/gemmaJSON): Nim JSON parser based on simdjson bindings.
- [simdjson-java](https://github.com/simdjson/simdjson-java): Java port.
- [mruby-fast-json](https://github.com/Asmod4n/mruby-fast-json): mruby binding with high API coverage.
About simdjson
--------------
@@ -210,10 +212,16 @@ We have an in-depth paper focused on the UTF-8 validation:
We also have an informal [blog post providing some background and context](https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of-json-per-second/).
For the video inclined, <br />
For the video inclined, we had a talk at QCon San Francisco 2019<br />
[![simdjson at QCon San Francisco 2019](http://img.youtube.com/vi/wlvKAT7SZIQ/0.jpg)](http://www.youtube.com/watch?v=wlvKAT7SZIQ)<br />
(It was the best voted talk, we're kinda proud of it.)
We also had a CppCon 2025 talk. We show how C++26 reflection allows for one-line serialization (to_json(player)) or deserialization—without invasive macros or manual mapping—using nothing but the C++ standard library. Whether youre a performance junkie or simply interested in the roadmap for the next decade of C++ development, watch our full talk!
[![simdjson at CppCon 2025](http://img.youtube.com/vi/Mcgk3CxHYMs/0.jpg)](http://www.youtube.com/watch?v=Mcgk3CxHYMs)<br />
Citing this work
-----------------
+1 -1
View File
@@ -171,7 +171,7 @@ We recommend Visual Studio 2019 or better on a 64-bit system.")
else()
target_compile_options(
simdjson-internal-flags INTERFACE
-Werror -Wall -Wextra -Weffc++ -Wsign-compare -Wshadow -Wwrite-strings
-Wall -Wextra -Weffc++ -Wsign-compare -Wshadow -Wwrite-strings
-Wpointer-arith -Winit-self -Wconversion -Wno-sign-conversion
)
if(CMAKE_CXX_STANDARD VERSION_GREATER_EQUAL 20)
+1 -1
View File
@@ -12,7 +12,7 @@ cmake_dependent_option(SIMDJSON_GOOGLE_BENCHMARKS "compile the Google Benchmark
if(SIMDJSON_GOOGLE_BENCHMARKS)
CPMAddPackage(
NAME google_benchmarks
URL https://github.com/google/benchmark/archive/refs/tags/v1.9.4.zip
URL https://github.com/google/benchmark/archive/refs/tags/v1.9.5.zip
OPTIONS
"BENCHMARK_ENABLE_TESTING OFF"
"BENCHMARK_ENABLE_INSTALL OFF"
+19
View File
@@ -271,6 +271,21 @@ Further, they may use the AreFileApisANSI function to determine whether
the filename is interpreted using the ANSI or the system default OEM
codepage, and they may call SetFileApisToOEM accordingly.
**Advanced feature:**
On non-Windows systems, you can use memory-file mapping to create a `simdjson::padded_string_view`
from a file on disk.
```cpp
// If the macro _WIN32 is defined, this will not work since we do not support memory-file mapping
// under Windows at this time.
simdjson::padded_memory_map map(myfilename);
if (!map.is_valid()) { /* handle error */ }
simdjson::padded_string_view view = map.view(); // view is usable while padded_memory_map is in scope
ondemand::document doc = parser.iterate(view); // parse the JSON
```
Documents are iterators
-----------------------
@@ -374,6 +389,10 @@ builds to disable additional runtime testing and get the best performance. We
disable these checks on a best-effort basis but the C++ standard does not provide
a direct way to check for a release build.
Warnign: Mixing debug and release simdjson code is unsafe: you either build all your code
using simdjson in release mode or all of it in debug mode.
Using the parsed JSON
---------------------
+16
View File
@@ -125,6 +125,22 @@ Further, they may use the AreFileApisANSI function to determine whether
the filename is interpreted using the ANSI or the system default OEM
codepage, and they may call SetFileApisToOEM accordingly.
**Advanced feature:**
On non-Windows systems, you can use memory-file mapping to create a `simdjson::padded_string_view`
from a file on disk.
```cpp
// if the macro _WIN32 is defined, this will not work since we do not support Windows
simdjson::padded_memory_map map(TWITTER_JSON);
if (!map.is_valid()) { /* handle error */ }
simdjson::padded_string_view view = map.view(); // view is usable while padded_memory_map is in scope
ondemand::document doc = parser.iterate(view); // parse the JSON
```
Using memory-file mapping requires some care. The file should not be modified while you are
accessing it.
Using the Parsed JSON
---------------------
+3
View File
@@ -33,6 +33,9 @@ compiles *all* the implementations into the executable. On Intel, it will includ
(icelake, haswell, westmere and fallback), on 64-bit ARM it will include just one since running dispatching is unnecessary, and on PPC
it will include 2 (ppc64 and fallback).
On Loongson processors, LASX runtime dispatching is only enabled on GCC 15+, not on LLVM or older versions of GCC.
Thus unless you compile specifically for LASX or use GCC 15+, you will not benefit from LASX support.
If you know more about where you're going to run and want to save the space, you can disable any of
these implementations at compile time with `-DSIMDJSON_IMPLEMENTATION_X=0` (where X is ICELAKE, HASWELL,
WESTMERE, ARM64, PPC64, LSX, LASX and FALLBACK).
+14
View File
@@ -154,6 +154,20 @@ for (auto doc : docs) {
See [basics.md](basics.md#newline-delimited-json-ndjson-and-json-lines) for an overview of the API.
**Advanced feature:**
On non-Windows systems, you can use memory-file mapping to create a `simdjson::padded_string_view`
from a file on disk.
```cpp
// If the macro _WIN32 is defined, this will not work since we do not support memory-file mapping
// under Windows at this time.
simdjson::padded_memory_map map(myfilename);
if (!map.is_valid()) { /* handle error */ }
simdjson::padded_string_view view = map.view(); // view is usable while padded_memory_map is in scope
ondemand::document doc = parser.iterate(view); // parse the JSON
```
## Use cases
From [jsonlines.org](http://jsonlines.org/examples/):
+14
View File
@@ -217,6 +217,20 @@ got full document at 29
```
**Advanced feature:**
On non-Windows systems, you can use memory-file mapping to create a `simdjson::padded_string_view`
from a file on disk.
```cpp
// If the macro _WIN32 is defined, this will not work since we do not support memory-file mapping
// under Windows at this time.
simdjson::padded_memory_map map(myfilename);
if (!map.is_valid()) { /* handle error */ }
simdjson::padded_string_view view = map.view(); // view is usable while padded_memory_map is in scope
ondemand::document doc = parser.iterate(view); // parse the JSON
```
Incomplete streams
-----------
@@ -2,13 +2,6 @@
function(add_quickstart_test TEST_NAME SOURCE_FILE)
cmake_parse_arguments(PARSE_ARGV 2 ARGS "NO_EXCEPTIONS;NATIVE_ARCH" "" "CXX_STANDARD;LABELS")
# Standard compiler flags
if (MSVC)
list(APPEND QUICKSTART_FLAGS /WX)
else()
list(APPEND QUICKSTART_FLAGS -Werror)
endif()
# Native architecture compiler flag
if (ARGS_NATIVE_ARCH)
if (MSVC)
-1
View File
@@ -119,5 +119,4 @@
#define SIMDJSON_CONSTEVAL 0
#endif // defined(__cpp_consteval) && __cpp_consteval >= 201811L && defined(__cpp_lib_constexpr_string) && __cpp_lib_constexpr_string >= 201907L
#endif // !defined(SIMDJSON_CONSTEVAL)
#endif // SIMDJSON_COMPILER_CHECK_H
@@ -41,6 +41,19 @@ simdjson_inline int leading_zeroes(uint64_t input_num) {
#endif// _MSC_VER
}
simdjson_inline int trailing_zeroes(uint64_t input_num) {
#ifdef _MSC_VER
unsigned long trailing_zero = 0;
// Search the mask data from least significant bit (LSB)
// to most significant bit (MSB) for a set bit (1).
if (_BitScanForward64(&trailing_zero, input_num))
return (int)trailing_zero;
else return 64;
#else
return __builtin_ctzll(input_num);
#endif// _MSC_VER
}
} // unnamed namespace
} // namespace fallback
} // namespace simdjson
@@ -27,6 +27,22 @@
#define SIMDJSON_EXPERIMENTAL_HAS_NEON 1
#endif
#endif
#if defined(__loongarch_sx)
#ifndef SIMDJSON_EXPERIMENTAL_HAS_LSX
#define SIMDJSON_EXPERIMENTAL_HAS_LSX 1
#endif
#endif
#if defined(__riscv_v_intrinsic) && __riscv_v_intrinsic >= 11000 && \
defined(__riscv_vector)
#ifndef SIMDJSON_EXPERIMENTAL_HAS_RVV
#define SIMDJSON_EXPERIMENTAL_HAS_RVV 1
#endif
#endif
#if (defined(__PPC64__) || defined(_M_PPC64)) && defined(__ALTIVEC__)
#ifndef SIMDJSON_EXPERIMENTAL_HAS_PPC64
#define SIMDJSON_EXPERIMENTAL_HAS_PPC64 1
#endif
#endif
#if SIMDJSON_EXPERIMENTAL_HAS_NEON
#include <arm_neon.h>
#ifdef _MSC_VER
@@ -39,6 +55,22 @@
#include <intrin.h>
#endif
#endif
#if SIMDJSON_EXPERIMENTAL_HAS_LSX
#include <lsxintrin.h>
#endif
#if SIMDJSON_EXPERIMENTAL_HAS_RVV
#include <riscv_vector.h>
#endif
#if SIMDJSON_EXPERIMENTAL_HAS_PPC64
#include <altivec.h>
#ifdef bool
#undef bool
#endif
#ifdef vector
#undef vector
#endif
#endif
namespace simdjson {
namespace SIMDJSON_IMPLEMENTATION {
@@ -139,6 +171,35 @@ simdjson_inline bool fast_needs_escaping(std::string_view view) {
}
return _mm_movemask_epi8(running) != 0;
}
#elif SIMDJSON_EXPERIMENTAL_HAS_PPC64
simdjson_inline bool fast_needs_escaping(std::string_view view) {
if (view.size() < 16) {
return simple_needs_escaping(view);
}
size_t i = 0;
__vector unsigned char running = vec_splats((unsigned char)0);
__vector unsigned char v34 = vec_splats((unsigned char)34);
__vector unsigned char v92 = vec_splats((unsigned char)92);
__vector unsigned char v32 = vec_splats((unsigned char)32);
for (; i + 15 < view.size(); i += 16) {
__vector unsigned char word =
vec_vsx_ld(0, reinterpret_cast<const unsigned char *>(view.data() + i));
running = vec_or(running, (__vector unsigned char)vec_cmpeq(word, v34));
running = vec_or(running, (__vector unsigned char)vec_cmpeq(word, v92));
running = vec_or(running,
(__vector unsigned char)vec_cmplt(word, v32));
}
if (i < view.size()) {
__vector unsigned char word = vec_vsx_ld(
0, reinterpret_cast<const unsigned char *>(view.data() + view.length() - 16));
running = vec_or(running, (__vector unsigned char)vec_cmpeq(word, v34));
running = vec_or(running, (__vector unsigned char)vec_cmpeq(word, v92));
running = vec_or(running,
(__vector unsigned char)vec_cmplt(word, v32));
}
return !vec_all_eq(running, vec_splats((unsigned char)0));
}
#else
simdjson_inline bool fast_needs_escaping(std::string_view view) {
return simple_needs_escaping(view);
@@ -182,28 +243,12 @@ find_next_json_quotable_character(const std::string_view view,
needs_escape = vorrq_u8(needs_escape, vceqq_u8(word, v92));
needs_escape = vorrq_u8(needs_escape, vcltq_u8(word, v32));
if (vmaxvq_u32(vreinterpretq_u32_u8(needs_escape)) != 0) {
// Found quotable character - extract exact byte position using ctz
uint64x2_t as64 = vreinterpretq_u64_u8(needs_escape);
uint64_t lo = vgetq_lane_u64(as64, 0);
uint64_t hi = vgetq_lane_u64(as64, 1);
const uint8x8_t res = vshrn_n_u16(vreinterpretq_u16_u8(needs_escape), 4);
const uint64_t mask = vget_lane_u64(vreinterpret_u64_u8(res), 0);
if(mask != 0) {
size_t offset = ptr - reinterpret_cast<const uint8_t *>(view.data());
#ifdef _MSC_VER
unsigned long trailing_zero = 0;
if (lo != 0) {
_BitScanForward64(&trailing_zero, lo);
return offset + trailing_zero / 8;
} else {
_BitScanForward64(&trailing_zero, hi);
return offset + 8 + trailing_zero / 8;
}
#else
if (lo != 0) {
return offset + __builtin_ctzll(lo) / 8;
} else {
return offset + 8 + __builtin_ctzll(hi) / 8;
}
#endif
auto trailing_zero = trailing_zeroes(mask);
return offset + (trailing_zero >> 2);
}
ptr += 16;
remaining -= 16;
@@ -241,13 +286,128 @@ find_next_json_quotable_character(const std::string_view view,
if (mask != 0) {
// Found quotable character - use trailing zero count to find position
size_t offset = ptr - reinterpret_cast<const uint8_t *>(view.data());
#ifdef _MSC_VER
unsigned long trailing_zero = 0;
_BitScanForward(&trailing_zero, mask);
return offset + trailing_zero;
return offset + trailing_zeroes(mask);
}
ptr += 16;
remaining -= 16;
}
// Scalar fallback for remaining bytes
size_t current = len - remaining;
return find_next_json_quotable_character_scalar(view, current);
}
#elif SIMDJSON_EXPERIMENTAL_HAS_LSX
simdjson_inline size_t
find_next_json_quotable_character(const std::string_view view,
size_t location) noexcept {
const size_t len = view.size();
const uint8_t *ptr =
reinterpret_cast<const uint8_t *>(view.data()) + location;
size_t remaining = len - location;
//SIMD constants for characters requiring escape
__m128i v34 = __lsx_vreplgr2vr_b(34); // '"'
__m128i v92 = __lsx_vreplgr2vr_b(92); // '\\'
__m128i v32 = __lsx_vreplgr2vr_b(32); // control char threshold
while (remaining >= 16){
__m128i word = __lsx_vld(ptr, 0);
//Check for the quotable characters: '"', '\\', or control char (<32)
__m128i needs_escape = __lsx_vseq_b(word, v34);
needs_escape = __lsx_vor_v(needs_escape, __lsx_vseq_b(word, v92));
needs_escape = __lsx_vor_v(needs_escape, __lsx_vslt_bu(word, v32));
if (!__lsx_bz_v(needs_escape)){
//Found quotable character - extract exact byte position
uint64_t lo = __lsx_vpickve2gr_du(needs_escape,0);
uint64_t hi = __lsx_vpickve2gr_du(needs_escape,1);
size_t offset = ptr - reinterpret_cast<const uint8_t *>(view.data());
if ( lo != 0) {
return offset + trailing_zeroes(lo) / 8;
} else {
return offset + 8 + trailing_zeroes(hi) / 8;
}
}
ptr += 16;
remaining -= 16;
}
size_t current = len - remaining;
return find_next_json_quotable_character_scalar(view, current);
}
#elif SIMDJSON_EXPERIMENTAL_HAS_RVV
simdjson_inline size_t
find_next_json_quotable_character(const std::string_view view,
size_t location) noexcept {
const size_t len = view.size();
const uint8_t *ptr =
reinterpret_cast<const uint8_t *>(view.data()) + location;
size_t remaining = len - location;
while (remaining > 0) {
size_t vl = __riscv_vsetvl_e8m1(remaining);
vuint8m1_t word = __riscv_vle8_v_u8m1(ptr, vl);
// Check for quotable characters: '"', '\\', or control chars (< 32)
vbool8_t needs_escape = __riscv_vmseq(word, (uint8_t)34, vl);
needs_escape = __riscv_vmor(needs_escape,
__riscv_vmseq(word, (uint8_t)92, vl), vl);
needs_escape = __riscv_vmor(needs_escape,
__riscv_vmsltu(word, (uint8_t)32, vl), vl);
long first = __riscv_vfirst(needs_escape, vl);
if (first >= 0) {
size_t offset = ptr - reinterpret_cast<const uint8_t *>(view.data());
return offset + first;
}
ptr += vl;
remaining -= vl;
}
return len;
}
#elif SIMDJSON_EXPERIMENTAL_HAS_PPC64
simdjson_inline size_t
find_next_json_quotable_character(const std::string_view view,
size_t location) noexcept {
const size_t len = view.size();
const uint8_t *ptr =
reinterpret_cast<const uint8_t *>(view.data()) + location;
size_t remaining = len - location;
// SIMD constants for characters requiring escape
__vector unsigned char v34 = vec_splats((unsigned char)34); // '"'
__vector unsigned char v92 = vec_splats((unsigned char)92); // '\\'
__vector unsigned char v32 = vec_splats((unsigned char)32); // control char threshold
// Bitmask for vec_vbpermq to extract one bit per byte
const __vector unsigned char perm_mask = {0x78, 0x70, 0x68, 0x60, 0x58, 0x50,
0x48, 0x40, 0x38, 0x30, 0x28, 0x20,
0x18, 0x10, 0x08, 0x00};
while (remaining >= 16) {
__vector unsigned char word =
vec_vsx_ld(0, reinterpret_cast<const unsigned char *>(ptr));
// Check for quotable characters: '"', '\\', or control chars (< 32)
__vector unsigned char needs_escape =
(__vector unsigned char)vec_cmpeq(word, v34);
needs_escape = vec_or(needs_escape,
(__vector unsigned char)vec_cmpeq(word, v92));
needs_escape = vec_or(needs_escape,
(__vector unsigned char)vec_cmplt(word, v32));
__vector unsigned long long result =
(__vector unsigned long long)vec_vbpermq(needs_escape, perm_mask);
#ifdef __LITTLE_ENDIAN__
unsigned int mask = static_cast<unsigned int>(result[1]);
#else
return offset + __builtin_ctz(mask);
unsigned int mask = static_cast<unsigned int>(result[0]);
#endif
if (mask != 0) {
size_t offset = ptr - reinterpret_cast<const uint8_t *>(view.data());
return offset + __builtin_ctz(mask);
}
ptr += 16;
remaining -= 16;
@@ -336,7 +496,7 @@ simdjson_inline bool string_builder::capacity_check(size_t upcoming_bytes) {
return true;
}
// check for overflow, most of the time there is no overflow
if (simdjson_likely(position + upcoming_bytes < position)) {
if (simdjson_unlikely(position + upcoming_bytes < position)) {
return false;
}
// We will rarely get here.
+60
View File
@@ -10,6 +10,14 @@
#include <climits>
#include <cwchar>
#ifndef _WIN32
#include <fcntl.h>
#include <stdio.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#endif
namespace simdjson {
namespace internal {
@@ -366,6 +374,57 @@ inline bool padded_string_builder::reserve(size_t additional) noexcept {
return true;
}
#ifndef _WIN32
simdjson_inline padded_memory_map::padded_memory_map(const char *filename) noexcept {
int fd = open(filename, O_RDONLY);
if (fd == -1) {
return; // file not found or cannot be opened, data will be nullptr
}
struct stat st;
if (fstat(fd, &st) == -1) {
close(fd);
return; // failed to get file size, data will be nullptr
}
size = static_cast<size_t>(st.st_size);
size_t total_size = size + simdjson::SIMDJSON_PADDING;
void *anon_map =
mmap(NULL, total_size, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (anon_map == MAP_FAILED) {
close(fd);
return; // failed to create anonymous mapping, data will be nullptr
}
void *file_map =
mmap(anon_map, size, PROT_READ, MAP_SHARED | MAP_FIXED, fd, 0);
if (file_map == MAP_FAILED) {
munmap(anon_map, total_size);
close(fd);
return; // failed to mmap file, data will be nullptr
}
data = static_cast<const char *>(file_map);
close(fd); // no longer needed after mapping
}
simdjson_inline padded_memory_map::~padded_memory_map() noexcept {
if (data != nullptr) {
munmap(const_cast<char *>(data), size + simdjson::SIMDJSON_PADDING);
}
}
simdjson_inline simdjson::padded_string_view padded_memory_map::view() const noexcept simdjson_lifetime_bound {
if(!is_valid()) {
return simdjson::padded_string_view(); // return an empty view if mapping failed
}
return simdjson::padded_string_view(data, size, size + simdjson::SIMDJSON_PADDING);
}
simdjson_inline bool padded_memory_map::is_valid() const noexcept {
return data != nullptr;
}
#endif // _WIN32
} // namespace simdjson
inline simdjson::padded_string operator ""_padded(const char *str, size_t len) {
@@ -376,4 +435,5 @@ inline simdjson::padded_string operator ""_padded(const char8_t *str, size_t len
return simdjson::padded_string(reinterpret_cast<const char *>(str), len);
}
#endif
#endif // SIMDJSON_PADDED_STRING_INL_H
+56
View File
@@ -277,6 +277,62 @@ inline std::ostream& operator<<(std::ostream& out, const padded_string& s) { ret
inline std::ostream& operator<<(std::ostream& out, simdjson_result<padded_string> &s) noexcept(false) { return out << s.value(); }
#endif
#ifndef _WIN32
/**
* A class representing a memory-mapped file with padding.
* It is only available on non-Windows platforms, as Windows has different APIs for memory mapping.
*/
class padded_memory_map {
public:
/**
* Create a new padded memory map for the given file.
* After creating the memory map, you can call view() to get a padded_string_view of the file content.
* The memory map will be automatically released when the padded_memory_map instance is destroyed.
* Note that the file content is not copied, so this is efficient for large files. However,
* the file must remain unchanged while the memory map is in use. In case of error (e.g., file not found,
* permission denied, etc.), the memory map will be invalid and view() will return an empty view.
* You can check if the memory map is valid by calling is_valid() before using view().
*
* @param filename the path to the file to memory-map.
*/
simdjson_inline padded_memory_map(const char *filename) noexcept;
/**
* Destroy the padded memory map and release any resources.
*/
simdjson_inline ~padded_memory_map() noexcept;
// lifetime of the view is tied to the memory map, so we can return a view
// directly
/**
* Get a view of the memory-mapped file. It always succeeds, but the view may be empty
* if the memory map is invalid (e.g., due to file not found, permission denied, etc.).
* You can check if the memory map is valid by calling is_valid() before using the view.
*
* Lifetime of the view is tied to the memory map, so the view should not be used after the
* padded_memory_map instance is destroyed.
*
* @return a padded_string_view representing the memory-mapped file, or an empty view if the memory map is invalid.
*/
simdjson_inline simdjson::padded_string_view view() const noexcept simdjson_lifetime_bound;
/**
* Check if the memory map is valid.
*
* @return true if the memory map is valid, false otherwise.
*/
simdjson_inline bool is_valid() const noexcept;
private:
padded_memory_map() = delete;
padded_memory_map(const padded_memory_map &) = delete;
padded_memory_map &operator=(const padded_memory_map &) = delete;
const char *data{nullptr};
size_t size{0};
};
#endif // _WIN32
} // namespace simdjson
// This is deliberately outside of simdjson so that people get it without having to use the namespace
+1 -1
View File
@@ -17,7 +17,7 @@ namespace simdjson {
*/
class padded_string_view : public std::string_view {
private:
size_t _capacity;
size_t _capacity{0};
public:
/** Create an empty padded_string_view. */
+19
View File
@@ -66,6 +66,25 @@ using std::size_t;
#define SIMDJSON_IS_LASX 1 // We can always run both
#elif defined(__loongarch_sx)
#define SIMDJSON_IS_LSX 1
// Adjust for runtime dispatching support.
#if defined(__GNUC__) && !defined(__clang__) && !defined(__INTEL_COMPILER) && !defined(__NVCOMPILER)
#if __GNUC__ > 15 || (__GNUC__ == 15 && __GNUC_MINOR__ >= 0)
// We are ok, we will support runtime dispatch for LASX.
#else
// We disable runtime dispatch for LASX, which means that we will not be able to use LASX
// even if it is supported by the hardware.
// Loongson users should update to GCC 15 or better.
#define SIMDJSON_IMPLEMENTATION_LASX 0
#endif
#else
// We are not using GCC, so we assume that we can support runtime dispatch for LASX.
// https://godbolt.org/z/jcMnrjYhs
#define SIMDJSON_IMPLEMENTATION_LASX 0
#endif
#endif
#elif defined(__PPC64__) || defined(_M_PPC64)
#define SIMDJSON_IS_PPC64 1
+3 -3
View File
@@ -4,7 +4,7 @@
#define SIMDJSON_SIMDJSON_VERSION_H
/** The version of simdjson being used (major.minor.revision) */
#define SIMDJSON_VERSION "4.3.0"
#define SIMDJSON_VERSION "4.4.2"
namespace simdjson {
enum {
@@ -15,11 +15,11 @@ enum {
/**
* The minor version (major.MINOR.revision) of simdjson being used.
*/
SIMDJSON_VERSION_MINOR = 3,
SIMDJSON_VERSION_MINOR = 4,
/**
* The revision (major.minor.REVISION) of simdjson being used.
*/
SIMDJSON_VERSION_REVISION = 0
SIMDJSON_VERSION_REVISION = 2
};
} // namespace simdjson
+46 -2
View File
@@ -1,4 +1,4 @@
/* auto-generated on 2026-02-18 20:08:53 -0500. version 4.3.0 Do not edit! */
/* auto-generated on 2026-03-20 11:11:38 -0400. version 4.4.2 Do not edit! */
/* including simdjson.cpp: */
/* begin file simdjson.cpp */
#define SIMDJSON_SRC_SIMDJSON_CPP
@@ -146,7 +146,6 @@
#define SIMDJSON_CONSTEVAL 0
#endif // defined(__cpp_consteval) && __cpp_consteval >= 201811L && defined(__cpp_lib_constexpr_string) && __cpp_lib_constexpr_string >= 201907L
#endif // !defined(SIMDJSON_CONSTEVAL)
#endif // SIMDJSON_COMPILER_CHECK_H
/* end file simdjson/compiler_check.h */
/* including simdjson/portability.h: #include "simdjson/portability.h" */
@@ -219,6 +218,25 @@ using std::size_t;
#define SIMDJSON_IS_LASX 1 // We can always run both
#elif defined(__loongarch_sx)
#define SIMDJSON_IS_LSX 1
// Adjust for runtime dispatching support.
#if defined(__GNUC__) && !defined(__clang__) && !defined(__INTEL_COMPILER) && !defined(__NVCOMPILER)
#if __GNUC__ > 15 || (__GNUC__ == 15 && __GNUC_MINOR__ >= 0)
// We are ok, we will support runtime dispatch for LASX.
#else
// We disable runtime dispatch for LASX, which means that we will not be able to use LASX
// even if it is supported by the hardware.
// Loongson users should update to GCC 15 or better.
#define SIMDJSON_IMPLEMENTATION_LASX 0
#endif
#else
// We are not using GCC, so we assume that we can support runtime dispatch for LASX.
// https://godbolt.org/z/jcMnrjYhs
#define SIMDJSON_IMPLEMENTATION_LASX 0
#endif
#endif
#elif defined(__PPC64__) || defined(_M_PPC64)
#define SIMDJSON_IS_PPC64 1
@@ -60938,6 +60956,19 @@ simdjson_inline int leading_zeroes(uint64_t input_num) {
#endif// _MSC_VER
}
simdjson_inline int trailing_zeroes(uint64_t input_num) {
#ifdef _MSC_VER
unsigned long trailing_zero = 0;
// Search the mask data from least significant bit (LSB)
// to most significant bit (MSB) for a set bit (1).
if (_BitScanForward64(&trailing_zero, input_num))
return (int)trailing_zero;
else return 64;
#else
return __builtin_ctzll(input_num);
#endif// _MSC_VER
}
} // unnamed namespace
} // namespace fallback
} // namespace simdjson
@@ -63169,6 +63200,19 @@ simdjson_inline int leading_zeroes(uint64_t input_num) {
#endif// _MSC_VER
}
simdjson_inline int trailing_zeroes(uint64_t input_num) {
#ifdef _MSC_VER
unsigned long trailing_zero = 0;
// Search the mask data from least significant bit (LSB)
// to most significant bit (MSB) for a set bit (1).
if (_BitScanForward64(&trailing_zero, input_num))
return (int)trailing_zero;
else return 64;
#else
return __builtin_ctzll(input_num);
#endif// _MSC_VER
}
} // unnamed namespace
} // namespace fallback
} // namespace simdjson
+1862 -249
View File
File diff suppressed because it is too large Load Diff
Binary file not shown.
+1
View File
@@ -11,6 +11,7 @@ link_libraries(simdjson)
add_cpp_test(unicode_tests LABELS dom acceptance per_implementation)
add_cpp_test(minify_tests LABELS other acceptance per_implementation)
add_cpp_test(padded_string_tests LABELS other acceptance )
add_cpp_test(memory_map_tests LABELS other acceptance )
add_cpp_test(prettify_tests LABELS other acceptance per_implementation)
add_cpp_test(fractured_json_tests LABELS other acceptance per_implementation)
+47
View File
@@ -0,0 +1,47 @@
#ifdef _WIN32
#include <cstdlib>
// This test is not supported on Windows because it relies on POSIX APIs like
// mmap. Please run it on a POSIX-compliant system.
int main() { return EXIT_SUCCESS; }
#else
#include "simdjson.h"
#include "test_macros.h"
#if SIMDJSON_EXCEPTIONS
bool test_memory_map_exception() {
TEST_START();
simdjson::padded_memory_map map(TWITTER_JSON);
ASSERT_TRUE(map.is_valid());
simdjson::padded_string_view view = map.view();
simdjson::ondemand::document doc = simdjson::ondemand::parser::get_parser().iterate(view);
ASSERT_TRUE(doc["search_metadata"]["count"].get_uint64() == 100);
TEST_SUCCEED();
}
#endif
bool test_memory_map_noexception() {
TEST_START();
simdjson::padded_memory_map map(TWITTER_JSON);
if (!map.is_valid()) {
std::cerr << "Failed to memory-map the file " << TWITTER_JSON << std::endl;
return false;
}
simdjson::padded_string_view view = map.view();
simdjson::ondemand::parser parser;
simdjson::ondemand::document doc;
ASSERT_SUCCESS( parser.iterate(view).get(doc) );
uint64_t count;
ASSERT_SUCCESS( doc["search_metadata"]["count"].get(count) );
ASSERT_EQUAL(count, 100);
TEST_SUCCEED();
}
int main() {
#if SIMDJSON_EXCEPTIONS
return (test_memory_map_exception() && test_memory_map_noexception()) ? EXIT_SUCCESS : EXIT_FAILURE;
#else
return test_memory_map_noexception() ? EXIT_SUCCESS : EXIT_FAILURE;
#endif
}
#endif
+1 -1
View File
@@ -764,7 +764,7 @@ bool basics_1() {
TEST_START();
ondemand::parser parser;
auto json = padded_string::load("twitter.json");
auto json = padded_string::load("twitter.json"); // this is a simdjson_result<padded_string>
ondemand::document doc = parser.iterate(json); // load and parse a file
simdjson_unused auto unused_doc = doc.get_object();
+21
View File
@@ -20,6 +20,26 @@ namespace twitter_tests {
TEST_SUCCEED();
}
#if SIMDJSON_EXCEPTIONS
bool twitter_result_example() {
TEST_START();
simdjson_result<padded_string> json = padded_string::load(TWITTER_JSON);
ondemand::parser parser;
auto doc = parser.iterate(json);
for (ondemand::object tweet : doc["statuses"]) {
uint64_t id = tweet["id"];
std::string_view text = tweet["text"];
std::string_view screen_name = tweet["user"]["screen_name"];
uint64_t retweets = tweet["retweet_count"];
uint64_t favorites = tweet["favorite_count"];
(void) id;
(void) text;
(void) retweets;
(void) favorites;
(void) screen_name;
}
TEST_SUCCEED();
}
bool twitter_example() {
TEST_START();
padded_string json;
@@ -182,6 +202,7 @@ namespace twitter_tests {
twitter_default_profile() &&
twitter_image_sizes() &&
#if SIMDJSON_EXCEPTIONS
twitter_result_example() &&
twitter_count_exception() &&
twitter_example() &&
twitter_default_profile_exception() &&