Daniel Lemire
c806e955c4
* Initial work on JSON builder * moving the files back to ondemand for now. * tweak * more later * update * minor edits * dropping vs arm (missing support) * adding tests. we still specialized write_string_escaped * tweaking * fix typo * tweaking the approach * minor fix * missing store * another missing store * Attempt at fixing failing serialization tests. (#2292) * Fixing appeand_float typo (#2294) * applying a couple of fixes * updating single header * fix for pre C++17 if constexpr * Fixing unused argument problem and updating the singleheader file * various pedantic fixes * Sketch of builder * reordering. * simplify * Adding draft of static reflection based deserialization * Updating simdjson singleheader * patching the automated deserialization. * automated * Adding support for smart pointers of user defined types. * Adding specialization for smart pointers for basic types. I think it is highly likely that this can be done in a more generic way. * Referncing a later version of rapidjson that fixed the issue related with assignment attempt of a const variable for GenericStringRef class. * guarding the tests * adding documentation for string_builder * saving * rename to 'append' * saving * non-functional benchmarks (#2342) * non-functional benchmarks * Fix typo * various fixes * tweaking --------- Co-authored-by: Daniel Lemire <dlemire@lemire.me> Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com> * tuning * various minor fixes * minor tweak * minor simplification * updating amal * adding a cast * update * fancy casting * removing dead code * Pushing latest changes. CITM benchmark is still not working. * Still not working, but now I am getting only 10 errors. * add static reflection benchmark to 'large random' benchmark and allows (#2349) deserialization (with static reflection) from objects and arrays. Co-authored-by: Daniel Lemire <dlemire@lemire.me> * Removing std::map from CitmCatalog definition, since that is not currently supported. * Added free to rust bench, segfault is still happening.. * The syntax changed: ^E became ^^E. (#2350) * The syntax changed: ^E became ^^E. * guarding --------- Co-authored-by: Daniel Lemire <dlemire@lemire.me> * Adding support for string_view_keyed_map types. * Adding concepts as a conditional include. * updating single-header * Adding concepts to ondemand deps * rust benchmark is finally working * Fixing small typo in docs. * adding docker config and instructions so that our users can test the static reflection (#2358) * adding docker config and instructions so that our users can test the static reflection * completing the instructions * pruning white spaces --------- Co-authored-by: Daniel Lemire <dlemire@lemire.me> * minor optimizations on the JSON builder branch * avoiding undef behaviour * saving * somewhat nicer builder * make it possible to run just one benchmark * adding linux perf * fixing minor issue * updating swar * Adding real world compilation benchmark (#2379) * Adding compilation benchmark for json parsing with and without reflection * Moving it to the benchmark folder, also reducing a bit the number of iterations. * Removing script from root folder. * Reducing number of iterations * Update benchmark/benchmark_reflection_usage_compilation.sh Co-authored-by: Daniel Lemire <daniel@lemire.me> * Update benchmark/benchmark_reflection_usage_compilation.sh Co-authored-by: Daniel Lemire <daniel@lemire.me> * Update benchmark/benchmark_reflection_usage_compilation.sh Co-authored-by: Daniel Lemire <daniel@lemire.me> * Making the script more customizable and also test whether the compiler being used supports reflection before actually running the benchmark --------- Co-authored-by: Daniel Lemire <daniel@lemire.me> * Using define_static_string from https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p3491r2.html (#2389) * Applying changes needed after latest reflection paper updates. * Working, but no template for yet. * Updating single-header to incldue the use of define_static_string. * copying over master --------- Co-authored-by: Daniel Lemire <dlemire@lemire.me> Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
Building the experimental LLVM compiler for static reflection
The simdjson has experimental support for static reflection, when the compiler supports it. We focus on the latest p2996 paper that is targeting C++26.
Current status
There are 2 versions of compiler that aim to support the C++26 reflection paper (p2996).
- Clang-p2996 llvm branch that open-source and available in the Compiler Explorer.
- EDG reflection branch that is only publicly available in the Compiler Explorer.
For now, we will resort to #1, since it is open-source.
We are excited to hear that this reflection proposal seems to be on-track for C++26. As per Herb Sutter.
Instructions for building simdjson for static reflection (using docker)
We are assuming that you are running Linux or macOS. We recommend that Windows users rely on WSL.
Follow the following two steps:
- Clone the project
git clone https://github.com/simdjson/simdjson.git
cd simdjson
As of March 2025, support for static reflection is available only in the json_builder_init branch, so you should switch:
git checkout -B json_builder_init
-
Make sure that you have docker installed and running on your system. Most Linux distributions support docker though some (like RedHat) have the equivalent (Podman). Users of Apple systems may want to consider OrbStack. You do not need to familiar with docker, you just need to make sure that you are have it running.
-
While inside the
simdjsonrepository, run the following bash script:
bash ./p2996/run_docker.sh bash
This will enter a bash shell with access to the repo directory. Note that this will take some time when running it for the first time, since the specific container image has to be built.
This step builds and executes a docker container defined by our Dockerfile which provides the necessary environment.
Importantly, we build the experimental LLVM compiler based on the current state of the
p2996 branch of the https://github.com/bloomberg/clang-p2996.git repository. It is possible that the build could fail. Furthermore, you may need to refresh the build from time to time. We provide a script which allows you to delete the docker image you have built (bash ./p2996/remove_docker.sh) so you can start anew.
- While inside the docker shell, configure the build system with cmake:
CXX=clang++ cmake -B buildreflect -D SIMDJSON_STATIC_REFLECTION=ON -DSIMDJSON_DEVELOPER_MODE=ON
This only needs to be done once.
- Build the code...
cmake --build buildreflect --target benchmark_serialization_citm_catalog benchmark_serialization_twitter
- Run the tests...
ctest --test-dir buildreflect --output-on-failure
- Run the benchmarks.
./buildreflect/benchmark/static_reflect/citm_catalog_benchmark/benchmark_serialization_citm_catalog
./buildreflect/benchmark/static_reflect/twitter_benchmark/benchmark_serialization_twitter
You can modify the source code with your favorite editor and run again steps 5 (Build the code) and 6 (Run the tests) and 7 (Run the benchmark). Importantly, you should remain in the docker shell.
You can create a new docker shell at any time by running step 3 (bash script).