- rapidjson_citm_catalog_data.h: Remove references to non-existent
fields (areaNames, topicNames, venueNames, etc.) and properly handle
std::optional fields with value checks before dereferencing
- benchmark_parsing_citm.cpp: Fix CITMPrice field names from abbreviated
form (audience, seat) to actual struct names (audienceSubCategoryId,
seatCategoryId) in nlohmann, rapidjson, and yyjson parsing code
Previously, the FFI overhead for Rust/serde was estimated at <1%. This
commit adds proper measurement infrastructure and reveals the actual
overhead is ~10%:
- CString conversion: ~5.4% (memory copy of 82KB string)
- FFI call mechanics: ~5.5%
Changes:
- Add measure_twitter_ffi_overhead() and measure_citm_ffi_overhead()
functions in lib.rs that measure pure serde vs FFI overhead
- Add FfiOverheadResult struct to serde_benchmark.h
- Add measure_rust_ffi_overhead() in benchmark_serialization_twitter.cpp
- Update benchmark_writeup.md with measured results and corrected claims
Key findings:
- Pure Rust serde_json: ~1,930 MB/s
- With FFI overhead: ~1,730 MB/s (as reported)
- simdjson vs pure Rust/serde: ~1.5x faster (not 2x)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
simdjson was designed for buffer reuse, so showing both variants:
- Buffer reuse: realistic production performance (~12-13% faster)
- Fresh allocation: fair comparison with other libraries
- Update yyjson serialization to match C++ CitmCatalog struct (only events + performances)
- Add yyjson include and benchmark function to CITM serialization
- Link yyjson to CITM serialization benchmark in CMakeLists.txt
- Output size matches other libraries: 496,682 bytes
The CITM benchmark was using simdjson::to_json_string() while Twitter
used simdjson::builder::to_json(). This caused misleading results
(532 MB/s vs 2780 MB/s). Now both use the same API for consistency.
Memory Allocation Fairness:
- Add "fair" benchmark variants that allocate fresh buffers each iteration
- Add "reuse" variants showing optimized API potential with buffer reuse
- Fair variants match allocation behavior of competing libraries
Rust/serde CITM Fix:
- Rewrite Rust CitmCatalog struct to match C++ exactly
- Now only serializes events + performances (matching C++ behavior)
- Output size now matches: 496,682 bytes for all libraries
Documentation:
- Add comprehensive benchmark_fairness.md report
- Document reflect-cpp output size discrepancy (optional field handling)
- Document Rust FFI overhead (negligible for this data size)
- Include reproducibility instructions
Results after fixes:
- Twitter: All libraries produce identical 81,927 byte output
- CITM: simdjson/nlohmann/Rust all produce 496,682 bytes
- CITM reflect-cpp: 476,270 bytes (documented optional handling difference)
- Fix typo: SIMDJSON_USER_RUST -> SIMDJSON_USE_RUST in CMakeLists.txt
- Update unified_benchmark.sh to use correct SIMDJSON_USE_RUST flag
- Add yyjson to Twitter serialization benchmark
- Fix CITM catalog field names to match JSON keys (audienceSubCategoryId, seatCategoryId)
- Fix CMake target names for yyjson and rapidjson in CITM benchmark
If you provide a custom serializer for range types it is currently never used due to the requires clause for string_builder::append with ranges is overly broad
Francisco Geiman Thiesen