admin/simdjson-simdjson
1
0
Fork 0
mirror of https://github.com/simdjson/simdjson synced 2026-06-08 17:27:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

adding a C++17 padded_input for convenience. (#2667)

Browse Source 
* adding a C++17 padded_input for convenience.

* adding header

* tuning.

* being explicit

* reworking the docu

* avoid windows.h

* Update include/simdjson/padded_string_view-inl.h

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/ondemand/ondemand_padded_input.cpp

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/ondemand/ondemand_readme_examples.cpp

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update doc/performance.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* minor update

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Daniel Lemire
2026-04-09 14:07:44 -04:00
committed by GitHub
parent bc48e72070
commit 85cadf4074
10 changed files with 531 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/basics.md
+90 -68
View File
@@ -161,54 +161,77 @@ The basics: loading and parsing JSON documents
----------------------------------------------
The simdjson library allows you to navigate and validate JSON documents ([RFC 8259](https://www.tbray.org/ongoing/When/201x/2017/12/14/rfc8259.html)).
As required by the standard, your JSON document should be in a Unicode (UTF-8) string. The whole
string, from the beginning to the end, needs to be valid: we do not attempt to tolerate bad
inputs before or after a document.
Your JSON document should be a valid Unicode (UTF-8) string.
For efficiency reasons, simdjson requires a string with a few bytes (`simdjson::SIMDJSON_PADDING`)
at the end, these bytes may be read but their content does not affect the parsing. In practice,
it means that the JSON inputs should be stored in a memory region with `simdjson::SIMDJSON_PADDING`
extra bytes at the end. You do not have to set these bytes to specific values though you may
want to if you want to avoid runtime warnings with some sanitizers. We expect the user
of the library to load the data (from disk or from the network) into a padded buffer. To make
this easy, we provide the `padded_string::load` function which loads files from disk in a padded buffer.
[You can similarly fetch a file from a URL to a padded string](https://github.com/simdjson/curltostring) using our `simdjson::padded_string_builder`. Advanced users may want to read the section Free Padding in [our performance notes](performance.md).
The simdjson library offers a tree-like [API](https://en.wikipedia.org/wiki/API), which you can
access by creating a `ondemand::parser` and calling the `iterate()` method. The iterate method
quickly indexes the input string and may detect some errors. The following example illustrates
how to get started with an input JSON file (`"twitter.json"`):
To parse JSON, create a `ondemand::parser` and call its `iterate()` method on a padded input.
The simplest way to load a JSON file is with `padded_string::load`:
```cpp
ondemand::parser parser;
auto json = padded_string::load("twitter.json"); // load JSON file 'twitter.json'.
ondemand::document doc = parser.iterate(json); // position a pointer at the beginning of the JSON data
auto json = padded_string::load("twitter.json");
ondemand::document doc = parser.iterate(json);
```
(Windows users compiling with C++17 or better may use `wchar_t` strings to support non-ASCII
filenames: `padded_string::load(L"twitter.json")`.)
For inline JSON strings, use the `_padded` suffix:
If you prefer not to create your own `ondemand::parser` instance, you can access
a thread-local version by calling `ondemand::parser.get_parser()`.
```cpp
ondemand::parser parser;
auto json = "[1,2,3]"_padded;
ondemand::document doc = parser.iterate(json);
```
If you are compiling with C++17 or better, you can use `simdjson::padded_input`
which accepts any string-like input and handles padding automatically:
```cpp
ondemand::parser parser;
std::string_view json = "[1,2,3]";
simdjson::padded_input input(json);
ondemand::document doc = parser.iterate(input);
// Also works with std::string, considering reserved capacity
std::string json_str = "[1,2,3]";
json_str.reserve(100); // Reserve extra space
simdjson::padded_input input2(json_str); // May avoid copying
ondemand::document doc2 = parser.iterate(input2);
```
The simdjson library also accepts `std::string` instances directly---if the provided
reference is non-const, it will allocate padding as needed:
```cpp
ondemand::parser parser;
std::string json = "[1,2,3]";
ondemand::document doc = parser.iterate(json);
```
By default, the simdjson library throws exceptions (`simdjson_error`) on errors. We omit `try`-`catch` clauses from our illustrating examples: if you omit `try`-`catch` in your code, an uncaught exception will halt your program. It is also possible to use simdjson without generating exceptions, and you may even build the library without exception support at all. See [Error handling](#error-handling) for details.
### Advanced input options
This section covers additional ways to provide JSON input to simdjson, including
options for fine-grained control over padding and memory.
**Thread-local parser.** If you prefer not to create your own `ondemand::parser` instance, you can access
a thread-local version by calling `ondemand::parser.get_parser()`:
```cpp
ondemand::document doc = ondemand::parser.get_parser().iterate(json);
```
However, you should be careful because a parser instance can only be used for one
document at a time, thus it is only applicable when you are only parsing one
A parser instance can only be used for one document at a time, so
the thread-local parser is only applicable when you parse one
document per thread at any one time.
You can also create a padded string---and call `iterate()`:
**`padded_input` details (C++17+).** The actual padding only occurs when the JSON string ends near the boundary of a memory page, which is
uncommon. Using a `simdjson::padded_input` is safe although sanitizers and tools like valgrind
might report illegal reads (which are safe in our case because they remain in the mapped page). You should avoid `simdjson::padded_input`
on systems without a page size of at least 4096: virtually all systems qualify except for
some niche embedded systems running custom operating systems. Standard Linux, Windows, macOS, Android, iOS, etc., are all fine. Note that, most times, a `simdjson::padded_input` instance will not copy the data and will only act
as a view (it does not own the memory).
```cpp
ondemand::parser parser;
auto json = "[1,2,3]"_padded; // The _padded suffix creates a simdjson::padded_string instance
ondemand::document doc = parser.iterate(json); // parse a string
```
If you have a buffer of your own with enough padding already (SIMDJSON_PADDING extra bytes allocated), you can use `padded_string_view` to pass it in:
**User-managed buffers.** If you have a buffer of your own with enough padding already (`SIMDJSON_PADDING` extra bytes allocated), you can use `padded_string_view` to pass it in:
```cpp
ondemand::parser parser;
@@ -217,73 +240,72 @@ strcpy(json, "[1]");
ondemand::document doc = parser.iterate(json, strlen(json), sizeof(json));
```
The simdjson library will also accept `std::string` instances. If the provided
reference is non-const, it will allocate padding as needed.
You can copy your data directly on a `simdjson::padded_string` as follows:
**Copying into a `padded_string`.** You can copy your data directly into a `simdjson::padded_string`:
```cpp
const char * data = "my data"; // 7 bytes
simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer
```
Or as follows...
Or from a `std::string`:
```cpp
std::string data = "my data";
simdjson::padded_string my_padded_data(data); // copies to a padded buffer
```
You can then parse the JSON data from the `simdjson::padded_string` instance:
```cpp
ondemand::document doc = parser.iterate(my_padded_data);
```
Whenever you pass an `std::string` reference to `parser::iterate`,
the parser will access the bytes beyond the end of
**`std::string` and sanitizer warnings.** Whenever you pass an `std::string` reference to `parser::iterate`,
the parser may access bytes beyond the end of
the string but before the end of the allocated memory (`std::string::capacity()`).
If you are using a sanitizer that checks for reading uninitialized bytes or `std::string`'s
container-overflow checks, you may encounter sanitizer warnings.
You can safely ignore these warnings. Or you can call `simdjson::pad(std::string&)` to pad the
string with `SIMDJSON_PADDING` spaces: this function returns a `simdjson::padding_string_view` which can be be passed to the parser's iterator function:
Sanitizers that check for reading uninitialized bytes may produce warnings.
You can safely ignore these warnings, or call `simdjson::pad(std::string&)` to pad the
string explicitly:
```cpp
std::string json = "[1]";
ondemand::document doc = parser.iterate(simdjson::pad(json));
```
We recommend against creating many `std::string` or many `std::padding_string` instances in your application to store your JSON data.
We recommend against creating many `std::string` or many `std::padded_string` instances in your application to store your JSON data.
Consider reusing the same buffers and limiting memory allocations.
By default, the simdjson library throws exceptions (`simdjson_error`) on errors. We omit `try`-`catch` clauses from our illustrating examples: if you omit `try`-`catch` in your code, an uncaught exception will halt your program. It is also possible to use simdjson without generating exceptions, and you may even build the library without exception support at all. See [Error handling](#error-handling) for details.
**Memory-file mapping (non-Windows).** You can use memory-file mapping to create a `simdjson::padded_string_view`
from a file on disk:
Some users may want to browse code along with the compiled assembly. You want to check out the following lists of examples:
```cpp
simdjson::padded_memory_map map(myfilename);
if (!map.is_valid()) { /* handle error */ }
simdjson::padded_string_view view = map.view();
ondemand::document doc = parser.iterate(view);
```
* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/98Kx9Kqjn)
* [simdjson examples with errors without exceptions](https://godbolt.org/z/PKG7GdbPo)
*Windows-specific*: Windows users who need to read files with
**Windows-specific notes.** Windows users compiling with C++17 or better may use `wchar_t` strings to support non-ASCII
filenames: `padded_string::load(L"twitter.json")`. Windows users who need to read files with
non-ANSI characters in the name should set their code page to
UTF-8 (65001). This should be the default with Windows 11 and better.
Further, they may use the AreFileApisANSI function to determine whether
the filename is interpreted using the ANSI or the system default OEM
codepage, and they may call SetFileApisToOEM accordingly.
Some users may want to browse code along with the compiled assembly:
**Advanced feature:**
On non-Windows systems, you can use memory-file mapping to create a `simdjson::padded_string_view`
from a file on disk.
* [simdjson examples with errors handled through exceptions](https://godbolt.org/z/98Kx9Kqjn)
* [simdjson examples with errors without exceptions](https://godbolt.org/z/PKG7GdbPo)
```cpp
// If the macro _WIN32 is defined, this will not work since we do not support memory-file mapping
// under Windows at this time.
simdjson::padded_memory_map map(myfilename);
if (!map.is_valid()) { /* handle error */ }
simdjson::padded_string_view view = map.view(); // view is usable while padded_memory_map is in scope
ondemand::document doc = parser.iterate(view); // parse the JSON
```
**Summary of input types:**
| Input Type / Method | Padding Requirement | How Padding is Handled | Ownership / Copying | Notes / Warnings |
|----------------------------------------------|-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|----------------------------------------------|----------------------------------------------------------------------------------|
| `padded_string::load("file.json")` | Automatic (SIMDJSON_PADDING extra bytes) | Library allocates padded buffer and loads file into it | Owned by `padded_string` | Recommended for files; safest and simplest. |
| `"...json..."_padded` literal | Automatic (built-in padding) | Creates `padded_string` with padding | Owned by `padded_string` | Convenient for small hardcoded JSON. |
| `simdjson::padded_input` (C++17+) | Automatic when needed | Adds padding **only** if the string ends near a memory page boundary. For `std::string`, considers `capacity()` | Usually a non-owning view (no copy most times) | Safe on standard OS (page size ≥ 4096). May trigger sanitizer/valgrind warnings (harmless). Avoid on niche embedded systems. |
| User buffer with explicit padding | Must have at least `SIMDJSON_PADDING` extra allocated bytes after JSON content | Pass via `iterate(ptr, json_length, total_allocated_size)` or `padded_string_view` | User-owned (no copy) | Use `char buf[len + SIMDJSON_PADDING]`. Library reads (but never writes) into padding. |
| `std::string` (non-const) | Library checks `capacity()` | If insufficient, library may allocate a padded copy | May copy (depends on capacity) | Can trigger sanitizer warnings on uninitialized bytes. Use `simdjson::pad(json)` to avoid. |
| `simdjson::pad(std::string&)` | Adds padding if needed | Returns `padded_string_view` pointing to the (possibly resized) string | References original string | Recommended to silence sanitizers when using `std::string`. |
| `padded_string(data, length)` or `padded_string(std::string)` | Automatic (copies into padded buffer) | Explicit copy into owned padded buffer | Owned by `padded_string` | Safe when you want full ownership and padding guaranteed. |
| `padded_string_view` (manual) | User guarantees `SIMDJSON_PADDING` extra bytes after the viewed length | User provides pointer + length + capacity | Non-owning view | Low-level; requires careful buffer management. |
| Memory-mapped file (`padded_memory_map`) | Automatic via mapping (non-Windows only) | Creates view with sufficient padding | Non-owning (tied to map lifetime) | Advanced; efficient for large files on Linux/macOS/etc. |
Documents are iterators
doc/dom.md
+13
View File
@@ -882,6 +882,19 @@ simdjson::dom::element element = parser.parse(padded_json_copy.get(), json_len,
Setting the `realloc_if_needed` parameter `false` in this manner may lead to better performance since copies are avoided, but it requires that the user takes more responsibilities: the simdjson library cannot verify that the input buffer was padded with SIMDJSON_PADDING extra bytes.
If you are compiling your project with C++17 or better, you can use a `simdjson::padded_input`:
```cpp
simdjson::dom::parser parser;
std::string_view json = "[1,2,3]";
simdjson::padded_input input(json); // Automatically pads if needed
simdjson::dom::element element = parser.parse(input);
```
The actual padding only occurs if the JSON string ends near the boundary of a memory page, which is uncommon. Using a `simdjson::padded_input` is safe although sanitizers and tools like valgrind might report illegal reads (which are safe in our case because they remain in the mapped page). You should avoid `simdjson::padded_input` on systems without a page size of at least 4096: virtually all systems qualify except for some niche embedded systems running custom operating systems. Standard Linux, Windows, macOS, Android, iOS, etc., are all fine. Note that, most times, an `simdjson::padded_input` instance will not copy the data and will only act
as a view (it does not own the memory).
Performance Tips
---------------------
doc/performance.md
+20 -4
View File
@@ -204,9 +204,25 @@ but can be significantly larger. E.g., Apple systems favour pages spanning 16 ki
In effect, it means that you can almost always read a few bytes beyond your current buffer---without
allocating extra memory. However, tools such as valgrind or memory sanitizers will flag such behavior as unsafe.
Nevertheless, you can still make sure of this capability in your code if you are an expert
programmer and you are willing to silence sanitizer warnings. The following code provides
a portable example.
You can still make sure of this capability in your code if you are an expert
programmer and you are willing to silence sanitizer warnings.
If you are building simdjson with C++17 or better, you can use `simdjson::padded_input`.
The `padded_input` struct automatically manages padding for you. It can be constructed from a `std::string_view` or a C-style string with length. If the input already has sufficient padding (up to the end of the memory page), it creates a view without copying. Otherwise, it copies the data into a `padded_string` with proper padding.
Example usage:
```cpp
std::string_view json = get_json_data();
simdjson::padded_input input(json); // Automatically pads if needed
auto result = parser.parse(input);
```
This simplifies padding management compared to manually checking and allocating.
More generally, the following code provides a portable example.
The conditional compilation checks for the `_MSC_VER` macro (indicating Microsoft Visual Studio)
@@ -250,7 +266,7 @@ long page_size() {
}
// Returns true if the buffer + len + simdjson::SIMDJSON_PADDING crosses the
// page boundary.
// page boundary. Assumes len != 0.
bool need_allocation(const char *buf, size_t len) {
return ((reinterpret_cast<uintptr_t>(buf + len - 1) % page_size())
+ simdjson::SIMDJSON_PADDING >= static_cast<uintptr_t>(page_size()));
include/simdjson/padded_string_view-inl.h
+85
View File
@@ -6,6 +6,15 @@
#include <cstring> /* memcmp */
// for page size computation.
#if defined(__unix__) || defined(__APPLE__) || defined(__linux__)
#include <unistd.h>
#if defined(__APPLE__)
#include <sys/sysctl.h>
#endif
#endif
namespace simdjson {
inline padded_string_view::padded_string_view(const char* s, size_t len, size_t capacity) noexcept
@@ -95,7 +104,83 @@ inline padded_string_view pad_with_reserve(std::string& s) noexcept {
return padded_string_view(s.data(), s.size(), s.capacity());
}
inline uint32_t get_page_size() noexcept {
#if defined(_WINDOWS_) // if and only if someone loaded Windows.h, we can get the page size from there.
// Otherwise, we assume 4096.
static const uint32_t cached = []() -> uint32_t {
SYSTEM_INFO si;
GetSystemInfo(&si);
return static_cast<std::uint32_t>(si.dwPageSize);
}();
return cached;
#elif defined(__unix__) || defined(__APPLE__) || defined(__linux__)
static const uint32_t cached = []() -> uint32_t {
long page_size = sysconf(_SC_PAGESIZE);
if (page_size > 0) {
return static_cast<uint32_t>(page_size);
}
return 4096; // fallback
}();
return cached;
#else
return 4096; // fallback
#endif
}
#if SIMDJSON_CPLUSPLUS17
inline padded_input::padded_input(std::string_view sv)
: storage(simdjson::padded_string_view{}) {
if (needs_allocation(sv.data(), sv.size())) {
storage = simdjson::padded_string(sv);
} else {
storage = simdjson::padded_string_view(
sv.data(), sv.size(), sv.size() + simdjson::SIMDJSON_PADDING);
}
}
inline padded_input::padded_input(const char *data, size_t length)
: storage(simdjson::padded_string_view{}) {
if (needs_allocation(data, length)) {
storage = simdjson::padded_string(data, length);
} else {
storage = simdjson::padded_string_view(
data, length, length + simdjson::SIMDJSON_PADDING);
}
}
inline padded_input::padded_input(const std::string &s)
: storage(simdjson::padded_string_view{}) {
const size_t len = s.size();
const size_t cap = s.capacity();
// Here we have the string content from data() to data() + size(),
// but the memory is accessible from data() to data() + capacity().
const size_t needed_padding = (cap - len) < simdjson::SIMDJSON_PADDING
? simdjson::SIMDJSON_PADDING - (cap - len) : 0;
if (needed_padding > 0 && needs_allocation(s.data(), cap, needed_padding)) {
storage = simdjson::padded_string(s);
} else {
storage = simdjson::padded_string_view(
s.data(), len, len + simdjson::SIMDJSON_PADDING);
}
}
inline bool padded_input::is_view() const noexcept {
return std::holds_alternative<simdjson::padded_string_view>(storage);
}
inline padded_input::operator simdjson::padded_string_view() const noexcept {
return std::visit([](const auto& p) -> simdjson::padded_string_view {
return p;
}, storage);
}
inline bool padded_input::needs_allocation(const char* buf, size_t len, size_t padding) noexcept {
if(len == 0) { return false; }
const auto page_size = get_page_size();
return ((reinterpret_cast<uintptr_t>(buf + len - 1) % page_size)
+ padding >= static_cast<uintptr_t>(page_size));
}
#endif // SIMDJSON_CPLUSPLUS17
} // namespace simdjson
include/simdjson/padded_string_view.h
+66
View File
@@ -9,6 +9,9 @@
#include <memory>
#include <string>
#include <ostream>
#if SIMDJSON_CPLUSPLUS17
#include <variant>
#endif
namespace simdjson {
@@ -73,6 +76,68 @@ public:
}; // padded_string_view
/**
* Get the system's memory page size. By default, we return
* 4096 bytes, which is the most common page size. On systems
* where the page size is not a multiple of 4096 bytes, and not
* a unix-like system, nor Windows, this function may return an
* incorrect value.
*
* @return The page size in bytes.
*/
inline uint32_t get_page_size() noexcept;
#if SIMDJSON_CPLUSPLUS17
/**
* A padded_input is a wrapper around either a padded_string_view or a padded_string.
* It will automatically pad a string_view if it does not have sufficient padding
* up to the end of the memory page. Note that a requirement for this method to
* make sense is to be on a system with a page size of at least 4096 (which is
* universal except on some embedded systems).
*/
struct padded_input {
/**
* Construct a padded_input from a string_view. If the string_view does not have sufficient padding,
* the data will be copied into a padded_string and the padded_string_view will point to the
* padded_string's data. Otherwise, the padded_string_view will point to the original string_view's data.
*/
inline explicit padded_input(std::string_view sv);
/**
* Construct a padded_input from a C-style string (length specified). If the string does not have sufficient padding,
* the data will be copied into a padded_string and the padded_string_view will point to the
* padded_string's data. Otherwise, the padded_string_view will point to the original string's data.
*/
inline explicit padded_input(const char *data, size_t length);
/**
* Construct a padded_input from a std::string. If the string does not have sufficient padding
* (considering its capacity), the data will be copied into a padded_string and the padded_string_view
* will point to the padded_string's data. Otherwise, the padded_string_view will point to the
* original string's data.
*/
inline explicit padded_input(const std::string &s);
/**
* Check if the padded_input is a view.
*
* @return true if the padded_input is a view, false otherwise.
*/
inline bool is_view() const noexcept;
/**
* Convert the padded_input to a padded_string_view.
*
* @return The padded_string_view.
*/
inline operator simdjson::padded_string_view() const noexcept;
private:
std::variant<simdjson::padded_string_view, simdjson::padded_string> storage;
// whether we cross a page boundary and need to allocate a new padded string.
static inline bool needs_allocation(const char* buf, size_t len, size_t padding = SIMDJSON_PADDING) noexcept;
};
#endif // SIMDJSON_CPLUSPLUS17
#if SIMDJSON_EXCEPTIONS
/**
* Send padded_string instance to an output stream.
@@ -105,6 +170,7 @@ inline padded_string_view pad(std::string& s) noexcept;
* @return The padded string.
*/
inline padded_string_view pad_with_reserve(std::string& s) noexcept;
} // namespace simdjson
#endif // SIMDJSON_PADDED_STRING_VIEW_H
tests/dom/readme_examples.cpp
+30
View File
@@ -1,5 +1,6 @@
#include <iostream>
#include "simdjson.h"
#include "simdjson/padded_string_view.h"
using namespace std;
using namespace simdjson;
@@ -531,6 +532,35 @@ void simplepad() {
if(error) { exit(-1); }
}
#if SIMDJSON_CPLUSPLUS17
void simpleinputpad_dom1() {
std::string_view json = "[1,2,3]";
simdjson::padded_input input(json);
dom::parser parser;
dom::element doc;
auto error = parser.parse(input).get(doc);
if(error) { exit(-1); }
}
void simpleinputpad_dom2() {
const char *jsonpointer = R"(
{
"key": "value"
}
)";
size_t len = strlen(jsonpointer);
simdjson::padded_input input(jsonpointer, len);
dom::parser parser;
dom::element doc;
auto error = parser.parse(input).get(doc);
if(error) { exit(-1); }
std::string_view val;
error = doc["key"].get(val);
if(error) { exit(-1); }
if(val != "value") { exit(-1); }
}
#endif // SIMDJSON_CPLUSPLUS17
void jsondollar() {
dom::parser parser;
auto json = R"( { "c" :{ "foo": { "a": [ 10, 20, 30 ] }}, "d": { "foo2": { "a": [ 10, 20, 30 ] }} , "e": 120 })"_padded;
tests/ondemand/CMakeLists.txt
+1
View File
@@ -40,6 +40,7 @@ add_cpp_test(ondemand_convert_tests LABELS ondemand acceptance
add_cpp_test(ondemand_unknown_tests LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_wildcard_tests LABELS ondemand acceptance per_implementation)
if(NOT SIMDJSON_SANITIZE)
add_cpp_test(ondemand_padded_input LABELS ondemand acceptance per_implementation)
add_cpp_test(ondemand_cacheline LABELS ondemand acceptance per_implementation)
endif()
tests/ondemand/ondemand_cacheline.cpp
+1 -1
View File
@@ -20,7 +20,7 @@ long page_size() {
}
// Returns true if the buffer + len + simdjson::SIMDJSON_PADDING crosses the
// page boundary.
// page boundary. Assumes len != 0.
bool need_allocation(const char *buf, size_t len) {
return ((reinterpret_cast<uintptr_t>(buf + len - 1) % page_size())
+ simdjson::SIMDJSON_PADDING >= static_cast<uintptr_t>(page_size()));
tests/ondemand/ondemand_padded_input.cpp
+197
View File
@@ -0,0 +1,197 @@
#include "simdjson.h"
#include "simdjson/padded_string_view.h"
#include <cstdio>
#include <cstring>
#include <cstdlib>
#include <string>
#include <string_view>
#if SIMDJSON_CPLUSPLUS17
bool test_padded_input_from_string_view() {
std::string_view json = "[1,2,3]";
simdjson::padded_input input(json);
simdjson::ondemand::parser parser;
simdjson::ondemand::document doc;
auto error = parser.iterate(input).get(doc);
if (error) {
printf("FAILED: test_padded_input_from_string_view (iterate error: %s)\n",
simdjson::error_message(error));
return false;
}
size_t count = 0;
for (auto val : doc) {
int64_t v;
error = val.get_int64().get(v);
if (error) {
printf("FAILED: test_padded_input_from_string_view (get_int64 error)\n");
return false;
}
count++;
}
if (count != 3) {
printf("FAILED: test_padded_input_from_string_view (expected 3 elements, got %zu)\n", count);
return false;
}
printf("OK: test_padded_input_from_string_view\n");
return true;
}
bool test_padded_input_from_cstring() {
const char *json = R"({"key": "value"})";
size_t len = std::strlen(json);
simdjson::padded_input input(json, len);
simdjson::ondemand::parser parser;
simdjson::ondemand::document doc;
auto error = parser.iterate(input).get(doc);
if (error) {
printf("FAILED: test_padded_input_from_cstring (iterate error: %s)\n",
simdjson::error_message(error));
return false;
}
std::string_view val;
error = doc["key"].get_string().get(val);
if (error) {
printf("FAILED: test_padded_input_from_cstring (get_string error)\n");
return false;
}
if (val != "value") {
printf("FAILED: test_padded_input_from_cstring (expected 'value', got '%.*s')\n",
(int)val.size(), val.data());
return false;
}
printf("OK: test_padded_input_from_cstring\n");
return true;
}
bool test_padded_input_is_view() {
// A short string at a "safe" position should typically be a view (no copy).
// We can't guarantee this in all environments, but we can at least test
// that is_view() returns a valid boolean and parsing works either way.
std::string_view json = "[1]";
simdjson::padded_input input(json);
bool view = input.is_view();
printf(" is_view() = %s\n", view ? "true" : "false");
simdjson::ondemand::parser parser;
simdjson::ondemand::document doc;
auto error = parser.iterate(input).get(doc);
if (error) {
printf("FAILED: test_padded_input_is_view (iterate error: %s)\n",
simdjson::error_message(error));
return false;
}
printf("OK: test_padded_input_is_view\n");
return true;
}
bool test_padded_input_nested_json() {
std::string_view json = R"({"a":{"b":[1,2,{"c":true}]}})";
simdjson::padded_input input(json);
simdjson::ondemand::parser parser;
simdjson::ondemand::document doc;
auto error = parser.iterate(input).get(doc);
if (error) {
printf("FAILED: test_padded_input_nested_json (iterate error: %s)\n",
simdjson::error_message(error));
return false;
}
bool c_val;
error = doc["a"]["b"].at(2)["c"].get_bool().get(c_val);
if (error) {
printf("FAILED: test_padded_input_nested_json (get_bool error)\n");
return false;
}
if (!c_val) {
printf("FAILED: test_padded_input_nested_json (expected true)\n");
return false;
}
printf("OK: test_padded_input_nested_json\n");
return true;
}
bool test_padded_input_dom() {
std::string_view json = "[1,2,3]";
simdjson::padded_input input(json);
simdjson::dom::parser parser;
simdjson::dom::element doc;
auto error = parser.parse(input).get(doc);
if (error) {
printf("FAILED: test_padded_input_dom (parse error: %s)\n",
simdjson::error_message(error));
return false;
}
printf("OK: test_padded_input_dom\n");
return true;
}
bool test_padded_input_dom_object() {
const char *json = R"({"name":"simdjson","version":42})";
size_t len = std::strlen(json);
simdjson::padded_input input(json, len);
simdjson::dom::parser parser;
simdjson::dom::element doc;
auto error = parser.parse(input).get(doc);
if (error) {
printf("FAILED: test_padded_input_dom_object (parse error: %s)\n",
simdjson::error_message(error));
return false;
}
std::string_view name;
error = doc["name"].get(name);
if (error || name != "simdjson") {
printf("FAILED: test_padded_input_dom_object (name mismatch)\n");
return false;
}
printf("OK: test_padded_input_dom_object\n");
return true;
}
bool test_padded_input_from_std_string() {
std::string json = R"({"key": "value"})";
// Reserve extra space to test capacity handling
json.reserve(json.size() + 100);
simdjson::padded_input input(json);
simdjson::ondemand::parser parser;
simdjson::ondemand::document doc;
auto error = parser.iterate(input).get(doc);
if (error) {
printf("FAILED: test_padded_input_from_std_string (iterate error: %s)\n",
simdjson::error_message(error));
return false;
}
std::string_view key;
error = doc["key"].get(key);
if (error || key != "value") {
printf("FAILED: test_padded_input_from_std_string (key/value mismatch)\n");
return false;
}
printf("OK: test_padded_input_from_std_string\n");
return true;
}
int main() {
bool ok = true;
ok = test_padded_input_from_string_view() && ok;
ok = test_padded_input_from_cstring() && ok;
ok = test_padded_input_from_std_string() && ok;
ok = test_padded_input_is_view() && ok;
ok = test_padded_input_nested_json() && ok;
ok = test_padded_input_dom() && ok;
ok = test_padded_input_dom_object() && ok;
if (ok) {
printf("\nAll padded_input tests passed.\n");
return EXIT_SUCCESS;
} else {
printf("\nSome padded_input tests FAILED.\n");
return EXIT_FAILURE;
}
}
#else // !SIMDJSON_CPLUSPLUS17
int main() {
printf("padded_input requires C++17 or better. Skipping.\n");
return EXIT_SUCCESS;
}
#endif // SIMDJSON_CPLUSPLUS17
tests/ondemand/ondemand_readme_examples.cpp
+28
View File
@@ -1,4 +1,5 @@
#include "simdjson.h"
#include "simdjson/padded_string_view.h"
#include "test_ondemand.h"
#if __cpp_lib_optional >= 201606L
#include <optional>
@@ -49,6 +50,33 @@ bool simplepad() {
return error == SUCCESS;
}
#if SIMDJSON_CPLUSPLUS17
bool simpleinputpad1() {
std::string json = "[1]";
simdjson::padded_input padded_json(json);
ondemand::parser parser;
ondemand::document doc;
auto error = parser.iterate(padded_json).get(doc);
return error == SUCCESS;
}
bool simpleinputpad2() {
const char *jsonpointer = R"(
{
"key": "value"
}
)";
size_t len = strlen(jsonpointer);
simdjson::padded_input padded_json(jsonpointer, len);
ondemand::parser parser;
ondemand::document doc;
auto error = parser.iterate(padded_json).get(doc);
return error == SUCCESS;
}
#endif // SIMDJSON_CPLUSPLUS17
bool string1() {
const char * data = "my data"; // 7 bytes
simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer