admin/simdjson-simdjson
1
0
Fork 0
mirror of https://github.com/simdjson/simdjson synced 2026-06-08 17:27:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Adding tests for issue 1745. (#1746)

Browse Source 
* Adding tests for issue 1745.

* Tweaking the documentation so that it is clearer.
This commit is contained in:
Daniel Lemire
2021-11-03 10:12:18 -04:00
committed by GitHub
parent 17f3148ac7
commit 1b01969bda
5 changed files with 135 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/basics.md
+25 -15
View File
@@ -200,7 +200,8 @@ allocations during parsing when using simdjson. [See our performance notes for d
C++11 Support and string_view
-------------
The simdjson library builds on compilers supporting the [C++11 standard](https://en.wikipedia.org/wiki/C%2B%2B11). It is also a strict requirement: we have no plan to support older C++ compilers.
The simdjson library builds on compilers supporting the [C++11 standard](https://en.wikipedia.org/wiki/C%2B%2B11).
It is also a strict requirement: we have no plan to support older C++ compilers.
We represent parsed Unicode (UTF-8) strings in simdjson using the `std::string_view` class. It avoids
the need to copy the data, as would be necessary with the `std::string` class. It also
@@ -218,7 +219,6 @@ is often best viewed as a temporary string value that is tied to the document yo
At the cost of some memory allocation, you may convert your `std::string_view` instances for long-term storage into `std::string` instances:
`std::string mycopy(view)` (C++17) or `std::string mycopy(view.begin(), view.end())` (prior to C++17).
The `std::string_view` class has become standard as part of C++17 but it is not always available
on compilers which only supports C++11. When we detect that `string_view` is natively
available, we define the macro `SIMDJSON_HAS_STRING_VIEW`.
@@ -236,6 +236,8 @@ transcode the UTF-8 strings produced by the simdjson library to other formats. S
Using the Parsed JSON
---------------------
We recommend that you first compile and run your code in Debug mode (with `NDEBUG`
undefined). When you do so, the simdjson library runs additional sanity tests on
your code to help ensure that you are using the library in a safe manner. Once
@@ -251,8 +253,10 @@ native types (`double`, `uint64_t`, `int64_t`, `bool`), we also access
Unicode (UTF-8) strings (`std::string_view`), objects (`simdjson::ondemand::object`)
and arrays (`simdjson::ondemand::array`).
We also have a generic type (`simdjson::ondemand::value`) which represents a potential
array or object, or scalar type (`double`, `uint64_t`, `int64_t`, `bool`, `null`, string) inside an array or an object. Both generic types (`simdjson::ondemand::document` and `simdjson::ondemand::value`) have a `type()` method returning
a `json_type` value describing the value (`json_type::array`, `json_type::object`, `json_type::number`, `json_type::string`, `json_type::boolean`, `json_type::null`).
array or object, or scalar type (`double`, `uint64_t`, `int64_t`, `bool`, `null`, string) inside
an array or an object. Both generic types (`simdjson::ondemand::document` and
`simdjson::ondemand::value`) have a `type()` method returning a `json_type` value describing the
value (`json_type::array`, `json_type::object`, `json_type::number`, `json_type::string`, `json_type::boolean`, `json_type::null`).
Advanced users who need to determine the number types (integer or float) dynamically,
should review our section [dynamic number types](#dynamic-number-types). Indeed,
@@ -260,10 +264,10 @@ we have an additional `ondemand::number` type which may represent either integer
or floating-point values, depending on how the numbers are formatted.
floating-point values followed by an integer.
While you are accessing the document, the `document` instance should remain in scope:
it is your "iterator" which keeps track of where you are in the JSON document.
By design, there is one and only one `document` instance per JSON document.
We invite you to keep the following rules in mind:
1. While you are accessing the document, the `document` instance should remain in scope: it is your "iterator" which keeps track of where you are in the JSON document. By design, there is one and only one `document` instance per JSON document.
2. Because On Demand is really just an iterator, you must fully consume the current object or array before accessing a sibling object or array.
3. Values can only be consumed once, you should get the values and store them if you plan to need them multiple times. You are expected to access the keys of an object just once. You are expected to go through the values of an array just once.
The following specific instructions indicate how to use the JSON when exceptions are enabled, but simdjson has full, idiomatic
support for users who avoid exceptions. See [the simdjson error handling documentation](basics.md#error-handling) for more.
@@ -273,18 +277,24 @@ support for users who avoid exceptions. See [the simdjson error handling documen
However, it is not fully validated. On Demand only fully validates the values you use and the
structure leading to it.
* **Extracting Values:** You can cast a JSON element to a native type:
`double(element)` or `double x = json_element`. This works for `std::string_view`, double, uint64_t, int64_t, bool,
ondemand::object and ondemand::array. At this point, the number, string or boolean will be parsed,
or the initial `[` or `{` will be verified. An exception is thrown if the cast is not possible.
`double(element)`. This works for `std::string_view`, double, uint64_t, int64_t, bool,
ondemand::object and ondemand::array. We also have explicit methods such as `get_string()`, `get_double()`,
`get_uint64()`, `get_int64()`, `get_bool()`, `get_object()` and `get_array()`. After a cast or an explicit method,
the number, string or boolean will be parsed, or the initial `[` or `{` will be verified. An exception is thrown if
the cast is not possible.
> IMPORTANT NOTE: values can only be parsed once. Since documents are *iterators*, once you have
> parsed a value (such as by casting to double), you cannot get at it again.
> parsed a value (such as by casting to double), you cannot get at it again. It is an error to call
> `get_string()` twice on an object (or to cast an object twice to `std::string_view`).
* **Field Access:** To get the value of the "foo" field in an object, use `object["foo"]`. This will
scan through the object looking for the field with the matching string, doing a character-by-character
comparison. For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do
not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. Furthermore, you can only consume one field at a time, on the same object. Thus
if you have retrieved `content["bids"].get_array()` and you later call `content["asks"].get_array()`, then the first array should no longer be accessed: it would be unsafe to do so.
You can detect such mistakes by first compiling and running the code in Debug mode: an OUT_OF_ORDER_ITERATION error is generated.
not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. If you consume an
object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]`, your code
is in error. Furthermore, you can only consume one field at a time, on the same object. Thus
if you have retrieved `content["bids"].get_array()` and you later call `content["asks"].get_array()`, then the
first array should no longer be accessed: it would be unsafe to do so. You can detect such mistakes by first
compiling and running the code in Debug mode: an OUT_OF_ORDER_ITERATION error is generated.
> NOTE: JSON allows you to escape characters in keys. E.g., the key `"date"` may be written as
> `"\u0064\u0061\u0074\u0065"`. By default, simdjson does *not* unescape keys when matching by default.
include/simdjson/generic/ondemand/document.h
+10
View File
@@ -94,6 +94,8 @@ public:
*
* The string is guaranteed to be valid UTF-8.
*
* Important: Calling get_string() twice on the same document is an error.
*
* @returns An UTF-8 string. The string is stored in the parser and will be invalidated the next
* time it parses a document or when it is destroyed.
* @returns INCORRECT_TYPE if the JSON value is not a string.
@@ -311,6 +313,10 @@ public:
* the code in Debug mode (or with the macro `SIMDJSON_DEVELOPMENT_CHECKS` set to 1): an
* OUT_OF_ORDER_ITERATION error is generated.
*
* You are expected to access keys only once. You should access the value corresponding to
* a key a single time. Doing object["mykey"].to_string()and then again object["mykey"].to_string()
* is an error.
*
* @param key The key to look up.
* @returns The value of the field, or NO_SUCH_FIELD if the field is not in the object.
*/
@@ -341,6 +347,10 @@ public:
* the code in Debug mode (or with the macro `SIMDJSON_DEVELOPMENT_CHECKS` set to 1): an
* OUT_OF_ORDER_ITERATION error is generated.
*
* You are expected to access keys only once. You should access the value corresponding to a key
* a single time. Doing object["mykey"].to_string() and then again object["mykey"].to_string()
* is an error.
*
* @param key The key to look up.
* @returns The value of the field, or NO_SUCH_FIELD if the field is not in the object.
*/
include/simdjson/generic/ondemand/object.h
+7
View File
@@ -44,6 +44,10 @@ public:
* the code in Debug mode (or with the macro `SIMDJSON_DEVELOPMENT_CHECKS` set to 1): an
* OUT_OF_ORDER_ITERATION error is generated.
*
* You are expected to access keys only once. You should access the value corresponding to a
* key a single time. Doing object["mykey"].to_string() and then again object["mykey"].to_string()
* is an error.
*
* @param key The key to look up.
* @returns The value of the field, or NO_SUCH_FIELD if the field is not in the object.
*/
@@ -77,6 +81,9 @@ public:
* the code in Debug mode (or with the macro `SIMDJSON_DEVELOPMENT_CHECKS` set to 1): an
* OUT_OF_ORDER_ITERATION error is generated.
*
* You are expected to access keys only once. You should access the value corresponding to a key
* a single time. Doing object["mykey"].to_string() and then again object["mykey"].to_string() is an error.
*
* @param key The key to look up.
* @returns The value of the field, or NO_SUCH_FIELD if the field is not in the object.
*/
include/simdjson/generic/ondemand/value.h
+3
View File
@@ -121,6 +121,9 @@ public:
*
* Equivalent to get<std::string_view>().
*
* Important: a value should be consumed once. Calling get_string() twice on the same value
* is an error.
*
* @returns An UTF-8 string. The string is stored in the parser and will be invalidated the next
* time it parses a document or when it is destroyed.
* @returns INCORRECT_TYPE if the JSON value is not a string.
tests/ondemand/ondemand_object_tests.cpp
+90
View File
@@ -6,6 +6,53 @@ using namespace simdjson;
namespace object_tests {
using namespace std;
using simdjson::ondemand::json_type;
bool issue1745() {
TEST_START();
auto json = R"({
"keys": [
{
"kid": "bbd2ac7c4c5eb8adc8eeffbc8f5a2dd6cf7545e4",
"e": "AQAB",
"alg": "RS256",
"use": "sig",
"n": "y930dtGTeMG52IPsKmMuEpPHLaxuYQlduZd6BqFVjc2-UFZR8fNqtnYzAjbXWJD_Tqxgdlj_MW4vogvX4sHwVpZONvdyeGoIyDQtis6iuGQhQamV85F_JbrEUnEw3QCO87Liz5UXG6BK2HRyPhDfMex1_tO0ROmySLFdCTS17D0wah71Ibpi0gI8LUi6kzVRjYDIC1oE-iK3Y9s88Bi4ZGYJxXAbnNwbwVkGOKCXja9k0jjBGRxZD-4KDuf493lFOOEGSLDA2Qp9rDqrURP12XYgvf_zJx_kSDipnr0gL6Vz2n3H4-XN4tA45zuzRkHoE7-XexPq-tv7kQ8pSjY2uQ",
"kty": "RSA"
},
{
"kty": "RSA",
"n": "zMHxWuxztMKXdBhv3rImlUvW_yp6nO03cVXPyA0Vyq0-M7LfOJJIF-OdNoRGdsFPHVKCFoo6qGhR8rBCmMxA4fM-Ubk5qKuUqCN9eP3yZJq8Cw9tUrt_qh7uW-qfMr0upcyeSHhC_zW1lTGs5sowDorKN_jQ1Sfh9hfBxfc8T7dQAAgEqqMcE3u-2J701jyhJz0pvurCfziiB3buY6SGREhBQwNwpnQjt_lE2U4km8FS0woPzt0ccE3zsGL2qM-LWZbOm9aXquSnqNJLt3tGVvShnev-GiJ1XfQ3EWm0f4w0TX9fTOkxstl0vo_vW_FjGQ0D1pXSjqb7n-hAdXwc9w",
"alg": "RS256",
"kid": "85828c59284a69b54b27483e487c3bd46cd2a2b3",
"e": "AQAB",
"use": "sig"
}
]
}
})"_padded;
ondemand::parser parser;
ondemand::document doc;
ASSERT_SUCCESS(parser.iterate(json).get(doc));
for (auto items_instance : doc["keys"]) {
ondemand::object items;
ASSERT_SUCCESS(items_instance.get_object().get(items));
std::string_view kty_value;
ASSERT_SUCCESS(items["kty"].get_string().get(kty_value));
std::string_view use_value;
ASSERT_SUCCESS(items["use"].get_string().get(use_value));
if (kty_value != "RSA" && use_value != "sig") { continue; }
std::string_view str;
ASSERT_SUCCESS(items["alg"].get_string().get(str));
ASSERT_EQUAL(str, "RS256");
ASSERT_EQUAL(use_value, "sig");
ASSERT_EQUAL(kty_value, "RSA");
ASSERT_SUCCESS(items["n"].get_string().get(str));
ASSERT_EQUAL(str.size(), 342);
ASSERT_SUCCESS(items["e"].get_string().get(str));
ASSERT_EQUAL(str, "AQAB");
}
TEST_SUCCEED();
}
bool issue1723() {
TEST_START();
@@ -178,6 +225,47 @@ namespace object_tests {
}
#if SIMDJSON_EXCEPTIONS
bool issue1745_with_exceptions() {
TEST_START();
auto json = R"({
"keys": [
{
"kid": "bbd2ac7c4c5eb8adc8eeffbc8f5a2dd6cf7545e4",
"e": "AQAB",
"alg": "RS256",
"use": "sig",
"n": "y930dtGTeMG52IPsKmMuEpPHLaxuYQlduZd6BqFVjc2-UFZR8fNqtnYzAjbXWJD_Tqxgdlj_MW4vogvX4sHwVpZONvdyeGoIyDQtis6iuGQhQamV85F_JbrEUnEw3QCO87Liz5UXG6BK2HRyPhDfMex1_tO0ROmySLFdCTS17D0wah71Ibpi0gI8LUi6kzVRjYDIC1oE-iK3Y9s88Bi4ZGYJxXAbnNwbwVkGOKCXja9k0jjBGRxZD-4KDuf493lFOOEGSLDA2Qp9rDqrURP12XYgvf_zJx_kSDipnr0gL6Vz2n3H4-XN4tA45zuzRkHoE7-XexPq-tv7kQ8pSjY2uQ",
"kty": "RSA"
},
{
"kty": "RSA",
"n": "zMHxWuxztMKXdBhv3rImlUvW_yp6nO03cVXPyA0Vyq0-M7LfOJJIF-OdNoRGdsFPHVKCFoo6qGhR8rBCmMxA4fM-Ubk5qKuUqCN9eP3yZJq8Cw9tUrt_qh7uW-qfMr0upcyeSHhC_zW1lTGs5sowDorKN_jQ1Sfh9hfBxfc8T7dQAAgEqqMcE3u-2J701jyhJz0pvurCfziiB3buY6SGREhBQwNwpnQjt_lE2U4km8FS0woPzt0ccE3zsGL2qM-LWZbOm9aXquSnqNJLt3tGVvShnev-GiJ1XfQ3EWm0f4w0TX9fTOkxstl0vo_vW_FjGQ0D1pXSjqb7n-hAdXwc9w",
"alg": "RS256",
"kid": "85828c59284a69b54b27483e487c3bd46cd2a2b3",
"e": "AQAB",
"use": "sig"
}
]
}
})"_padded;
ondemand::parser parser;
ondemand::document doc = parser.iterate(json);
for (auto items : doc["keys"]) {
std::string_view kty_value = items["kty"].get_string();
std::string_view use_value = items["use"].get_string();
if (kty_value != "RSA" && use_value != "sig") { continue; }
std::string_view str = items["alg"].get_string();
if(str != "RS256") { return false; }
if(use_value != "sig") { return false; }
if(kty_value != "RSA") { return false; }
str = items["n"].get_string();
if(str.size() != 342) { return false; }
str = items["e"].get_string();
if(str != "AQAB") { return false; }
}
TEST_SUCCEED();
}
bool issue1723_except() {
TEST_START();
@@ -988,6 +1076,7 @@ namespace object_tests {
bool run() {
return
issue1745() &&
issue1742() &&
issue1742_value() &&
issue1723() &&
@@ -997,6 +1086,7 @@ namespace object_tests {
missing_keys() &&
missing_keys_for_empty_top_level_object() &&
#if SIMDJSON_EXCEPTIONS
issue1745_with_exceptions() &&
issue1723_except() &&
fixed_broken_issue_1521() &&
issue_1521() &&