diff --git a/doc/basics.md b/doc/basics.md index 1f57b9540..9e49b5042 100644 --- a/doc/basics.md +++ b/doc/basics.md @@ -200,7 +200,8 @@ allocations during parsing when using simdjson. [See our performance notes for d C++11 Support and string_view ------------- -The simdjson library builds on compilers supporting the [C++11 standard](https://en.wikipedia.org/wiki/C%2B%2B11). It is also a strict requirement: we have no plan to support older C++ compilers. +The simdjson library builds on compilers supporting the [C++11 standard](https://en.wikipedia.org/wiki/C%2B%2B11). +It is also a strict requirement: we have no plan to support older C++ compilers. We represent parsed Unicode (UTF-8) strings in simdjson using the `std::string_view` class. It avoids the need to copy the data, as would be necessary with the `std::string` class. It also @@ -218,7 +219,6 @@ is often best viewed as a temporary string value that is tied to the document yo At the cost of some memory allocation, you may convert your `std::string_view` instances for long-term storage into `std::string` instances: `std::string mycopy(view)` (C++17) or `std::string mycopy(view.begin(), view.end())` (prior to C++17). - The `std::string_view` class has become standard as part of C++17 but it is not always available on compilers which only supports C++11. When we detect that `string_view` is natively available, we define the macro `SIMDJSON_HAS_STRING_VIEW`. @@ -236,6 +236,8 @@ transcode the UTF-8 strings produced by the simdjson library to other formats. S Using the Parsed JSON --------------------- + + We recommend that you first compile and run your code in Debug mode (with `NDEBUG` undefined). When you do so, the simdjson library runs additional sanity tests on your code to help ensure that you are using the library in a safe manner. Once @@ -251,8 +253,10 @@ native types (`double`, `uint64_t`, `int64_t`, `bool`), we also access Unicode (UTF-8) strings (`std::string_view`), objects (`simdjson::ondemand::object`) and arrays (`simdjson::ondemand::array`). We also have a generic type (`simdjson::ondemand::value`) which represents a potential -array or object, or scalar type (`double`, `uint64_t`, `int64_t`, `bool`, `null`, string) inside an array or an object. Both generic types (`simdjson::ondemand::document` and `simdjson::ondemand::value`) have a `type()` method returning -a `json_type` value describing the value (`json_type::array`, `json_type::object`, `json_type::number`, `json_type::string`, `json_type::boolean`, `json_type::null`). +array or object, or scalar type (`double`, `uint64_t`, `int64_t`, `bool`, `null`, string) inside +an array or an object. Both generic types (`simdjson::ondemand::document` and +`simdjson::ondemand::value`) have a `type()` method returning a `json_type` value describing the +value (`json_type::array`, `json_type::object`, `json_type::number`, `json_type::string`, `json_type::boolean`, `json_type::null`). Advanced users who need to determine the number types (integer or float) dynamically, should review our section [dynamic number types](#dynamic-number-types). Indeed, @@ -260,10 +264,10 @@ we have an additional `ondemand::number` type which may represent either integer or floating-point values, depending on how the numbers are formatted. floating-point values followed by an integer. -While you are accessing the document, the `document` instance should remain in scope: -it is your "iterator" which keeps track of where you are in the JSON document. -By design, there is one and only one `document` instance per JSON document. - +We invite you to keep the following rules in mind: +1. While you are accessing the document, the `document` instance should remain in scope: it is your "iterator" which keeps track of where you are in the JSON document. By design, there is one and only one `document` instance per JSON document. +2. Because On Demand is really just an iterator, you must fully consume the current object or array before accessing a sibling object or array. +3. Values can only be consumed once, you should get the values and store them if you plan to need them multiple times. You are expected to access the keys of an object just once. You are expected to go through the values of an array just once. The following specific instructions indicate how to use the JSON when exceptions are enabled, but simdjson has full, idiomatic support for users who avoid exceptions. See [the simdjson error handling documentation](basics.md#error-handling) for more. @@ -273,18 +277,24 @@ support for users who avoid exceptions. See [the simdjson error handling documen However, it is not fully validated. On Demand only fully validates the values you use and the structure leading to it. * **Extracting Values:** You can cast a JSON element to a native type: - `double(element)` or `double x = json_element`. This works for `std::string_view`, double, uint64_t, int64_t, bool, - ondemand::object and ondemand::array. At this point, the number, string or boolean will be parsed, - or the initial `[` or `{` will be verified. An exception is thrown if the cast is not possible. + `double(element)`. This works for `std::string_view`, double, uint64_t, int64_t, bool, + ondemand::object and ondemand::array. We also have explicit methods such as `get_string()`, `get_double()`, + `get_uint64()`, `get_int64()`, `get_bool()`, `get_object()` and `get_array()`. After a cast or an explicit method, + the number, string or boolean will be parsed, or the initial `[` or `{` will be verified. An exception is thrown if + the cast is not possible. > IMPORTANT NOTE: values can only be parsed once. Since documents are *iterators*, once you have - > parsed a value (such as by casting to double), you cannot get at it again. + > parsed a value (such as by casting to double), you cannot get at it again. It is an error to call + > `get_string()` twice on an object (or to cast an object twice to `std::string_view`). * **Field Access:** To get the value of the "foo" field in an object, use `object["foo"]`. This will scan through the object looking for the field with the matching string, doing a character-by-character comparison. For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do - not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. Furthermore, you can only consume one field at a time, on the same object. Thus - if you have retrieved `content["bids"].get_array()` and you later call `content["asks"].get_array()`, then the first array should no longer be accessed: it would be unsafe to do so. - You can detect such mistakes by first compiling and running the code in Debug mode: an OUT_OF_ORDER_ITERATION error is generated. + not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. If you consume an + object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]`, your code + is in error. Furthermore, you can only consume one field at a time, on the same object. Thus + if you have retrieved `content["bids"].get_array()` and you later call `content["asks"].get_array()`, then the + first array should no longer be accessed: it would be unsafe to do so. You can detect such mistakes by first + compiling and running the code in Debug mode: an OUT_OF_ORDER_ITERATION error is generated. > NOTE: JSON allows you to escape characters in keys. E.g., the key `"date"` may be written as > `"\u0064\u0061\u0074\u0065"`. By default, simdjson does *not* unescape keys when matching by default. diff --git a/include/simdjson/generic/ondemand/document.h b/include/simdjson/generic/ondemand/document.h index 7d7b0f1ec..0e9afa838 100644 --- a/include/simdjson/generic/ondemand/document.h +++ b/include/simdjson/generic/ondemand/document.h @@ -94,6 +94,8 @@ public: * * The string is guaranteed to be valid UTF-8. * + * Important: Calling get_string() twice on the same document is an error. + * * @returns An UTF-8 string. The string is stored in the parser and will be invalidated the next * time it parses a document or when it is destroyed. * @returns INCORRECT_TYPE if the JSON value is not a string. @@ -311,6 +313,10 @@ public: * the code in Debug mode (or with the macro `SIMDJSON_DEVELOPMENT_CHECKS` set to 1): an * OUT_OF_ORDER_ITERATION error is generated. * + * You are expected to access keys only once. You should access the value corresponding to + * a key a single time. Doing object["mykey"].to_string()and then again object["mykey"].to_string() + * is an error. + * * @param key The key to look up. * @returns The value of the field, or NO_SUCH_FIELD if the field is not in the object. */ @@ -341,6 +347,10 @@ public: * the code in Debug mode (or with the macro `SIMDJSON_DEVELOPMENT_CHECKS` set to 1): an * OUT_OF_ORDER_ITERATION error is generated. * + * You are expected to access keys only once. You should access the value corresponding to a key + * a single time. Doing object["mykey"].to_string() and then again object["mykey"].to_string() + * is an error. + * * @param key The key to look up. * @returns The value of the field, or NO_SUCH_FIELD if the field is not in the object. */ diff --git a/include/simdjson/generic/ondemand/object.h b/include/simdjson/generic/ondemand/object.h index 9c01b9c1d..64a3434f6 100644 --- a/include/simdjson/generic/ondemand/object.h +++ b/include/simdjson/generic/ondemand/object.h @@ -44,6 +44,10 @@ public: * the code in Debug mode (or with the macro `SIMDJSON_DEVELOPMENT_CHECKS` set to 1): an * OUT_OF_ORDER_ITERATION error is generated. * + * You are expected to access keys only once. You should access the value corresponding to a + * key a single time. Doing object["mykey"].to_string() and then again object["mykey"].to_string() + * is an error. + * * @param key The key to look up. * @returns The value of the field, or NO_SUCH_FIELD if the field is not in the object. */ @@ -77,6 +81,9 @@ public: * the code in Debug mode (or with the macro `SIMDJSON_DEVELOPMENT_CHECKS` set to 1): an * OUT_OF_ORDER_ITERATION error is generated. * + * You are expected to access keys only once. You should access the value corresponding to a key + * a single time. Doing object["mykey"].to_string() and then again object["mykey"].to_string() is an error. + * * @param key The key to look up. * @returns The value of the field, or NO_SUCH_FIELD if the field is not in the object. */ diff --git a/include/simdjson/generic/ondemand/value.h b/include/simdjson/generic/ondemand/value.h index 4de67995b..24686e3f2 100644 --- a/include/simdjson/generic/ondemand/value.h +++ b/include/simdjson/generic/ondemand/value.h @@ -121,6 +121,9 @@ public: * * Equivalent to get(). * + * Important: a value should be consumed once. Calling get_string() twice on the same value + * is an error. + * * @returns An UTF-8 string. The string is stored in the parser and will be invalidated the next * time it parses a document or when it is destroyed. * @returns INCORRECT_TYPE if the JSON value is not a string. diff --git a/tests/ondemand/ondemand_object_tests.cpp b/tests/ondemand/ondemand_object_tests.cpp index b71f4c229..ddb550003 100644 --- a/tests/ondemand/ondemand_object_tests.cpp +++ b/tests/ondemand/ondemand_object_tests.cpp @@ -6,6 +6,53 @@ using namespace simdjson; namespace object_tests { using namespace std; using simdjson::ondemand::json_type; + bool issue1745() { + TEST_START(); + auto json = R"({ + "keys": [ + { + "kid": "bbd2ac7c4c5eb8adc8eeffbc8f5a2dd6cf7545e4", + "e": "AQAB", + "alg": "RS256", + "use": "sig", + "n": "y930dtGTeMG52IPsKmMuEpPHLaxuYQlduZd6BqFVjc2-UFZR8fNqtnYzAjbXWJD_Tqxgdlj_MW4vogvX4sHwVpZONvdyeGoIyDQtis6iuGQhQamV85F_JbrEUnEw3QCO87Liz5UXG6BK2HRyPhDfMex1_tO0ROmySLFdCTS17D0wah71Ibpi0gI8LUi6kzVRjYDIC1oE-iK3Y9s88Bi4ZGYJxXAbnNwbwVkGOKCXja9k0jjBGRxZD-4KDuf493lFOOEGSLDA2Qp9rDqrURP12XYgvf_zJx_kSDipnr0gL6Vz2n3H4-XN4tA45zuzRkHoE7-XexPq-tv7kQ8pSjY2uQ", + "kty": "RSA" + }, + { + "kty": "RSA", + "n": "zMHxWuxztMKXdBhv3rImlUvW_yp6nO03cVXPyA0Vyq0-M7LfOJJIF-OdNoRGdsFPHVKCFoo6qGhR8rBCmMxA4fM-Ubk5qKuUqCN9eP3yZJq8Cw9tUrt_qh7uW-qfMr0upcyeSHhC_zW1lTGs5sowDorKN_jQ1Sfh9hfBxfc8T7dQAAgEqqMcE3u-2J701jyhJz0pvurCfziiB3buY6SGREhBQwNwpnQjt_lE2U4km8FS0woPzt0ccE3zsGL2qM-LWZbOm9aXquSnqNJLt3tGVvShnev-GiJ1XfQ3EWm0f4w0TX9fTOkxstl0vo_vW_FjGQ0D1pXSjqb7n-hAdXwc9w", + "alg": "RS256", + "kid": "85828c59284a69b54b27483e487c3bd46cd2a2b3", + "e": "AQAB", + "use": "sig" + } + ] +} + + })"_padded; + ondemand::parser parser; + ondemand::document doc; + ASSERT_SUCCESS(parser.iterate(json).get(doc)); + for (auto items_instance : doc["keys"]) { + ondemand::object items; + ASSERT_SUCCESS(items_instance.get_object().get(items)); + std::string_view kty_value; + ASSERT_SUCCESS(items["kty"].get_string().get(kty_value)); + std::string_view use_value; + ASSERT_SUCCESS(items["use"].get_string().get(use_value)); + if (kty_value != "RSA" && use_value != "sig") { continue; } + std::string_view str; + ASSERT_SUCCESS(items["alg"].get_string().get(str)); + ASSERT_EQUAL(str, "RS256"); + ASSERT_EQUAL(use_value, "sig"); + ASSERT_EQUAL(kty_value, "RSA"); + ASSERT_SUCCESS(items["n"].get_string().get(str)); + ASSERT_EQUAL(str.size(), 342); + ASSERT_SUCCESS(items["e"].get_string().get(str)); + ASSERT_EQUAL(str, "AQAB"); + } + TEST_SUCCEED(); + } bool issue1723() { TEST_START(); @@ -178,6 +225,47 @@ namespace object_tests { } #if SIMDJSON_EXCEPTIONS + bool issue1745_with_exceptions() { + TEST_START(); + auto json = R"({ + "keys": [ + { + "kid": "bbd2ac7c4c5eb8adc8eeffbc8f5a2dd6cf7545e4", + "e": "AQAB", + "alg": "RS256", + "use": "sig", + "n": "y930dtGTeMG52IPsKmMuEpPHLaxuYQlduZd6BqFVjc2-UFZR8fNqtnYzAjbXWJD_Tqxgdlj_MW4vogvX4sHwVpZONvdyeGoIyDQtis6iuGQhQamV85F_JbrEUnEw3QCO87Liz5UXG6BK2HRyPhDfMex1_tO0ROmySLFdCTS17D0wah71Ibpi0gI8LUi6kzVRjYDIC1oE-iK3Y9s88Bi4ZGYJxXAbnNwbwVkGOKCXja9k0jjBGRxZD-4KDuf493lFOOEGSLDA2Qp9rDqrURP12XYgvf_zJx_kSDipnr0gL6Vz2n3H4-XN4tA45zuzRkHoE7-XexPq-tv7kQ8pSjY2uQ", + "kty": "RSA" + }, + { + "kty": "RSA", + "n": "zMHxWuxztMKXdBhv3rImlUvW_yp6nO03cVXPyA0Vyq0-M7LfOJJIF-OdNoRGdsFPHVKCFoo6qGhR8rBCmMxA4fM-Ubk5qKuUqCN9eP3yZJq8Cw9tUrt_qh7uW-qfMr0upcyeSHhC_zW1lTGs5sowDorKN_jQ1Sfh9hfBxfc8T7dQAAgEqqMcE3u-2J701jyhJz0pvurCfziiB3buY6SGREhBQwNwpnQjt_lE2U4km8FS0woPzt0ccE3zsGL2qM-LWZbOm9aXquSnqNJLt3tGVvShnev-GiJ1XfQ3EWm0f4w0TX9fTOkxstl0vo_vW_FjGQ0D1pXSjqb7n-hAdXwc9w", + "alg": "RS256", + "kid": "85828c59284a69b54b27483e487c3bd46cd2a2b3", + "e": "AQAB", + "use": "sig" + } + ] +} + + })"_padded; + ondemand::parser parser; + ondemand::document doc = parser.iterate(json); + for (auto items : doc["keys"]) { + std::string_view kty_value = items["kty"].get_string(); + std::string_view use_value = items["use"].get_string(); + if (kty_value != "RSA" && use_value != "sig") { continue; } + std::string_view str = items["alg"].get_string(); + if(str != "RS256") { return false; } + if(use_value != "sig") { return false; } + if(kty_value != "RSA") { return false; } + str = items["n"].get_string(); + if(str.size() != 342) { return false; } + str = items["e"].get_string(); + if(str != "AQAB") { return false; } + } + TEST_SUCCEED(); + } bool issue1723_except() { TEST_START(); @@ -988,6 +1076,7 @@ namespace object_tests { bool run() { return + issue1745() && issue1742() && issue1742_value() && issue1723() && @@ -997,6 +1086,7 @@ namespace object_tests { missing_keys() && missing_keys_for_empty_top_level_object() && #if SIMDJSON_EXCEPTIONS + issue1745_with_exceptions() && issue1723_except() && fixed_broken_issue_1521() && issue_1521() &&