admin/simdjson-simdjson
1
0
Fork 0
mirror of https://github.com/simdjson/simdjson synced 2026-06-08 17:27:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Improving documentation. (#2020)

Browse Source 
* Improving documentation.

* Fix typo

* Moving the definition.

* [skip ci] typo
This commit is contained in:
Daniel Lemire
2023-06-20 13:51:53 -04:00
committed by GitHub
parent 92174c927d
commit 2934892c0b
4 changed files with 79 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/basics.md
+12 -6
View File
@@ -335,8 +335,8 @@ support for users who avoid exceptions. See [the simdjson error handling documen
`double(element)`. This works for `std::string_view`, double, uint64_t, int64_t, bool,
ondemand::object and ondemand::array. We also have explicit methods such as `get_string()`, `get_double()`,
`get_uint64()`, `get_int64()`, `get_bool()`, `get_object()` and `get_array()`. After a cast or an explicit method,
the number, string or boolean will be parsed, or the initial `{` or `[` will be verified for ondemand::object and ondemand::array. An exception is thrown if
the cast is not possible. Importantly, when getting an ondemand::object or ondemand::array instance, its content is
the number, string or boolean will be parsed, or the initial `{` or `[` will be verified for `ondemand::object` and `ondemand::array`. An exception may be thrown if
the cast is not possible: there error code is `simdjson::INCORRECT_TYPE` (see [Error Handling](#error-handling)). Importantly, when getting an ondemand::object or ondemand::array instance, its content is
not validated: you are only guaranteed that the corresponding initial character (`{` or `[`) is present. Thus,
for example, you could have an ondemand::object instance pointing at the invalid JSON `{ "this is not a valid object" }`: the validation occurs as you access the content.
The `get_string()` returns a valid UTF-8 string, after
@@ -354,7 +354,7 @@ support for users who avoid exceptions. See [the simdjson error handling documen
> `get_string()` twice on an object (or to cast an object twice to `std::string_view`).
* **Field Access:** To get the value of the "foo" field in an object, use `object["foo"]`. This will
scan through the object looking for the field with the matching string, doing a character-by-character
comparison. For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do
comparison. It may generate the error `simdjson::NO_SUCH_FIELD` if there is no such key in the object, it may throw an exception (see [Error Handling](#error-handling)). For efficiency reason, you should avoid looking up the same field repeatedly: e.g., do
not do `object["foo"]` followed by `object["foo"]` with the same `object` instance. Keep in mind that On Demand does not buffer or save the result of the parsing: if you repeatedly access `object["foo"]`, then it must repeatedly seek the key and parse the content. The library does not provide a distinct function to check if a key is present, instead we recommend you attempt to access the key: e.g., by doing `ondemand::value val{}; if(!object["foo"].get(val)) {...}`, you have that `val` contains the requested value inside the if clause. It is your responsability as a user to temporarily keep a reference to the value (`auto v = object["foo"]`), or to consume the content and store it in your own data structures. If you consume an
object twice: `std::string_view(object["foo"]` followed by `std::string_view(object["foo"]` then your code
is in error. Furthermore, you can only consume one field at a time, on the same object. The
@@ -887,10 +887,17 @@ auto error = parser.iterate(json).get(doc);
if (error) { cerr << error << endl; exit(1); }
```
When there is no error, the error code simdjson::SUCCESS is returned: it evaluates as false as a Boolean.
When there is no error, the error code `simdjson::SUCCESS`is returned: it evaluates as false as a Boolean.
We have several error codes to indicate errors, they all evaluate to true as a Boolean: your software should not generally not depend on exact
error codes. We may change the error codes in future releases and the exact error codes could vary depending on your system.
Some errors are recoverable:
* You may get the error `simdjson::INCORRECT_TYPE` after trying to convert a value to an incorrect type: e.g., you expected a number and try to convert the value to a number, but it is an array.
* You may query a key from an object, but the key is missing in which case you get the error `simdjson::NO_SUCH_FIELD`: e.g., you call `obj["myname"]` and the object does not have a key `"myname"`.
Other errors (e.g., `simdjson::INCOMPLETE_ARRAY_OR_OBJECT`) may indicate a fatal error and often follow from the fact that the document is not valid JSON. In which case, it is no longer possible to continue accessing the document: calling the method `is_alive()` on the document instance returns false. All following accesses will keep returning the same fatal error (e.g., `simdjson::INCOMPLETE_ARRAY_OR_OBJECT`).
When you use the code without exceptions, it is your responsibility to check for error before using the
result: if there is an error, the result value will not be valid and using it will caused undefined behavior. Most compilers should be able to help you if you activate the right
set of warnings: they can identify variables that are written to but never otherwise accessed.
@@ -1422,10 +1429,9 @@ This parser can't support a document that big
If your documents are large (e.g., larger than a megabyte), then the `iterate_many` function is maybe ill-suited. It is really meant to support reading efficiently streams of relatively small documents (e.g., a few kilobytes each). If you have larger documents, you should use other functions like `iterate`.
We also provide some support for comma-separated documents and other advanced features.
See [iterate_many.md](iterate_many.md) for detailed information and design.
Parsing Numbers Inside Strings
------------------------------
doc/iterate_many.md
+45 -2
View File
@@ -129,6 +129,18 @@ Some official formats **(non-exhaustive list)**:
API
---
Example:
```c++
auto json = R"({ "foo": 1 } { "foo": 2 } { "foo": 3 } )"_padded;
ondemand::parser parser;
ondemand::document_stream docs = parser.iterate_many(json);
for (auto doc : docs) {
std::cout << doc["foo"] << std::endl;
}
// Prints 1 2 3
```
See [basics.md](basics.md#newline-delimited-json-ndjson-and-json-lines) for an overview of the API.
## Use cases
@@ -238,7 +250,38 @@ This will print:
Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
Comma separated documents
Comma-separated documents
-----------
`iterate_many` also takes in an option to allow parsing of comma separated documents. In this mode, the entire buffer is processed in 1 batch and batch size will be increased to be as large as the JSON passed. Therefore, the capacity of the parser has to be sufficient to support the batch size set.
We also support comma-separated documents, but with some performance limitations. The `iterate_many` function takes in an option to allow parsing of comma separated documents (which defaults on false). In this mode, the entire buffer is processed in one batch. Therefore, the total size of the document should not exceed the maximal capacity of the parser (4 GB). This mode also effectively disallow multithreading. It is therefore mostly suitable for not "very large" inputs. In this mode, the batch_size parameter
is effectively ignored, as it is set to at least the document size.
Example:
```C++
auto json = R"( 1, 2, 3, 4, "a", "b", "c", {"hello": "world"} , [1, 2, 3])"_padded;
ondemand::parser parser;
ondemand::document_stream doc_stream;
// We pass '32' as the batch size, but it is a bogus parameter because, since
// we pass 'true' to the allow_comma parameter, the batch size will be set to at least
// the document size.
auto error = parser.iterate_many(json, 32, true).get(doc_stream);
if(error) { std::cerr << error << std::endl; return; }
for (auto doc : doc_stream) {
std::cout << doc.type() << std::endl;
}
```
This will print:
```
number
number
number
number
string
string
string
object
array
```
include/simdjson/generic/ondemand/parser.h
+5
View File
@@ -212,6 +212,11 @@ public:
* spot is cache-related: small enough to fit in cache, yet big enough to
* parse as many documents as possible in one tight loop.
* Defaults to 10MB, which has been a reasonable sweet spot in our tests.
* @param allow_comma_separated (defaults on false) This allows a mode where the documents are
* separated by commas instead of whitespace. It comes with a performance
* penalty because the entire document is indexed at once (and the document must be
* less than 4 GB), and there is no multithreading. In this mode, the batch_size parameter
* is effectively ignored, as it is set to at least the document size.
* @return The stream, or an error. An empty input will yield 0 documents rather than an EMPTY error. Errors:
* - MEMALLOC if the parser does not have enough capacity and memory allocation fails
* - CAPACITY if the parser does not have enough capacity and batch_size > max_capacity.
tests/ondemand/ondemand_readme_examples.cpp
+17 -1
View File
@@ -1037,6 +1037,22 @@ int load_example_except_morecomplete(void) {
}
return EXIT_SUCCESS;
}
bool allow_comma_separated_example() {
TEST_START();
auto json = R"( 1, 2, 3, 4, "a", "b", "c", {"hello": "world"} , [1, 2, 3])"_padded;
ondemand::parser parser;
ondemand::document_stream doc_stream;
// We pass '32' as the batch size, but it is a bogus parameter because, since
// we pass 'true' to the allow_comma parameter, the batch size will be set to at least
// the document size.
auto error = parser.iterate_many(json, 32, true).get(doc_stream);
if(error) { std::cerr << error << std::endl; return false; }
for (auto doc : doc_stream) {
std::cout << doc.type() << std::endl;
}
TEST_SUCCEED();
}
#endif
bool test_load_example() {
TEST_START();
@@ -1067,7 +1083,6 @@ bool current_location_tape_error() {
TEST_SUCCEED();
}
bool current_location_user_error() {
TEST_START();
auto json = R"( [1,2,3] )"_padded;
@@ -1301,6 +1316,7 @@ bool run() {
&& gen_raw1() && gen_raw2() && gen_raw3()
&& at_end()
&& example1956() && example1958()
&& allow_comma_separated_example()
// && basics_1() // Fails because twitter.json isn't in current directory. Compile test only.
&& basics_treewalk()
&& basics_treewalk_breakline()