mirror of
https://github.com/simdjson/simdjson
synced 2026-06-08 17:27:07 +00:00
lemire
577 lines
47 KiB
HTML
577 lines
47 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
|
|
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
|
|
<meta name="generator" content="Doxygen 1.9.1"/>
|
|
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
|
<title>simdjson: The Document-Object-Model (DOM) front-end</title>
|
|
<link href="tabs.css" rel="stylesheet" type="text/css"/>
|
|
<script type="text/javascript" src="jquery.js"></script>
|
|
<script type="text/javascript" src="dynsections.js"></script>
|
|
<link href="navtree.css" rel="stylesheet" type="text/css"/>
|
|
<script type="text/javascript" src="resize.js"></script>
|
|
<script type="text/javascript" src="navtreedata.js"></script>
|
|
<script type="text/javascript" src="navtree.js"></script>
|
|
<link href="search/search.css" rel="stylesheet" type="text/css"/>
|
|
<script type="text/javascript" src="search/searchdata.js"></script>
|
|
<script type="text/javascript" src="search/search.js"></script>
|
|
<link href="doxygen.css" rel="stylesheet" type="text/css" />
|
|
<link href="doxygen-awesome.css" rel="stylesheet" type="text/css"/>
|
|
<link href="doxygen-awesome-sidebar-only.css" rel="stylesheet" type="text/css"/>
|
|
<link href="doxygen-awesome-sidebar-only-darkmode-toggle.css" rel="stylesheet" type="text/css"/>
|
|
</head>
|
|
<body>
|
|
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
|
|
<div id="titlearea">
|
|
<table cellspacing="0" cellpadding="0">
|
|
<tbody>
|
|
<tr style="height: 56px;">
|
|
<td id="projectlogo"><img alt="Logo" src="logotiny.png"/></td>
|
|
<td id="projectalign" style="padding-left: 0.5em;">
|
|
<div id="projectname">simdjson
|
|
 <span id="projectnumber">3.1.5</span>
|
|
</div>
|
|
<div id="projectbrief">Ridiculously Fast JSON</div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<!-- end header part -->
|
|
<!-- Generated by Doxygen 1.9.1 -->
|
|
<script type="text/javascript">
|
|
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2 */
|
|
var searchBox = new SearchBox("searchBox", "search",false,'Search','.html');
|
|
/* @license-end */
|
|
</script>
|
|
<script type="text/javascript" src="menudata.js"></script>
|
|
<script type="text/javascript" src="menu.js"></script>
|
|
<script type="text/javascript">
|
|
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2 */
|
|
$(function() {
|
|
initMenu('',true,false,'search.php','Search');
|
|
$(document).ready(function() { init_search(); });
|
|
});
|
|
/* @license-end */</script>
|
|
<div id="main-nav"></div>
|
|
</div><!-- top -->
|
|
<div id="side-nav" class="ui-resizable side-nav-resizable">
|
|
<div id="nav-tree">
|
|
<div id="nav-tree-contents">
|
|
<div id="nav-sync" class="sync"></div>
|
|
</div>
|
|
</div>
|
|
<div id="splitbar" style="-moz-user-select:none;"
|
|
class="ui-resizable-handle">
|
|
</div>
|
|
</div>
|
|
<script type="text/javascript">
|
|
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2 */
|
|
$(document).ready(function(){initNavTree('md_doc_dom.html',''); initResizable(); });
|
|
/* @license-end */
|
|
</script>
|
|
<div id="doc-content">
|
|
<!-- window showing the filter options -->
|
|
<div id="MSearchSelectWindow"
|
|
onmouseover="return searchBox.OnSearchSelectShow()"
|
|
onmouseout="return searchBox.OnSearchSelectHide()"
|
|
onkeydown="return searchBox.OnSearchSelectKey(event)">
|
|
</div>
|
|
|
|
<!-- iframe showing the search results (closed by default) -->
|
|
<div id="MSearchResultsWindow">
|
|
<iframe src="javascript:void(0)" frameborder="0"
|
|
name="MSearchResults" id="MSearchResults">
|
|
</iframe>
|
|
</div>
|
|
|
|
<div class="PageDoc"><div class="header">
|
|
<div class="headertitle">
|
|
<div class="title">The Document-Object-Model (DOM) front-end </div> </div>
|
|
</div><!--header-->
|
|
<div class="contents">
|
|
<div class="textblock"><p>An overview of what you need to know to use simdjson, with examples.</p>
|
|
<ul>
|
|
<li><a href="#dom-vs-on-demand">DOM vs On Demand</a></li>
|
|
<li><a href="#the-basics-loading-and-parsing-json-documents-using-the-dom-front-end">The Basics: Loading and Parsing JSON Documents</a></li>
|
|
<li><a href="#using-the-parsed-json">Using the Parsed JSON</a></li>
|
|
<li><a href="#c17-support">C++17 Support</a></li>
|
|
<li><a href="#json-pointer">JSON Pointer</a></li>
|
|
<li><a href="#error-handling">Error Handling</a><ul>
|
|
<li><a href="#error-handling-example">Error Handling Example</a></li>
|
|
<li><a href="#exceptions">Exceptions</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a href="#tree-walking-and-json-element-types">Tree Walking and JSON Element Types</a></li>
|
|
<li><a href="#reusing-the-parser-for-maximum-efficiency">Reusing the parser for maximum efficiency</a></li>
|
|
<li><a href="#server-loops-long-running-processes-and-memory-capacity">Server Loops: Long-Running Processes and Memory Capacity</a></li>
|
|
<li><a href="#best-use-of-the-dom-api">Best Use of the DOM API</a></li>
|
|
<li><a href="#padding-and-temporary-copies">Padding and Temporary Copies</a></li>
|
|
</ul>
|
|
<h1><a class="anchor" id="autotoc_md29"></a>
|
|
DOM vs On Demand</h1>
|
|
<p>The simdjson library offers two distinct approaches on how to access a JSON document. We support a conventional Document-Object-Model (DOM) front-end. In such a scenario, the JSON document is entirely parsed, validated and materialized in memory as the first step. The programmer may then access the parsed data using this in-memory model.</p>
|
|
<h1><a class="anchor" id="autotoc_md30"></a>
|
|
The Basics: Loading and Parsing JSON Documents using the DOM front-end</h1>
|
|
<p>The simdjson library offers a simple DOM tree API, which you can access by creating a <code>dom::parser</code> and calling the <code>load()</code> method:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line">dom::element doc = parser.load(filename); // load and parse a file</div>
|
|
</div><!-- fragment --><p>Or by creating a padded string (for efficiency reasons, simdjson requires a string with SIMDJSON_PADDING bytes at the end) and calling <code>parse()</code>:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line">dom::element doc = parser.parse("[1,2,3]"_padded); // parse a string, the _padded suffix creates a simdjson::padded_string instance</div>
|
|
</div><!-- fragment --><p>The parsed document resulting from the <code>parser.load</code> and <code>parser.parse</code> calls depends on the <code>parser</code> instance. Thus the <code>parser</code> instance must remain in scope. Furthermore, you must have at most one parsed document in play per <code>parser</code> instance. You cannot copy a <code>parser</code> instance, you may only move it.</p>
|
|
<p>If you need to keep a document around long term, you can keep or move the parser instance. Note that moving a parser instance, or keeping one in a movable data structure like vector or map, can cause any outstanding <code>element</code>, <code>object</code> or <code>array</code> instances to be invalidated. The <code>element</code>, <code>object</code> or <code>array</code> instances are mere thin wrappers akin to an <code>std::vector<int>::iterator</code>: they are invalid when default constructed, they must be tied to a valid document instance. If you need to store a parser in a movable data structure, you should use a <code>std::unique_ptr</code> to avoid this invalidation(e.g., <code>std::unique_ptr<dom::parser> parser(new dom::parser{})</code>).</p>
|
|
<p>During the<code>load</code> or <code>parse</code> calls, neither the input file nor the input string are ever modified. After calling <code>load</code> or <code>parse</code>, the source (either a file or a string) can be safely discarded. All of the JSON data is stored in the <code>parser</code> instance. The parsed document is also immutable in simdjson: you do not modify it by accessing it.</p>
|
|
<p>For best performance, a <code>parser</code> instance should be reused over several files: otherwise you will needlessly reallocate memory, an expensive process. It is also possible to avoid entirely memory allocations during parsing when using simdjson. <a class="el" href="md_doc_performance.html">See our performance notes for details</a>.</p>
|
|
<p>If you need a lower-level interface, you may call the function <code>parser.parse(const char * p, size_t l)</code> on a pointer <code>p</code> while specifying the length of your input <code>l</code> in bytes. To see how to get the very best performance from a low-level approach, you way want to read our <a href="https://github.com/simdjson/simdjson/blob/master/doc/performance.md#padding-and-temporary-copies">performance notes</a> on this topic (see the Padding and Temporary Copies section).</p>
|
|
<h1><a class="anchor" id="autotoc_md31"></a>
|
|
Using the Parsed JSON</h1>
|
|
<p>Once you have an element, you can navigate it with idiomatic C++ iterators, operators and casts.</p>
|
|
<ul>
|
|
<li><b>Extracting Values (with exceptions):</b> You can cast a JSON element to a native type: <code>double(element)</code> or <code>double x = json_element</code>. This works for double, uint64_t, int64_t, bool, dom::object and dom::array. An exception (<code><a class="el" href="structsimdjson_1_1simdjson__error.html" title="Exception thrown when an exception-supporting simdjson method is called.">simdjson::simdjson_error</a></code>) is thrown if the cast is not possible.</li>
|
|
<li><b>Extracting Values (without exceptions):</b> You can use a variant usage of <code>get()</code> with error codes to avoid exceptions. You first declare the variable of the appropriate type (<code>double</code>, <code>uint64_t</code>, <code>int64_t</code>, <code>bool</code>, <code>std::string_view</code>, <code>dom::object</code> and <code>dom::array</code>) and pass it by reference to <code>get()</code> which gives you back an error code: e.g., <div class="fragment"><div class="line">{c++}</div>
|
|
<div class="line"> simdjson::error_code error;</div>
|
|
<div class="line"> simdjson::padded_string numberstring = "1.2"_padded; // our JSON input ("1.2")</div>
|
|
<div class="line"> simdjson::dom::parser parser;</div>
|
|
<div class="line"> double value; // variable where we store the value to be parsed</div>
|
|
<div class="line"> error = parser.parse(numberstring).get(value);</div>
|
|
<div class="line"> if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }</div>
|
|
<div class="line"> std::cout << "I parsed " << value << " from " << numberstring.data() << std::endl;</div>
|
|
</div><!-- fragment --> The strings contain unescaped valid UTF-8 strings: no unmatched surrogate is allowed.</li>
|
|
<li><b>Field Access:</b> To get the value of the "foo" field in an object, use <code>object["foo"]</code>.</li>
|
|
<li><b>Array Iteration:</b> To iterate through an array, use <code>for (auto value : array) { ... }</code>. If you know the type of the value, you can cast it right there, too! <code>for (double value : array) { ... }</code></li>
|
|
<li><b>Object Iteration:</b> You can iterate through an object's fields, too: <code>for (auto [key, value] : object)</code></li>
|
|
<li><b>Array Index:</b> To get at an array value by index, use the at() method: <code>array.at(0)</code> gets the first element. The at() method has linear-time complexity so it should not be used to iterate over the values of an array. <blockquote class="doxtable">
|
|
<p>Note that array[0] does not compile, because implementing [] gives the impression indexing is a O(1) operation, which it is not presently in simdjson. Instead, you should iterate over the elements using a for-loop, as in our examples. </p>
|
|
</blockquote>
|
|
* <b>Array and Object size</b> Given an array or an object, you can get its size (number of elements or keys) with the <code>size()</code> method.</li>
|
|
<li><b>Checking an Element Type:</b> You can check an element's type with <code>element.type()</code>. It returns an <code>element_type</code> with values such as <code>simdjson::dom::element_type::ARRAY</code>, <code>simdjson::dom::element_type::OBJECT</code>, <code>simdjson::dom::element_type::INT64</code>, <code>simdjson::dom::element_type::UINT64</code>,<code>simdjson::dom::element_type::DOUBLE</code>, <code>simdjson::dom::element_type::STRING</code>, <code>simdjson::dom::element_type::BOOL</code> or, <code>simdjson::dom::element_type::NULL_VALUE</code>.</li>
|
|
<li><b>Output to streams and strings:</b> Given a document or an element (or node) out of a JSON document, you can output a minified string version using the C++ stream idiom (<code>out << element</code>). You can also request the construction of a minified string version (<code>simdjson::minify(element)</code>). Numbers are serialized as 64-bit floating-point numbers (<code>double</code>).</li>
|
|
</ul>
|
|
<h2><a class="anchor" id="autotoc_md32"></a>
|
|
Examples</h2>
|
|
<p>The following code illustrates all of the above:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">auto cars_json = R"( [</div>
|
|
<div class="line"> { "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },</div>
|
|
<div class="line"> { "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },</div>
|
|
<div class="line"> { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }</div>
|
|
<div class="line">] )"_padded;</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line"> </div>
|
|
<div class="line">// Iterating through an array of objects</div>
|
|
<div class="line">for (dom::object car : parser.parse(cars_json)) {</div>
|
|
<div class="line"> // Accessing a field by name</div>
|
|
<div class="line"> cout << "Make/Model: " << car["make"] << "/" << car["model"] << endl;</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> // Casting a JSON element to an integer</div>
|
|
<div class="line"> uint64_t year = car["year"];</div>
|
|
<div class="line"> cout << "- This car is " << 2020 - year << "years old." << endl;</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> // Iterating through an array of floats</div>
|
|
<div class="line"> double total_tire_pressure = 0;</div>
|
|
<div class="line"> for (double tire_pressure : car["tire_pressure"]) {</div>
|
|
<div class="line"> total_tire_pressure += tire_pressure;</div>
|
|
<div class="line"> }</div>
|
|
<div class="line"> cout << "- Average tire pressure: " << (total_tire_pressure / 4) << endl;</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> // Writing out all the information about the car</div>
|
|
<div class="line"> for (auto field : car) {</div>
|
|
<div class="line"> cout << "- " << field.key << ": " << field.value << endl;</div>
|
|
<div class="line"> }</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>Here is a different example illustrating the same ideas:</p>
|
|
<div class="fragment"><div class="line"> {C++}</div>
|
|
<div class="line">auto abstract_json = R"( [</div>
|
|
<div class="line"> { "12345" : {"a":12.34, "b":56.78, "c": 9998877} },</div>
|
|
<div class="line"> { "12545" : {"a":11.44, "b":12.78, "c": 11111111} }</div>
|
|
<div class="line"> ] )"_padded;</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line"> </div>
|
|
<div class="line">// Parse and iterate through an array of objects</div>
|
|
<div class="line">for (dom::object obj : parser.parse(abstract_json)) {</div>
|
|
<div class="line"> for(const auto key_value : obj) {</div>
|
|
<div class="line"> cout << "key: " << key_value.key << " : ";</div>
|
|
<div class="line"> dom::object innerobj = key_value.value;</div>
|
|
<div class="line"> cout << "a: " << double(innerobj["a"]) << ", ";</div>
|
|
<div class="line"> cout << "b: " << double(innerobj["b"]) << ", ";</div>
|
|
<div class="line"> cout << "c: " << int64_t(innerobj["c"]) << endl;</div>
|
|
<div class="line"> }</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>And another one:</p>
|
|
<div class="fragment"><div class="line">{C++}</div>
|
|
<div class="line"> auto abstract_json = R"(</div>
|
|
<div class="line"> { "str" : { "123" : {"abc" : 3.14 } } } )"_padded;</div>
|
|
<div class="line"> dom::parser parser;</div>
|
|
<div class="line"> double v = parser.parse(abstract_json)["str"]["123"]["abc"];</div>
|
|
<div class="line"> cout << "number: " << v << endl;</div>
|
|
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md33"></a>
|
|
C++17 Support</h1>
|
|
<p>While the simdjson library can be used in any project using C++ 11 and above, field iteration has special support C++ 17's destructuring syntax. For example:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">padded_string json = R"( { "foo": 1, "bar": 2 } )"_padded;</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line">dom::object object; // invalid until the get() succeeds</div>
|
|
<div class="line">auto error = parser.parse(json).get(object);</div>
|
|
<div class="line">if (error) { cerr << error << endl; return; }</div>
|
|
<div class="line">for (auto [key, value] : object) {</div>
|
|
<div class="line"> cout << key << " = " << value << endl;</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>For comparison, here is the C++ 11 version of the same code:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">// C++ 11 version for comparison</div>
|
|
<div class="line">padded_string json = R"( { "foo": 1, "bar": 2 } )"_padded;</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line">dom::object object; // invalid until the get() succeeds</div>
|
|
<div class="line">auto error = parser.parse(json).get(object);</div>
|
|
<div class="line">if (error) { cerr << error << endl; return; }</div>
|
|
<div class="line">for (dom::key_value_pair field : object) {</div>
|
|
<div class="line"> cout << field.key << " = " << field.value << endl;</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md34"></a>
|
|
JSON Pointer</h1>
|
|
<p>The simdjson library also supports <a href="https://tools.ietf.org/html/rfc6901">JSON pointer</a> through the <code>at_pointer()</code> method, letting you reach further down into the document in a single call:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">auto cars_json = R"( [</div>
|
|
<div class="line"> { "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },</div>
|
|
<div class="line"> { "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },</div>
|
|
<div class="line"> { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }</div>
|
|
<div class="line">] )"_padded;</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line">dom::element cars = parser.parse(cars_json);</div>
|
|
<div class="line">cout << cars.at_pointer("/0/tire_pressure/1") << endl; // Prints 39.9</div>
|
|
</div><!-- fragment --><p>A JSON Path is a sequence of segments each starting with the '/' character. Within arrays, an integer index allows you to select the indexed node. Within objects, the string value of the key allows you to select the value. If your keys contain the characters '/' or '~', they must be escaped as '~1' and '~0' respectively. An empty JSON Path refers to the whole document.</p>
|
|
<p>We also extend the JSON Pointer support to include <em>relative</em> paths. You can apply a JSON path to any node and the path gets interpreted relatively, as if the current node were a whole JSON document.</p>
|
|
<p>Consider the following example:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">auto cars_json = R"( [</div>
|
|
<div class="line"> { "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },</div>
|
|
<div class="line"> { "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },</div>
|
|
<div class="line"> { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }</div>
|
|
<div class="line">] )"_padded;</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line">dom::element cars = parser.parse(cars_json);</div>
|
|
<div class="line">cout << cars.at_pointer("/0/tire_pressure/1") << endl; // Prints 39.9</div>
|
|
<div class="line">for (dom::element car_element : cars) {</div>
|
|
<div class="line"> dom::object car; // invalid until the get() succeeds</div>
|
|
<div class="line"> simdjson::error_code error;</div>
|
|
<div class="line"> if ((error = car_element.get(car))) { std::cerr << error << std::endl; return; }</div>
|
|
<div class="line"> double x = car.at_pointer("/tire_pressure/1");</div>
|
|
<div class="line"> cout << x << endl; // Prints 39.9, 31 and 30</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md35"></a>
|
|
Error Handling</h1>
|
|
<p>All simdjson APIs that can fail return <code>simdjson_result<T></code>, which is a <value, error_code> pair. You can retrieve the value with .get(), like so:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">dom::element doc;</div>
|
|
<div class="line">auto error = parser.parse(json).get(doc);</div>
|
|
<div class="line">if (error) { cerr << error << endl; exit(1); }</div>
|
|
</div><!-- fragment --><p>When there is no error, the error code <a class="el" href="namespacesimdjson.html#a7b735a3a50ba79e3f7f14df5f77d8da9aa4ffc800804829a88c5b52ac3c4c6681" title="No error.">simdjson::SUCCESS</a> is returned: it evaluates as false as a Boolean. We have several error codes to indicate errors, they all evaluate to true as a Boolean: your software should not generally not depend on exact error codes. We may change the error codes in future releases and the exact error codes could vary depending on your system.</p>
|
|
<p>When you use the code without exceptions, it is your responsibility to check for error before using the result: if there is an error, the result value will not be valid and using it will caused undefined behavior.</p>
|
|
<p>We can write a "quick start" example where we attempt to parse the following JSON file and access some data, without triggering exceptions: </p><div class="fragment"><div class="line">{</div>
|
|
<div class="line"> "statuses": [</div>
|
|
<div class="line"> {</div>
|
|
<div class="line"> "id": 505874924095815700</div>
|
|
<div class="line"> },</div>
|
|
<div class="line"> {</div>
|
|
<div class="line"> "id": 505874922023837700</div>
|
|
<div class="line"> }</div>
|
|
<div class="line"> ],</div>
|
|
<div class="line"> "search_metadata": {</div>
|
|
<div class="line"> "count": 100</div>
|
|
<div class="line"> }</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>Our program loads the file, selects value corresponding to key "search_metadata" which expected to be an object, and then it selects the key "count" within that object.</p>
|
|
<div class="fragment"><div class="line"> {C++}</div>
|
|
<div class="line">#include <iostream></div>
|
|
<div class="line">#include "simdjson.h"</div>
|
|
<div class="line"> </div>
|
|
<div class="line">int main(void) {</div>
|
|
<div class="line"> simdjson::dom::parser parser;</div>
|
|
<div class="line"> simdjson::dom::element tweets; // invalid until the get() succeeds</div>
|
|
<div class="line"> auto error = parser.load("twitter.json").get(tweets);</div>
|
|
<div class="line"> if (error) { std::cerr << error << std::endl; return EXIT_FAILURE; }</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> simdjson::dom::element res; // invalid until the get() succeeds</div>
|
|
<div class="line"> if ((error = tweets["search_metadata"]["count"].get(res))) {</div>
|
|
<div class="line"> std::cerr << "could not access keys" << std::endl;</div>
|
|
<div class="line"> return EXIT_FAILURE;</div>
|
|
<div class="line"> }</div>
|
|
<div class="line"> std::cout << res << " results." << std::endl;</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>The following is a similar example where one wants to get the id of the first tweet without triggering exceptions. To do this, we use <code>["statuses"].at(0)["id"]</code>. We break that expression down:</p>
|
|
<ul>
|
|
<li>Get the list of tweets (the <code>"statuses"</code> key of the document) using <code>["statuses"]</code>). The result is expected to be an array.</li>
|
|
<li>Get the first tweet using <code>.at(0)</code>. The result is expected to be an object.</li>
|
|
<li>Get the id of the tweet using ["id"]. We expect the value to be a non-negative integer.</li>
|
|
</ul>
|
|
<p>Observe how we use the <code>at</code> method when querying an index into an array, and not the bracket operator.</p>
|
|
<div class="fragment"><div class="line"> {C++}</div>
|
|
<div class="line">#include <iostream></div>
|
|
<div class="line">#include "simdjson.h"</div>
|
|
<div class="line"> </div>
|
|
<div class="line">int main(void) {</div>
|
|
<div class="line"> simdjson::dom::parser parser;</div>
|
|
<div class="line"> simdjson::dom::element tweets; // invalid until the get() succeeds</div>
|
|
<div class="line"> auto error = parser.load("twitter.json").get(tweets);</div>
|
|
<div class="line"> if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }</div>
|
|
<div class="line"> uint64_t identifier;</div>
|
|
<div class="line"> error = tweets["statuses"].at(0)["id"].get(identifier);</div>
|
|
<div class="line"> if(error) { std::cerr << error << std::endl; return EXIT_FAILURE; }</div>
|
|
<div class="line"> std::cout << identifier << std::endl;</div>
|
|
<div class="line"> return EXIT_SUCCESS;</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>The <code>at()</code> method has linear-time complexity: it should not be used to iterate over the content of an array.</p>
|
|
<h2><a class="anchor" id="autotoc_md36"></a>
|
|
Error Handling Example</h2>
|
|
<p>This is how the example in "Using the Parsed JSON" could be written using only error code checking:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">auto cars_json = R"( [</div>
|
|
<div class="line"> { "make": "Toyota", "model": "Camry", "year": 2018, "tire_pressure": [ 40.1, 39.9, 37.7, 40.4 ] },</div>
|
|
<div class="line"> { "make": "Kia", "model": "Soul", "year": 2012, "tire_pressure": [ 30.1, 31.0, 28.6, 28.7 ] },</div>
|
|
<div class="line"> { "make": "Toyota", "model": "Tercel", "year": 1999, "tire_pressure": [ 29.8, 30.0, 30.2, 30.5 ] }</div>
|
|
<div class="line">] )"_padded;</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line">dom::array cars; // invalid until the get() succeeds</div>
|
|
<div class="line">auto error = parser.parse(cars_json).get(cars);</div>
|
|
<div class="line">if (error) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> </div>
|
|
<div class="line">// Iterating through an array of objects</div>
|
|
<div class="line">for (dom::element car_element : cars) {</div>
|
|
<div class="line"> dom::object car;</div>
|
|
<div class="line"> if ((error = car_element.get(car))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> // Accessing a field by name</div>
|
|
<div class="line"> std::string_view make, model;</div>
|
|
<div class="line"> if ((error = car["make"].get(make))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> if ((error = car["model"].get(model))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> cout << "Make/Model: " << make << "/" << model << endl;</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> // Casting a JSON element to an integer</div>
|
|
<div class="line"> uint64_t year;</div>
|
|
<div class="line"> if ((error = car["year"].get(year))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> cout << "- This car is " << 2020 - year << "years old." << endl;</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> // Iterating through an array of floats</div>
|
|
<div class="line"> double total_tire_pressure = 0;</div>
|
|
<div class="line"> dom::array tire_pressure_array;</div>
|
|
<div class="line"> if ((error = car["tire_pressure"].get(tire_pressure_array))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> for (dom::element tire_pressure_element : tire_pressure_array) {</div>
|
|
<div class="line"> double tire_pressure;</div>
|
|
<div class="line"> if ((error = tire_pressure_element.get(tire_pressure))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> total_tire_pressure += tire_pressure;</div>
|
|
<div class="line"> }</div>
|
|
<div class="line"> cout << "- Average tire pressure: " << (total_tire_pressure / 4) << endl;</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> // Writing out all the information about the car</div>
|
|
<div class="line"> for (auto field : car) {</div>
|
|
<div class="line"> cout << "- " << field.key << ": " << field.value << endl;</div>
|
|
<div class="line"> }</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>Here is another example:</p>
|
|
<div class="fragment"><div class="line"> {C++}</div>
|
|
<div class="line">auto abstract_json = R"( [</div>
|
|
<div class="line"> { "12345" : {"a":12.34, "b":56.78, "c": 9998877} },</div>
|
|
<div class="line"> { "12545" : {"a":11.44, "b":12.78, "c": 11111111} }</div>
|
|
<div class="line"> ] )"_padded;</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line">dom::array array; // invalid until after the next line</div>
|
|
<div class="line">auto error = parser.parse(abstract_json).get(array);</div>
|
|
<div class="line">if (error) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line">// Iterate through an array of objects</div>
|
|
<div class="line">for (dom::element elem : array) {</div>
|
|
<div class="line"> dom::object obj;</div>
|
|
<div class="line"> if ((error = elem.get(obj))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> for (auto & key_value : obj) {</div>
|
|
<div class="line"> cout << "key: " << key_value.key << " : ";</div>
|
|
<div class="line"> dom::object innerobj;</div>
|
|
<div class="line"> if ((error = key_value.value.get(innerobj))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> double va, vb;</div>
|
|
<div class="line"> if ((error = innerobj["a"].get(va))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> cout << "a: " << va << ", ";</div>
|
|
<div class="line"> if ((error = innerobj["b"].get(vc))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> cout << "b: " << vb << ", ";</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> int64_t vc;</div>
|
|
<div class="line"> if ((error = innerobj["c"].get(vc))) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> cout << "c: " << vc << endl;</div>
|
|
<div class="line"> }</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>And another one:</p>
|
|
<div class="fragment"><div class="line">{C++}</div>
|
|
<div class="line"> auto abstract_json = R"(</div>
|
|
<div class="line"> { "str" : { "123" : {"abc" : 3.14 } } } )"_padded;</div>
|
|
<div class="line"> dom::parser parser;</div>
|
|
<div class="line"> double v;</div>
|
|
<div class="line"> auto error = parser.parse(abstract_json)["str"]["123"]["abc"].get(v);</div>
|
|
<div class="line"> if (error) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> cout << "number: " << v << endl;</div>
|
|
</div><!-- fragment --><p>Notice how we can string several operations (<code>parser.parse(abstract_json)["str"]["123"]["abc"].get(v)</code>) and only check for the error once, a strategy we call <em>error chaining</em>.</p>
|
|
<p>The next two functions will take as input a JSON document containing an array with a single element, either a string or a number. They return true upon success.</p>
|
|
<div class="fragment"><div class="line"> {C++}</div>
|
|
<div class="line">simdjson::dom::parser parser{};</div>
|
|
<div class="line"> </div>
|
|
<div class="line">bool parse_double(const char *j, double &d) {</div>
|
|
<div class="line"> auto error = parser.parse(j, std::strlen(j))</div>
|
|
<div class="line"> .at(0)</div>
|
|
<div class="line"> .get(d, error);</div>
|
|
<div class="line"> if (error) { return false; }</div>
|
|
<div class="line"> return true;</div>
|
|
<div class="line">}</div>
|
|
<div class="line"> </div>
|
|
<div class="line">bool parse_string(const char *j, std::string &s) {</div>
|
|
<div class="line"> std::string_view answer;</div>
|
|
<div class="line"> auto error = parser.parse(j, strlen(j))</div>
|
|
<div class="line"> .at(0)</div>
|
|
<div class="line"> .get(answer, error);</div>
|
|
<div class="line"> if (error) { return false; }</div>
|
|
<div class="line"> s.assign(answer.data(), answer.size());</div>
|
|
<div class="line"> return true;</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><p>To ensure you don't write any code that uses exceptions, compile with <code>SIMDJSON_EXCEPTIONS=OFF</code>. For example, if including the project via cmake:</p>
|
|
<div class="fragment"><div class="line">target_compile_definitions(simdjson PUBLIC SIMDJSON_EXCEPTIONS=OFF)</div>
|
|
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md37"></a>
|
|
Exceptions</h2>
|
|
<p>Users more comfortable with an exception flow may choose to directly cast the <code>simdjson_result<T></code> to the desired type:</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">dom::element doc = parser.parse(json); // Throws an exception if there was an error!</div>
|
|
</div><!-- fragment --><p>When used this way, a <code>simdjson_error</code> exception will be thrown if an error occurs, preventing the program from continuing if there was an error.</p>
|
|
<p>If one is willing to trigger exceptions, it is possible to write simpler code:</p>
|
|
<div class="fragment"><div class="line"> {C++}</div>
|
|
<div class="line">#include <iostream></div>
|
|
<div class="line">#include "simdjson.h"</div>
|
|
<div class="line"> </div>
|
|
<div class="line">int main(void) {</div>
|
|
<div class="line"> simdjson::dom::parser parser;</div>
|
|
<div class="line"> simdjson::dom::element tweets = parser.load("twitter.json");</div>
|
|
<div class="line"> std::cout << "ID: " << tweets["statuses"].at(0)["id"] << std::endl;</div>
|
|
<div class="line"> return EXIT_SUCCESS;</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md38"></a>
|
|
Tree Walking and JSON Element Types</h1>
|
|
<p>Sometimes you don't necessarily have a document with a known type, and are trying to generically inspect or walk over JSON elements. To do that, you can use iterators and the type() method. For example, here's a quick and dirty recursive function that verbosely prints the JSON document as JSON (* ignoring nuances like trailing commas and escaping strings, for brevity's sake):</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">void print_json(dom::element element) {</div>
|
|
<div class="line"> switch (element.type()) {</div>
|
|
<div class="line"> case dom::element_type::ARRAY:</div>
|
|
<div class="line"> cout << "[";</div>
|
|
<div class="line"> for (dom::element child : dom::array(element)) {</div>
|
|
<div class="line"> print_json(child);</div>
|
|
<div class="line"> cout << ",";</div>
|
|
<div class="line"> }</div>
|
|
<div class="line"> cout << "]";</div>
|
|
<div class="line"> break;</div>
|
|
<div class="line"> case dom::element_type::OBJECT:</div>
|
|
<div class="line"> cout << "{";</div>
|
|
<div class="line"> for (dom::key_value_pair field : dom::object(element)) {</div>
|
|
<div class="line"> cout << "\"" << field.key << "\": ";</div>
|
|
<div class="line"> print_json(field.value);</div>
|
|
<div class="line"> }</div>
|
|
<div class="line"> cout << "}";</div>
|
|
<div class="line"> break;</div>
|
|
<div class="line"> case dom::element_type::INT64:</div>
|
|
<div class="line"> cout << int64_t(element) << endl;</div>
|
|
<div class="line"> break;</div>
|
|
<div class="line"> case dom::element_type::UINT64:</div>
|
|
<div class="line"> cout << uint64_t(element) << endl;</div>
|
|
<div class="line"> break;</div>
|
|
<div class="line"> case dom::element_type::DOUBLE:</div>
|
|
<div class="line"> cout << double(element) << endl;</div>
|
|
<div class="line"> break;</div>
|
|
<div class="line"> case dom::element_type::STRING:</div>
|
|
<div class="line"> cout << std::string_view(element) << endl;</div>
|
|
<div class="line"> break;</div>
|
|
<div class="line"> case dom::element_type::BOOL:</div>
|
|
<div class="line"> cout << bool(element) << endl;</div>
|
|
<div class="line"> break;</div>
|
|
<div class="line"> case dom::element_type::NULL_VALUE:</div>
|
|
<div class="line"> cout << "null" << endl;</div>
|
|
<div class="line"> break;</div>
|
|
<div class="line"> }</div>
|
|
<div class="line">}</div>
|
|
<div class="line"> </div>
|
|
<div class="line">void basics_treewalk_1() {</div>
|
|
<div class="line"> dom::parser parser;</div>
|
|
<div class="line"> print_json(parser.load("twitter.json"));</div>
|
|
<div class="line">}</div>
|
|
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md39"></a>
|
|
Reusing the parser for maximum efficiency</h1>
|
|
<p>If you're using simdjson to parse multiple documents, or in a loop, you should make a parser once and reuse it. The simdjson library will allocate and retain internal buffers between parses, keeping buffers hot in cache and keeping memory allocation and initialization to a minimum. In this manner, you can parse terabytes of JSON data without doing any new allocation.</p>
|
|
<div class="fragment"><div class="line"> {c++}</div>
|
|
<div class="line">dom::parser parser;</div>
|
|
<div class="line"> </div>
|
|
<div class="line">// This initializes buffers and a document big enough to handle this JSON.</div>
|
|
<div class="line">dom::element doc = parser.parse("[ true, false ]"_padded);</div>
|
|
<div class="line">cout << doc << endl;</div>
|
|
<div class="line"> </div>
|
|
<div class="line">// This reuses the existing buffers, and reuses and *overwrites* the old document</div>
|
|
<div class="line">doc = parser.parse("[1, 2, 3]"_padded);</div>
|
|
<div class="line">cout << doc << endl;</div>
|
|
<div class="line"> </div>
|
|
<div class="line">// This also reuses the existing buffers, and reuses and *overwrites* the old document</div>
|
|
<div class="line">dom::element doc2 = parser.parse("true"_padded);</div>
|
|
<div class="line">// Even if you keep the old reference around, doc and doc2 refer to the same document.</div>
|
|
<div class="line">cout << doc << endl;</div>
|
|
<div class="line">cout << doc2 << endl;</div>
|
|
</div><!-- fragment --><p>It's not just internal buffers though. The simdjson library reuses the document itself. The dom::element, dom::object and dom::array instances are <em>references</em> to the internal document. You are only <em>borrowing</em> the document from simdjson, which purposely reuses and overwrites it each time you call parse. This prevent wasteful and unnecessary memory allocation in 99% of cases where JSON is just read, used, and converted to native values or thrown away.</p>
|
|
<blockquote class="doxtable">
|
|
<p><b>You are only borrowing the document from the simdjson parser. Don't keep it long term!</b> </p>
|
|
</blockquote>
|
|
<p>This is key: don't keep the <code>document&</code>, <code>dom::element</code>, <code>dom::array</code>, <code>dom::object</code> or <code>string_view</code> objects you get back from the API. Convert them to C++ native values, structs and arrays that you own.</p>
|
|
<h1><a class="anchor" id="autotoc_md40"></a>
|
|
Server Loops: Long-Running Processes and Memory Capacity</h1>
|
|
<p>The simdjson library automatically expands its memory capacity when larger documents are parsed, so that you don't unexpectedly fail. In a short process that reads a bunch of files and then exits, this works pretty flawlessly.</p>
|
|
<p>Server loops, though, are long-running processes that will keep the parser around forever. This means that if you encounter a really, really large document, simdjson will not resize back down. The simdjson library lets you adjust your allocation strategy to prevent your server from growing without bound:</p>
|
|
<ul>
|
|
<li>You can set a <em>max capacity</em> when constructing a parser:</li>
|
|
</ul>
|
|
<div class="fragment"><div class="line">{c++}</div>
|
|
<div class="line"> dom::parser parser(1000*1000); // Never grow past documents > 1MB</div>
|
|
<div class="line"> for (web_request request : listen()) {</div>
|
|
<div class="line"> dom::element doc;</div>
|
|
<div class="line"> auto error = parser.parse(request.body).get(doc);</div>
|
|
<div class="line"> // If the document was above our limit, emit 413 = payload too large</div>
|
|
<div class="line"> if (error == CAPACITY) { request.respond(413); continue; }</div>
|
|
<div class="line"> // ...</div>
|
|
<div class="line"> }</div>
|
|
</div><!-- fragment --><p>This parser will grow normally as it encounters larger documents, but will never pass 1MB.</p>
|
|
<ul>
|
|
<li>You can set a <em>fixed capacity</em> that never grows, as well, which can be excellent for predictability and reliability, since simdjson will never call malloc after startup!</li>
|
|
</ul>
|
|
<div class="fragment"><div class="line">{c++}</div>
|
|
<div class="line"> dom::parser parser(0); // This parser will refuse to automatically grow capacity</div>
|
|
<div class="line"> auto error = parser.allocate(1000*1000); // This allocates enough capacity to handle documents <= 1MB</div>
|
|
<div class="line"> if (error) { cerr << error << endl; exit(1); }</div>
|
|
<div class="line"> </div>
|
|
<div class="line"> for (web_request request : listen()) {</div>
|
|
<div class="line"> dom::element doc;</div>
|
|
<div class="line"> error = parser.parse(request.body).get(doc);</div>
|
|
<div class="line"> // If the document was above our limit, emit 413 = payload too large</div>
|
|
<div class="line"> if (error == CAPACITY) { request.respond(413); continue; }</div>
|
|
<div class="line"> // ...</div>
|
|
<div class="line"> }</div>
|
|
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md41"></a>
|
|
Best Use of the DOM API</h1>
|
|
<p>The simdjson API provides access to the JSON DOM (document-object-model) content as a tree of <code>dom::element</code> instances, each representing an object, an array or an atomic type (null, true, false, number). These <code>dom::element</code> instances are lightweight objects (e.g., spanning 16 bytes) and it might be advantageous to pass them by value, as opposed to passing them by reference or by pointer.</p>
|
|
<h1><a class="anchor" id="autotoc_md42"></a>
|
|
Padding and Temporary Copies</h1>
|
|
<p>The simdjson function <code>parser.parse</code> reads data from a padded buffer, containing SIMDJSON_PADDING extra bytes added at the end. If you are passing a <code>padded_string</code> to <code>parser.parse</code> or loading the JSON directly from disk (<code>parser.load</code>), padding is automatically handled. When calling <code>parser.parse</code> on a pointer (e.g., <code>parser.parse(my_char_pointer, my_length_in_bytes)</code>) a temporary copy is made by default with adequate padding and you, again, do not need to be concerned with padding.</p>
|
|
<p>Some users may not be able use our <code>padded_string</code> class or to load the data directly from disk (<code>parser.load</code>). They may need to pass data pointers to the library. If these users wish to avoid temporary copies and corresponding temporary memory allocations, they may want to call <code>parser.parse</code> with the <code>realloc_if_needed</code> parameter set to false (e.g., <code>parser.parse(my_char_pointer, my_length_in_bytes, false)</code>). In such cases, they need to ensure that there are at least SIMDJSON_PADDING extra bytes at the end that can be safely accessed and read. They do not need to initialize the padded bytes to any value in particular. The following example is safe:</p>
|
|
<p>```C++ const char *json = R"({"key":"value"})"; const size_t json_len = std::strlen(json); std::unique_ptr<char[]> padded_json_copy{new char[json_len + SIMDJSON_PADDING]}; memcpy(padded_json_copy.get(), json, json_len); memset(padded_json_copy.get() + json_len, 0, SIMDJSON_PADDING); <a class="el" href="classsimdjson_1_1dom_1_1parser.html" title="A persistent document parser.">simdjson::dom::parser</a> parser; <a class="el" href="classsimdjson_1_1dom_1_1element.html" title="A JSON element.">simdjson::dom::element</a> element = parser.parse(padded_json_copy.get(), json_len, false); ````</p>
|
|
<p>Setting the <code>realloc_if_needed</code> parameter <code>false</code> in this manner may lead to better performance since copies are avoided, but it requires that the user takes more responsibilities: the simdjson library cannot verify that the input buffer was padded with SIMDJSON_PADDING extra bytes. </p>
|
|
</div></div><!-- contents -->
|
|
</div><!-- PageDoc -->
|
|
</div><!-- doc-content -->
|
|
<!-- start footer part -->
|
|
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
|
|
<ul>
|
|
<li class="footer">Generated by <a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.9.1 </li>
|
|
</ul>
|
|
</div>
|
|
</body>
|
|
</html>
|