A JSON fragment iterator. More...

#include <parser.h>

Public Member Functions
	parser (size_t max_capacity=SIMDJSON_MAXSIZE_BYTES) noexcept
	Create a JSON parser.

	parser (parser &&other) noexcept=default

simdjson_inline	parser (const parser &other)=delete

simdjson_inline parser &	operator= (const parser &other)=delete

simdjson_inline parser &	operator= (parser &&other) noexcept=default

	~parser () noexcept=default
	Deallocate the JSON parser.

simdjson_warn_unused simdjson_result< document >	iterate (padded_string_view json) &noexcept
	Start iterating an on-demand JSON document.

simdjson_warn_unused simdjson_result< document >	iterate (const char *json, size_t len, size_t capacity) &noexcept

simdjson_warn_unused simdjson_result< document >	iterate (const uint8_t *json, size_t len, size_t capacity) &noexcept

simdjson_warn_unused simdjson_result< document >	iterate (std::string_view json, size_t capacity) &noexcept

simdjson_warn_unused simdjson_result< document >	iterate (const std::string &json) &noexcept

simdjson_warn_unused simdjson_result< document >	iterate (std::string &json) &noexcept

simdjson_warn_unused simdjson_result< document >	iterate (const simdjson_result< padded_string > &json) &noexcept

simdjson_warn_unused simdjson_result< document >	iterate (const simdjson_result< padded_string_view > &json) &noexcept

simdjson_warn_unused simdjson_result< document >	iterate (padded_string &&json) &noexcept=delete

simdjson_result< document_stream >	iterate_many (const uint8_t *buf, size_t len, size_t batch_size=DEFAULT_BATCH_SIZE, bool allow_comma_separated=false) noexcept
	Parse a buffer containing many JSON documents.

simdjson_result< document_stream >	iterate_many (padded_string_view json, size_t batch_size=DEFAULT_BATCH_SIZE, bool allow_comma_separated=false) noexcept

simdjson_result< document_stream >	iterate_many (const char *buf, size_t len, size_t batch_size=DEFAULT_BATCH_SIZE, bool allow_comma_separated=false) noexcept

simdjson_result< document_stream >	iterate_many (const std::string &s, size_t batch_size=DEFAULT_BATCH_SIZE, bool allow_comma_separated=false) noexcept

simdjson_result< document_stream >	iterate_many (std::string &s, size_t batch_size=DEFAULT_BATCH_SIZE, bool allow_comma_separated=false) noexcept

simdjson_result< document_stream >	iterate_many (const padded_string &s, size_t batch_size=DEFAULT_BATCH_SIZE, bool allow_comma_separated=false) noexcept

simdjson_pure simdjson_inline size_t	capacity () const noexcept
	The capacity of this parser (the largest document it can process).

simdjson_pure simdjson_inline size_t	max_capacity () const noexcept
	The maximum capacity of this parser (the largest document it is allowed to process).

simdjson_inline void	set_max_capacity (size_t max_capacity) noexcept

simdjson_pure simdjson_inline size_t	max_depth () const noexcept
	The maximum depth of this parser (the most deeply nested objects and arrays it can process).

simdjson_warn_unused error_code	allocate (size_t capacity, size_t max_depth=DEFAULT_MAX_DEPTH) noexcept
	Ensure this parser has enough memory to process JSON documents up to `capacity` bytes in length and `max_depth` depth.

simdjson_inline simdjson_result< std::string_view >	unescape (raw_json_string in, uint8_t *&dst, bool allow_replacement=false) const noexcept
	Unescape this JSON string, replacing \ with \, with newline, etc.

simdjson_inline simdjson_result< std::string_view >	unescape_wobbly (raw_json_string in, uint8_t *&dst) const noexcept
	Unescape this JSON string, replacing \ with \, with newline, etc.

Static Public Member Functions
static simdjson_inline simdjson_warn_unused ondemand::parser &	get_parser ()
	Get a unique parser instance corresponding to the current thread.

static simdjson_inline bool	release_parser ()
	Release the parser instance initialized by get_parser() and all the associated resources (memory).

Public Attributes
bool	threaded {false}
	When SIMDJSON_THREADS_ENABLED is not defined, the parser instance cannot use threads.

Detailed Description

A JSON fragment iterator.

This holds the actual iterator as well as the buffer for writing strings.

Definition at line 37 of file parser.h.

Constructor & Destructor Documentation

◆ parser()

simdjson_inline simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::parser ( size_t max_capacity = SIMDJSON_MAXSIZE_BYTES )

inlineexplicitnoexcept

Create a JSON parser.

The new parser will have zero capacity.

Definition at line 20 of file parser-inl.h.

Member Function Documentation

◆ allocate()

simdjson_warn_unused simdjson_inline error_code simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::allocate	(	size_t	capacity,
		size_t	max_depth = `DEFAULT_MAX_DEPTH`
	)

noexcept

Ensure this parser has enough memory to process JSON documents up to capacity bytes in length and max_depth depth.

The max_depth parameter is only relevant when the macro SIMDJSON_DEVELOPMENT_CHECKS is set to true. The document's instance current_depth() method should be used to monitor the parsing depth and limit it if desired.

Parameters

capacity	The new capacity.
max_depth	The new max_depth. Defaults to DEFAULT_MAX_DEPTH.

Returns: The error, if there is one.

Definition at line 24 of file parser-inl.h.

◆ capacity()

simdjson_pure simdjson_inline size_t simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::capacity ( ) const

noexcept

The capacity of this parser (the largest document it can process).

Definition at line 164 of file parser-inl.h.

◆ get_parser()

simdjson_inline simdjson_warn_unused ondemand::parser & simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::get_parser ( )

static

Get a unique parser instance corresponding to the current thread.

This instance can be safely used within the current thread, but it should not be passed to other threads.

A parser should only be used for one document at a time.

Our simdjson::from functions use this parser instance.

You can free the related parser by calling release_parser().

Definition at line 198 of file parser-inl.h.

◆ iterate() [1/8]

simdjson_warn_unused simdjson_inline simdjson_result< document > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate	(	const char *	json,
		size_t	len,
		size_t	capacity
	)		&

noexcept

Definition at line 87 of file parser-inl.h.

◆ iterate() [2/8]

simdjson_warn_unused simdjson_inline simdjson_result< document > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate ( const simdjson_result< padded_string > & json ) &

noexcept

Definition at line 114 of file parser-inl.h.

◆ iterate() [3/8]

simdjson_warn_unused simdjson_inline simdjson_result< document > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate ( const simdjson_result< padded_string_view > & json ) &

noexcept

Definition at line 107 of file parser-inl.h.

◆ iterate() [4/8]

simdjson_warn_unused simdjson_inline simdjson_result< document > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate ( const std::string & json ) &

noexcept

Definition at line 103 of file parser-inl.h.

◆ iterate() [5/8]

simdjson_warn_unused simdjson_inline simdjson_result< document > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate	(	const uint8_t *	json,
		size_t	len,
		size_t	capacity
	)		&

noexcept

Definition at line 91 of file parser-inl.h.

◆ iterate() [6/8]

simdjson_warn_unused simdjson_inline simdjson_result< document > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate ( padded_string_view json ) &

noexcept

Start iterating an on-demand JSON document.

ondemand::parser parser; document doc = parser.iterate(json);

It is expected that the content is a valid UTF-8 file, containing a valid JSON document. Otherwise the iterate method may return an error. In particular, the whole input should be valid: we do not attempt to tolerate incorrect content either before or after a JSON document. If there is a UTF-8 BOM, the parser skips it.

IMPORTANT: Validate what you use

Calling iterate on an invalid JSON document may not immediately trigger an error. The call to iterate does not parse and validate the whole document.

IMPORTANT: Buffer Lifetime

Because parsing is done while you iterate, you must keep the JSON buffer around at least as long as the document iteration.

IMPORTANT: Document Lifetime

Only one iteration at a time can happen per parser, and the parser must be kept alive during iteration to ensure intermediate buffers can be accessed. Any document must be destroyed before you call parse() again or destroy the parser.

REQUIRED: Buffer Padding

The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what those bytes are initialized to, as long as they are allocated. These bytes will be read: if you using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the SIMDJSON_PADDING bytes to avoid runtime warnings.

std::string references

If you pass a mutable std::string reference (std::string&), the parser will seek to extend its capacity to SIMDJSON_PADDING bytes beyond the end of the string.

Whenever you pass an std::string reference, the parser will access the bytes beyond the end of the string but before the end of the allocated memory (std::string::capacity()). If you are using a sanitizer that checks for reading uninitialized bytes or std::string's container-overflow checks, you may encounter sanitizer warnings. You can safely ignore these warnings. Or you can call simdjson::pad(std::string&) to pad the string with SIMDJSON_PADDING spaces: this function returns a simdjson::padding_string_view which can be be passed to the parser's iterate function:

std::string json = R"({ "foo": 1 } { "foo": 2 } { "foo": 3 } )"; document doc = parser.iterate(simdjson::pad(json));

Parameters

json	The JSON to parse.
len	The length of the JSON.
capacity	The number of bytes allocated in the JSON (must be at least len+SIMDJSON_PADDING).

Returns

The document, or an error:

INSUFFICIENT_PADDING if the input has less than SIMDJSON_PADDING extra bytes.
MEMALLOC if realloc_if_needed the parser does not have enough capacity, and memory allocation fails.
EMPTY if the document is all whitespace.
UTF8_ERROR if the document is not valid UTF-8.
UNESCAPED_CHARS if a string contains control characters that must be escaped
UNCLOSED_STRING if there is an unclosed string in the document.

Definition at line 51 of file parser-inl.h.

◆ iterate() [7/8]

simdjson_warn_unused simdjson_inline simdjson_result< document > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate ( std::string & json ) &

noexcept

Definition at line 99 of file parser-inl.h.

◆ iterate() [8/8]

simdjson_warn_unused simdjson_inline simdjson_result< document > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate	(	std::string_view	json,
		size_t	capacity
	)		&

noexcept

Definition at line 95 of file parser-inl.h.

◆ iterate_many() [1/6]

simdjson_result< document_stream > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate_many	(	const char *	buf,
		size_t	len,
		size_t	batch_size = `DEFAULT_BATCH_SIZE`,
		bool	allow_comma_separated = `false`
	)

inlinenoexcept

Definition at line 147 of file parser-inl.h.

◆ iterate_many() [2/6]

simdjson_result< document_stream > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate_many	(	const padded_string &	s,
		size_t	batch_size = `DEFAULT_BATCH_SIZE`,
		bool	allow_comma_separated = `false`
	)

inlinenoexcept

Definition at line 155 of file parser-inl.h.

◆ iterate_many() [3/6]

simdjson_result< document_stream > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate_many	(	const std::string &	s,
		size_t	batch_size = `DEFAULT_BATCH_SIZE`,
		bool	allow_comma_separated = `false`
	)

inlinenoexcept

Definition at line 158 of file parser-inl.h.

◆ iterate_many() [4/6]

simdjson_result< document_stream > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate_many	(	const uint8_t *	buf,
		size_t	len,
		size_t	batch_size = `DEFAULT_BATCH_SIZE`,
		bool	allow_comma_separated = `false`
	)

inlinenoexcept

Parse a buffer containing many JSON documents.

auto json = R"({ "foo": 1 } { "foo": 2 } { "foo": 3 } )"_padded; ondemand::parser parser; ondemand::document_stream docs = parser.iterate_many(json); for (auto & doc : docs) { std::cout << doc["foo"] << std::endl; } // Prints 1 2 3

No copy of the input buffer is made.

The function is lazy: it may be that no more than one JSON document at a time is parsed.

The caller is responsabile to ensure that the input string data remains unchanged and is not deleted during the loop.

Format

The buffer must contain a series of one or more JSON documents, concatenated into a single buffer, separated by ASCII whitespace. It effectively parses until it has a fully valid document, then starts parsing the next document at that point. (It does this with more parallelism and lookahead than you might think, though.)

documents that consist of an object or array may omit the whitespace between them, concatenating with no separator. Documents that consist of a single primitive (i.e. documents that are not arrays or objects) MUST be separated with ASCII whitespace.

The characters inside a JSON document, and between JSON documents, must be valid Unicode (UTF-8). If there is a UTF-8 BOM, the parser skips it.

The documents must not exceed batch_size bytes (by default 1MB) or they will fail to parse. Setting batch_size to excessively large or excessively small values may impact negatively the performance.

Threads

When compiled with SIMDJSON_THREADS_ENABLED, this method will use a single thread under the hood to do some lookahead.

REQUIRED: Buffer Padding

The buffer must have at least SIMDJSON_PADDING extra allocated bytes. It does not matter what those bytes are initialized to, as long as they are allocated. These bytes will be read: if you using a sanitizer that verifies that no uninitialized byte is read, then you should initialize the SIMDJSON_PADDING bytes to avoid runtime warnings.

This is checked automatically with all iterate_many function calls, except for the two that take pointers (const char* or const uint8_t*).

Threads

Parser Capacity

If the parser's current capacity is less than batch_size, it will allocate enough capacity to handle it (up to max_capacity).

Parameters

buf	The concatenated JSON to parse.
len	The length of the concatenated JSON.
batch_size	The batch size to use. MUST be larger than the largest document. The sweet spot is cache-related: small enough to fit in cache, yet big enough to parse as many documents as possible in one tight loop. Defaults to 10MB, which has been a reasonable sweet spot in our tests.
allow_comma_separated	(defaults on false) This allows a mode where the documents are separated by commas instead of whitespace. It comes with a performance penalty because the entire document is indexed at once (and the document must be less than 4 GB), and there is no multithreading. In this mode, the batch_size parameter is effectively ignored, as it is set to at least the document size.

Returns

The stream, or an error. An empty input will yield 0 documents rather than an EMPTY error. Errors:

MEMALLOC if the parser does not have enough capacity and memory allocation fails
CAPACITY if the parser does not have enough capacity and batch_size > max_capacity.
other json errors if parsing fails. You should not rely on these errors to always the same for the same document: they may vary under runtime dispatch (so they may vary depending on your system and hardware).

Definition at line 136 of file parser-inl.h.

◆ iterate_many() [5/6]

simdjson_result< document_stream > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate_many	(	padded_string_view	json,
		size_t	batch_size = `DEFAULT_BATCH_SIZE`,
		bool	allow_comma_separated = `false`
	)

inlinenoexcept

Definition at line 151 of file parser-inl.h.

◆ iterate_many() [6/6]

simdjson_result< document_stream > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::iterate_many	(	std::string &	s,
		size_t	batch_size = `DEFAULT_BATCH_SIZE`,
		bool	allow_comma_separated = `false`
	)

inlinenoexcept

Definition at line 161 of file parser-inl.h.

◆ max_capacity()

simdjson_pure simdjson_inline size_t simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::max_capacity ( ) const

noexcept

The maximum capacity of this parser (the largest document it is allowed to process).

Definition at line 167 of file parser-inl.h.

◆ max_depth()

simdjson_pure simdjson_inline size_t simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::max_depth ( ) const

noexcept

The maximum depth of this parser (the most deeply nested objects and arrays it can process).

This parameter is only relevant when the macro SIMDJSON_DEVELOPMENT_CHECKS is set to true. The document's instance current_depth() method should be used to monitor the parsing depth and limit it if desired.

Definition at line 170 of file parser-inl.h.

◆ release_parser()

simdjson_inline bool simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::release_parser ( )

static

Release the parser instance initialized by get_parser() and all the associated resources (memory).

Returns true if a parser instance was released.

Definition at line 202 of file parser-inl.h.

◆ set_max_capacity()

simdjson_inline void simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::set_max_capacity ( size_t max_capacity )

noexcept

Definition at line 174 of file parser-inl.h.

◆ unescape()

simdjson_inline simdjson_warn_unused simdjson_result< std::string_view > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::unescape	(	raw_json_string	in,
		uint8_t *&	dst,
		bool	allow_replacement = `false`
	)		const

noexcept

Unescape this JSON string, replacing \ with \,
with newline, etc.

to a user-provided buffer. The result must be valid UTF-8. The provided pointer is advanced to the end of the string by reference, and a string_view instance is returned. You can ensure that your buffer is large enough by allocating a block of memory at least as large as the input JSON plus SIMDJSON_PADDING and then unescape all strings to this one buffer.

This unescape function is a low-level function. If you want a more user-friendly approach, you should avoid raw_json_string instances (e.g., by calling unescaped_key() instead of key() or get_string() instead of get_raw_json_string()).

IMPORTANT: string_view lifetime

The string_view is only valid as long as the bytes in dst.

Parameters

raw_json_string	input
dst	A pointer to a buffer at least large enough to write this string as well as an additional SIMDJSON_PADDING bytes.
allow_replacement	Whether we allow a replacement if the input string contains unmatched surrogate pairs.

Returns: A string_view pointing at the unescaped string in dst @error STRING_ERROR if escapes are incorrect.

Definition at line 182 of file parser-inl.h.

◆ unescape_wobbly()

simdjson_inline simdjson_warn_unused simdjson_result< std::string_view > simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::unescape_wobbly	(	raw_json_string	in,
		uint8_t *&	dst
	)		const

noexcept

Unescape this JSON string, replacing \ with \,
with newline, etc.

to a user-provided buffer. The result may not be valid UTF-8. See https://simonsapin.github.io/wtf-8/ The provided pointer is advanced to the end of the string by reference, and a string_view instance is returned. You can ensure that your buffer is large enough by allocating a block of memory at least as large as the input JSON plus SIMDJSON_PADDING and then unescape all strings to this one buffer.

This unescape function is a low-level function. If you want a more user-friendly approach, you should avoid raw_json_string instances (e.g., by calling unescaped_key() instead of key() or get_string() instead of get_raw_json_string()).

IMPORTANT: string_view lifetime

The string_view is only valid as long as the bytes in dst.

Parameters

raw_json_string	input
dst	A pointer to a buffer at least large enough to write this string as well as an additional SIMDJSON_PADDING bytes.

Returns: A string_view pointing at the unescaped string in dst @error STRING_ERROR if escapes are incorrect.

Definition at line 190 of file parser-inl.h.

Member Data Documentation

◆ threaded

bool simdjson::SIMDJSON_IMPLEMENTATION::ondemand::parser::threaded {false}

When SIMDJSON_THREADS_ENABLED is not defined, the parser instance cannot use threads.

Definition at line 311 of file parser.h.

The documentation for this class was generated from the following files:

include/simdjson/generic/ondemand/parser.h
include/simdjson/generic/ondemand/parser-inl.h

Public Member Functions

Static Public Member Functions

Public Attributes

Detailed Description

Constructor & Destructor Documentation

◆ parser()

Member Function Documentation

◆ allocate()

◆ capacity()

◆ get_parser()

◆ iterate() [1/8]

◆ iterate() [2/8]

◆ iterate() [3/8]

◆ iterate() [4/8]

◆ iterate() [5/8]

◆ iterate() [6/8]

IMPORTANT: Validate what you use

IMPORTANT: Buffer Lifetime

IMPORTANT: Document Lifetime

REQUIRED: Buffer Padding

std::string references

◆ iterate() [7/8]

◆ iterate() [8/8]

◆ iterate_many() [1/6]

◆ iterate_many() [2/6]

◆ iterate_many() [3/6]

◆ iterate_many() [4/6]

Format

Threads

REQUIRED: Buffer Padding

Threads

Parser Capacity

◆ iterate_many() [5/6]

◆ iterate_many() [6/6]

◆ max_capacity()

◆ max_depth()

◆ release_parser()

◆ set_max_capacity()

◆ unescape()

IMPORTANT: string_view lifetime

◆ unescape_wobbly()

IMPORTANT: string_view lifetime

Member Data Documentation

◆ threaded