documenting fatal errors

doc/basics.md

@@ -1531,7 +1531,20 @@ Some errors are recoverable:
 * You may get the error `simdjson::INCORRECT_TYPE` after trying to convert a value to an incorrect type: e.g., you expected a number and try to convert the value to a number, but it is an array.
 * You may query a key from an object, but the key is missing in which case you get the error `simdjson::NO_SUCH_FIELD`: e.g., you call `obj["myname"]` and the object does not have a key `"myname"`.
 
-Other errors (e.g., `simdjson::INCOMPLETE_ARRAY_OR_OBJECT`) may indicate a fatal error and often follow from the fact that the document is not valid JSON. In which case, it is no longer possible to continue accessing the document: calling the method `is_alive()` on the document instance returns false. All following accesses will keep returning the same fatal error (e.g., `simdjson::INCOMPLETE_ARRAY_OR_OBJECT`).
+Other errors (`simdjson::INCOMPLETE_ARRAY_OR_OBJECT` and `simdjson::TAPE_ERROR`)  indicate a fatal error and follow from the fact that the document is not valid JSON. These errors are not recoverable: you cannot continue. In which case, it is no longer safe to continue accessing the document: calling the method `is_alive()` on the document instance returns false. It is your responsability as a user to stop using the simdjson
+document after encountering these fatal errors. Consider the following example, after
+the fatal error, the document instance cannot be used. Observe how the JSON input is invalid.
+```cpp
+  simdjson::padded_string badjson = R"( { "make": "Toyota", "model": "Camry",  "year"})"_padded;
+  simdjson::ondemand::parser parser;
+  simdjson::ondemand::document doc;
+  auto errordoc = parser.iterate(badjson).get(doc);
+  // errordoc == simdjson::SUCCESS
+  simdjson::ondemand::value v;
+  auto error = doc.get_object()["year"].get(v);
+  // simdjson::is_fatal(error)) is true!
+  // doc.is_alive() is false
+```
 
 When you use the code without exceptions, it is your responsibility to check for error before using the
 result: if there is an error, the result value will not be valid and using it will caused undefined behavior. Most compilers should be able to help you if you activate the right

include/simdjson/error-inl.h

@@ -6,6 +6,11 @@
 #include <iostream>
 
 namespace simdjson {
+
+inline bool is_fatal(error_code error) noexcept {
+  return error == TAPE_ERROR || error == INCOMPLETE_ARRAY_OR_OBJECT;
+}
+
 namespace internal {
   // We store the error code so we can validate the error message is associated with the right code
   struct error_code_info {

include/simdjson/error.h

@@ -20,7 +20,7 @@ enum error_code {
   SUCCESS = 0,                ///< No error
   CAPACITY,                   ///< This parser can't support a document that big
   MEMALLOC,                   ///< Error allocating memory, most likely out of memory
-  TAPE_ERROR,                 ///< Something went wrong, this is a generic error
+  TAPE_ERROR,                 ///< Something went wrong, this is a generic error. Fatal/unrecoverable error.
   DEPTH_ERROR,                ///< Your document exceeds the user-specified depth limitation
   STRING_ERROR,               ///< Problem while parsing a string
   T_ATOM_ERROR,               ///< Problem while parsing an atom starting with the letter 't'
@@ -45,13 +45,21 @@ enum error_code {
   PARSER_IN_USE,              ///< parser is already in use.
   OUT_OF_ORDER_ITERATION,     ///< tried to iterate an array or object out of order (checked when SIMDJSON_DEVELOPMENT_CHECKS=1)
   INSUFFICIENT_PADDING,       ///< The JSON doesn't have enough padding for simdjson to safely parse it.
-  INCOMPLETE_ARRAY_OR_OBJECT, ///< The document ends early.
+  INCOMPLETE_ARRAY_OR_OBJECT, ///< The document ends early. Fatal/unrecoverable error.
   SCALAR_DOCUMENT_AS_VALUE,   ///< A scalar document is treated as a value.
   OUT_OF_BOUNDS,              ///< Attempted to access location outside of document.
   TRAILING_CONTENT,           ///< Unexpected trailing content in the JSON input
   NUM_ERROR_CODES
 };
 
+/**
+ * Some errors are fatal and invalidate the document. This function returns true if the
+ * error is fatal. It returns true for TAPE_ERROR and INCOMPLETE_ARRAY_OR_OBJECT.
+ * Once a fatal error is encountered, the on-demand document is no longer valid and
+ * processing should stop.
+ */
+ inline bool is_fatal(error_code error) noexcept;
+
 /**
  * It is the convention throughout the code that  the macro SIMDJSON_DEVELOPMENT_CHECKS determines whether
  * we check for OUT_OF_ORDER_ITERATION. The logic behind it is that these errors only occurs when the code

include/simdjson/generic/ondemand/value_iterator.h

@@ -125,7 +125,8 @@ public:
    *
    * @returns Whether the object had any fields (returns false for empty).
    * @error INCOMPLETE_ARRAY_OR_OBJECT If there are no more tokens (implying the *parent*
-   *        array or object is incomplete).
+   *        array or object is incomplete). An INCOMPLETE_ARRAY_OR_OBJECT is an unrecoverable error that
+   *        invalidates the document.
    */
   simdjson_warn_unused simdjson_inline simdjson_result<bool> started_object() noexcept;
   /**
@@ -135,7 +136,8 @@ public:
    *
    * @returns Whether the object had any fields (returns false for empty).
    * @error INCOMPLETE_ARRAY_OR_OBJECT If there are no more tokens (implying the *parent*
-   *        array or object is incomplete).
+   *        array or object is incomplete). An INCOMPLETE_ARRAY_OR_OBJECT is an unrecoverable error that
+   *        invalidates the document.
    */
   simdjson_warn_unused simdjson_inline simdjson_result<bool> started_root_object() noexcept;
 
@@ -257,7 +259,8 @@ public:
    *
    * @returns Whether the array had any elements (returns false for empty).
    * @error INCOMPLETE_ARRAY_OR_OBJECT If there are no more tokens (implying the *parent*
-   *        array or object is incomplete).
+   *        array or object is incomplete). An INCOMPLETE_ARRAY_OR_OBJECT is an unrecoverable error that
+   *        invalidates the document.
    */
   simdjson_warn_unused simdjson_inline simdjson_result<bool> started_array() noexcept;
   /**
@@ -267,7 +270,8 @@ public:
    *
    * @returns Whether the array had any elements (returns false for empty).
    * @error INCOMPLETE_ARRAY_OR_OBJECT If there are no more tokens (implying the *parent*
-   *        array or object is incomplete).
+   *        array or object is incomplete). An INCOMPLETE_ARRAY_OR_OBJECT is an unrecoverable error that
+   *        invalidates the document.
    */
   simdjson_warn_unused simdjson_inline simdjson_result<bool> started_root_array() noexcept;
 

singleheader/simdjson.cpp

@@ -4696,7 +4696,7 @@ namespace internal {
     { SUCCESS, "SUCCESS: No error" },
     { CAPACITY, "CAPACITY: This parser can't support a document that big" },
     { MEMALLOC, "MEMALLOC: Error allocating memory, we're most likely out of memory" },
-    { TAPE_ERROR, "TAPE_ERROR: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc." },
+    { TAPE_ERROR, "TAPE_ERROR: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc. This is a fatal and unrecoverable error." },
     { DEPTH_ERROR, "DEPTH_ERROR: The JSON document was too deep (too many nested objects and arrays)" },
     { STRING_ERROR, "STRING_ERROR: Problem while parsing a string" },
     { T_ATOM_ERROR, "T_ATOM_ERROR: Problem while parsing an atom starting with the letter 't'" },

src/internal/error_tables.cpp

@@ -11,7 +11,7 @@ namespace internal {
     { SUCCESS, "SUCCESS: No error" },
     { CAPACITY, "CAPACITY: This parser can't support a document that big" },
     { MEMALLOC, "MEMALLOC: Error allocating memory, we're most likely out of memory" },
-    { TAPE_ERROR, "TAPE_ERROR: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc." },
+    { TAPE_ERROR, "TAPE_ERROR: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc.  This is a fatal and unrecoverable error." },
     { DEPTH_ERROR, "DEPTH_ERROR: The JSON document was too deep (too many nested objects and arrays)" },
     { STRING_ERROR, "STRING_ERROR: Problem while parsing a string" },
     { T_ATOM_ERROR, "T_ATOM_ERROR: Problem while parsing an atom starting with the letter 't'" },
@@ -36,7 +36,7 @@ namespace internal {
     { PARSER_IN_USE, "PARSER_IN_USE: Cannot parse a new document while a document is still in use." },
     { OUT_OF_ORDER_ITERATION, "OUT_OF_ORDER_ITERATION: Objects and arrays can only be iterated when they are first encountered." },
     { INSUFFICIENT_PADDING, "INSUFFICIENT_PADDING: simdjson requires the input JSON string to have at least SIMDJSON_PADDING extra bytes allocated, beyond the string's length. Consider using the simdjson::padded_string class if needed." },
-    { INCOMPLETE_ARRAY_OR_OBJECT, "INCOMPLETE_ARRAY_OR_OBJECT: JSON document ended early in the middle of an object or array." },
+    { INCOMPLETE_ARRAY_OR_OBJECT, "INCOMPLETE_ARRAY_OR_OBJECT: JSON document ended early in the middle of an object or array. This is a fatal and unrecoverable error." },
     { SCALAR_DOCUMENT_AS_VALUE, "SCALAR_DOCUMENT_AS_VALUE: A JSON document made of a scalar (number, Boolean, null or string) is treated as a value. Use get_bool(), get_double(), etc. on the document instead. "},
     { OUT_OF_BOUNDS, "OUT_OF_BOUNDS: Attempt to access location outside of document."},
     { TRAILING_CONTENT, "TRAILING_CONTENT: Unexpected trailing content in the JSON input."}

tests/ondemand/ondemand_readme_examples.cpp

@@ -10,6 +10,20 @@ using namespace std;
 using namespace simdjson;
 using error_code = simdjson::error_code;
 
+bool fatal_error() {
+  TEST_START();
+  padded_string badjson = R"( { "make": "Toyota", "model": "Camry",  "year"})"_padded;
+  ondemand::parser parser;
+  ondemand::document doc;
+  auto errordoc = parser.iterate(badjson).get(doc);
+  if(errordoc != simdjson::SUCCESS) { return false; }
+  simdjson::ondemand::value v;
+  auto error = doc.get_object()["year"].get(v);
+  ASSERT_TRUE(simdjson::is_fatal(error));
+  ASSERT_FALSE(doc.is_alive());
+  TEST_SUCCEED();
+}
+
 bool simplepad() {
   std::string json = "[1]";
   ondemand::parser parser;
@@ -22,14 +36,14 @@ bool string1() {
   const char * data = "my data"; // 7 bytes
   simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer
   std::cout << my_padded_data << std::endl;
-  return true;
+  TEST_SUCCEED();
 }
 
 bool string2() {
   std::string data = "my data";
   simdjson::padded_string my_padded_data(data); // copies to a padded buffer
   std::cout << my_padded_data << std::endl;
-  return true;
+  TEST_SUCCEED();
 }
 
 bool to_string_example_no_except() {
@@ -1893,6 +1907,7 @@ bool value_raw_json_object() {
 #endif
 bool run() {
   return true
+    && fatal_error()
 #if SIMDJSON_EXCEPTIONS
 #if SIMDJSON_CPLUSPLUS17
     && big_int_array()