mirror of
https://github.com/simdjson/simdjson
synced 2026-06-08 17:27:07 +00:00
c806e955c4
* Initial work on JSON builder * moving the files back to ondemand for now. * tweak * more later * update * minor edits * dropping vs arm (missing support) * adding tests. we still specialized write_string_escaped * tweaking * fix typo * tweaking the approach * minor fix * missing store * another missing store * Attempt at fixing failing serialization tests. (#2292) * Fixing appeand_float typo (#2294) * applying a couple of fixes * updating single header * fix for pre C++17 if constexpr * Fixing unused argument problem and updating the singleheader file * various pedantic fixes * Sketch of builder * reordering. * simplify * Adding draft of static reflection based deserialization * Updating simdjson singleheader * patching the automated deserialization. * automated * Adding support for smart pointers of user defined types. * Adding specialization for smart pointers for basic types. I think it is highly likely that this can be done in a more generic way. * Referncing a later version of rapidjson that fixed the issue related with assignment attempt of a const variable for GenericStringRef class. * guarding the tests * adding documentation for string_builder * saving * rename to 'append' * saving * non-functional benchmarks (#2342) * non-functional benchmarks * Fix typo * various fixes * tweaking --------- Co-authored-by: Daniel Lemire <dlemire@lemire.me> Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com> * tuning * various minor fixes * minor tweak * minor simplification * updating amal * adding a cast * update * fancy casting * removing dead code * Pushing latest changes. CITM benchmark is still not working. * Still not working, but now I am getting only 10 errors. * add static reflection benchmark to 'large random' benchmark and allows (#2349) deserialization (with static reflection) from objects and arrays. Co-authored-by: Daniel Lemire <dlemire@lemire.me> * Removing std::map from CitmCatalog definition, since that is not currently supported. * Added free to rust bench, segfault is still happening.. * The syntax changed: ^E became ^^E. (#2350) * The syntax changed: ^E became ^^E. * guarding --------- Co-authored-by: Daniel Lemire <dlemire@lemire.me> * Adding support for string_view_keyed_map types. * Adding concepts as a conditional include. * updating single-header * Adding concepts to ondemand deps * rust benchmark is finally working * Fixing small typo in docs. * adding docker config and instructions so that our users can test the static reflection (#2358) * adding docker config and instructions so that our users can test the static reflection * completing the instructions * pruning white spaces --------- Co-authored-by: Daniel Lemire <dlemire@lemire.me> * minor optimizations on the JSON builder branch * avoiding undef behaviour * saving * somewhat nicer builder * make it possible to run just one benchmark * adding linux perf * fixing minor issue * updating swar * Adding real world compilation benchmark (#2379) * Adding compilation benchmark for json parsing with and without reflection * Moving it to the benchmark folder, also reducing a bit the number of iterations. * Removing script from root folder. * Reducing number of iterations * Update benchmark/benchmark_reflection_usage_compilation.sh Co-authored-by: Daniel Lemire <daniel@lemire.me> * Update benchmark/benchmark_reflection_usage_compilation.sh Co-authored-by: Daniel Lemire <daniel@lemire.me> * Update benchmark/benchmark_reflection_usage_compilation.sh Co-authored-by: Daniel Lemire <daniel@lemire.me> * Making the script more customizable and also test whether the compiler being used supports reflection before actually running the benchmark --------- Co-authored-by: Daniel Lemire <daniel@lemire.me> * Using define_static_string from https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p3491r2.html (#2389) * Applying changes needed after latest reflection paper updates. * Working, but no template for yet. * Updating single-header to incldue the use of define_static_string. * copying over master --------- Co-authored-by: Daniel Lemire <dlemire@lemire.me> Co-authored-by: Francisco Geiman Thiesen <franciscogthiesen@gmail.com>
248 lines
8.3 KiB
C++
248 lines
8.3 KiB
C++
#ifndef SIMDJSON_IMPLEMENTATION_H
|
|
#define SIMDJSON_IMPLEMENTATION_H
|
|
|
|
#include "simdjson/internal/atomic_ptr.h"
|
|
#include "simdjson/internal/dom_parser_implementation.h"
|
|
|
|
#include <memory>
|
|
|
|
namespace simdjson {
|
|
|
|
/**
|
|
* Validate the UTF-8 string.
|
|
*
|
|
* @param buf the string to validate.
|
|
* @param len the length of the string in bytes.
|
|
* @return true if the string is valid UTF-8.
|
|
*/
|
|
simdjson_warn_unused bool validate_utf8(const char * buf, size_t len) noexcept;
|
|
/**
|
|
* Validate the UTF-8 string.
|
|
*
|
|
* @param sv the string_view to validate.
|
|
* @return true if the string is valid UTF-8.
|
|
*/
|
|
simdjson_inline simdjson_warn_unused bool validate_utf8(const std::string_view sv) noexcept {
|
|
return validate_utf8(sv.data(), sv.size());
|
|
}
|
|
|
|
/**
|
|
* Write the string to the output buffer while escaping double-quote, backlash and ascii control characters.
|
|
*
|
|
* @param input the string_view to escape
|
|
* @param out output buffer (for escaped string): to be safe, it should have 6 * input.size() allocated bytes.
|
|
* @return number of bytes written
|
|
*/
|
|
simdjson_warn_unused size_t write_string_escaped(const std::string_view input, char *out) noexcept;
|
|
|
|
/**
|
|
* Validate the UTF-8 string.
|
|
*
|
|
* @param p the string to validate.
|
|
* @return true if the string is valid UTF-8.
|
|
*/
|
|
simdjson_inline simdjson_warn_unused bool validate_utf8(const std::string& s) noexcept {
|
|
return validate_utf8(s.data(), s.size());
|
|
}
|
|
|
|
/**
|
|
* An implementation of simdjson for a particular CPU architecture.
|
|
*
|
|
* Also used to maintain the currently active implementation. The active implementation is
|
|
* automatically initialized on first use to the most advanced implementation supported by the host.
|
|
*/
|
|
class implementation {
|
|
public:
|
|
|
|
/**
|
|
* The name of this implementation.
|
|
*
|
|
* const implementation *impl = simdjson::get_active_implementation();
|
|
* cout << "simdjson is optimized for " << impl->name() << "(" << impl->description() << ")" << endl;
|
|
*
|
|
* @return the name of the implementation, e.g. "haswell", "westmere", "arm64".
|
|
*/
|
|
virtual std::string name() const { return std::string(_name); }
|
|
|
|
/**
|
|
* The description of this implementation.
|
|
*
|
|
* const implementation *impl = simdjson::get_active_implementation();
|
|
* cout << "simdjson is optimized for " << impl->name() << "(" << impl->description() << ")" << endl;
|
|
*
|
|
* @return the description of the implementation, e.g. "Intel/AMD AVX2", "Intel/AMD SSE4.2", "ARM NEON".
|
|
*/
|
|
virtual std::string description() const { return std::string(_description); }
|
|
|
|
/**
|
|
* The instruction sets this implementation is compiled against
|
|
* and the current CPU match. This function may poll the current CPU/system
|
|
* and should therefore not be called too often if performance is a concern.
|
|
*
|
|
* @return true if the implementation can be safely used on the current system (determined at runtime).
|
|
*/
|
|
bool supported_by_runtime_system() const;
|
|
|
|
/**
|
|
* @private For internal implementation use
|
|
*
|
|
* The instruction sets this implementation is compiled against.
|
|
*
|
|
* @return a mask of all required `internal::instruction_set::` values.
|
|
*/
|
|
virtual uint32_t required_instruction_sets() const { return _required_instruction_sets; }
|
|
|
|
/**
|
|
* @private For internal implementation use
|
|
*
|
|
* const implementation *impl = simdjson::get_active_implementation();
|
|
* cout << "simdjson is optimized for " << impl->name() << "(" << impl->description() << ")" << endl;
|
|
*
|
|
* @param capacity The largest document that will be passed to the parser.
|
|
* @param max_depth The maximum JSON object/array nesting this parser is expected to handle.
|
|
* @param dst The place to put the resulting parser implementation.
|
|
* @return the error code, or SUCCESS if there was no error.
|
|
*/
|
|
virtual error_code create_dom_parser_implementation(
|
|
size_t capacity,
|
|
size_t max_depth,
|
|
std::unique_ptr<internal::dom_parser_implementation> &dst
|
|
) const noexcept = 0;
|
|
|
|
/**
|
|
* @private For internal implementation use
|
|
*
|
|
* Minify the input string assuming that it represents a JSON string, does not parse or validate.
|
|
*
|
|
* Overridden by each implementation.
|
|
*
|
|
* @param buf the json document to minify.
|
|
* @param len the length of the json document.
|
|
* @param dst the buffer to write the minified document to. *MUST* be allocated up to len + SIMDJSON_PADDING bytes.
|
|
* @param dst_len the number of bytes written. Output only.
|
|
* @return the error code, or SUCCESS if there was no error.
|
|
*/
|
|
simdjson_warn_unused virtual error_code minify(const uint8_t *buf, size_t len, uint8_t *dst, size_t &dst_len) const noexcept = 0;
|
|
|
|
|
|
/**
|
|
* Validate the UTF-8 string.
|
|
*
|
|
* Overridden by each implementation.
|
|
*
|
|
* @param buf the string to validate.
|
|
* @param len the length of the string in bytes.
|
|
* @return true if and only if the string is valid UTF-8.
|
|
*/
|
|
simdjson_warn_unused virtual bool validate_utf8(const char *buf, size_t len) const noexcept = 0;
|
|
|
|
/**
|
|
* Write the string to the output buffer while escaping double-quote, backlash and ascii control characters.
|
|
*
|
|
* @param input the string_view to escape
|
|
* @param out output buffer (for escaped string): to be safe, it should have 6 * input.size() allocated bytes.
|
|
* @return number of bytes written
|
|
*/
|
|
simdjson_warn_unused virtual size_t write_string_escaped(const std::string_view input, char *out) const noexcept = 0;
|
|
protected:
|
|
/** @private Construct an implementation with the given name and description. For subclasses. */
|
|
simdjson_inline implementation(
|
|
std::string_view name,
|
|
std::string_view description,
|
|
uint32_t required_instruction_sets
|
|
) :
|
|
_name(name),
|
|
_description(description),
|
|
_required_instruction_sets(required_instruction_sets)
|
|
{
|
|
}
|
|
protected:
|
|
~implementation() = default;
|
|
|
|
private:
|
|
/**
|
|
* The name of this implementation.
|
|
*/
|
|
std::string_view _name;
|
|
|
|
/**
|
|
* The description of this implementation.
|
|
*/
|
|
std::string_view _description;
|
|
|
|
/**
|
|
* Instruction sets required for this implementation.
|
|
*/
|
|
const uint32_t _required_instruction_sets;
|
|
};
|
|
|
|
/** @private */
|
|
namespace internal {
|
|
|
|
/**
|
|
* The list of available implementations compiled into simdjson.
|
|
*/
|
|
class available_implementation_list {
|
|
public:
|
|
/** Get the list of available implementations compiled into simdjson */
|
|
simdjson_inline available_implementation_list() {}
|
|
/** Number of implementations */
|
|
size_t size() const noexcept;
|
|
/** STL const begin() iterator */
|
|
const implementation * const *begin() const noexcept;
|
|
/** STL const end() iterator */
|
|
const implementation * const *end() const noexcept;
|
|
|
|
/**
|
|
* Get the implementation with the given name.
|
|
*
|
|
* Case sensitive.
|
|
*
|
|
* const implementation *impl = simdjson::get_available_implementations()["westmere"];
|
|
* if (!impl) { exit(1); }
|
|
* if (!imp->supported_by_runtime_system()) { exit(1); }
|
|
* simdjson::get_active_implementation() = impl;
|
|
*
|
|
* @param name the implementation to find, e.g. "westmere", "haswell", "arm64"
|
|
* @return the implementation, or nullptr if the parse failed.
|
|
*/
|
|
const implementation * operator[](const std::string_view &name) const noexcept {
|
|
for (const implementation * impl : *this) {
|
|
if (impl->name() == name) { return impl; }
|
|
}
|
|
return nullptr;
|
|
}
|
|
|
|
/**
|
|
* Detect the most advanced implementation supported by the current host.
|
|
*
|
|
* This is used to initialize the implementation on startup.
|
|
*
|
|
* const implementation *impl = simdjson::available_implementation::detect_best_supported();
|
|
* simdjson::get_active_implementation() = impl;
|
|
*
|
|
* @return the most advanced supported implementation for the current host, or an
|
|
* implementation that returns UNSUPPORTED_ARCHITECTURE if there is no supported
|
|
* implementation. Will never return nullptr.
|
|
*/
|
|
const implementation *detect_best_supported() const noexcept;
|
|
};
|
|
|
|
} // namespace internal
|
|
|
|
/**
|
|
* The list of available implementations compiled into simdjson.
|
|
*/
|
|
extern SIMDJSON_DLLIMPORTEXPORT const internal::available_implementation_list& get_available_implementations();
|
|
|
|
/**
|
|
* The active implementation.
|
|
*
|
|
* Automatically initialized on first use to the most advanced implementation supported by this hardware.
|
|
*/
|
|
extern SIMDJSON_DLLIMPORTEXPORT internal::atomic_ptr<const implementation>& get_active_implementation();
|
|
|
|
} // namespace simdjson
|
|
|
|
#endif // SIMDJSON_IMPLEMENTATION_H
|