fd44c2a2ff
Merge pull request #927 from simdjson/dlemire/exposingthestringminifier
...
Exposing the string minifier.
2020-06-13 07:47:20 -07:00
a86a82b39c
Rename minify class to minifier so the minify() method is cleared up
2020-06-12 17:05:25 -07:00
4dfbf98e4e
Using a worker instead of a thread per batch ( #920 )
...
In the parse_many function, we have one thread doing the stage 1, while the main thread does stage 2. So if stage 1 and stage 2 take half the time, the parse_many could run at twice the speed. It is unlikely to do so. Still, we see benefits of about 40% due to threading.
To achieve this interleaving, we load the data in batches (blocks) of some size. In the current code (master), we create a new thread for each batch. Thread creation is expensive so our approach only works over sizeable batches. This PR improves things and makes parse_many faster when using small batches.
This fixes our parse_stream benchmark which is just busted.
This replaces the one-thread per batch routine by a worker object that reuses the same thread. In benchmarks, this allows us to get the same maximal speed, but with smaller processing blocks. It does not help much with larger blocks because the cost of the thread create gets amortized efficiently.
This PR makes parse_many beneficial over small datasets. It also makes us less dependent on the thread creation time.
Unfortunately, it is going to be difficult to say anything definitive in general. The cost of creating a thread varies widely depending on the OS. On some systems, it might be cheap, in others very expensive. It should be expected that the new code will depend less drastically on the performances of the underlying system, since we create juste one thread.
Co-authored-by: John Keiser <john@johnkeiser.com >
Co-authored-by: Daniel Lemire <lemire@gmai.com >
2020-06-12 16:51:18 -04:00
fe01da077e
Make threaded version work again
2020-06-07 16:21:00 -07:00
ef63a84a3e
Move document stream state to implementation
2020-06-07 16:20:44 -07:00
7a69da16e4
Fixing issue 906 ( #912 )
...
* Fixing issue 906
* Safe patching.
* Now with explanations.
* Bumping up memory allocation.
* Putting the patch back.
* fallback fixes.
Co-authored-by: Daniel Lemire <lemire@gmai.com >
2020-06-05 15:37:09 -04:00
1aab4752e2
Store all parser state in the implementation
2020-06-01 12:15:54 -07:00
a3a9bde83e
Move DOM parsing into concrete interface implementation
2020-06-01 12:14:09 -07:00
40d57da83c
fixes issue 891 ( #893 )
2020-05-20 11:54:53 -04:00
e6c9dfbd91
Make include files more fine-grained
2020-05-19 14:42:04 -07:00
7ad4020829
Make main compilation chunks into .cpp files
2020-05-19 13:32:35 -07:00
a476531524
Share ref_address everywhere it's used
2020-05-19 13:30:34 -07:00
2a6e6b3dbd
Cleaning string_view ( #872 )
...
* Cleaning string_view
* Corrected typo
* Alignment.
2020-05-10 16:05:52 -04:00
1d06624d38
Unset /D_CRT_SECURE_NO_WARNINGS
...
- Also localize DISABLE_DEPRECATED_WARNING so that we catch other
deprecations
2020-05-04 11:35:05 -07:00
d40069a018
Disable deprecation warnings for VS builds
...
fopen/getenv are standard c++ that are not deprecated.
2020-05-04 11:34:00 -07:00
fc1ddcd2f8
Faster case-insensitive comparisons. ( #837 )
...
* Faster case-insensitive comparisons.
2020-04-30 15:52:28 -04:00
0f9dbf84b7
Fix incorrect check for case insensitive key lookup ( #824 )
2020-04-29 13:55:28 -04:00
d4239aaa8f
default initialisaiton ( #779 )
...
* padded_string.* default initialisation
parsedjson_iterator - copy constructor; depth_index not necessary
2020-04-23 18:32:11 -04:00
87acab0846
elimination of most of g++ -Weffc++ warnings ( #764 )
...
Co-authored-by: Matjaž Ostroveršnik <ostri@localhost.localdomain >
Co-authored-by: Daniel Lemire <lemire@gmail.com >
2020-04-23 10:06:44 -04:00
d4a37f6ef5
Enable conversion warnings on Linux and Windows
2020-04-22 14:21:30 -07:00
3e9e14f4d6
Reenable deprecation warnings on Windows
2020-04-22 08:53:19 -07:00
289cc3e7a0
Treat warnings as errors during compilation
2020-04-15 19:59:38 -07:00
7480b87e07
Merge pull request #693 from simdjson/jkeiser/cmake-quickstartcpp
...
Add C++11 tests to cmake
2020-04-15 19:53:14 -07:00
befa6423be
This massively improves the performance of tight loops relying on a type() call. ( #721 )
...
* This massively improves the performance of tight loops relying on a type() call.
* Adding a few more benchmarks
2020-04-15 20:45:40 -04:00
fd418f568c
Fix c++11 warnings on clang
...
- namespace x::y is C++17
- static_assert requires message in C++11
2020-04-15 17:27:48 -07:00
326c175dcb
Massive performance boost for get<double>. ( #719 )
...
* Massive performance boost for get<double>.
2020-04-15 20:09:45 -04:00
b523c43927
Can we provide a size() function to arrays and objects? (eager approach) [TO BE MERGED] ( #690 )
...
* This is an implementation of "size()" for arrays and objects.
* Adding benchmark
* Adding a size() remark in the documentation.
* Extending size() to result types.
2020-04-15 10:15:48 -04:00
75545ff70d
ref qualify parser methods to avoid use of dangling objects ( #703 )
...
To avoid using data belonging to a temporary, the parse functions are ref qualified to get a compile error if used on an rvalue. See https://github.com/simdjson/simdjson/issues/696
Compilation tests are also added, to make sure bad usage fails to compile.
Reviewed by jkeiser.
2020-04-15 09:57:52 +02:00
1e30b6e334
Compile under C++ 11
2020-04-08 14:00:13 -07:00
406240bae3
Support C++ 14
2020-04-08 14:00:13 -07:00
12ed6336b1
remove three trailing semicolons that -pedantic dislikes ( #673 )
2020-04-02 21:06:25 -04:00
13aee51011
Add element.type() for type switching
2020-04-02 14:07:19 -07:00
d93af1161d
Remove set_capacity, replace with allocate
...
Makes allocation point more predictable
2020-03-30 13:49:54 -07:00
434776db1a
Deprecate more things
2020-03-30 13:48:43 -07:00
7ed65e42d7
Add actual examples from basics.md to readme_examples
2020-03-29 16:28:29 -07:00
ea8a5020e2
Remove array indexer, make object indexer key lookup
2020-03-28 15:56:43 -07:00
622d9c9480
Replace as_X and is_X with get<T> and is<T>
2020-03-28 15:29:53 -07:00
62da98aef6
Rename dom::stream to dom::document_stream
2020-03-28 13:42:24 -07:00
03746b966b
Move document/element/etc. under dom
2020-03-28 13:42:21 -07:00
836e1fc330
Use simdjson_result for all _result classes
2020-03-28 12:03:05 -07:00
e836c28008
Deprecate parser error code methods
...
- Also make competitions compile without warnings
2020-03-28 10:13:20 -07:00
748df8d109
Use string_view instead of string/char* where possible
2020-03-27 13:11:41 -07:00
5ad405006c
Return document::element from parse, load, parse_many, load_many
2020-03-27 12:24:41 -07:00
e3efbcddc1
Cast padded_string to string_view instead of string
2020-03-27 09:13:11 -07:00
c14b2fb36c
Remove const char* variants for at_key()
...
- Remove const char * variants for at_key(), string_view covers them
- Add at_key_case_insensitive variants on *_result
- Add at(), at_key(), at_key_case_insensitive() tests
2020-03-27 09:09:08 -07:00
f0f111b387
Make ParsedJson::Iterator backcompat test
2020-03-27 09:07:39 -07:00
abb0bf9247
Fixed basictests
2020-03-26 19:40:29 -04:00
56841bcede
Fix conversion error on Windows
2020-03-26 12:48:07 -07:00
006cc2ed60
Remove simdjson_move_result
2020-03-26 12:48:03 -07:00
2e420169c3
Remove document::parse and document::load
2020-03-26 10:13:09 -07:00
John Keiser