From 772a5dc3d57eeb7665d46f4b569a3dd459316fd7 Mon Sep 17 00:00:00 2001 From: Daniel Lemire Date: Mon, 23 Mar 2020 14:06:10 -0400 Subject: [PATCH 1/6] Updated script so that we generate a small and a large file. --- scripts/ruby/kostya_large.rb | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/scripts/ruby/kostya_large.rb b/scripts/ruby/kostya_large.rb index 542664d44..8633fbf88 100755 --- a/scripts/ruby/kostya_large.rb +++ b/scripts/ruby/kostya_large.rb @@ -1,6 +1,21 @@ #!/usr/bin/env ruby require 'json' +y = [] + +10000.times do + h = { + 'x' => rand, + 'y' => rand, + 'z' => rand, + 'name' => ('a'..'z').to_a.shuffle[0..5].join + ' ' + rand(10000).to_s, + 'opts' => {'1' => [1, true]}, + } + y << h +end + +File.open("2.json", 'w') { |f| f.write JSON.pretty_generate('coordinates' => y, 'info' => "some info") } + x = [] 524288.times do From 6c2ab064cb18fac6ae8ca809a6b585b7d7cbd47b Mon Sep 17 00:00:00 2001 From: John Keiser Date: Mon, 23 Mar 2020 08:28:44 -0700 Subject: [PATCH 2/6] Correct quick start Not sure how this update didn't make it in, but it's necessary. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 25207470e..80303e9dd 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ simdjson is easily consumable with a single .h and .cpp file. std::cout << tweets["search_metadata"]["count"] << " results." << std::endl; } ``` -3. `g++ -o parser parser.cpp` (or clang++) +3. `g++ -o parser parser.cpp simdjson.cpp -std=c++17` 4. `./parser` ``` 100 results. From ceee00b276d22c9ac9a35daa804f27afc422f6a2 Mon Sep 17 00:00:00 2001 From: John Keiser Date: Mon, 23 Mar 2020 10:00:48 -0700 Subject: [PATCH 3/6] Use c++ instead of g++ in quick start for clang --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 80303e9dd..d9e03c02a 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ simdjson is easily consumable with a single .h and .cpp file. std::cout << tweets["search_metadata"]["count"] << " results." << std::endl; } ``` -3. `g++ -o parser parser.cpp simdjson.cpp -std=c++17` +3. `c++ -o parser parser.cpp simdjson.cpp -std=c++17` 4. `./parser` ``` 100 results. From d1eef242c69ba56425cf43837c78409eb76310c0 Mon Sep 17 00:00:00 2001 From: Daniel Lemire Date: Mon, 23 Mar 2020 18:04:22 -0400 Subject: [PATCH 4/6] Trying to give better guidance regarding large files. (#594) --- README.md | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d9e03c02a..fc137c198 100644 --- a/README.md +++ b/README.md @@ -146,11 +146,27 @@ The json stream parser is threaded, using exactly two threads. ## Large files -If you are processing large files (e.g., 100 MB), it is likely that the performance of simdjson will be limited by page misses and/or page allocation. [On some systems, memory allocation runs far slower than we can parse (e.g., 1.4GB/s).](https://lemire.me/blog/2020/01/14/how-fast-can-you-allocate-a-large-block-of-memory-in-c/) +If you are processing large files (e.g., 100 MB), it is possible that the performance of simdjson will be limited by page misses and/or page allocation. [On some systems, memory allocation runs far slower than we can parse (e.g., 1.4GB/s).](https://lemire.me/blog/2020/01/14/how-fast-can-you-allocate-a-large-block-of-memory-in-c/) -You will get best performance with large or huge pages. Under Linux, you can enable transparent huge pages with a command like `echo always > /sys/kernel/mm/transparent_hugepage/enabled` (root access may be required). We recommend that you report performance numbers with and without huge pages. +A viable strategy is to amortize the cost of page allocation by reusing the same `parser` object over several files: + +```C++ +// create one parser +simdjson::document::parser parser; +... +// the parser is going to pay a memory allocation price +auto [doc1, error1] = parser.parse(largestring1); +... +// use again the same parser, it will be faster +auto [doc2, error2] = parser.parse(largestring2); +... +auto [doc3, error3] = parser.load("largefilename"); +``` + +If you cannot reuse the same parser instance, maybe because your application just processes one large document once, you will get best performance with large or huge pages. Under Linux, you can enable transparent huge pages with a command like `echo always > /sys/kernel/mm/transparent_hugepage/enabled` (root access may be required). It may be more difficult to achieve the same result under other systems like macOS or Windows. + +In general, when running benchmarks over large files, we recommend that you report performance numbers with and without huge pages if possible. Furthermore, you should amortize the parsing (e.g., by parsing several large files) to distinguish the time spent parsing from the time spent allocating memory. -Another strategy is to reuse pre-allocated buffers. That is, you avoid reallocating memory. You just allocate memory once and reuse the blocks of memory. ## Including simdjson From 5af0dfb031ea989507660c168869af69e58f4a2b Mon Sep 17 00:00:00 2001 From: Daniel Lemire Date: Tue, 24 Mar 2020 05:50:27 -0400 Subject: [PATCH 5/6] Putting back the fuzzer badge and extending the documentation. (#587) --- README.md | 1 + fuzz/Fuzzing.md | 12 +++++++++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index fc137c198..182da975d 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ This library is part of the [Awesome Modern C++](https://awesomecpp.com) list. [![Build Status](https://cloud.drone.io/api/badges/simdjson/simdjson/status.svg)](https://cloud.drone.io/simdjson/simdjson) [![CircleCI](https://circleci.com/gh/simdjson/simdjson.svg?style=svg)](https://circleci.com/gh/simdjson/simdjson) +[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/simdjson.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&q=proj%3Asimdjson&can=2) [![Build status](https://ci.appveyor.com/api/projects/status/ae77wp5v3lebmu6n/branch/master?svg=true)](https://ci.appveyor.com/project/lemire/simdjson-jmmti/branch/master) [![][license img]][license] diff --git a/fuzz/Fuzzing.md b/fuzz/Fuzzing.md index 281215137..b5a61e76e 100644 --- a/fuzz/Fuzzing.md +++ b/fuzz/Fuzzing.md @@ -6,9 +6,15 @@ - https://github.com/lemire/simdjson/issues/351 - https://github.com/lemire/simdjson/issues/345 -Simdjson tries to follow [fuzzing best practises](https://google.github.io/oss-fuzz/advanced-topics/ideal-integration/#summary). +The simdjson library tries to follow [fuzzing best practises](https://google.github.io/oss-fuzz/advanced-topics/ideal-integration/#summary). + +The simdjson library is continuously fuzzed on [oss-fuzz](https://github.com/google/oss-fuzz). + + +## Currently open bugs + +You can find the currently opened bugs, if any at [bugs.chromium.org](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&q=proj%3Asimdjson&can=2): make sure not to miss the "Open Issues" selector. Bugs that are fixed by follow-up commits are automatically closed. -Simdjson is continuously fuzzed on [oss-fuzz](https://github.com/google/oss-fuzz). ## Fuzzing as a CI job @@ -29,7 +35,7 @@ The corpus will grow over time and easy to find bugs will be detected already du ## Corpus -Simdjson does not benefit from a corpus as much as other projects, because the library is very fast and explores the input space very well. With that said, it is still beneficial to have one. The CI job stores the corpus on bintray between runs, and is available here: https://dl.bintray.com/pauldreik/simdjson-fuzz-corpus/corpus/corpus.tar +The simdjson library does not benefit from a corpus as much as other projects, because the library is very fast and explores the input space very well. With that said, it is still beneficial to have one. The CI job stores the corpus on bintray between runs, and is available here: https://dl.bintray.com/pauldreik/simdjson-fuzz-corpus/corpus/corpus.tar One can also grab the corpus as an artifact from the github actions job. Pick a run, then go to artifacts and download. From 5514ae3879a8f6aebd3953fbbb51e9d2eb6d88fb Mon Sep 17 00:00:00 2001 From: Daniel Lemire Date: Tue, 24 Mar 2020 09:41:32 -0400 Subject: [PATCH 6/6] Adding another user. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 182da975d..737696363 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,7 @@ simdjson is easily consumable with a single .h and .cpp file. - [Microsoft FishStore](https://github.com/microsoft/FishStore) - [Yandex ClickHouse](https://github.com/yandex/ClickHouse) - [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer) +- [azul](https://github.com/tudelft3d/azul) If you are planning to use simdjson in a product, please work from one of our releases.