admin/coderforlife-ms-compress
1
0
Fork 0
mirror of https://github.com/coderforlife/ms-compress synced 2026-06-08 13:34:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • a0fcab84a7 Merge pull request #34 from nemequ/master master Jeffrey Bush 2020-01-05 14:12:32 -05:00
  • 51ca7e19ea Merge pull request #35 from mingkuang-Chuyu/master Jeffrey Bush 2020-01-05 14:10:59 -05:00
  • b92e6b9c4a add VS ARM64 support mingkuang 2019-12-20 10:59:41 +08:00
  • 0ffbda3fa6 Adaptive different VS versions mingkuang 2019-12-20 10:18:13 +08:00
  • f3cc32fa1f More fixes for the trailing null-byte support for LZNT1. Now works when output buffer is larger than needed and during streaming. Jeffrey Bush 2017-06-23 17:30:29 -04:00
  • bbe5176e76 Allow LZNT1 compressed data to end with a single trailing null-byte and still work. Jeffrey Bush 2017-06-23 16:51:14 -04:00
  • 9e696b96f9 build.sh: allow overriding the compiler and flags Evan Nemerson 2017-03-21 13:01:11 -07:00
  • 4811c1b567 Fix a warning from GCC about using a possibly uninitialized variable Evan Nemerson 2017-03-21 12:53:28 -07:00
  • 82b3971669 Fix detection of GCC >= 4.7 in diagnostic handling macros Evan Nemerson 2017-03-21 12:51:36 -07:00
  • f05faa7226 Add macros for disabling warnings on ICC Evan Nemerson 2017-03-21 12:36:27 -07:00
  • e5d8cc5f43 Fixed big endian handling when running without unaligned access. Additionally removed PDP endian code as there is no way to test it and may have been wrong. Also properly indented various other defines in the internal header. Jeffrey Bush 2016-07-20 16:51:30 -07:00
  • d70aefea63 Fixed #31. Was using the ReadBits_Fast method instead of ReadBits in the full bounds-checking loop. Jeffrey Bush 2016-06-21 14:12:27 -07:00
  • 158667afbc GCC didn't have maybe-uninitialized before v4.7. Fixes #30. Jeffrey Bush 2016-06-15 16:14:30 -07:00
  • 881f63999a Merge pull request #29 from nemequ/master Jeffrey Bush 2016-03-04 16:18:17 -08:00
  • f59818a94c Check that size is not 0 before calling memcpy. Evan Nemerson 2016-02-02 10:08:13 -08:00
  • 741130c5e5 Added Array.h. Jeffrey Bush 2016-01-19 23:12:02 -08:00
  • bda1423f7f Decided to have Xpress compression also use a minimal amount of stack memory. Jeffrey Bush 2016-01-19 22:05:51 -08:00
  • 805467dd07 Now the Xpress Dictionary uses an array that chooses to be either stack-based or heap-based, reducing the stack memory usage of Xpress Huffman compression by up to 1.25 MB when MSCOMP_WITHOUT_LARGE_STACK is defined. Jeffrey Bush 2016-01-19 22:00:50 -08:00
  • e2f37457b4 Added LARGE_STACK feature that when disabled (using #define MSCOMP_WITHOUT_LARGE_STACK) will cause various large variables to be heap-based instead of stack-based. Now in use by LZNT1Dictionary (512-768 kb reduction in stack usage) and HuffmanEncoder::CreateCodesSlow (512 kb reduction in stack usage, but only in very rare cases). Jeffrey Bush 2016-01-19 21:46:42 -08:00
  • 35271887b5 Change the checks in the Huffman Decoder so that invalid data doesn't cause buffer overflows. Fixes #28. Jeffrey Bush 2016-01-19 20:12:32 -08:00
  • 0afd4775e0 Increased the Xpress Huffman max compressed size just slightly, it is the final necessary increase. Fixes #27. Jeffrey Bush 2016-01-19 12:19:53 -08:00
  • b3f657bd76 Reverting some changes accidentally made by the merge. Jeffrey Bush 2016-01-19 11:58:43 -08:00
  • 8fe5ffdd36 Merge branch 'master' of github.com:coderforlife/ms-compress Jeffrey Bush 2016-01-19 11:53:52 -08:00
  • 60ef3852db Fixed the calculation for the max compressed size of Xpress Huffman for the last chunk. Jeffrey Bush 2016-01-19 11:49:53 -08:00
  • 461ede1050 Added a needed cast. Jeffrey Bush 2016-01-19 11:48:49 -08:00
  • fa6e9f2c3e Merge pull request #26 from nemequ/master Jeffrey Bush 2015-11-15 12:35:22 -08:00
  • 8e45dbf200 Don't compare a signed value with an unsigned one Evan Nemerson 2015-11-14 22:23:19 -08:00
  • 4e263b55e3 Don't redefine common preprocessor macros Evan Nemerson 2015-11-14 20:27:08 -08:00
  • 3b82292208 Merge pull request #24 from nemequ/master Jeffrey Bush 2015-11-04 16:55:36 -08:00
  • 303a7d5e2e Fix types for debug builds which were broken by 778e8bbb Evan Nemerson 2015-11-04 14:53:03 -08:00
  • 43d33ab842 Merge pull request #23 from nemequ/master Jeffrey Bush 2015-11-04 13:14:11 -08:00
  • 778e8bbbf4 Avoid ODR violations. Evan Nemerson 2015-11-04 12:51:04 -08:00
  • 73ce174b24 Additional checks in Xpress Dictionary Fill and Add. Should fix #20. Jeffrey Bush 2015-09-28 03:32:56 -07:00
  • 4fca365f4c Allow null input/output buffers if available is 0, fixing #19. Jeffrey Bush 2015-09-28 03:25:57 -07:00
  • 8ba2fb0949 Possible fix to #17. Jeffrey Bush 2015-09-15 23:31:44 -07:00
  • 90829b848d Fixed whitespace. Jeffrey Bush 2015-09-15 23:01:24 -07:00
  • e22df7d307 Added a buffer offerflow check fixing #18. Jeffrey Bush 2015-09-15 22:50:33 -07:00
  • db99414411 Updating to Visual Studio 2015 Jeffrey Bush 2015-09-15 22:49:18 -07:00
  • 043aaaf56c Update README.md Jeffrey Bush 2015-09-11 12:08:35 -07:00
  • 78101d2eba Improved byte-order detection and usage. coderforlife 2015-09-11 12:01:17 -07:00
  • 254f814428 Fixed problem with general header compling in regular C. coderforlife 2015-09-11 12:00:35 -07:00
  • a179b8d46c Removed one of the restricted pointers that caused GCC pre-4.9.2 to fail to compile. coderforlife 2015-05-28 19:09:18 -07:00
  • 39f51004b8 Added restrict to many pointers in LZNT1. Does not seem to have helped or hurt performance with GCC so leaving them in. May try to add them to other algorithms eventually. Jeffrey Bush 2015-05-25 16:32:25 -07:00
  • 73af8f022c Fixed problems with non-unaligned access methods that produced the wrong byte order and updated Xpress Huffman compression to be able to use non-unaligned accesses. Jeffrey Bush 2015-04-17 16:24:02 -07:00
  • c493a62623 Updated build scripts. Jeffrey Bush 2015-04-17 15:23:46 -07:00
  • 618ede3c46 Updated readme with latest timings. Jeffrey Bush 2015-04-17 15:20:58 -07:00
  • 3ab0ab7c9f Minor update. Jeffrey Bush 2015-04-17 15:20:05 -07:00
  • 91a4e53b38 Adjusted the speed tester to use the second-to-last core to make sure it uses a real core and not a hyper-threaded core. Jeffrey Bush 2015-04-17 15:18:18 -07:00
  • 837a292aec Added a new compile time configuration option MSCOMP_WITHOUT_UNALIGNED_ACCESS that makes it so there are no unaligned accesses to work on #15. This is not yet complete - Xpress Huffman compression still has many unaligned accesses. Jeffrey Bush 2015-04-17 15:14:13 -07:00
  • 1cfac269a8 Finally fixed #4. The problem is that on Windows the stack is 4-byte aligned and GCC assumes (for some optimizations) 16-byte alignment. A GCC stack alignment fix was added to functions that needed it. Jeffrey Bush 2015-03-19 20:52:22 -07:00
  • 0dac29e4f8 Spacing changes in test script. Jeffrey Bush 2015-03-19 20:43:29 -07:00
  • 1095f73d1e Addressed some bugs with XPRESS Huffman compression of uncompromisable data and decompression of invalid data. This partially helps #14. Jeffrey Bush 2015-03-18 14:24:17 -07:00
  • 66e93e7581 Fixing some warnings with gcc -Wextra -pedantic -Wuninitialized. The remaining warnings are mostly about variadic macros. Jeffrey Bush 2015-03-18 12:55:41 -07:00
  • 74101ccad3 Added MaxCompressedSize function to OpenSrc compressor. Jeffrey Bush 2015-03-18 12:37:00 -07:00
  • 941fbb4860 Minor changes. Jeffrey Bush 2015-03-18 12:02:06 -07:00
  • ce1d61cf62 Forgot to commit sorting.h when re-adding package-merge algorithm. Fixes #12. Jeffrey Bush 2015-03-18 11:47:21 -07:00
  • e877ca5554 Corrected XPRESS Huffman compression to max size and added checks to make sure it always stays under the max size. In addition many other changes were made for improved performance. Jeffrey Bush 2015-03-17 22:39:05 -07:00
  • 14d766a224 The SSE copy method was actually incorrect (more needed to be copied initially and the input needed to be shifted back). Once these were fixed it was slower than the original fast copy so it was removed. Jeffrey Bush 2015-03-17 22:35:41 -07:00
  • dce8c3ac6a Played around with SSE optimizations like prefetch and 128-bit copies. I couldn't get prefetching to really help during decompression but 128-bit copies helped in XPRESS (Huffman) decompression. Jeffrey Bush 2015-03-16 15:01:56 -07:00
  • fdc815abef Removed old code from HuffmanDecoder Jeffrey Bush 2015-03-16 14:59:02 -07:00
  • 3697228090 Test speed script now tries to make the process high-priority and have CPU affinity. It also supports different number of repeats for compression and decompression (so if testing new decompressions you don't have to wait long for compressions). Jeffrey Bush 2015-03-16 14:18:32 -07:00
  • e99e328faf Removing Peek_Not0 and updating Peek to work really fast for 0 and non-0. Few other minor Huffman decoder changes. Jeffrey Bush 2015-03-13 18:34:09 -07:00
  • 3b3e29ed69 Adding comment about bzip2 inspiration and removing sorting library. Jeffrey Bush 2015-03-13 18:32:35 -07:00
  • 7b7ed8cbf4 Completely changed the Huffman encoder to use an algorithm inspired by bzip2. This increases compression speed by more than 100%! That is its twice as fast! Downside is that compression ratio is slightly worse (but just barely: 32.792% to 32.853% or 167kb in 269MB). It also uses less stack (total 17kb). Jeffrey Bush 2015-03-13 18:25:24 -07:00
  • 345d9752b4 Improved the package-merge algorithm on the Huffman Encoder to use stack memory (its a lot of it - 519 kb now!). This improves its speed quite a bit (compression speed increases by 25%). Jeffrey Bush 2015-03-13 18:20:07 -07:00
  • c65aaf2bdb Very minor changes. Jeffrey Bush 2015-03-13 00:44:19 -07:00
  • 511e1215ae More improvements to XPRESS Huffman decompression speed. Jeffrey Bush 2015-03-13 00:43:59 -07:00
  • 2f995338d1 Added xpress_huff_max_compressed_size function. Jeffrey Bush 2015-03-12 22:47:45 -07:00
  • 229725e55e Updating readme with new speeds. Jeffrey Bush 2015-03-12 21:10:33 -07:00
  • 5308b1b2f8 Made FAST_COPY a little more compatible and fixed a new bug in XPRESS decompression. Jeffrey Bush 2015-03-12 21:06:55 -07:00
  • b4264bb650 Improved the speed of the XPRESS Huffman decompressor, mainly by adding a fast decompression loop, along improving HuffmanDecoder and InputBitstream. Jeffrey Bush 2015-03-12 20:43:14 -07:00
  • 8efdeb1012 Updated comments in internal.h to describe undefined values from the intrinsic functions and many other things. Jeffrey Bush 2015-03-12 20:40:39 -07:00
  • 3dfd9a2fd1 Changed XPRESS to only switch to checked-lengths after maxed out full-byte length (instead of half-byte). Also updated OUT_NEAR_END to use the FAST_COPY_ROOM constant. Jeffrey Bush 2015-03-12 20:39:55 -07:00
  • 34bffc992f Fix bug in Xpress-Huffman compression that was taking the log2 of 0 (only showed up after PGO!). Jeffrey Bush 2015-03-12 20:38:25 -07:00
  • da4fd73e24 Updated comments slightly. Jeffrey Bush 2015-03-12 17:57:06 -07:00
  • 6fa0409de5 Generalized the fast-copy routine used by decompressors (along with fixing some length-checks). Also added ASSERT_ALWAYS which does both assert and ALWAYS on the given command. Jeffrey Bush 2015-03-12 17:55:04 -07:00
  • bb8562fe8b Made highbit/log2 function use compiler intrinsics if available. If not available it uses a slower method than before (but no static table). Jeffrey Bush 2015-03-11 21:54:48 -07:00
  • f305590ff2 WindowsPos should always be inlined. Jeffrey Bush 2015-03-11 20:01:45 -07:00
  • 32a8ce209b Found which loop was causing the access violation with GCC tree-vectorization and put it into the source so tree-vectorization is not disabled globally. Also improved Huffman encoder slightly, mainly by changing merge sort to use insertion sort when "small". Jeffrey Bush 2015-03-11 20:01:04 -07:00
  • 4fea9774b9 Fixed #10. Jeffrey Bush 2015-03-05 23:40:39 -08:00
  • 424f800e76 Fixed nested lists in readme. Jeffrey Bush 2015-03-05 18:51:44 -08:00
  • c36b257fa9 The improved improved XPRESS dictionary sped up the XPRESS compressors but their compression ratios got slightly worse (but were designed to still be slightly better than RTL). XPRESS got much faster - 3.2x faster than it was! - but still is not as fast as RTL. Jeffrey Bush 2015-03-05 18:40:58 -08:00
  • 5c030344fc Putting -fno-tree-vectorize back into build scripts because of bug #4. Jeffrey Bush 2015-03-05 17:01:02 -08:00
  • b24b7b8bf4 Improved XpressDictionary to skip pathological cases (hoping to find a match after searching a bunch) and a new hash function. These were inspired by zlib. Jeffrey Bush 2015-03-05 16:03:58 -08:00
  • 0eb4373f86 Updated readme to include a larger corpus, other MS algorithms not yet implemented, and stats. Jeffrey Bush 2015-03-04 22:22:56 -08:00
  • 84c3d91db6 Updated default LZNT1 dictionary to be simpler, match the API of the new SA LZNT1 dictionary (except returns indicating memory allocation problems), and even slightly faster. Updated comments. Jeffrey Bush 2015-03-04 21:45:55 -08:00
  • c2f56006ea Fixed bad import on Linux Jeffrey Bush 2015-03-04 17:30:58 -08:00
  • 9859e00249 Made test scripts executable on Linux. Jeffrey Bush 2015-03-04 17:25:25 -08:00
  • 4937a619b1 Fixed a bug with the generalized decompressor and expanded which warnings are ignored by uninit variable used. Jeffrey Bush 2015-03-04 16:19:34 -08:00
  • 549737a5ba Updated some comments and some minor pieces of code. Jeffrey Bush 2015-03-04 00:06:11 -08:00
  • 7fa1790ab4 Made default LZNT1 dictionary the non-SA one. Jeffrey Bush 2015-03-03 20:57:04 -08:00
  • 7df6abf11a Added a new dictionary for LZNT1. It is slower than the old one (which is still default) but uses significantly less memory and no dynamic allocations. It is still being worked on and may get "up to speed" one day. Jeffrey Bush 2015-03-03 20:40:47 -08:00
  • 8cd4d3bffc Removed a temporary file. Jeffrey Bush 2015-03-03 20:36:05 -08:00
  • dd1a5eaf9c Added import for the always/never debugging. Jeffrey Bush 2015-03-03 20:31:23 -08:00
  • ebb694fc36 Updated Windows build script to support double-clicking or command line use. Jeffrey Bush 2015-03-03 20:29:02 -08:00
  • b3381fb014 Fixed bugs with the new streamable interface (never returning STREAM_END) and the testing framework not setting the right flag along with various other small bug fixes and optimizations. Jeffrey Bush 2015-02-27 03:45:24 -08:00
  • f3cfb83d11 The tree-vectorization optimization of GCC no longer seems to be an issue. Jeffrey Bush 2015-02-13 18:23:13 -08:00
  • 2418d0979e Fixed some small errors. Jeffrey Bush 2015-02-12 17:09:44 -08:00
  • 8c2b0a8c6b Removed xxx_uncompressed_size functions (since they were not constant-time and basically did decompression without writing). Jeffrey Bush 2015-02-12 17:01:30 -08:00
  • da2673ccaf Minor clarification. Jeffrey Bush 2015-02-12 13:22:07 -08:00