22

I have a large file of size 500 mb to compress in a minute with the best possible compression ratio. I have found out these algorithms to be suitable for my use.

  1. lz4
  2. lz4_hc
  3. snappy
  4. quicklz
  5. blosc

Can someone give a comparison of speed and compression ratios between these algorithms?

2
  • Have you found a comparison between lz4 and fastlz?
    – roalz
    Sep 15, 2017 at 9:21
  • I'm interested too, it seems lz4 is more efficient than fastlz but I couldn't find stats proving that fact.
    – rraallvv
    Apr 22, 2018 at 18:48

4 Answers 4

38

Yann Collet's lz4, hands down.

enter image description here

2
  • 1
    What is you recommendation for embedded systems? which is the efficient compression and decompression algorithm regarding and space and time. Jan 19, 2018 at 4:21
  • 1
    Often people don't know about the large-window brotli and perform large corpus benchmarking with the small-window brotli. Brotli's HTTP content encoding variant is small-window to allow decompression on cheap mobile phones. Other compressors (particularly so lzma and zstd) don't do that limitation and should be compared with large-window brotli, not small-window brotli. Typically you can see 10 % density improvements (within 0.6 % of lzma) using the large-window brotli, while keeping the high decompression speed. Mar 27, 2019 at 15:25
5

This migth help you: (lz4 vs snappy) http://java-performance.info/performance-general-compression/ (benchmarks for lz4, snappy, lz4hc, blosc) https://web.archive.org/web/20170706065303/http://blosc.org:80/synthetic-benchmarks.html (now not available on http://www.blosc.org/synthetic-benchmarks.html)

2

If you are only aiming for high compression density, you want to look at LZMA and large-window Brotli. These two algorithms give the best compression density from the widely available open-sourced algorithms. Brotli is slower at compression, but ~5x faster at decompression.

2

Like most questions, the answer usually ends up being: It depends :)

The other answers gave you good pointers, but another thing to take into account is RAM usage in both compression and decompression stages, as well as decompression speed in MB/s.

Decompression speed is typically inversely proportional to the compression ratio, so you may think you chose the perfect algorithm to save some bandwidth/disk storage, but then whatever is consuming that data downstream now has to spend much more time, CPU cycles and/or RAM to decompress. And RAM usage might seem inconsequential, but maybe the downstream system is an embedded/low-voltage system? Maybe RAM is plentiful, but CPU is limited? All those things need to be taken into account.

Here's an example of a suite of benchmarks done on various algorithms, taking a lot of these considerations into account:

https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

1
  • That was one of the interesting things about H265 vs H264 as well. H264 encodes faster, but results in a larger size. H265 results in a smaller result, and the decoder, while more complex, actually runs at similar speeds because the resulting data it needs to work on is significantly less.
    – Pyro
    Nov 27, 2020 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.