1

I'm creating data dumps from my site for others to download and analyze. Each dump will be a giant XML file.

I'm trying to figure out the best compression algorithm that:

  • Compresses efficiently (CPU-wise)
  • Makes the smallest possible file
  • Is fairly common

I know the basics of compression, but haven't a clue as to which algo fits the bill. I'll be using MySQL and Python to generate the dump, so I'll need something with a good python library.

1
  • CPU-wise and smallest possible file are mutually exclusive I think. May 20, 2011 at 5:38

1 Answer 1

3

GZIP with standard compression level should be fine for most cases. Higher compression levels=more CPU time. BZ2 is packing better but is also slower. Well, there is always a trade-off between CPU consumption/running time and compression efficiency...all compressions with default compression levels should be fine.

2
  • +1 For large text files, where speed is important, just use gzip on fastest setting.
    – samplebias
    May 20, 2011 at 6:08
  • Ended up using this. Mark Pilgrim's Dive into Python 3.0 gives excellent instructions for it as well, which is always handy.
    – mlissner
    May 23, 2011 at 1:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.