Re: pack operation is thrashing my server

On 9/6/08, Linus Torvalds [off-list ref] wrote:

 I'm starting to hate zlib. I actually spent almost a week trying to clean
 up the zlib source code and make it something that gcc can compile into
 clean code, but the fact is, zlib isn't amenable to that. The whole "shift
 <n> bits in from the buffer" approach means that there is no way to make
 zlib generate good code unless you are an insanely competent assembly
 hacker or have tons of registers to keep all the temporaries live in.

 Now, I still do think that all my reasons for choosing zlib were pretty
 solid (it's a well-tested piece of code and it is _everywhere_ and easy to
 use), but boy do I wish there had been alternatives.

Some alternative algorithms are here...
http://cs.fit.edu/~mmahoney/compression
It is possible to beat zlib by 2x at the cost of CPU time and memory.

Of course switching to these algorithms would involve a lot of testing
and benchmarking. I'm also not sure how PAQ would fare on lots of
small git objects instead of large files.

Turning a 500MB packfile into a 250MB has lots of advantages in IO
reduction so it is worth some CPU/memory to create it.

You can even win 50'000€ for a better algorithm.
http://prize.hutter1.net/

-- 
Jon Smirl
jonsmirl@gmail.com