Re: pack operation is thrashing my server

On Sat, Sep 06, 2008 at 06:46:29PM -0700, Linus Torvalds wrote:

On Sat, 6 Sep 2008, Junio C Hamano wrote:

quoted

This is reproducible  "rev-list --objects --all" in my copy of the kernel
repo takes around 47-48 seconds user time, and with the (idiotic) patch it
is cut down to 41-42 seconds.

So I had forgotten about that patch since nobody reacted to it.

I think the patch is wrong, please don't apply it, even though it does 
help performance.

The reason? 

Right now we depend on "avail_out" also making zlib understand to stop 
looking at the input stream. Sad, but true - we don't know or care about 
the compressed size of the object, only the uncompressed size. So in 
unpack_compressed_entry(), we simply set the output length, and expect 
zlib to stop when it's sufficient.

Which it does - but the patch kind of violates that whole design.

Now, it so happens that things seem to work, probably because the zlib 
format does have enough synchronization in it to not try to continue past 
the end _anyway_, but I think this makes the patch be of debatable value.

I'm starting to hate zlib. I actually spent almost a week trying to clean 
up the zlib source code and make it something that gcc can compile into 
clean code, but the fact is, zlib isn't amenable to that. The whole "shift 
<n> bits in from the buffer" approach means that there is no way to make 
zlib generate good code unless you are an insanely competent assembly 
hacker or have tons of registers to keep all the temporaries live in.

Now, I still do think that all my reasons for choosing zlib were pretty 
solid (it's a well-tested piece of code and it is _everywhere_ and easy to 
use), but boy do I wish there had been alternatives. 

I know at least 7-zip has its own gzip compression/decompression code
(though it's C++). Maybe some other tools have theirs too.

Anyways, if it can make a speed difference, it might be worth having a
minimalist custom gzip compression/decompression "library" embedded
withing git.

Mike