Re: pack operation is thrashing my server

Linus Torvalds [off-list ref] writes:

Interestingly, it turns out that git also hits a sad performance downside 
of using zlib.

We always tend to set "stream.avail_out" to the exact size of the expected 
output. And it turns out that that means that the fast-path case of 
inffast.c doesn't trigger as often as it could. This (idiotic) patch 
actually seems to help performance on git rev-list by about 5%.

But maybe it's just me seeing things. But I did this because of the entry 
assumptions in inflate_fast(), that code only triggers for the case of 
strm->avail_out >= 258.

Sad, if true.

This is reproducible  "rev-list --objects --all" in my copy of the kernel
repo takes around 47-48 seconds user time, and with the (idiotic) patch it
is cut down to 41-42 seconds.

(with patch)
41.41user 0.51system 0:41.93elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+134411minor)pagefaults 0swaps

(without patch)
47.21user 0.64system 0:47.85elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+134935minor)pagefaults 0swaps

One funny thing about your patch is that it also reduces the number of
minor faults; I would have expected that the additional memory wastage
(even though most of the allocated object buffer memory would be freed
immediately as soon as the caller is done with it) would result in larger
number of faults, not smaller, which is puzzling.