Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

2009/8/1 J.H. [off-list ref]:

Mark A Rada wrote:

quoted

 >Note that for me the above results are not aligned in table.
 >This is a cosmetic issue.

The table formatting issue was due to a bad habit of mixing tabs and
spaces,
I decided to go with spaces this time :)


 >One thing that would concern me greatly, is not so much the CPU time
(though that's a *huge* change in comparison to gz) but the memory usage.
 Where gzip and bzip2 are chewing 4M and 13M respectively, xz chews 102M.
 >From a 'beefy' server perspective chewing up that much memory per snapshot
for that long could be bad.  This is likely something that needs to have
some sort of enable/disable switch if it's going to be included.

True, and there are two solutions I can think of for this "problem".

   1. My tests were at the default compression level, the XZ documentation
   says that at lower levels you will get resource usage and compression
   ratios that are comparable to BZip2. However, I'm not sure where you
   would change the compression level variable for this (globally for the
   system, somewhere in $GITWEB_CONFIG, a git config variable). Does
   someone know the correct answer here?

Well you can always call xz with -[1-9] to change the compression level
(same as gzip and bzip2) though I think a full disabling would be 'more'
preferable, though I'm not sure I like Jakub's suggestion of just deleting
it after the fact, it would work.

quoted

   2. Implement snapshot caching for Gitweb.

I think it's slightly broken in my version (binary files don't work right
apparently, it's on my todo list to fix in my upcoming update) but both Lea
and I have done this long ago (caching layers in Gitweb), which would be an
acceptable workaround for this, create once and serve many - though this has
the downside of trading cpu for diskspace.  At least with xz there's less
diskspace used ;-)

I think more my concern is more what's enabled by default, and since xz is
still new (as was pointed out) it's probably worth only enabling if the
admin selects it to be enabled.

FWIW the perl project ripped out all the snapshot generation logic
from gitweb, and replaced it with a tool that generates snapshots
correctly for our requirements (if the build process needs additional
files /currently/ git-archive does not support adding them), this
includes a disk level cache for the snapshots since creating the tar,
adding the additional files, then gziping is quite slow.

If its interesting to people I can post it and the other changes here,
although its not a "nice" change, as I literally ripped out the
existing code.

cheers,
Yves





-- 
perl -Mre=debug -e "/just|another|perl|hacker/"