[PATCHv5 3/3] gitweb: add support for XZ compressed snapshots

Made message more succinct, no changes to source.

--
Mark A Rada (ferrous26)
marada@uwaterloo.ca


--->8---
From: Mark Rada <redacted>
Date: Wed, 5 Aug 2009 23:08:49 -0400
Subject: [PATCH 3/3] gitweb: add support for XZ compressed snapshots

The XZ compression format uses the LZMA2 compression algorithm, which
often yields higher compression ratios than both GZip and BZip2 at the
cost of using more CPU time and RAM. XZ is the slowest for compression,
but still much faster than BZip2 for decompression, almost comparable
to GZip (see benchmarks below).

I ran some simple benchmarks, starting with an already tarball'd archive
of the repos listed below. Memory usage seemed to be consistent for any
given algorithm at their default compression level.

CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)
Virtual Memory Usage
        GZip: 4152K        BZip2: 13352K        XZ: 102M

Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)  349M
gzip    23.70s user   0.47s system  99% cpu   24.227 total    76M
gunzip  3.74s user    0.74s system  94% cpu   4.741 total
bzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M
bunzip2 31.05s user   1.02s system  99% cpu   32.355 total
xz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M
unxz    7.67s user    0.80s system  98% cpu   8.607 total

Git (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M
gzip    0.77s user    0.03s system  99% cpu   0.792 total    2.5M
gunzip  0.12s user    0.02s system  98% cpu   0.142 total
bzip2   3.42s user    0.02s system  99% cpu   3.454 total    2.1M
bunzip2 0.95s user    0.03s system  99% cpu   0.984 total
xz      12.88s user   0.14s system  98% cpu   13.239 total   1.9M
unxz    0.27s user    0.03s system  99% cpu   0.298 total

XZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                1.8M
gzip    0.12s user    0.00s system  95% cpu   0.132 total    442K
gunzip  0.02s user    0.00s system  97% cpu   0.027 total
bzip2   1.28s user    0.01s system  99% cpu   1.298 total    363K
bunzip2 0.15s user    0.01s system  100% cpu  0.157 total
xz      1.62s user    0.03s system  99% cpu   1.652 total    347K
unxz    0.05s user    0.00s system  99% cpu   0.058 total

 From a time and memory perspective, nothing compares to GZip, but if
you have an average upload speed of 20KB/s, it would take ~400 seconds
longer to transfer the BZip2'd kernel snapshot than the XZ snapshot;
the transfer time difference is even greater between GZip and XZ. The
real time savings are relatively the same for all test cases, but less
dramatic for smaller repositories.

XZ decompresses about 1.8-2 times slower than GZip, and 2.7-3.75 times
faster than BZip2, with XZ getting relatively faster as snapshots get
larger. However, it takes relatively longer to compress as snapshots
get larger.

The downside for XZ'd snapshots is the large CPU and memory load put on
the server to generate the compressed snapshot. XZ will eventually have
threading support, and the real clock time for making XZ'd snapshots
would decrease if the server had a beefy multi-core CPU.

XZ compression is disabled by default because the current default is
GZip, and XZ is only really competitive with BZip2; the CPU and memory
requirements will be an issue for high load or lightweight servers.
Also, the XZ format is still new (the format was declared stable ~6
months ago), and there have been no "stable" releases of the utils yet.

Signed-off-by: Mark Rada <redacted>
---
  gitweb/gitweb.perl |    9 +++++++++
  1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index a0cdf31..84659f5 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl

@@ -177,6 +177,14 @@ our %known_snapshot_formats = (
  		'format' => 'tar',
  		'compressor' => ['bzip2']},

+	'txz' => {
+		'display' => 'tar.xz',
+		'type' => 'application/x-xz',
+		'suffix' => '.tar.xz',
+		'format' => 'tar',
+		'compressor' => ['xz'],
+		'disabled' => 1},
+
  	'zip' => {
  		'display' => 'zip',
  		'type' => 'application/x-zip',

@@ -189,6 +197,7 @@ our %known_snapshot_formats = (
  our %known_snapshot_format_aliases = (
  	'gzip'  => 'tgz',
  	'bzip2' => 'tbz2',
+	'xz'    => 'txz',

  	# backward compatibility: legacy gitweb config support
  	'x-gzip' => undef, 'gz' => undef,

-- 
1.6.4