Add Gitweb support for LZMA compressed snapshots

Hi,

I thought I would submit this little patch I made to my gitweb. I am  
on a relatively slow
connection, and so LZMA compression time is less of a  concern than  
bandwidth---I'm
guessing that I am not the only person who suffers from slow internet  
connection
syndrome.


--
Mark A Rada (ferrous26)
marada@uwaterloo.ca

Hi,

On Thu, 30 Jul 2009, Mark A Rada wrote:

I thought I would submit this little patch I made to my gitweb. I am on 
a relatively slow connection, and so LZMA compression time is less of a 
concern than bandwidth---I'm guessing that I am not the only person who 
suffers from slow internet connection syndrome.

That's great!

FWIW on this mailing list, we prefer to have patches inlined so that it is 
easier for reviewers to comment (it is almost impossible to get any patch 
into git.git without review, and because reviewing is such an ungratifying 
job you want to make it easier).

The patch is pretty straight-forward, but I'd love to see some information 
in the commit message about:

- performance numbers on the _server_ (i.e. how does the RAM and CPU load 
  compare to, say, bzip2?)

- how to disable/enable it (yes, it's in the documentation, but you can 
  make things easier still)

- whether it is turned on by default, and why.

Thanks,
Dscho

Mark A Rada [off-list ref] writes:

I thought I would submit this little patch I made to my gitweb. I am
on a relatively slow connection, and so LZMA compression time is
less of a concern than bandwidth---I'm guessing that I am not the
only person who suffers from slow internet connection syndrome.

First, Documentation/SubmittingPatches states that we prefer inline
patches, with additional comments (like the one above) either between
"---\n" line and diffstat, or like I did here before patch with patch
separated by comment by e.g. "-- >8 --\n" (scissors) line.

If posting patch inline isn't possible, and you have to use
attachement (for example because mailer you use wraps lines), please
at least use 'text/plain' mimetype so patch can be viewed without need
to save it in separate file (this might require changing suffix from
'.patch' to '.txt').


Second, if you are using 'txz' as internal name for new snapshot
format, why not use 'XZ utils' instead of 'LZMA Utils', and use
'display' => 'txz', 'type' => 'application/x-xz' and 
'suffix' => '.txz' (BTW. shouldn't suffix in your case be '.tar.lzma'
and not only '.lzma'?) and of course 'compressor' => ['xz']?


BTW. I wonder if it would be good idea to add support for this format
directly to git-archive...  OTOH it would mean additional dependency.

-- 
Jakub Narebski
Git User's Survey 2009: http://tinyurl.com/GitSurvey2009

-- >8 --
From: Mark Rada <redacted>
Subject: [PATCH] Add Gitweb support for LZMA compressed snapshots

Signed-off-by: Mark Rada <redacted>
---
 gitweb/gitweb.perl |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 38492d0..e131ea4 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl

@@ -184,6 +184,13 @@ our %known_snapshot_formats = (
 		'format' => 'tar',
 		'compressor' => ['bzip2']},
 
+	'txz' => {
+		'display' => 'tar.lzma',
+		'type' => 'application/x-lzma',
+		'suffix' => '.lzma',
+		'format' => 'tar',
+		'compressor' => ['lzma']},
+
 	'zip' => {
 		'display' => 'zip',
 		'type' => 'application/x-zip',

@@ -196,6 +203,7 @@ our %known_snapshot_formats = (
 our %known_snapshot_format_aliases = (
 	'gzip'  => 'tgz',
 	'bzip2' => 'tbz2',
+	'lzma'  => 'txz',
 
 	# backward compatibility: legacy gitweb config support
 	'x-gzip' => undef, 'gz' => undef,

-- 
1.6.4

Mark A Rada wrote:

Hi,

I thought I would submit this little patch I made to my gitweb. I am on 
a relatively slow
connection, and so LZMA compression time is less of a  concern than 
bandwidth---I'm
guessing that I am not the only person who suffers from slow internet 
connection
syndrome.


-- 
Mark A Rada (ferrous26)
marada@uwaterloo.ca

Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it 
uses the lzma2 format which is, by far, preferable to what 'lzma' 
outputs.  Same compression (lzma) just different file format.

- John 'Warthog9' Hawley

Hi,

On Thu, 30 Jul 2009, Jakub Narebski wrote:

BTW. I wonder if it would be good idea to add support for this format
directly to git-archive...  OTOH it would mean additional dependency.

I don't think it would be a good idea; we do not have bzip2 support 
either.

The only reason we have inbuilt gzip and zip support is because the format 
is so similar to Git's own compression.

Ciao,
Dscho

On Thu, Jul 30, 2009 at 8:43 AM, Johannes
Schindelin[off-list ref] wrote:

Hi,

On Thu, 30 Jul 2009, Jakub Narebski wrote:

quoted

BTW. I wonder if it would be good idea to add support for this format
directly to git-archive...  OTOH it would mean additional dependency.

I don't think it would be a good idea; we do not have bzip2 support
either.

bzip2 has no advantages whatsoever. AFAIK xz is superior to other
formats and it would be nice to see git make a technological stance
encouraging xz.

The only reason we have inbuilt gzip and zip support is because the format
is so similar to Git's own compression.

Personally I don't see the point of having zip support.

-- 
Felipe Contreras

On Thu, Jul 30, 2009 at 09:44, J.H.[off-list ref] wrote:

Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses

The host seems to be down.

the lzma2 format which is, by far, preferable to what 'lzma' outputs.  Same
compression (lzma) just different file format.

And the whole project seems to be very young (listen on ohloh since 2007).
And it is not in current stable distributions. Exotic thing.

Hi,

On Fri, 31 Jul 2009, Felipe Contreras wrote:

On Thu, Jul 30, 2009 at 8:43 AM, Johannes
Schindelin[off-list ref] wrote:

quoted

Hi,

On Thu, 30 Jul 2009, Jakub Narebski wrote:

quoted

BTW. I wonder if it would be good idea to add support for this format
directly to git-archive...  OTOH it would mean additional dependency.

I don't think it would be a good idea; we do not have bzip2 support
either.

bzip2 has no advantages whatsoever.

Bzzzzt.  Wrong.  Just because you cannot see them does not mean it has no 
advantages.

AFAIK xz is superior to other formats and it would be nice to see git 
make a technological stance encouraging xz.

Bzzzt. Wrong again.  Git's mission in life is not to encourage one 
compression over another.

If at all, the only compression Git actually does promote in a sense is 
zlib compression.

quoted

The only reason we have inbuilt gzip and zip support is because the 
format is so similar to Git's own compression.

Personally I don't see the point of having zip support.

Personally, I see the point of having zip support.  It makes things easy 
for Windows users.  And it's an established format, much more so than 
tar.gz.

Ciao,
Dscho

On Sat, Aug 1, 2009 at 12:34 PM, Johannes
Schindelin[off-list ref] wrote:

Hi,

On Fri, 31 Jul 2009, Felipe Contreras wrote:

quoted

On Thu, Jul 30, 2009 at 8:43 AM, Johannes
Schindelin[off-list ref] wrote:

quoted

Hi,

On Thu, 30 Jul 2009, Jakub Narebski wrote:

quoted

BTW. I wonder if it would be good idea to add support for this format
directly to git-archive...  OTOH it would mean additional dependency.

I don't think it would be a good idea; we do not have bzip2 support
either.

bzip2 has no advantages whatsoever.

Bzzzzt.  Wrong.  Just because you cannot see them does not mean it has no
advantages.

All right, most the time you compress once, and multiple people
uncompress multiple times. Therefore the path that should be optimized
is the decompression, that leaves bzip2 out of the picture.

You can think of advantages of bzip2 in more esoteric cases where the
compression time is also important. I don't think that's the case
here. Actually it might be if gitweb is not caching the tarballs, but
if that's the case I wouldn't say it's an advantage of bzip2.

quoted

AFAIK xz is superior to other formats and it would be nice to see git
make a technological stance encouraging xz.

Bzzzt. Wrong again.  Git's mission in life is not to encourage one
compression over another.

Git's mission in life is not promoting asciidot either, but it's doing
that unintentionally by merely using it.

If at all, the only compression Git actually does promote in a sense is
zlib compression.

quoted

quoted

The only reason we have inbuilt gzip and zip support is because the
format is so similar to Git's own compression.

Personally I don't see the point of having zip support.

Personally, I see the point of having zip support.  It makes things easy
for Windows users.  And it's an established format, much more so than
tar.gz.

Windows can't extract .tar.gz?

-- 
Felipe Contreras

On Sat, Aug 1, 2009 at 3:10 PM, Felipe
Contreras[off-list ref] wrote:

Windows can't extract .tar.gz?

Not without installing 3. party software. Zip-support is built into
the operating system.

-- 
Erik "kusma" Faye-Lund
kusmabite@gmail.com
(+47) 986 59 656

On Fri, Jul 31, 2009 at 03:45:02PM +0000, Felipe Contreras wrote:

bzip2 has no advantages whatsoever. AFAIK xz is superior to other
formats and it would be nice to see git make a technological stance
encouraging xz.

git is not about encouraging any compression. If you have not noticed,
git-archive produces tar and not tgz (though it would not be difficult
to implement it, because we use zlib anyway). If you like xz and it is
installed on your system, you can always compress tar with it:

git archive HEAD | xz > my-archive.tar.xz

So, I see no reason to have this compression coupled with git.

quoted

The only reason we have inbuilt gzip and zip support is because the format
is so similar to Git's own compression.

Personally I don't see the point of having zip support.

zip is still a popular format on Windows. If git archive was not able to
produce then creating would require creating tar first, extracting it to
some temporary directory and then creating a zip file. It is not very
efficient process and requires extra efforts, so because we use zlib
compression anyway, it makes perfect sense that we can create zip file
directly.

Dmitry

On Sat, Aug 01, 2009 at 09:43:39AM +0200, Alex Riesen wrote:

And the whole project seems to be very young (listen on ohloh since 2007).
And it is not in current stable distributions. Exotic thing.

The project seems to be it is developed together with Igor Pavlov, who
is the author of 7-Zip (and now 7-Zip also supports LZMA2 and XZ
format). Accordingly to Igor Pavlov:

   LZMA2 provides the following advantages over LZMA:
   1) Better compression ratio for data than can't be compressed.
   It can store such blocks of data in uncompressed form.
   Also it decompresses such data faster.
   2) Better multithreading support. If you compress big file, LZMA2 can split
   that file to chunks and compress these chunks in multiple threads.

Clearly XZ is not very popular yet, but I would not call it as exotic.
At least, I expect it to be supported by future distributives.


Dmitry

On Sat, Aug 1, 2009 at 4:43 AM, Alex Riesen[off-list ref] wrote:

On Thu, Jul 30, 2009 at 09:44, J.H.[off-list ref] wrote:

quoted

Don't use 'lzma' the command, use 'xz' ( http://tukaani.org/xz/ ) as it uses

The host seems to be down.

You can clone it here: git://ctrl.tukaani.org/xz.git

Best regards,
Andre

On 1-Aug-09, at 10:04 AM, Erik Faye-Lund wrote:

On Sat, Aug 1, 2009 at 3:10 PM, Felipe
Contreras[off-list ref] wrote:

quoted

Windows can't extract .tar.gz?

Not without installing 3. party software. Zip-support is built into
the operating system.

-- 
Erik "kusma" Faye-Lund
kusmabite@gmail.com
(+47) 986 59 656
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

FWIW, Windows doesn't seem to come with a whole lot, and you need
3rd party software for a number of things, BUT that is also one of the
primary reasons that I need to be able to  quickly get snapshots from
my Git repositories at home, and using zip format is convenient if I am
in an extremely restrictive environment and cannot get a smaller XZ
compressed snapshot :).


--
Mark A Rada (ferrous26)
marada@uwaterloo.ca

On Sat, Aug 1, 2009 at 6:07 PM, Mark A Rada[off-list ref] wrote:

FWIW, Windows doesn't seem to come with a whole lot, and you need
3rd party software for a number of things

Tell that to the US Department of Justice ;)

-- 
Erik "kusma" Faye-Lund
kusmabite@gmail.com
(+47) 986 59 656