[PATCHv5] Add Gitweb support for XZ compressed snapshots

Subsystems: the rest

9 messages, 4 authors, 2016-06-15 · open the first message on its own page

[PATCHv5] Add Gitweb support for XZ compressed snapshots

From: Mark A Rada <hidden>
Date: 2016-06-15 22:47:08

 >Note that for me the above results are not aligned in table.
 >This is a cosmetic issue.

The table formatting issue was due to a bad habit of mixing tabs and  
spaces,
I decided to go with spaces this time :)


 >One thing that would concern me greatly, is not so much the CPU time  
(though that's a *huge* change in comparison to gz) but the memory  
usage.  Where gzip and bzip2 are chewing 4M and 13M respectively, xz  
chews 102M.  >From a 'beefy' server perspective chewing up that much  
memory per snapshot for that long could be bad.  This is likely  
something that needs to have some sort of enable/disable switch if  
it's going to be included.

True, and there are two solutions I can think of for this "problem".

	1. My tests were at the default compression level, the XZ documentation
	says that at lower levels you will get resource usage and compression
	ratios that are comparable to BZip2. However, I'm not sure where you
	would change the compression level variable for this (globally for the
	system, somewhere in $GITWEB_CONFIG, a git config variable). Does
	someone know the correct answer here?

	2. Implement snapshot caching for Gitweb.


--
Mark A Rada (ferrous26)
marada@uwaterloo.ca


From: Mark Rada <redacted>
Date: Thu, 30 Jul 2009 08:56:42 -0400
Subject: [PATCH] Add Gitweb support for XZ compressed snapshots

The XZ compression format uses the LZMA2 compression algorithm, which
often yields higher compression ratios than both GZip and BZip2 at the
cost of using more CPU time and RAM. Though, while XZ is the slowest
for compression, it is much faster than BZip2 for decompression, almost
comparable to GZip (see benchmarks below).

You can enable XZ compressed snapshots by adding 'txz' to the list of
default options for snapshots in your $GITWEB_CONFIG.

I did some simple benchmarks, starting with an already tarballed
archive of the repos listed below. Memory usage seemed to be consistent
for any given algorithm at their default compression level. Timings were
gathered using the `time' command.

CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)
Virtual Memory Usage
	GZip: 4152K	BZip2: 13352K	XZ: 102M

Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)   349M
gzip    23.70s user   0.47s system  99% cpu   24.227 total    76M
gunzip  3.74s user    0.74s system  94% cpu   4.741 total
bzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M
bunzip2 31.05s user   1.02s system  99% cpu   32.355 total
xz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M
unxz    7.67s user    0.80s system  98% cpu   8.607 total

Git (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M
gzip    0.77s user    0.03s system  99% cpu   0.792 total     2.5M
gunzip  0.12s user    0.02s system  98% cpu   0.142 total
bzip2   3.42s user    0.02s system  99% cpu   3.454 total     2.1M
bunzip2 0.95s user    0.03s system  99% cpu   0.984 total
xz      12.88s user   0.14s system  98% cpu   13.239 total    1.9M
unxz    0.27s user    0.03s system  99% cpu   0.298 total

XZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                 1.8M
gzip    0.12s user    0.00s system  95% cpu   0.132 total     442K
gunzip  0.02s user    0.00s system  97% cpu   0.027 tota
bzip2   1.28s user    0.01s system  99% cpu   1.298 total     363K
bunzip2 0.15s user    0.01s system  100% cpu  0.157 total
xz      1.62s user    0.03s system  99% cpu   1.652 total     347K
unxz    0.05s user    0.00s system  99% cpu   0.058 total

Purely from a time and memory perspective, nothing compares to GZip in
each of the three tests. Though, if you have an average upload speed of
20KB/s, it would take ~400 seconds longer to transfer the kernel
snapshot that was BZip2 compressed than it would the XZ compressed
snapshot, the transfer time difference is even greater when compared to
the GZip compressed snapshot. The wall clock time savings are
relatively the same for all test cases, but less dramatic for the
smaller repositories.

The obvious downside for XZ compressed snapshots is the large CPU and
memory load put on the server to actualy generate the snapshot. Though
XZ will eventually have good threading support, and I suspect then that
the wall clock time for making an XZ compressed snapshot would go down
considerably if the server had a beefy multi-core CPU.

I have not enabled XZ compression by default because the current default
is GZip, and XZ is only really competitive with BZip2. Also, the XZ
format is still fairly new (the format was declared stable about 6
months ago), and there have been no "stable" releases of the utils yet.

Signed-off-by: Mark Rada <redacted>
---
  gitweb/gitweb.perl |    8 ++++++++
  1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 7fbd5ff..3398163 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -176,6 +176,13 @@ our %known_snapshot_formats = (
  		'format' => 'tar',
  		'compressor' => ['bzip2']},

+	'txz' => {
+		'display' => 'tar.xz',
+		'type' => 'application/x-xz',
+		'suffix' => '.tar.xz',
+		'format' => 'tar',
+		'compressor' => ['xz']},
+
  	'zip' => {
  		'display' => 'zip',
  		'type' => 'application/x-zip',
@@ -188,6 +195,7 @@ our %known_snapshot_formats = (
  our %known_snapshot_format_aliases = (
  	'gzip'  => 'tgz',
  	'bzip2' => 'tbz2',
+	'xz'    => 'txz',

  	# backward compatibility: legacy gitweb config support
  	'x-gzip' => undef, 'gz' => undef,
-- 
1.6.4

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

From: J.H. <hidden>
Date: 2016-06-15 22:47:08

Mark A Rada wrote:
 >Note that for me the above results are not aligned in table.
 >This is a cosmetic issue.

The table formatting issue was due to a bad habit of mixing tabs and 
spaces,
I decided to go with spaces this time :)


 >One thing that would concern me greatly, is not so much the CPU time 
(though that's a *huge* change in comparison to gz) but the memory 
usage.  Where gzip and bzip2 are chewing 4M and 13M respectively, xz 
chews 102M.  >From a 'beefy' server perspective chewing up that much 
memory per snapshot for that long could be bad.  This is likely 
something that needs to have some sort of enable/disable switch if it's 
going to be included.

True, and there are two solutions I can think of for this "problem".

    1. My tests were at the default compression level, the XZ documentation
    says that at lower levels you will get resource usage and compression
    ratios that are comparable to BZip2. However, I'm not sure where you
    would change the compression level variable for this (globally for the
    system, somewhere in $GITWEB_CONFIG, a git config variable). Does
    someone know the correct answer here?
Well you can always call xz with -[1-9] to change the compression level 
(same as gzip and bzip2) though I think a full disabling would be 'more' 
preferable, though I'm not sure I like Jakub's suggestion of just 
deleting it after the fact, it would work.
    2. Implement snapshot caching for Gitweb.
I think it's slightly broken in my version (binary files don't work 
right apparently, it's on my todo list to fix in my upcoming update) but 
both Lea and I have done this long ago (caching layers in Gitweb), which 
would be an acceptable workaround for this, create once and serve many - 
though this has the downside of trading cpu for diskspace.  At least 
with xz there's less diskspace used ;-)

I think more my concern is more what's enabled by default, and since xz 
is still new (as was pointed out) it's probably worth only enabling if 
the admin selects it to be enabled.

- John 'Warthog9' Hawley

quoted hunk
-- 
Mark A Rada (ferrous26)
marada@uwaterloo.ca


From: Mark Rada <redacted>
Date: Thu, 30 Jul 2009 08:56:42 -0400
Subject: [PATCH] Add Gitweb support for XZ compressed snapshots

The XZ compression format uses the LZMA2 compression algorithm, which
often yields higher compression ratios than both GZip and BZip2 at the
cost of using more CPU time and RAM. Though, while XZ is the slowest
for compression, it is much faster than BZip2 for decompression, almost
comparable to GZip (see benchmarks below).

You can enable XZ compressed snapshots by adding 'txz' to the list of
default options for snapshots in your $GITWEB_CONFIG.

I did some simple benchmarks, starting with an already tarballed
archive of the repos listed below. Memory usage seemed to be consistent
for any given algorithm at their default compression level. Timings were
gathered using the `time' command.

CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)
Virtual Memory Usage
    GZip: 4152K    BZip2: 13352K    XZ: 102M

Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)   349M
gzip    23.70s user   0.47s system  99% cpu   24.227 total    76M
gunzip  3.74s user    0.74s system  94% cpu   4.741 total
bzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M
bunzip2 31.05s user   1.02s system  99% cpu   32.355 total
xz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M
unxz    7.67s user    0.80s system  98% cpu   8.607 total

Git (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M
gzip    0.77s user    0.03s system  99% cpu   0.792 total     2.5M
gunzip  0.12s user    0.02s system  98% cpu   0.142 total
bzip2   3.42s user    0.02s system  99% cpu   3.454 total     2.1M
bunzip2 0.95s user    0.03s system  99% cpu   0.984 total
xz      12.88s user   0.14s system  98% cpu   13.239 total    1.9M
unxz    0.27s user    0.03s system  99% cpu   0.298 total

XZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                 1.8M
gzip    0.12s user    0.00s system  95% cpu   0.132 total     442K
gunzip  0.02s user    0.00s system  97% cpu   0.027 tota
bzip2   1.28s user    0.01s system  99% cpu   1.298 total     363K
bunzip2 0.15s user    0.01s system  100% cpu  0.157 total
xz      1.62s user    0.03s system  99% cpu   1.652 total     347K
unxz    0.05s user    0.00s system  99% cpu   0.058 total

Purely from a time and memory perspective, nothing compares to GZip in
each of the three tests. Though, if you have an average upload speed of
20KB/s, it would take ~400 seconds longer to transfer the kernel
snapshot that was BZip2 compressed than it would the XZ compressed
snapshot, the transfer time difference is even greater when compared to
the GZip compressed snapshot. The wall clock time savings are
relatively the same for all test cases, but less dramatic for the
smaller repositories.

The obvious downside for XZ compressed snapshots is the large CPU and
memory load put on the server to actualy generate the snapshot. Though
XZ will eventually have good threading support, and I suspect then that
the wall clock time for making an XZ compressed snapshot would go down
considerably if the server had a beefy multi-core CPU.

I have not enabled XZ compression by default because the current default
is GZip, and XZ is only really competitive with BZip2. Also, the XZ
format is still fairly new (the format was declared stable about 6
months ago), and there have been no "stable" releases of the utils yet.

Signed-off-by: Mark Rada <redacted>
---
 gitweb/gitweb.perl |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 7fbd5ff..3398163 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -176,6 +176,13 @@ our %known_snapshot_formats = (
         'format' => 'tar',
         'compressor' => ['bzip2']},

+    'txz' => {
+        'display' => 'tar.xz',
+        'type' => 'application/x-xz',
+        'suffix' => '.tar.xz',
+        'format' => 'tar',
+        'compressor' => ['xz']},
+
     'zip' => {
         'display' => 'zip',
         'type' => 'application/x-zip',
@@ -188,6 +195,7 @@ our %known_snapshot_formats = (
 our %known_snapshot_format_aliases = (
     'gzip'  => 'tgz',
     'bzip2' => 'tbz2',
+    'xz'    => 'txz',

     # backward compatibility: legacy gitweb config support
     'x-gzip' => undef, 'gz' => undef,

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

From: demerphq <hidden>
Date: 2016-06-15 22:47:08

2009/8/1 J.H. [off-list ref]:
Mark A Rada wrote:
quoted
 >Note that for me the above results are not aligned in table.
 >This is a cosmetic issue.

The table formatting issue was due to a bad habit of mixing tabs and
spaces,
I decided to go with spaces this time :)


 >One thing that would concern me greatly, is not so much the CPU time
(though that's a *huge* change in comparison to gz) but the memory usage.
 Where gzip and bzip2 are chewing 4M and 13M respectively, xz chews 102M.
 >From a 'beefy' server perspective chewing up that much memory per snapshot
for that long could be bad.  This is likely something that needs to have
some sort of enable/disable switch if it's going to be included.

True, and there are two solutions I can think of for this "problem".

   1. My tests were at the default compression level, the XZ documentation
   says that at lower levels you will get resource usage and compression
   ratios that are comparable to BZip2. However, I'm not sure where you
   would change the compression level variable for this (globally for the
   system, somewhere in $GITWEB_CONFIG, a git config variable). Does
   someone know the correct answer here?
Well you can always call xz with -[1-9] to change the compression level
(same as gzip and bzip2) though I think a full disabling would be 'more'
preferable, though I'm not sure I like Jakub's suggestion of just deleting
it after the fact, it would work.
quoted
   2. Implement snapshot caching for Gitweb.
I think it's slightly broken in my version (binary files don't work right
apparently, it's on my todo list to fix in my upcoming update) but both Lea
and I have done this long ago (caching layers in Gitweb), which would be an
acceptable workaround for this, create once and serve many - though this has
the downside of trading cpu for diskspace.  At least with xz there's less
diskspace used ;-)

I think more my concern is more what's enabled by default, and since xz is
still new (as was pointed out) it's probably worth only enabling if the
admin selects it to be enabled.
FWIW the perl project ripped out all the snapshot generation logic
from gitweb, and replaced it with a tool that generates snapshots
correctly for our requirements (if the build process needs additional
files /currently/ git-archive does not support adding them), this
includes a disk level cache for the snapshots since creating the tar,
adding the additional files, then gziping is quite slow.

If its interesting to people I can post it and the other changes here,
although its not a "nice" change, as I literally ripped out the
existing code.

cheers,
Yves





-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

From: Jakub Narebski <hidden>
Date: 2016-06-15 22:47:08

"J.H." [off-list ref] writes:
Well you can always call xz with -[1-9] to change the compression
level (same as gzip and bzip2) though I think a full disabling would
be 'more' preferable, though I'm not sure I like Jakub's suggestion of
just deleting it after the fact, it would work.
[...]

The problem is that 'keys %known_snapshot_formats' serves also as list
of allowed snapshot formats, if project specific override is enabled.
We can add another optional flag ('disabled' => 1) if you don't want
to delete from %known_snapshot_formats in $GITWEB_CONFIG, though I
don't know if it is worth it.  Anyway such mechanism can be added, and
IMHO should be added, in a separate commit.
I think more my concern is more what's enabled by default, and since
xz is still new (as was pointed out) it's probably worth only enabling
if the admin selects it to be enabled.
By default (i.e. as in gitweb in git.git) 'snapshot' feature has
disabled projects specific override.  Which means only 'tgz' snapshot
is enabled / used.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

From: Jakub Narebski <hidden>
Date: 2016-06-15 22:47:08

demerphq [off-list ref] writes:
FWIW the perl project ripped out all the snapshot generation logic
from gitweb, and replaced it with a tool that generates snapshots
correctly for our requirements (if the build process needs additional
files /currently/ git-archive does not support adding them), this
includes a disk level cache for the snapshots since creating the tar,
adding the additional files, then gziping is quite slow.

If its interesting to people I can post it and the other changes here,
although its not a "nice" change, as I literally ripped out the
existing code.
Do you mean gitweb serving http://perl5.git.perl.org/ ?

Well, at least publish this version of gitweb somewhere (snapshot or
better git repository), and add it to the list of gitweb forks at
http://git.or.cz/gitwiki/Gitweb wiki page.

Sidenote: snapshot support appeared, as far as I know, first at fork of
gitweb that had been used by XMMS2, but the snapshot was generated by
a separate snapshot.cgi script... written in Python.  (Now they use
stock (if old) gitweb, I think.)

P.S. Perhaps you or other Perl Mongers would be interested in creating
yet another git web interface, in Perl, but contrary to (current)
gitweb modular and using more of CPAN modules?  For example convert
SVN::Web or Insurrection (or other Perl module) from Subversion (or
other SCM) to Git (and rename it to Git::Web, or ???).

P.P.S. Could you per chance post announcement of Git User's Survey 2009
on http://perl5.git.perl.org, just like it is done on http://repo.or.cz
and http://git.kernel.org?  TIA.

-- 
Jakub Narebski
ShadeHawk on #git

Git User's Survey 2009
http://tinyurl.com/GitSurvey2009

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

From: demerphq <hidden>
Date: 2016-06-15 22:47:08

2009/8/1 Jakub Narebski [off-list ref]:
demerphq [off-list ref] writes:
quoted
FWIW the perl project ripped out all the snapshot generation logic
from gitweb, and replaced it with a tool that generates snapshots
correctly for our requirements (if the build process needs additional
files /currently/ git-archive does not support adding them), this
includes a disk level cache for the snapshots since creating the tar,
adding the additional files, then gziping is quite slow.

If its interesting to people I can post it and the other changes here,
although its not a "nice" change, as I literally ripped out the
existing code.
Do you mean gitweb serving http://perl5.git.perl.org/ ?
Yes. I updated it to a relatively recent version just the other week.
Well, at least publish this version of gitweb somewhere (snapshot or
better git repository), and add it to the list of gitweb forks at
http://git.or.cz/gitwiki/Gitweb wiki page.
Sure, we have been discussing doing that recently.

I need to cleanup how I set up the repository for it tho.

Also probably there are one or two patches that should be pushed back to you.
Sidenote: snapshot support appeared, as far as I know, first at fork of
gitweb that had been used by XMMS2, but the snapshot was generated by
a separate snapshot.cgi script... written in Python.  (Now they use
stock (if old) gitweb, I think.)
Ah, I'm not generating it with a separate cgi script, I just replaced
the call to git-archive to be a call to my own tool.
P.S. Perhaps you or other Perl Mongers would be interested in creating
yet another git web interface, in Perl, but contrary to (current)
gitweb modular and using more of CPAN modules?  For example convert
SVN::Web or Insurrection (or other Perl module) from Subversion (or
other SCM) to Git (and rename it to Git::Web, or ???).
I think that there is some interest in doing that and likely over time
there will be more. It has come up at least a few times in our
discussion forums. I think you could easily make a public call for
support on places like Perlmonks to get more action tho.

A better git api toolset for perl would make things a lot easier.
Especially an XS one.
P.P.S. Could you per chance post announcement of Git User's Survey 2009
on http://perl5.git.perl.org, just like it is done on http://repo.or.cz
and http://git.kernel.org?  TIA.
Sure.

cheers,
Yves
-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

From: Jakub Narebski <hidden>
Date: 2016-06-15 22:47:08

On Sat, 1 Aug 2009, demerphq wrote:
2009/8/1 Jakub Narebski [off-list ref]:
quoted
demerphq [off-list ref] writes:
quoted
FWIW the perl project ripped out all the snapshot generation logic
from gitweb, and replaced it with a tool that generates snapshots
correctly for our requirements (if the build process needs additional
files /currently/ git-archive does not support adding them), this
includes a disk level cache for the snapshots since creating the tar,
adding the additional files, then gziping is quite slow.

If its interesting to people I can post it and the other changes here,
although its not a "nice" change, as I literally ripped out the
existing code.
Do you mean gitweb serving http://perl5.git.perl.org/ ?
Yes. I updated it to a relatively recent version just the other week.
By the way, if you don't mind, how do you keep your changes on top
of changing target, i.e. on top of updating baseline to recent 
versions?
quoted
Well, at least publish this version of gitweb somewhere (snapshot or
better git repository), and add it to the list of gitweb forks at
http://git.or.cz/gitwiki/Gitweb wiki page.
Sure, we have been discussing doing that recently.

I need to cleanup how I set up the repository for it tho.
O.K.
Also probably there are one or two patches that should be pushed back
to you. 
What do those patches consist of?
 
quoted
Sidenote: snapshot support appeared, as far as I know, first at fork of
gitweb that had been used by XMMS2, but the snapshot was generated by
a separate snapshot.cgi script... written in Python.  (Now they use
stock (if old) gitweb, I think.)
Ah, I'm not generating it with a separate cgi script, I just replaced
the call to git-archive to be a call to my own tool.
O.K.  

By the way, this is one of the only two places where we have to
use quote_command and 3-argument form of open, instead of list for of
magic open.  It is because of pipeline, piping git-archive output
into compressor.  The other such place is git_object, to redirect error
stream to /dev/null (to discard stderr).

I thought about replacing it by list form of open somewhat (you can
find it in git mailing list archive), but it is not easy.  And IPC::Run
is IMVHO a bit of overkill, especially for "minimal dependencies"
gitweb (perhaps for Git::Web?).
quoted
P.S. Perhaps you or other Perl Mongers would be interested in creating
yet another git web interface, in Perl, but contrary to (current)
gitweb modular and using more of CPAN modules?  For example convert
SVN::Web or Insurrection (or other Perl module) from Subversion (or
other SCM) to Git (and rename it to Git::Web, or ???).
I think that there is some interest in doing that and likely over time
there will be more. It has come up at least a few times in our
discussion forums. I think you could easily make a public call for
support on places like Perlmonks to get more action tho.
Thanks in advance.
A better git api toolset for perl would make things a lot easier.
Especially an XS one.
Well, in the beginnings of Git.pm there was XS interface (to 
git-hash-object or git-cat-file), but it was dropped because it
used -fPIC, which is not portable enough for Git (this is not present
in perl/Git.pm history in git.git repository).  Git.pm was created
by Petr 'Pasky' Baudis, author of Cogito porcelain, and creator and
admin of repo.or.cz (and also creator of first Git homepage).  
But he is not a Perl hacker (correct me if I am wrong, Pasky); both
Cogito and repo.or.cz duct tape (under name of Girocco) are written
in bash, not in Perl.  Which can be seen for example by using 
Error::Simple in Git.pm...

Also I don't think that Pasky has time for maintaining Git.pm, 
nevermind modernizing it / adding new features.

There was another attempt to make Perl interface for Git, more 
object-oriented, by Lea Wiemann during GSoC 2008 project 'gitweb caching'.
You can see results in Lea repository at repo.or.cz[1], and also in 
git.kernel.org gitweb code[2].

[1]: http://repo.or.cz/w/git/gitweb-caching.git
[2]: http://git.kernel.org/?p=git/warthog9/gitweb.git
 
quoted
P.P.S. Could you per chance post announcement of Git User's Survey 2009
on http://perl5.git.perl.org, just like it is done on http://repo.or.cz
and http://git.kernel.org?  TIA.
Sure.
Hmmm... I don't see "Git User's Survey 2009" announcement at 
http://perl5.git.perl.org/.  Neither at Planet Perl Iron Man, nor at
Perlsphere

-- 
Jakub Narebski

Git User's Survey 2009:
http://tinyurl.com/GitSurvey2009

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

From: demerphq <hidden>
Date: 2016-06-15 22:47:08

2009/8/3 Jakub Narebski [off-list ref]:
On Sat, 1 Aug 2009, demerphq wrote:
quoted
2009/8/1 Jakub Narebski [off-list ref]:
quoted
demerphq [off-list ref] writes:
quoted
FWIW the perl project ripped out all the snapshot generation logic
from gitweb, and replaced it with a tool that generates snapshots
correctly for our requirements (if the build process needs additional
files /currently/ git-archive does not support adding them), this
includes a disk level cache for the snapshots since creating the tar,
adding the additional files, then gziping is quite slow.

If its interesting to people I can post it and the other changes here,
although its not a "nice" change, as I literally ripped out the
existing code.
Do you mean gitweb serving http://perl5.git.perl.org/ ?
Yes. I updated it to a relatively recent version just the other week.
By the way, if you don't mind, how do you keep your changes on top
of changing target, i.e. on top of updating baseline to recent
versions?
Things broke. I fixed them. :-)

quoted
Also probably there are one or two patches that should be pushed back
to you.
What do those patches consist of?
One has to do with layout problems in the blame, and I think one deals
with broken line number anchor links.

Ill look closer when i have more time.
By the way, this is one of the only two places where we have to
use quote_command and 3-argument form of open, instead of list for of
magic open.  It is because of pipeline, piping git-archive output
into compressor.  The other such place is git_object, to redirect error
stream to /dev/null (to discard stderr).

I thought about replacing it by list form of open somewhat (you can
find it in git mailing list archive), but it is not easy.  And IPC::Run
is IMVHO a bit of overkill, especially for "minimal dependencies"
gitweb (perhaps for Git::Web?).
Lately I have been playing with Capture::Tiny, which seems to fill
some nice niches in this respect.
quoted
A better git api toolset for perl would make things a lot easier.
Especially an XS one.
Well, in the beginnings of Git.pm there was XS interface (to
git-hash-object or git-cat-file), but it was dropped because it
used -fPIC, which is not portable enough for Git (this is not present
in perl/Git.pm history in git.git repository).  Git.pm was created
by Petr 'Pasky' Baudis, author of Cogito porcelain, and creator and
admin of repo.or.cz (and also creator of first Git homepage).
But he is not a Perl hacker (correct me if I am wrong, Pasky); both
Cogito and repo.or.cz duct tape (under name of Girocco) are written
in bash, not in Perl.  Which can be seen for example by using
Error::Simple in Git.pm...

Also I don't think that Pasky has time for maintaining Git.pm,
nevermind modernizing it / adding new features.

There was another attempt to make Perl interface for Git, more
object-oriented, by Lea Wiemann during GSoC 2008 project 'gitweb caching'.
You can see results in Lea repository at repo.or.cz[1], and also in
git.kernel.org gitweb code[2].

[1]: http://repo.or.cz/w/git/gitweb-caching.git
[2]: http://git.kernel.org/?p=git/warthog9/gitweb.git
Yeah I recall that being discussed. I see that Git.pm has recevied
some attention since the last time I looked. Which is good to see.

Having lurked on the list I understand that there is some kind of
effort for a git library.  Perhaps at some point that will end up with
an XS interface.
quoted
quoted
P.P.S. Could you per chance post announcement of Git User's Survey 2009
on http://perl5.git.perl.org, just like it is done on http://repo.or.cz
and http://git.kernel.org?  TIA.
Sure.
Hmmm... I don't see "Git User's Survey 2009" announcement at
http://perl5.git.perl.org/.  Neither at Planet Perl Iron Man, nor at
Perlsphere
Ah, sorry, I posted it to the perl5-porters mailing list and asked the
relevant people to forward the request on or post it on their site. I
know at least one perl committer already did it. However tomorrow the
main european perl conference starts (YAPC::EU in Lisbon) and I think
a big chunk of the "active" Perl community will be there, so probably
there wont be much action from them until they get back.

To be honest I'm not familiar with Planet Perl Iron Man nor
Perlsphere. Ill have to check them out i guess. :-)

I will try to remember to put up a note on the website as soon as possible.

cheers,
Yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots

From: Jakub Narebski <hidden>
Date: 2016-06-15 22:47:08

On Mon, 3 August 2009, demerphq wrote:
2009/8/3 Jakub Narebski [off-list ref]:
quoted
On Sat, 1 Aug 2009, demerphq wrote:
quoted
2009/8/1 Jakub Narebski [off-list ref]:
quoted
quoted
quoted
Do you mean gitweb serving http://perl5.git.perl.org/ ?
Yes. I updated it to a relatively recent version just the other week.
By the way, if you don't mind, how do you keep your changes on top
of changing target, i.e. on top of updating baseline to recent
versions?
Things broke. I fixed them. :-)
Err... I meant here to ask if you use merging into your branch, rebasing
your branch on top of upstream, using some patch management interface like
StGIT, Guilt ot Quilt, or topic branch management tool namely TopGit?
 
quoted
quoted
Also probably there are one or two patches that should be pushed back
to you.
What do those patches consist of?
One has to do with layout problems in the blame, and I think one deals
with broken line number anchor links.

Ill look closer when i have more time.
About the one with broken 'linenr' links: you should look at patch which
makes gitweb use information in "previous" header from git-blame -p, 
which was send to git mailing list and is currently in 'pu':
  [PATCHv2 03/10] gitweb: Use "previous" header of git-blame -p in 'blame' view
  Message-Id: [off-list ref]
  http://article.gmane.org/gmane.comp.version-control.git/123961


What layout problems in 'blame' view did you notice? 
quoted
By the way, this is one of the only two places where we have to
use quote_command and 3-argument form of open, instead of list for of
magic open.  It is because of pipeline, piping git-archive output
into compressor.  The other such place is git_object, to redirect error
stream to /dev/null (to discard stderr).

I thought about replacing it by list form of open somewhat (you can
find it in git mailing list archive), but it is not easy.  And IPC::Run
is IMVHO a bit of overkill, especially for "minimal dependencies"
gitweb (perhaps for Git::Web?).
Lately I have been playing with Capture::Tiny, which seems to fill
some nice niches in this respect.
I don't quite see how you plan to use Capture::Tiny here.  The output
of git-archive has to be passed to compressor, and then dumped to stdout,
not captured.

-- 
Jakub Narebski
Poland
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help