Incremental cvsimports

Dear Git,

I have a 3Gb repository (10 years of history) which I want to import into git.

I have tested little pieces of that and a 23Mb directory expands to 120Mb prior
to packing/pruning.  Therefore I don't have a big enough disk to import the
whole repository at once. So I figure I could do it in pieces with a pack after
each piece.

So I tried something like:

 1.      $ git cvsimport  -o austrics -v -d /cvsroot -C /GitRepo /cvsroot/A

followed by

 2.     $ git cvsimport  -o austrics -v -d /cvsroot -C /GitRepo /cvsroot/B

After 1, /GitRepo/.git is set up but /GitRepo has the members of A,
not A itself.
After 2, /GitRepo also has members of B. This isn't quite what I wanted!

Is there a way to do this?

Cheers,
Geoff Russell

On 5/24/06, Geoff Russell [off-list ref] wrote:

Dear Git,

Dear Geoff,

if you look at the list archive for the last couple of days, you'll
see there's been quite a bit of activity in tuning cvsimport so that
it scales better with large imports like yours. We have been playing
with a gentoo cvs repo with 300K commits / 1.6GB uncompressed.

Don't split up the tree... that'll lead to something rather ackward.
Instead, fetch and build git from Junio's 'master' branch which seems
to have collected most (all?) of the patches posted, including one
from Linus that will repack the repo every 1K commits -- keeping the
import size down.

You _will_ need a lot of memory though, as cvsps grows large (working
on a workaround now) and cvsimport grows a bit over time (where is
that last leak?!). And a fast machine -- specially fast IO. I've just
switched from an old test machine to an AMD64 with fast disks, and
it's importing around 10K commits per hour.

You will probably want to run cvsps by hand, and later use the -P flag.

cheers,


martin

On 5/24/06, Martin Langhoff [off-list ref] wrote:

You _will_ need a lot of memory though, as cvsps grows large (working
on a workaround now)

While the workaround is worked around, you can create the cvsps file
using something like

CVSROOT=~/tmp/gentoorepo/ cvsps --norc -x -u -A gentoo-x86  >
gentoo.cvsps 2> gentoo.err

and then pass a -P gentoo.cvsps to cvsimport. s/gentoo/myproject/ ;-)


martin

Dear Martin,


On 5/24/06, Martin Langhoff [off-list ref] wrote:

On 5/24/06, Geoff Russell [off-list ref] wrote:

quoted

Dear Git,

Dear Geoff,

if you look at the list archive for the last couple of days, you'll
see there's been quite a bit of activity in tuning cvsimport so that
it scales better with large imports like yours. We have been playing
with a gentoo cvs repo with 300K commits / 1.6GB uncompressed.

Don't split up the tree... that'll lead to something rather ackward.
Instead, fetch and build git from Junio's 'master' branch which seems
to have collected most (all?) of the patches posted, including one
from Linus that will repack the repo every 1K commits -- keeping the
import size down.

I got the latest git and yes, the size is kept down. I've only tried with
a smaller repository but it looks promising. When I ran git-cvsimport without a
CVS-module name (wanting the entire repository), it gave me a Usage message
indicating that the CVS-module name was optional - but it isn't :)

I did have to change
2 lines in git-cvsimport to get it to run with my 5.8.0 perl (problems with
POSIX errno). I've attached a patch but my work around isn't as quick as
what it replaced.

Many thanks, I'll have a go with the big repository at work tomorrow!

Cheers,
Geoff Russell

P.S. I've just started to look with git. We have wanted a cvs replacement for
a while but have been too scared to change (until now).

You _will_ need a lot of memory though, as cvsps grows large (working
on a workaround now) and cvsimport grows a bit over time (where is
that last leak?!). And a fast machine -- specially fast IO. I've just
switched from an old test machine to an AMD64 with fast disks, and
it's importing around 10K commits per hour.

I

You will probably want to run cvsps by hand, and later use the -P flag.

cheers,


martin

On Wed, May 24, 2006 at 08:49:03PM +0930, Geoff Russell wrote:

I did have to change 2 lines in git-cvsimport to get it to run with my
5.8.0 perl (problems with POSIX errno). I've attached a patch but my
work around isn't as quick as what it replaced.

Can you describe your problem in more detail? The POSIX errno constants
have been available since long before 5.8.0, so we should be able to use
them.

(btw, the change was introduced in my commit() cleanups:
  e73aefe4fdba0d161d9878642c69b40d83a0204c).

-Peff

Dear Jeff,

See below.

On 5/24/06, Jeff King [off-list ref] wrote:

On Wed, May 24, 2006 at 08:49:03PM +0930, Geoff Russell wrote:

quoted

I did have to change 2 lines in git-cvsimport to get it to run with my
5.8.0 perl (problems with POSIX errno). I've attached a patch but my
work around isn't as quick as what it replaced.

Can you describe your problem in more detail? The POSIX errno constants
have been available since long before 5.8.0, so we should be able to use
them.

   $ ./git-cvsimport

   ":errno_h" is not exported by the POSIX module
   Can't continue after import errors at
/usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
    BEGIN failed--compilation aborted at ./git-cvsimport line 26.

When I deleted ":errno_h" I needed to patch the place it was used (as per patch
I attached in original post).

Cheers,
Geoff Russell

(btw, the change was introduced in my commit() cleanups:
  e73aefe4fdba0d161d9878642c69b40d83a0204c).

-Peff
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Wed, May 24, 2006 at 10:03:44PM +0930, Geoff Russell wrote:

  ":errno_h" is not exported by the POSIX module
  Can't continue after import errors at
/usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
   BEGIN failed--compilation aborted at ./git-cvsimport line 26.

Hmm. It looks like something is nonstandard in your setup. I just compiled
5.8.0 from source and the :errno_h tag works fine. What is your
platform?  Can you try the following and let me know which work:
  $ perl -e 'use POSIX qw(:errno_h)'
  $ perl -e 'use POSIX qw(errno_h)'
  $ perl -e 'use Errno'

-Peff

Hi Jeff,

On 5/24/06, Jeff King [off-list ref] wrote:

On Wed, May 24, 2006 at 10:03:44PM +0930, Geoff Russell wrote:

quoted

  ":errno_h" is not exported by the POSIX module
  Can't continue after import errors at
/usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
   BEGIN failed--compilation aborted at ./git-cvsimport line 26.

Hmm. It looks like something is nonstandard in your setup. I just compiled
5.8.0 from source and the :errno_h tag works fine. What is your
platform?  Can you try the following and let me know which work:

I compiled perl from source on Mandrake 9.1.

  $ perl -e 'use POSIX qw(:errno_h)'
  $ perl -e 'use POSIX qw(errno_h)'
  $ perl -e 'use Errno'

All 3 work.  But if I add a second tag before the ':errno_h", then I
get an error.

The "use" line that makes git-cvsimport compile for me is:

        use POSIX qw(strftime dup2 ENOENT);

Which just imports the required symbol and not the full tag list.

Cheers,
Geoff.

-Peff
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Wed, May 24, 2006 at 11:17:32PM +0930, Geoff Russell wrote:

All 3 work.  But if I add a second tag before the ':errno_h", then I
get an error.

The "use" line that makes git-cvsimport compile for me is:

       use POSIX qw(strftime dup2 ENOENT);

Odd. It's either a bug with importing tags in older versions, or there's
some deep perl voodoo that I don't understand (either way, it is "fixed"
in more recent versions).  Importing ENOENT directly is reasonable.

Junio, can you apply the following fix?

diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index af331d9..76f6246 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl

@@ -23,7 +23,7 @@ use File::Basename qw(basename dirname);
 use Time::Local;
 use IO::Socket;
 use IO::Pipe;
-use POSIX qw(strftime dup2 :errno_h);
+use POSIX qw(strftime dup2 ENOENT);
 use IPC::Open2;
 
 $SIG{'PIPE'}="IGNORE";