git-svnimport failed and now git-repack hates me

So I'm using git 1.4.1, and I have been experimenting with importing
the KDE sources from Subversion using git-svnimport.

First issue I ran into: On a machine with 4GB of RAM, when I tried to
do a full import, git-svnimport died after 309906 revisions, saying
that it couldn't fork.

Checking `top` and `ps` revealed that there were no git-svnimport
processes doing anything, but all of my 4G of RAM was still marked as
used by the kernel. I had to do sysctl -w vm.drop_caches=3 to get it
to free all the RAM that the svn import had used up.

Now, after that, I tried doing `git-repack -a` because I wanted to see
how small the packed archive would be (before trying to continue
importing the rest of the revisions. There are at least another 100k
revisions that I should be able to import, eventually.)

The repack finished after about nine hours, but when I try to do a
git-verify-pack on it, it dies with this error message:

error: Packfile
.git/objects/pack/pack-540263fe66ab9398cc796f000d52531a5c6f3df3.pack
SHA1 mismatch with itself

I get the same message from git-prune.

Any ideas?

On Wed, 3 Jan 2007, Chris Lee wrote:

So I'm using git 1.4.1, and I have been experimenting with importing
the KDE sources from Subversion using git-svnimport.

As one single _huge_ import? All the sub-projects together? I have to say, 
that sounds pretty horrid.

First issue I ran into: On a machine with 4GB of RAM, when I tried to
do a full import, git-svnimport died after 309906 revisions, saying
that it couldn't fork.

Checking `top` and `ps` revealed that there were no git-svnimport
processes doing anything, but all of my 4G of RAM was still marked as
used by the kernel. I had to do sysctl -w vm.drop_caches=3 to get it
to free all the RAM that the svn import had used up.

I think that was just all cached, and all ok. The reason you didn't see 
any git-svnimport was that it had died off already, and all your memory 
was just caches. You could just have left it alone, and the kernel would 
have started re-using the memory for other things even without any 
"drop_caches". 

But what you did there didn't make anything worse, it was just likely had 
no real impact.

However, it does sound like git-svnimport probably acts like git-cvsimport 
used to, and just keeps too much in memory - so it's never going to act 
really nicely..

It also looks like git-svnimport never repacks the repo, which is 
absolutely horrible for performance on all levels. The CVS importer 
repacks every one thousand commits or something like that.

Now, after that, I tried doing `git-repack -a` because I wanted to see
how small the packed archive would be (before trying to continue
importing the rest of the revisions. There are at least another 100k
revisions that I should be able to import, eventually.)

I suspect you'd have been better off just re-starting, and using something 
like

	while :
	do
		git svnimport -l 1000 <...>
		.. figure out some way to decide if it's all done ..
		git repack -d
	done

which would make svnimport act a bit  more sanely, and repack 
incrementally. That should make both the import much faster, _and_ avoid 
any insane big repack at the end (well, you'd still want to do a "git 
repack -a -d" at the end to turn the many smaller packs into a bigger one, 
but it would be nicer).

However, I don't know what the proper magic is for svnimport to do that 
sane "do it in chunks and tell when you're all done". Or even better - to 
just make it repack properly and not keep everything in memory.

The repack finished after about nine hours, but when I try to do a
git-verify-pack on it, it dies with this error message:

error: Packfile
.git/objects/pack/pack-540263fe66ab9398cc796f000d52531a5c6f3df3.pack
SHA1 mismatch with itself

That sounds suspiciously like the bug we had in out POWER sha1 
implementation that would generate the wrong SHA1 for any pack-file that 
was over 512MB in size, due to an overflow in 32 bits (SHA1 does some 
counting in _bits_, so 512MB is 4G _bits_),

Now, I assume you're not on POWER (and we fixed that bug anyway - and I 
think long before 1.4.1 too), but I could easily imagine the same bug in 
some other SHA1 implementation (or perhaps _another_ overflow at the 1GB 
or 2GB mark..). I assume that the pack-file you had was something horrid..

I hope this is with a 64-bit kernel and a 64-bit user space? That should 
limit _some_ of the issues. But I would still not be surprised if your 
SHA1 libraries had some 32-bit ("unsigned int") or 31-bit ("int") limits 
in them somewhere - very few people do SHA1's over huge areas, and even 
when you do SHA1 on something like a DVD image (which is easily over any 
4GB limit), that tends to be done as many smaller calls to the SHA1 
library routines.

Junio - I suspect "pack-check.c" really shouldn't try to do it as one 
single humungous "SHA1_Update()" call. It showed one bug on PPC, I 
wouldn't be surprised if it's implicated now on some other architecture. 

Shawn - does the pack-file-windowing thing already change that? I'm too 
lazy to check..

As to who knows how to fix git-svnimport to do something saner, I have no 
clue.. Sasha seems to have touched it last. Sasha?

		Linus

Linus Torvalds [off-list ref] wrote:

Junio - I suspect "pack-check.c" really shouldn't try to do it as one 
single humungous "SHA1_Update()" call. It showed one bug on PPC, I 
wouldn't be surprised if it's implicated now on some other architecture. 

It used to do it as one big SHA1_Update() call...
 

Shawn - does the pack-file-windowing thing already change that? I'm too 
lazy to check..

But with the mmap window thing in `next` it does it in window
units only.  Which the user could configure to be huge, or could
configure to be sane.  The default when using mmap() is 32 MiB;
1 MiB when using pread() and git_mmap().

-- 
Shawn.

Linus Torvalds [off-list ref] wrote:

On Wed, 3 Jan 2007, Chris Lee wrote:

quoted

First issue I ran into: On a machine with 4GB of RAM, when I tried to
do a full import, git-svnimport died after 309906 revisions, saying
that it couldn't fork.

Managing memory with the Perl SVN libraries has been very painful in my
experience.

Part of it is Perl, which (as far as I know) never frees allocated
memory back to the OS (although Perl can reuse the allocated memory for
other things).  I'm CC-ing the resident Perl guru on this...

I'm also fairly certain that most higher-level languages have this
problem.

I suspect you'd have been better off just re-starting, and using something 
like

	while :
	do
		git svnimport -l 1000 <...>
		.. figure out some way to decide if it's all done ..
		git repack -d
	done

However, I don't know what the proper magic is for svnimport to do that 
sane "do it in chunks and tell when you're all done". Or even better - to 
just make it repack properly and not keep everything in memory.

<shameless self-promotion>
	git-svn already does this chunking internally

	Just set the repack interval to something smaller than 1000;
	(--repack=100) if you experience timeouts.
</shameless self-promotion>

-- 
Eric Wong

"Shawn O. Pearce" [off-list ref] wrote:

Linus Torvalds [off-list ref] wrote:

quoted

Junio - I suspect "pack-check.c" really shouldn't try to do it as one 
single humungous "SHA1_Update()" call. It showed one bug on PPC, I 
wouldn't be surprised if it's implicated now on some other architecture. 

It used to do it as one big SHA1_Update() call...
 

quoted

Shawn - does the pack-file-windowing thing already change that? I'm too 
lazy to check..

But with the mmap window thing in `next` it does it in window
units only.  Which the user could configure to be huge, or could
configure to be sane.  The default when using mmap() is 32 MiB;
1 MiB when using pread() and git_mmap().

I should also point out that my git-fastimport hack that we used
on the huge Mozilla import may be helpful here.  Its _very_ fast
as it goes right to a pack file, but there's no SVN frontend for
it at this time.

-- 
Shawn.

On 1/3/07, Shawn O. Pearce [off-list ref] wrote:

I should also point out that my git-fastimport hack that we used
on the huge Mozilla import may be helpful here.  Its _very_ fast
as it goes right to a pack file, but there's no SVN frontend for
it at this time.

I would be *really* interested in playing with that. Where do I get it?

quoted

quoted

quoted

quoted

"Eric" == Eric Wong [off-list ref] writes:

Eric> Part of it is Perl, which (as far as I know) never frees allocated
Eric> memory back to the OS (although Perl can reuse the allocated memory for
Eric> other things).

It does on Linux, of all things.  That's because Linux has a smarter
malloc/free that uses mmap(2) for the large chunks.  On Linux, Perl memory
size can apparently grow and shrink nicely.  The "old school" advice about
Perl comes from sbrk(2)-driven malloc/free.

Try:

        $x[1e6] = "0";
        sleep 10; # do a ps here
        @x = ();
        sleep 30; # do a ps here

and watch the process on Linux.  If I'm right, this should show a large
process,  then a smaller one.

If you're getting a growing process though, you probably have a circular data
reference.  Maybe you have a tree with backpointers, and those backpointers
should have been weakened?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
[off-list ref] <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Chris Lee [off-list ref] wrote:

On 1/3/07, Shawn O. Pearce [off-list ref] wrote:

quoted

I should also point out that my git-fastimport hack that we used
on the huge Mozilla import may be helpful here.  Its _very_ fast
as it goes right to a pack file, but there's no SVN frontend for
it at this time.

I would be *really* interested in playing with that. Where do I get it?

Its a fork of git.git on repo.or.cz; the gitweb can be seen here:

  http://repo.or.cz/w/git/fastimport.git

the clone url is:

  git://repo.or.cz/git/fastimport.git
  http://repo.or.cz/r/git/fastimport.git

The entire code is in fast-import.c.  The input stream it consumes
comes in on STDIN and is documented in a large comment at the top
of the file.

All that's needed is to get data from SVN in a way that it can be
fed into git-fastimport.

-- 
Shawn.

On 1/3/07, Shawn O. Pearce [off-list ref] wrote:

Chris Lee [off-list ref] wrote:

quoted

On 1/3/07, Shawn O. Pearce [off-list ref] wrote:

quoted

I should also point out that my git-fastimport hack that we used
on the huge Mozilla import may be helpful here.  Its _very_ fast
as it goes right to a pack file, but there's no SVN frontend for
it at this time.

I would be *really* interested in playing with that. Where do I get it?

Its a fork of git.git on repo.or.cz; the gitweb can be seen here:

  http://repo.or.cz/w/git/fastimport.git

the clone url is:

  git://repo.or.cz/git/fastimport.git
  http://repo.or.cz/r/git/fastimport.git

The entire code is in fast-import.c.  The input stream it consumes
comes in on STDIN and is documented in a large comment at the top
of the file.

Neat. How do I do that?

[cc: list modified to remove folks who probably aren't immediately
 interested in git-fastimport]

Chris Lee [off-list ref] wrote:

On 1/3/07, Shawn O. Pearce [off-list ref] wrote:

quoted

the clone url is:

 git://repo.or.cz/git/fastimport.git
 http://repo.or.cz/r/git/fastimport.git

The entire code is in fast-import.c.  The input stream it consumes
comes in on STDIN and is documented in a large comment at the top
of the file.

Neat. How do I do that?

I'm not sure I understand the question...

-- 
Shawn.

Uh... somehow, it lost this part:

All that's needed is to get data from SVN in a way that it can be
fed into git-fastimport.

That's what I meant - I assume that someone already has the
svn-repo-to-gfi piece working? Where's that available from?

Chris Lee [off-list ref] wrote:

Uh... somehow, it lost this part:

quoted

All that's needed is to get data from SVN in a way that it can be
fed into git-fastimport.

That's what I meant - I assume that someone already has the
svn-repo-to-gfi piece working? Where's that available from?

No.  That hasn't been written.

In theory someone could take the SVN dump library (its a chunk of
C code which parses SVN dump files) and write a tool which translates
it into git-fastimport.

One could also use the SVN client library to suck data from SVN
and pump it into git-fastimport.

Jon Smirl attempted to create a CVS-->git-fastimport program in
Python by starting with the cvs2svn codebase, but that doesn't
do anything about importing *from* SVN.  Jon was able to import
the entire Mozilla CVS repository (250k commits, about 3 GiB
input) in 2 hours using his hacked up cvs2svn and git-fastimport.
The resulting pack was ~900 MiB.  He recompressed that using
`git repack -a -d --window=50 --depth=1000` (which is insane) in
about an hour.

-- 
Shawn.

On 1/3/07, Chris Lee [off-list ref] wrote:

Uh... somehow, it lost this part:

quoted

All that's needed is to get data from SVN in a way that it can be
fed into git-fastimport.

That's what I meant - I assume that someone already has the
svn-repo-to-gfi piece working? Where's that available from?

Right, and I'm an idiot! Awesome.

I obviously didn't comprehend the part where you wrote:

I should also point out that my git-fastimport hack that we used
on the huge Mozilla import may be helpful here.  Its _very_ fast
as it goes right to a pack file, but there's no SVN frontend for
it at this time.

Anyway. Thanks for the pointers, I'll see if I can't hack something up.

"Randal L. Schwartz" [off-list ref] wrote:

quoted

quoted

quoted

quoted

quoted

"Eric" == Eric Wong [off-list ref] writes:

Eric> Part of it is Perl, which (as far as I know) never frees allocated
Eric> memory back to the OS (although Perl can reuse the allocated memory for
Eric> other things).

It does on Linux, of all things.  That's because Linux has a smarter
malloc/free that uses mmap(2) for the large chunks.  On Linux, Perl memory
size can apparently grow and shrink nicely.  The "old school" advice about
Perl comes from sbrk(2)-driven malloc/free.

Try:

        $x[1e6] = "0";
        sleep 10; # do a ps here
        @x = ();
        sleep 30; # do a ps here

and watch the process on Linux.  If I'm right, this should show a large
process,  then a smaller one.

Nope, not happening to me.  I'm using Perl 5.8.8-7 and glibc 2.3.6.ds1-8
on a Debian Etch machine.  The kernel is a vanilla 2.6.18.1 from
kernel.org.

strace shows an mmap2 call, but no corresponding mumap.  I've added a
sleep loop to the end of the above program and had it print
something every 10 seconds; but so far, there's still no munmap.

while (1) {
        print "hi\n" if ((time % 10) == 0);
	sleep 1;
}

Trying to allocate a bigger chunk (1e7) doesn't show anything different,
either.  I've also conducted similar experiments with Ruby in the past
and noticed the same things...

-- 
Eric Wong

On 1/3/07, Linus Torvalds [off-list ref] wrote:

quoted

So I'm using git 1.4.1, and I have been experimenting with importing
the KDE sources from Subversion using git-svnimport.

As one single _huge_ import? All the sub-projects together? I have to say,
that sounds pretty horrid.

Unfortunately, that's how the KDE repo is organized. (I tried arguing
against this when they were going to do the original import, but I
lost the argument.) And git-svnimport doesn't appear to have any sort
of method for splitting a gigantic svn repo into several smaller git
repos.

quoted

First issue I ran into: On a machine with 4GB of RAM, when I tried to
do a full import, git-svnimport died after 309906 revisions, saying
that it couldn't fork.

Checking `top` and `ps` revealed that there were no git-svnimport
processes doing anything, but all of my 4G of RAM was still marked as
used by the kernel. I had to do sysctl -w vm.drop_caches=3 to get it
to free all the RAM that the svn import had used up.

I think that was just all cached, and all ok. The reason you didn't see
any git-svnimport was that it had died off already, and all your memory
was just caches. You could just have left it alone, and the kernel would
have started re-using the memory for other things even without any
"drop_caches".

But what you did there didn't make anything worse, it was just likely had
no real impact.

I got the tip about drop_caches from davej. Normally, when a process
taking up a huge amount of memory exits, it shows a bunch of free
memory in `top` and friends. I was a little bit surprised when that
didn't happen this time.

However, it does sound like git-svnimport probably acts like git-cvsimport
used to, and just keeps too much in memory - so it's never going to act
really nicely..

It also looks like git-svnimport never repacks the repo, which is
absolutely horrible for performance on all levels. The CVS importer
repacks every one thousand commits or something like that.

Yeah. I haven't bothered hacking git-svnimport yet - but it looks like
having it automatically repack every thousand revisions or so would
probably be a pretty big win.

quoted

Now, after that, I tried doing `git-repack -a` because I wanted to see
how small the packed archive would be (before trying to continue
importing the rest of the revisions. There are at least another 100k
revisions that I should be able to import, eventually.)

I suspect you'd have been better off just re-starting, and using something
like

        while :
        do
                git svnimport -l 1000 <...>
                .. figure out some way to decide if it's all done ..
                git repack -d
        done

which would make svnimport act a bit  more sanely, and repack
incrementally. That should make both the import much faster, _and_ avoid
any insane big repack at the end (well, you'd still want to do a "git
repack -a -d" at the end to turn the many smaller packs into a bigger one,
but it would be nicer).

However, I don't know what the proper magic is for svnimport to do that
sane "do it in chunks and tell when you're all done". Or even better - to
just make it repack properly and not keep everything in memory.

You can pass limits to svnimport to give it a revision to start at and
another one to end at, so that wouldn't be too bad - I was thinking
about working around it like that (so that i don't have to go poking
around in the Perl code behind the svn importer).

By default, if I had, say, one pack with the first 1000 revisions, and
I imported another 1000, running 'git-repack' on its own would leave
the first pack alone and create a new pack with just the second 1000
revisions, right?

quoted

The repack finished after about nine hours, but when I try to do a
git-verify-pack on it, it dies with this error message:

error: Packfile
.git/objects/pack/pack-540263fe66ab9398cc796f000d52531a5c6f3df3.pack
SHA1 mismatch with itself

That sounds suspiciously like the bug we had in out POWER sha1
implementation that would generate the wrong SHA1 for any pack-file that
was over 512MB in size, due to an overflow in 32 bits (SHA1 does some
counting in _bits_, so 512MB is 4G _bits_),

Now, I assume you're not on POWER (and we fixed that bug anyway - and I
think long before 1.4.1 too), but I could easily imagine the same bug in
some other SHA1 implementation (or perhaps _another_ overflow at the 1GB
or 2GB mark..). I assume that the pack-file you had was something horrid..

I hope this is with a 64-bit kernel and a 64-bit user space? That should
limit _some_ of the issues. But I would still not be surprised if your
SHA1 libraries had some 32-bit ("unsigned int") or 31-bit ("int") limits
in them somewhere - very few people do SHA1's over huge areas, and even
when you do SHA1 on something like a DVD image (which is easily over any
4GB limit), that tends to be done as many smaller calls to the SHA1
library routines.

This is on a dual-CPU dual-core Opteron, running the AMD64 variant of
Ubuntu's Edgy release (64-bit kernel, 64-bit native userland). The
pack-file was around 2.3GB.

Accidentally sent this to just Linus instead of the list...

On 1/3/07, Linus Torvalds [off-list ref] wrote:

quoted

So I'm using git 1.4.1, and I have been experimenting with importing
the KDE sources from Subversion using git-svnimport.

As one single _huge_ import? All the sub-projects together? I have to say,
that sounds pretty horrid.

Unfortunately, that's how the KDE repo is organized. (I tried arguing
against this when they were going to do the original import, but I
lost the argument.) And git-svnimport doesn't appear to have any sort
of method for splitting a gigantic svn repo into several smaller git
repos.

quoted

First issue I ran into: On a machine with 4GB of RAM, when I tried to
do a full import, git-svnimport died after 309906 revisions, saying
that it couldn't fork.

Checking `top` and `ps` revealed that there were no git-svnimport
processes doing anything, but all of my 4G of RAM was still marked as
used by the kernel. I had to do sysctl -w vm.drop_caches=3 to get it
to free all the RAM that the svn import had used up.

I think that was just all cached, and all ok. The reason you didn't see
any git-svnimport was that it had died off already, and all your memory
was just caches. You could just have left it alone, and the kernel would
have started re-using the memory for other things even without any
"drop_caches".

But what you did there didn't make anything worse, it was just likely had
no real impact.

I got the tip about drop_caches from davej. Normally, when a process
taking up a huge amount of memory exits, it shows a bunch of free
memory in `top` and friends. I was a little bit surprised when that
didn't happen this time.

However, it does sound like git-svnimport probably acts like git-cvsimport
used to, and just keeps too much in memory - so it's never going to act
really nicely..

It also looks like git-svnimport never repacks the repo, which is
absolutely horrible for performance on all levels. The CVS importer
repacks every one thousand commits or something like that.

Yeah. I haven't bothered hacking git-svnimport yet - but it looks like
having it automatically repack every thousand revisions or so would
probably be a pretty big win.

quoted

Now, after that, I tried doing `git-repack -a` because I wanted to see
how small the packed archive would be (before trying to continue
importing the rest of the revisions. There are at least another 100k
revisions that I should be able to import, eventually.)

I suspect you'd have been better off just re-starting, and using something
like

        while :
        do
                git svnimport -l 1000 <...>
                .. figure out some way to decide if it's all done ..
                git repack -d
        done

which would make svnimport act a bit  more sanely, and repack
incrementally. That should make both the import much faster, _and_ avoid
any insane big repack at the end (well, you'd still want to do a "git
repack -a -d" at the end to turn the many smaller packs into a bigger one,
but it would be nicer).

However, I don't know what the proper magic is for svnimport to do that
sane "do it in chunks and tell when you're all done". Or even better - to
just make it repack properly and not keep everything in memory.

You can pass limits to svnimport to give it a revision to start at and
another one to end at, so that wouldn't be too bad - I was thinking
about working around it like that (so that i don't have to go poking
around in the Perl code behind the svn importer).

By default, if I had, say, one pack with the first 1000 revisions, and
I imported another 1000, running 'git-repack' on its own would leave
the first pack alone and create a new pack with just the second 1000
revisions, right?

quoted

The repack finished after about nine hours, but when I try to do a
git-verify-pack on it, it dies with this error message:

error: Packfile
.git/objects/pack/pack-540263fe66ab9398cc796f000d52531a5c6f3df3.pack
SHA1 mismatch with itself

That sounds suspiciously like the bug we had in out POWER sha1
implementation that would generate the wrong SHA1 for any pack-file that
was over 512MB in size, due to an overflow in 32 bits (SHA1 does some
counting in _bits_, so 512MB is 4G _bits_),

Now, I assume you're not on POWER (and we fixed that bug anyway - and I
think long before 1.4.1 too), but I could easily imagine the same bug in
some other SHA1 implementation (or perhaps _another_ overflow at the 1GB
or 2GB mark..). I assume that the pack-file you had was something horrid..

I hope this is with a 64-bit kernel and a 64-bit user space? That should
limit _some_ of the issues. But I would still not be surprised if your
SHA1 libraries had some 32-bit ("unsigned int") or 31-bit ("int") limits
in them somewhere - very few people do SHA1's over huge areas, and even
when you do SHA1 on something like a DVD image (which is easily over any
4GB limit), that tends to be done as many smaller calls to the SHA1
library routines.

This is on a dual-CPU dual-core Opteron, running the AMD64 variant of
Ubuntu's Edgy release (64-bit kernel, 64-bit native userland). The
pack-file was around 2.3GB.

On Thu, 4 Jan 2007, Chris Lee wrote:

Unfortunately, that's how the KDE repo is organized. (I tried arguing
against this when they were going to do the original import, but I
lost the argument.) And git-svnimport doesn't appear to have any sort
of method for splitting a gigantic svn repo into several smaller git
repos.

Well, the good news is, I think we could probably split it up from within 
git. It's not fundamentally hard, although it is pretty damn expensive 
(and it would require the subproject support to do really well).

So ignore that issue for now. I'd love to see the end result, if only 
because it sounds like you have a test-case for git that is four times 
bigger than the mozilla archive - even if it's just because of some really 
really stupid design decisions from the KDE SVN maintainers ;)

(But I would actually expect that KDE SVN uses SVN subprojects, so 
hopefully it's not _really_ one big repository. Of course, I don't know if 
SVN really does subprojects or how well it does them, so that's just a 
total guess).

The real problem with a SVN import is that I think SVN doesn't do merges 
right, so you can't import merge history properly (well, you can, if you 
decide that "properly" really means "SVN can't merge, so we can't really 
show it as merges in git either").

I think both git-svn and git-svnimport can _guess_ about merges, but it's 
just a heuristic, afaik. Whether it's a good one, I don't know.

Yeah. I haven't bothered hacking git-svnimport yet - but it looks like
having it automatically repack every thousand revisions or so would
probably be a pretty big win.

That, or making it use the same "fastimport" that the hacked-up CVS 
importer was made to use. Either way, somebody who understands SVN 
intimately (and probably perl) would need to work on it. 

That would not be me, so I can't really help ;)

By default, if I had, say, one pack with the first 1000 revisions, and
I imported another 1000, running 'git-repack' on its own would leave
the first pack alone and create a new pack with just the second 1000
revisions, right?

Yes. It's _probably_ better to do a full re-pack every once in a while 
(because if you have a lot of pack-files, eventually that ends up being 
problematic too), but as a first approximation, it's probably fine to just 
do a plain "git repack" every thousand commits, and then do a full big 
repack at the end.

The big repack will still be pretty expensive, but it should be less 
painful than having everything unpacked. And at least the import won't 
have run with millions and millions of loose objects.

So doing a "git repack -a -d" at the end is a good idea, and _maybe_ it 
could be done in the middle too for really big packs.

Again, doing what fastimport does avoids most of the whole issue, since it 
just generates a pack up-front instead. But that requires the importer to 
specifically understand about that kind of setup.

This is on a dual-CPU dual-core Opteron, running the AMD64 variant of
Ubuntu's Edgy release (64-bit kernel, 64-bit native userland). The
pack-file was around 2.3GB.

Ok, that should all be fine. A 31-bit thing in OpenSSL would explain it, 
and doesn't sound unlikely. Just somebody using "int" somewhere, and it 
would never have been triggered by any sane user of SHA1_Update(). The git 
pack-check.c usage really _is_ very odd, even if it happens to make sense 
in that particular schenario.

		Linus

On 1/4/07, Linus Torvalds [off-list ref] wrote:

Well, the good news is, I think we could probably split it up from within
git. It's not fundamentally hard, although it is pretty damn expensive
(and it would require the subproject support to do really well).

I was hoping that'd be possible at some point. I really want to split
the submodules back out into first-class modules - one of my biggest
misgivings about the current KDE repository setup is how everything is
part of one gigantic repository.

So ignore that issue for now. I'd love to see the end result, if only
because it sounds like you have a test-case for git that is four times
bigger than the mozilla archive - even if it's just because of some really
really stupid design decisions from the KDE SVN maintainers ;)

The full on-disk size of the KDE SVN repo is about 37GB, last time I
checked. It may be up to 38 or 39GB now - I last ran rsync against the
svn repo a few weeks ago. I'm only focusing on importing the first
409k revisions at the moment, because that comprises the commits that
originally came from CVS and were imported into SVN. Almost
immediately after the CVS import, coolo made some changes - moving all
of the core KDE modules into /trunk/KDE, and their branches and tags
into /branches/KDE and /tags/KDE respectively. This, I suspect will
end up making things "fun" for the other part of the import, which is
another 200k revisions, give or take.

So, yes, I suspect it's quite a bit larger than Mozilla. I'm doing the
conversion to git as a test so that I can show some numbers to the KDE
guys; I'm not trying to campaign for a transition to git, but I think
it's definitely worth exploring what such a world would look like. But
in order for me to try to make a compelling argument for an eventual
project move to git, the git win32 support would need to be really
good. (In KDE4, we're supporting Windows and OS X as well as X11 as
first-class platforms.)

(But I would actually expect that KDE SVN uses SVN subprojects, so
hopefully it's not _really_ one big repository. Of course, I don't know if
SVN really does subprojects or how well it does them, so that's just a
total guess).

I don't think so, but I'll ask coolo (the KDE SVN administrator).

The real problem with a SVN import is that I think SVN doesn't do merges
right, so you can't import merge history properly (well, you can, if you
decide that "properly" really means "SVN can't merge, so we can't really
show it as merges in git either").

I think both git-svn and git-svnimport can _guess_ about merges, but it's
just a heuristic, afaik. Whether it's a good one, I don't know.

Not too worried about the merges right now - as long as I have a rough
approximation of what the original looked like, I'm pretty happy.

quoted

Yeah. I haven't bothered hacking git-svnimport yet - but it looks like
having it automatically repack every thousand revisions or so would
probably be a pretty big win.

That, or making it use the same "fastimport" that the hacked-up CVS
importer was made to use. Either way, somebody who understands SVN
intimately (and probably perl) would need to work on it.

That would not be me, so I can't really help ;)

Well, Shawn pointed me at the fastimport stuff, and I happen to know
Perl reasonably well (I think) so I'll take a stab at trying it that
way.

quoted

By default, if I had, say, one pack with the first 1000 revisions, and
I imported another 1000, running 'git-repack' on its own would leave
the first pack alone and create a new pack with just the second 1000
revisions, right?

Yes. It's _probably_ better to do a full re-pack every once in a while
(because if you have a lot of pack-files, eventually that ends up being
problematic too), but as a first approximation, it's probably fine to just
do a plain "git repack" every thousand commits, and then do a full big
repack at the end.

Sounds like a good idea. Also sounds like it would be much less
painful than the current situation, where it takes over nine hours to
pack up all these revisions. :)

The big repack will still be pretty expensive, but it should be less
painful than having everything unpacked. And at least the import won't
have run with millions and millions of loose objects.

So doing a "git repack -a -d" at the end is a good idea, and _maybe_ it
could be done in the middle too for really big packs.

Okay, good to know.

Again, doing what fastimport does avoids most of the whole issue, since it
just generates a pack up-front instead. But that requires the importer to
specifically understand about that kind of setup.

I'll definitely be investigating the fastimport option. Looks like
I'll get to crack open some of my Perl books - haven't had to do that
in a while. :)

On 1/3/07, Linus Torvalds [off-list ref] wrote:

quoted

Checking `top` and `ps` revealed that there were no git-svnimport
processes doing anything, but all of my 4G of RAM was still marked as
used by the kernel. I had to do sysctl -w vm.drop_caches=3 to get it
to free all the RAM that the svn import had used up.

I think that was just all cached, and all ok. The reason you didn't see
any git-svnimport was that it had died off already, and all your memory
was just caches. You could just have left it alone, and the kernel would
have started re-using the memory for other things even without any
"drop_caches".

But what you did there didn't make anything worse, it was just likely had
no real impact.

Thought it was worth mentioning this:

When I checked top, the numbers it showed me were:
Mem:   4059332k total,  3216480k used,   842852k free,    40824k buffers
Swap:        0k total,        0k used,        0k free,    37364k cached

40MB in buffers, 37MB in cache, and 3GB used.

Seems like *something* was definitely lost there. The 'used' number
didn't go down at all when I started doing other things; it went up as
the new programs started, then they used up some RAM, and then when
they exited they'd free whatever resources they'd used. However, until
I did the drop_caches, that number stayed pretty damn big.

The system has been up since then, doing lots of things, and still
seems pretty stable, so I think it's okay, but I thought that it was
worth mentioning that something seemed to be leaky.

On Thu, 4 Jan 2007, Chris Lee wrote:

Seems like *something* was definitely lost there. The 'used' number
didn't go down at all when I started doing other things; it went up as
the new programs started

The 'used' number basically _never_ goes down as long as there is memory 
free. The kernel simply doesn't have any reason to free any of its caches, 
even if those caches end up not being very useful.

What happened is almost certainly that with your big unpacked repository, 
the kernel ended up using a lot of memory on filename caching. In other 
words, I'd have expected that if you were to do 

	cat /proc/slabinfo

you'd have seen a _lot_ of memory being used for dentries ("dentry_cache") 
and inodes ("ext3_inode_cache" assuming you're an ext3 user).

The kernel can easily drop those caches on demand, but "free" isn't quite 
smart enough to know about them as being caches, so they will just show up 
as "used".

That said, since you didn't want them, dropping them by hand with sysctl 
certainly didn't hurt. Manual control can often be better than automatic 
heuristics..

So the reason why repacking is so useful is that it gets rid of all these 
millions of individual files. They all take up space on the disk, but they 
also do end up having a lot of caches associated with them.

Btw, you may find that despite your 4GB of RAM, you might still be 
better off with a swapfile. It gives the kernel a certain amount of 
freedom in choosing how to allocate memory, and perhaps more importantly, 
even when the kernel doesn't actively use it, it means that IF the kernel 
runs out of totally free memory (because it has decided to keep a lot of 
stuff in the dentry cache), it gives the kernel choices, and a certain 
"buffer" for making the right decision.

What often happens is that the memory management heuristics don't make the 
"perfect" choice (partly because it's theoretically impossible anyway, but 
largely just because it's just a damn hard problem to even get all that 
*close* to perfect), and having a swap partition or even a swap file just 
allows the kernel to make some mistakes without it hitting a hard wall of 
"oh, I can't do anything at all about this particular page".

So that buffer zone can be helpful in avoiding bad situations, but it can 
actually also end up improving performance - it doesn't sound like the 
case in this particular situation, but in some other loads there really 
are a lot of dirty pages that aren't all that useful and where the memory 
really could be better used for other things if the largely unused dirty 
page could just be written to disk.

			Linus

On 17:59 Wed 03 Jan     , Linus Torvalds wrote:

However, I don't know what the proper magic is for svnimport to do that 
sane "do it in chunks and tell when you're all done". Or even better - to 
just make it repack properly and not keep everything in memory.

As to who knows how to fix git-svnimport to do something saner, I have no 
clue.. Sasha seems to have touched it last. Sasha?

I guess it should not be hard to do svnimport in incrementally with
repacking. Like this:

diff --git a/git-svnimport.perl b/git-svnimport.perl
index 071777b..afbbe63 100755
--- a/git-svnimport.perl
+++ b/git-svnimport.perl

@@ -31,12 +31,13 @@ $SIG{'PIPE'}="IGNORE";
 $ENV{'TZ'}="UTC";
 
 our($opt_h,$opt_o,$opt_v,$opt_u,$opt_C,$opt_i,$opt_m,$opt_M,$opt_t,$opt_T,
-    $opt_b,$opt_r,$opt_I,$opt_A,$opt_s,$opt_l,$opt_d,$opt_D,$opt_S,$opt_F,$opt_P);
+    $opt_b,$opt_r,$opt_I,$opt_A,$opt_s,$opt_l,$opt_d,$opt_D,$opt_S,$opt_F,
+    $opt_P,$opt_R);
 
 sub usage() {
 	print STDERR <<END;
 Usage: ${\basename $0}     # fetch/update GIT from SVN
-       [-o branch-for-HEAD] [-h] [-v] [-l max_rev]
+       [-o branch-for-HEAD] [-h] [-v] [-l max_rev] [-R repack_each_revs]
        [-C GIT_repository] [-t tagname] [-T trunkname] [-b branchname]
        [-d|-D] [-i] [-u] [-r] [-I ignorefilename] [-s start_chg]
        [-m] [-M regex] [-A author_file] [-S] [-F] [-P project_name] [SVN_URL]

@@ -44,7 +45,7 @@ END
 	exit(1);
 }
 
-getopts("A:b:C:dDFhiI:l:mM:o:rs:t:T:SP:uv") or usage();
+getopts("A:b:C:dDFhiI:l:mM:o:rs:t:T:SP:R:uv") or usage();
 usage if $opt_h;
 
 my $tag_name = $opt_t || "tags";

@@ -52,6 +53,7 @@ my $trunk_name = $opt_T || "trunk";
 my $branch_name = $opt_b || "branches";
 my $project_name = $opt_P || "";
 $project_name = "/" . $project_name if ($project_name);
+my $repack_after = $opt_R || 1000;
 
 @ARGV == 1 or @ARGV == 2 or usage();
 

@@ -938,11 +940,27 @@ if ($opt_l < $current_rev) {
     exit;
 }
 
-print "Fetching from $current_rev to $opt_l ...\n" if $opt_v;
+print "Processing from $current_rev to $opt_l ...\n" if $opt_v;
 
-my $pool=SVN::Pool->new;
-$svn->{'svn'}->get_log("/",$current_rev,$opt_l,0,1,1,\&commit_all,$pool);
-$pool->clear;
+my $from_rev;
+my $to_rev = $current_rev;
+
+while ($to_rev < $opt_l) {
+	$from_rev = $to_rev;
+	$to_rev = $from_rev + $repack_after;
+	$to_rev = $opt_l if $opt_l < $to_rev;
+	print "Fetching from $from_rev to $to_rev ...\n" if $opt_v;
+	my $pool=SVN::Pool->new;
+	$svn->{'svn'}->get_log("/",$from_rev,$to_rev,0,1,1,\&commit_all,$pool);
+	$pool->clear;
+	my $pid = fork();
+	die "Fork: $!\n" unless defined $pid;
+	unless($pid) {
+		exec("git-repack", "-d")
+			or die "Cannot repack: $!\n";
+	}
+	waitpid($pid, 0);
+}
 
 
 unlink($git_index);

Chris, it works fine for me with small repository (~9000 revisions), but
I don't have such huge one as yours. Could you try? Thanks.

Sasha

Chris, it works fine for me with small repository (~9000 revisions), but
I don't have such huge one as yours. Could you try? Thanks.

Patch looks like it makes sense. I can definitely try it later.

Back to work for now...

Since fetch reforks itself at most every 1000 revisions, we
need to update the counter in the parent process to have a
working count if we set our repack interval to be > ~1000
revisions.  multi-fetch has always done this correctly
because of an extra process; now fetch uses the extra process;
as well.

While we're at it, only compile the $sha1 regex that checks for
repacking once.

Signed-off-by: Eric Wong <redacted>
---

I wrote:

	Just set the repack interval to something smaller than 1000;
	(--repack=100) if you experience timeouts.

Chris: you shouldn't get timeouts (at least not across HTTP(s)).
Also, don't worry about repack=100 either; there was a bug that
was triggered only in 'fetch' not 'multi-fetch' (you should use
'multi-fetch').  This patch fixes the 'fetch' bug.

 git-svn.perl |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 0fc386a..5377762 100755
--- a/git-svn.perl
+++ b/git-svn.perl

@@ -102,7 +102,7 @@ my %cmt_opts = ( 'edit|e' => \$_edit,
 );
 
 my %cmd = (
-	fetch => [ \&fetch, "Download new revisions from SVN",
+	fetch => [ \&cmd_fetch, "Download new revisions from SVN",
 			{ 'revision|r=s' => \$_revision, %fc_opts } ],
 	init => [ \&init, "Initialize a repo for tracking" .
 			  " (requires URL argument)",

@@ -293,6 +293,10 @@ sub init {
 	setup_git_svn();
 }
 
+sub cmd_fetch {
+	fetch_child_id($GIT_SVN, @_);
+}
+
 sub fetch {
 	check_upgrade_needed();
 	$SVN_URL ||= file_to_s("$GIT_SVN_DIR/info/url");

@@ -836,7 +840,6 @@ sub fetch_child_id {
 	my $ref = "$GIT_DIR/refs/remotes/$id";
 	defined(my $pid = open my $fh, '-|') or croak $!;
 	if (!$pid) {
-		$_repack = undef;
 		$GIT_SVN = $ENV{GIT_SVN_ID} = $id;
 		init_vars();
 		fetch(@_);

@@ -844,7 +847,7 @@ sub fetch_child_id {
 	}
 	while (<$fh>) {
 		print $_;
-		check_repack() if (/^r\d+ = $sha1/);
+		check_repack() if (/^r\d+ = $sha1/o);
 	}
 	close $fh or croak $?;
 }

@@ -1407,7 +1410,6 @@ sub git_commit {
 
 	# this output is read via pipe, do not change:
 	print "r$log_msg->{revision} = $commit\n";
-	check_repack();
 	return $commit;
 }
 

-- 
1.5.0.rc0.g0d67

This adds ability to do import "in chunks" (default 1000 revisions),
after each chunk git repo will be repacked. The option -R is used to
change default value of chunk size (or how often repository will
repacked).

Signed-off-by: Sasha Khapyorsky <redacted>
---

Chris reported successful test with this patch.

 Documentation/git-svnimport.txt |   10 +++++++++-
 git-svnimport.perl              |   32 +++++++++++++++++++++++++-------
 2 files changed, 34 insertions(+), 8 deletions(-)

diff --git a/Documentation/git-svnimport.txt b/Documentation/git-svnimport.txt
index 2c7c7da..b166cf3 100644
--- a/Documentation/git-svnimport.txt
+++ b/Documentation/git-svnimport.txt

@@ -15,7 +15,7 @@ SYNOPSIS
 		[ -b branch_subdir ] [ -T trunk_subdir ] [ -t tag_subdir ]
 		[ -s start_chg ] [ -m ] [ -r ] [ -M regex ]
 		[ -I <ignorefile_name> ] [ -A <author_file> ]
-		[ -P <path_from_trunk> ]
+		[ -R <repack_each_revs>] [ -P <path_from_trunk> ]
 		<SVN_repository_URL> [ <path> ]
 
 

@@ -108,6 +108,14 @@ repository without -A.
 Formerly, this option controlled how many revisions to pull,
 due to SVN memory leaks. (These have been worked around.)
 
+-R <repack_each_revs>::
+	Specify how often git repository should be repacked.
++
+The default value is 1000. git-svnimport will do import in chunks of 1000
+revisions, after each chunk git repository will be repacked. To disable
+this behavior specify some big value here which is mote than number of
+revisions to import.
+
 -P <path_from_trunk>::
 	Partial import of the SVN tree.
 +

diff --git a/git-svnimport.perl b/git-svnimport.perl
index 071777b..afbbe63 100755
--- a/git-svnimport.perl
+++ b/git-svnimport.perl

@@ -31,12 +31,13 @@ $SIG{'PIPE'}="IGNORE";
 $ENV{'TZ'}="UTC";
 
 our($opt_h,$opt_o,$opt_v,$opt_u,$opt_C,$opt_i,$opt_m,$opt_M,$opt_t,$opt_T,
-    $opt_b,$opt_r,$opt_I,$opt_A,$opt_s,$opt_l,$opt_d,$opt_D,$opt_S,$opt_F,$opt_P);
+    $opt_b,$opt_r,$opt_I,$opt_A,$opt_s,$opt_l,$opt_d,$opt_D,$opt_S,$opt_F,
+    $opt_P,$opt_R);
 
 sub usage() {
 	print STDERR <<END;
 Usage: ${\basename $0}     # fetch/update GIT from SVN
-       [-o branch-for-HEAD] [-h] [-v] [-l max_rev]
+       [-o branch-for-HEAD] [-h] [-v] [-l max_rev] [-R repack_each_revs]
        [-C GIT_repository] [-t tagname] [-T trunkname] [-b branchname]
        [-d|-D] [-i] [-u] [-r] [-I ignorefilename] [-s start_chg]
        [-m] [-M regex] [-A author_file] [-S] [-F] [-P project_name] [SVN_URL]

@@ -44,7 +45,7 @@ END
 	exit(1);
 }
 
-getopts("A:b:C:dDFhiI:l:mM:o:rs:t:T:SP:uv") or usage();
+getopts("A:b:C:dDFhiI:l:mM:o:rs:t:T:SP:R:uv") or usage();
 usage if $opt_h;
 
 my $tag_name = $opt_t || "tags";

@@ -52,6 +53,7 @@ my $trunk_name = $opt_T || "trunk";
 my $branch_name = $opt_b || "branches";
 my $project_name = $opt_P || "";
 $project_name = "/" . $project_name if ($project_name);
+my $repack_after = $opt_R || 1000;
 
 @ARGV == 1 or @ARGV == 2 or usage();
 

@@ -938,11 +940,27 @@ if ($opt_l < $current_rev) {
     exit;
 }
 
-print "Fetching from $current_rev to $opt_l ...\n" if $opt_v;
+print "Processing from $current_rev to $opt_l ...\n" if $opt_v;
 
-my $pool=SVN::Pool->new;
-$svn->{'svn'}->get_log("/",$current_rev,$opt_l,0,1,1,\&commit_all,$pool);
-$pool->clear;
+my $from_rev;
+my $to_rev = $current_rev;
+
+while ($to_rev < $opt_l) {
+	$from_rev = $to_rev;
+	$to_rev = $from_rev + $repack_after;
+	$to_rev = $opt_l if $opt_l < $to_rev;
+	print "Fetching from $from_rev to $to_rev ...\n" if $opt_v;
+	my $pool=SVN::Pool->new;
+	$svn->{'svn'}->get_log("/",$from_rev,$to_rev,0,1,1,\&commit_all,$pool);
+	$pool->clear;
+	my $pid = fork();
+	die "Fork: $!\n" unless defined $pid;
+	unless($pid) {
+		exec("git-repack", "-d")
+			or die "Cannot repack: $!\n";
+	}
+	waitpid($pid, 0);
+}
 
 
 unlink($git_index);

-- 
1.5.0.rc0.g2484-dirty

On 1/6/07, Sasha Khapyorsky [off-list ref] wrote:

This adds ability to do import "in chunks" (default 1000 revisions),
after each chunk git repo will be repacked. The option -R is used to
change default value of chunk size (or how often repository will
repacked).

Actually, I just noticed an issue here with this - it appears to be
double-importing the edge revisions.

So if I started with -s 349000 and tell it to repack every 1000
revisions, it's now importing every thousandth revision twice.

Off-by-one?

On 10:12 Sun 07 Jan     , Chris Lee wrote:

On 1/6/07, Sasha Khapyorsky [off-list ref] wrote:

quoted

This adds ability to do import "in chunks" (default 1000 revisions),
after each chunk git repo will be repacked. The option -R is used to
change default value of chunk size (or how often repository will
repacked).

Actually, I just noticed an issue here with this - it appears to be
double-importing the edge revisions.

So if I started with -s 349000 and tell it to repack every 1000
revisions, it's now importing every thousandth revision twice.

Indeed. There is the fix:

diff --git a/git-svnimport.perl b/git-svnimport.perl
index afbbe63..f1f1a7d 100755
--- a/git-svnimport.perl
+++ b/git-svnimport.perl

@@ -943,10 +943,10 @@ if ($opt_l < $current_rev) {
 print "Processing from $current_rev to $opt_l ...\n" if $opt_v;
 
 my $from_rev;
-my $to_rev = $current_rev;
+my $to_rev = $current_rev - 1;
 
 while ($to_rev < $opt_l) {
-	$from_rev = $to_rev;
+	$from_rev = $to_rev + 1;
 	$to_rev = $from_rev + $repack_after;
 	$to_rev = $opt_l if $opt_l < $to_rev;
 	print "Fetching from $from_rev to $to_rev ...\n" if $opt_v;

Sasha

This fixes newly introduced bug when the incremental cycle edge revisions
are imported twice.

Signed-off-by: Sasha Khapyorsky <redacted>
---
 git-svnimport.perl |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/git-svnimport.perl b/git-svnimport.perl
index afbbe63..f1f1a7d 100755
--- a/git-svnimport.perl
+++ b/git-svnimport.perl

@@ -943,10 +943,10 @@ if ($opt_l < $current_rev) {
 print "Processing from $current_rev to $opt_l ...\n" if $opt_v;
 
 my $from_rev;
-my $to_rev = $current_rev;
+my $to_rev = $current_rev - 1;
 
 while ($to_rev < $opt_l) {
-	$from_rev = $to_rev;
+	$from_rev = $to_rev + 1;
 	$to_rev = $from_rev + $repack_after;
 	$to_rev = $opt_l if $opt_l < $to_rev;
 	print "Fetching from $from_rev to $to_rev ...\n" if $opt_v;

-- 
1.5.0.rc0.g2484-dirty