Re: Memory issue with fast-import, why track branches?

My first response was along the lines of "Why the heck are you storing
sha1's like that!?", until I realised that you're not storing actual git
sha1's, but mtn's hashes, which does make sense.

I'm doing something very similar with my perforce scripts, however I am
doing a bit more magic instead of making so many branches.

Instead of making branches, I make a tag instead, for each and every
changeset.  Every time I make a new git commit, if I need to do it from
a tag, I first read the tag and determine the sha1 I should use, and use
that instead.

Alternatively, you could choose to manage your mapping yourself, and
write them to a .git/mtg-git-map file.

On Sun, 2008-12-21 at 07:54 +0200, Felipe Contreras wrote:

Hi,

I tracked down an issue I have when importing a big repository. For
some reason memory usage keeps increasing until there is no more
memory.

Here is what valgrind shows:
==21034== 471,080,280 bytes in 114,517 blocks are still reachable in
loss record 8 of 8
==21034==    at 0x4004BA2: calloc (vg_replace_malloc.c:397)
==21034==    by 0x806A340: xcalloc (wrapper.c:75)
==21034==    by 0x8063BC1: use_pack (sha1_file.c:808)
==21034==    by 0x8063DA9: unpack_object_header (sha1_file.c:1443)
==21034==    by 0x8064F4F: unpack_entry (sha1_file.c:1736)
==21034==    by 0x8065393: cache_or_unpack_entry (sha1_file.c:1606)
==21034==    by 0x8065464: read_packed_sha1 (sha1_file.c:2000)
==21034==    by 0x80655E5: read_object (sha1_file.c:2090)
==21034==    by 0x8065677: read_sha1_file (sha1_file.c:2106)
==21034==    by 0x8056AE9: parse_object (object.c:190)
==21034==    by 0x805E90A: write_ref_sha1 (refs.c:1214)
==21034==    by 0x804CC4F: update_branch (fast-import.c:1558)

After looking at the code my guess is that I have a humongous amount
of branches.

Actually they are not really branches, but refs. For each git commit
there's an original mtn ref that I store in 'refs/mtn/sha1', but since
I'm using 'commit refs/mtn/sha1' to store it, a branch is created for
every commit.

I guess there are many ways to fix the issue, but for starters I
wonder why is fast-import keeping track of all the branches? In my
case I would like fast-import to work exactly the same if I specify
branches or not (I'll update them later).

Cheers.