git-fast-export hg mutt (24M vs 184M)

6 messages, 4 authors, 2016-06-15 · open the first message on its own page

git-fast-export hg mutt (24M vs 184M)

From: Thomas Glanzmann <hidden>
Date: 2016-06-15 22:43:08

Hello,
I just tried to vendor track / import the mutt hg repository into git.
git-fast-export-hg is quiet amazing but the resulting git repository
blows up in size and I have not the slightes clue why.

        git clone git://repo.or.cz/fast-export.git
        hg clone http://dev.mutt.org/hg/mutt

Could someone have a look at this? I used Debian Etch with a git version
I build myself using an ugly script. I had to upgrade mecurial as well
because the version didn't had the cmdlog.py. I used the unstable debian
package for that.

        (thinkpad) [~/work/mutt] git-init-db
        Initialized empty Git repository in .git/
        (thinkpad) [~/work/mutt] ~/work/fast-export/hg-fast-export.sh -r /tmp/mutt

git version 1.5.2.rc0.56.g6169a

        Thomas

Re: git-fast-export hg mutt (24M vs 184M)

From: Thomas Glanzmann <hidden>
Date: 2016-06-15 22:43:08

Hello,
git-repack -a -d -f got it down to 19M. I missed the -f parameter
before. Sorry for the noise.

        Thomas

Re: git-fast-export hg mutt (24M vs 184M)

From: Pierre Habouzit <hidden>
Date: 2016-06-15 22:43:08

On Thu, May 03, 2007 at 09:17:16PM +0200, Thomas Glanzmann wrote:
Hello,
git-repack -a -d -f got it down to 19M. I missed the -f parameter
before. Sorry for the noise.
  You may want to use git gc that does that (and a bit more) for you.

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

Re: git-fast-export hg mutt (24M vs 184M)

From: Shawn O. Pearce <hidden>
Date: 2016-06-15 22:43:08

Pierre Habouzit [off-list ref] wrote:
On Thu, May 03, 2007 at 09:17:16PM +0200, Thomas Glanzmann wrote:
quoted
Hello,
git-repack -a -d -f got it down to 19M. I missed the -f parameter
before. Sorry for the noise.
  You may want to use git gc that does that (and a bit more) for you.
Actually, in this case, no.

git-gc by default doesn't use the -f option.  -f to git-repack
means "no reuse deltas".  That particular feature of git-repack is
basically required to be used after running git-fast-import with
anything sizeable.

The reason you need -f is git-fast-import does not write optimally
compressed blobs (file revisions) when it creates the packfile.
Instead it does a reasonable best effort while using a minimum
amount of memory.  The Git packfiles get most of their compression
benefits from being able to see all of a project's data at once;
this is impossible in fast-import as we're only seeing a small part
of the incoming data stream at any single point in time.

If you had a lot of tags imported you might want to also use `git
pack-refs` (one of the chores that git-gc does), or `git pack-refs
--all` if you have a lot of dangling branches imported.  The other
chores in git-gc aren't actually useful after running fast-import
(reflog expire, prune, rerere gc).

-- 
Shawn.

Re: git-fast-export hg mutt (24M vs 184M)

From: Pierre Habouzit <hidden>
Date: 2016-06-15 22:43:08

On Thu, May 03, 2007 at 05:18:24PM -0400, Shawn O. Pearce wrote:
Pierre Habouzit [off-list ref] wrote:
quoted
On Thu, May 03, 2007 at 09:17:16PM +0200, Thomas Glanzmann wrote:
quoted
Hello,
git-repack -a -d -f got it down to 19M. I missed the -f parameter
before. Sorry for the noise.
  You may want to use git gc that does that (and a bit more) for you.
Actually, in this case, no.

git-gc by default doesn't use the -f option.  -f to git-repack
means "no reuse deltas".  That particular feature of git-repack is
basically required to be used after running git-fast-import with
anything sizeable.
  okay, so why git fast-import does not let some note somewhere (to be
picked by git gc later) "a fast-import has been run, use -f for next
repack if you want best compression" ?

  I'd think that would make a lot of sense, and that users that now
naively (like me) think git-gc would always be enough would not be
dramatically wrong ? :)

  I mean it's nothing *very* important but some 
  `touch $GIT_DIR/info/unpacked-fast-import` in fast-import then:
  if test -f $GIT_DIR/info/unpacked-fast-import; then
      REPACK_OPTIONS=$REPACK_OPTIONS\ -f
  fi
  // do the repack
  rm -f $GIT_DIR/info/unpacked-fast-import

  would do the trick, wouldn't it ?

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

Re: git-fast-export hg mutt (24M vs 184M)

From: Nicolas Pitre <hidden>
Date: 2016-06-15 22:43:08

On Fri, 4 May 2007, Pierre Habouzit wrote:
On Thu, May 03, 2007 at 05:18:24PM -0400, Shawn O. Pearce wrote:
quoted
Pierre Habouzit [off-list ref] wrote:
quoted
On Thu, May 03, 2007 at 09:17:16PM +0200, Thomas Glanzmann wrote:
quoted
Hello,
git-repack -a -d -f got it down to 19M. I missed the -f parameter
before. Sorry for the noise.
  You may want to use git gc that does that (and a bit more) for you.
Actually, in this case, no.

git-gc by default doesn't use the -f option.  -f to git-repack
means "no reuse deltas".  That particular feature of git-repack is
basically required to be used after running git-fast-import with
anything sizeable.
  okay, so why git fast-import does not let some note somewhere (to be
picked by git gc later) "a fast-import has been run, use -f for next
repack if you want best compression" ?
Nah.

The conversion script should do it itself directly after it is done with 
fast-import.


Nicolas
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help