parsecvs fails even on simple input

One of the patches recently merged into parsecvs master, namely

commit f5b3cb849517adfd7790c1bfa84bbb84220e3e7b
Author: Al Viro [off-list ref]
Date:   Tue Jan 16 04:15:35 2007 -0500

    [PATCH] generate tree objects just as we calculate changesets
    
    ... and don't store the fsckloads of rev_file in ExportGit mode; they
    are only needed (for now) in ExportGraph one.
    
    Tree generation is done directly, without hitting on-disk index.  Very fast
    now.

broke parsecvs entirely.  The reproducer (attached) is very simple:
initial commit of a just added file.  parsecvs now barfs on it:

# parsecvs a,v
Initialized empty shared Git repository in .git/
Load:                                 a,v ....................*     1 of     1
Pack pack-6b955e2d966143fc957ccd272e9dd822ceaccf25 created
Removing unused objects 81%...
Removing unused objects 100%...
Done.

error: invalid object d0141680ee5324d51a558a0a48c8a867cbc6a47c
error: writing tree
Authors: No such file or directory
Save:                              master ....................*     1 of     1


The problem is the following: after that commit parsecvs tries to add
objects to the git tree on its own via calls to libgit; however, in
between it runs git-pack-objects.  Thus objects move to pack files
without libgit being aware of it; this results in 'ivalid object'
errors.

However, the object with that hash exists but is stored on the pack
file; if parsecvs is run in the same directory for the second time it
finds it there and happily completes.

I haven't yet had the time to dig deeper into this problem and code a
patch; however, the whole idea of doing part of the job through the
(unpublished) libgit API and the rest via callouts to git utilities
looks like calling for trouble.  Wouldn't it be better to teach parsecvs
to speak git-fast-import language instead?

Roman.

On Fri, 2007-06-22 at 15:36 +0400, Roman Kagan wrote:

The problem is the following: after that commit parsecvs tries to add
objects to the git tree on its own via calls to libgit; however, in
between it runs git-pack-objects.  Thus objects move to pack files
without libgit being aware of it; this results in 'ivalid object'
errors.

Sticking a call to reprepare_packed_git() after the pack creation fixes
this nicely.

 Wouldn't it be better to teach parsecvs
to speak git-fast-import language instead?

Avoiding fork/exec is rather important for parsecvs perforamance.

-- 
keith.packard@intel.com

Keith Packard [off-list ref] wrote:

On Fri, 2007-06-22 at 15:36 +0400, Roman Kagan wrote:

quoted

 Wouldn't it be better to teach parsecvs
to speak git-fast-import language instead?

Avoiding fork/exec is rather important for parsecvs perforamance.

That sort of thing is the entire point behind fast-import.  Its only
one fork+exec to setup the fast-import "daemon" in the background,
and you do everything over a pipe to its stdin.  Including forcing
it to finish its current packfile and open a new one on the next
object (the `checkpoint` command).

fast-import is fast, its input language is fairly simple, and its
quite stable.  And its only one fork+exec.  That's peanuts compared
to the disk IO involved in any sizable import process.

-- 
Shawn.

On Sun, Jun 24, 2007 at 10:31:07PM +0100, Keith Packard wrote:

On Fri, 2007-06-22 at 15:36 +0400, Roman Kagan wrote:

quoted

The problem is the following: after that commit parsecvs tries to add
objects to the git tree on its own via calls to libgit; however, in
between it runs git-pack-objects.  Thus objects move to pack files
without libgit being aware of it; this results in 'ivalid object'
errors.

Sticking a call to reprepare_packed_git() after the pack creation fixes
this nicely.

Ehm sort of...  Except that I woudn't call that extern declaration
nice.

I'm now tracking down another problem which I didn't see before:
parsecvs apparently doesn't close .git-cvs/log-XXX files and ends up
exhausting the open file descriptor limit.  I'll update when I have more
info.

quoted

 Wouldn't it be better to teach parsecvs
to speak git-fast-import language instead?

Avoiding fork/exec is rather important for parsecvs perforamance.

Avoiding _one_ fork/exec is certainly not.

OTOH git-fast-import seems to be essentially the public API for the
parsecvs kind of tasks.  It may be wiser from the maintenance POV to use
that instead of direct libgit calls (unless parsecvs is going to land in
the git tree).  I'll try to find the time and take a look at this
somewhere next week.

Roman.

On Wed, 2007-06-27 at 19:33 +0400, Roman Kagan wrote:

OTOH git-fast-import seems to be essentially the public API for the
parsecvs kind of tasks.  It may be wiser from the maintenance POV to use
that instead of direct libgit calls (unless parsecvs is going to land in
the git tree).  I'll try to find the time and take a look at this
somewhere next week.

Yeah, I didn't quite understand how git-fast-import worked. Looks like
it aligns with parsecvs's structure fairly well. Let me know if you get
it working.

-- 
keith.packard@intel.com