git-svn failure when symlink added in svn

Hi,

A few weeks ago I reported a symlink related error with git-svn and
I've now had a chance to track down a few more details.  The trigger
seems to be if a file is removed from svn and then later added as a
symlink.  The error I get is:

  error: git-checkout-index: unable to create symlink foo.txt (Invalid argument)

This is from the call to symlink(new, path) in entry.c and it seems
that new is ''.

Here is a recipe to reproduce:

## First create an svn repository
  svnadmin create SVN123-repos
  svn co file:///Users/seth/temp/SVN123-repos SVN123
  cd SVN123
  echo 123 > foo.txt
  svn add foo.txt 
  svn ci -m "add a file"

## Now mirror using git-svn
  cd ..
  mkdir GIT123
  cd GIT123/
  git svn init file:///Users/seth/temp/SVN123-repos
  git svn fetch

## Next remove and add a file as a symlink
  cd ..
  cd SVN123
  echo 123 > bar.txt
  svn add bar.txt 
  svn ci -m"add bar"
  svn rm foo.txt 
  svn ci -m "remove foo"
  ln -s bar.txt foo.txt
  svn add foo.txt 
  svn ci -m"add foo as symlink"

## Finally, try to rebase
  cd ../GIT123/
  git svn rebase

git version 1.5.1.53.g77e6f
svn 1.4.0


+ seth

Seth Falcon [off-list ref] wrote:

A few weeks ago I reported a symlink related error with git-svn and
I've now had a chance to track down a few more details.  The trigger
seems to be if a file is removed from svn and then later added as a
symlink.  The error I get is:

  error: git-checkout-index: unable to create symlink foo.txt (Invalid argument)

This is from the call to symlink(new, path) in entry.c and it seems
that new is ''.

I can't reproduce it on Linux with ext3.  I translated your recipe into
a test script in the patch below.  Anybody familiar with OSX and/or HFS
know if there's a workaround or fix for this?

From: Eric Wong <redacted>
Date: Sat, 14 Apr 2007 13:03:24 -0700
Subject: [PATCH] git-svn: add test to handle conversion from file to symlink

This does not trigger any failures on my Linux machine with ext3,
but it may fail on OSX and/or HFS.

This test is based on a bug report by Seth Falcon:
  http://permalink.gmane.org/gmane.comp.version-control.git/44445

Signed-off-by: Eric Wong <redacted>
---
 t/t9112-git-svn-file-to-symlink.sh |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
 create mode 100755 t/t9112-git-svn-file-to-symlink.sh

diff --git a/t/t9112-git-svn-file-to-symlink.sh b/t/t9112-git-svn-file-to-symlink.sh
new file mode 100755
index 0000000..f94310f
--- /dev/null
+++ b/t/t9112-git-svn-file-to-symlink.sh

@@ -0,0 +1,36 @@
+#!/bin/sh
+# Copyright (c) 2007 Eric Wong
+test_description='git-svn file to symlink'
+. ./lib-git-svn.sh
+
+test_expect_success 'create file in svn repository' "
+	svn co '$svnrepo' svn &&
+	cd svn &&
+		echo 123 > foo.txt &&
+		svn add foo.txt &&
+		svn commit -m 'add a file'
+		cd ..
+	"
+
+test_expect_success 'clone with git-svn' "pwd && git svn clone '$svnrepo' git"
+
+test_expect_success 'remove and add file as symlink in svn' "
+	cd svn &&
+		echo 123 > bar.txt &&
+		svn add bar.txt &&
+		svn commit -m 'add bar' &&
+		svn rm foo.txt &&
+		svn commit -m 'remove foo' &&
+		ln -s bar.txt foo.txt &&
+		svn add foo.txt &&
+		svn ci -m 'add foo as symlink'
+		cd ..
+	"
+
+test_expect_success 'rebase in git-svn' "
+	cd git &&
+		git svn rebase
+		cd ..
+	"
+
+test_done

-- 
Eric Wong

Eric Wong [off-list ref] writes:

I can't reproduce it on Linux with ext3.  I translated your recipe into
a test script in the patch below.  Anybody familiar with OSX and/or HFS
know if there's a workaround or fix for this?

Thanks for sending the test case.  It doesn't properly fail for me on
OSX, but if I run it with -v then I do see the error (so it is failing
on OSX and, as you found, not on Linux).

I added a silly print statement to see the symlink args:

diff --git a/entry.c b/entry.c
index d72f811..70f6402 100644
--- a/entry.c
+++ b/entry.c

@@ -129,6 +129,7 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
                                return error("git-checkout-index: unable to write file %s",
                                        path);
                } else {
+                        fprintf(stderr, "symlink: '%s' => '%s'\n", path, new);
                        wrote = symlink(new, path);
                        free(new);
                        if (wrote)

And so then on Linux with -v I get (after snipping most of the
output):

   * expecting success: 
           cd git &&
                   git svn rebase
                   cd ..
   
           A       bar.txt
   r2 = 31e734669e3fe4dbbd375e5a9f5af828a5b7ba92 (git-svn)
           D       foo.txt
   r3 = bd3b318730e8efc77235976abb18d04bc927bf9e (git-svn)
           A       foo.txt
   r4 = 2376eedcfec1de7cbe69b2bbad1c5de231a0ed0d (git-svn)
   First, rewinding head to replay your work on top of it...
   symlink: 'foo.txt' => 'bar.txt'
   HEAD is now at 2376eed... add foo as symlink
   Fast-forwarded master to refs/remotes/git-svn.
   *   ok 4: rebase in git-svn
   
   * passed all 4 test(s)

On my OSX laptop I get:

   * expecting success: 
           cd git &&
                   git svn rebase
                   cd ..
   
           A       bar.txt
   r2 = 4964f302b94ede0301b33faf5f4242c4bab3108b (git-svn)
           D       foo.txt
   r3 = 178a9ff3c7013d4ad8ec7defa93b91a1080c1e53 (git-svn)
           A       foo.txt
   r4 = 9f0bc38df8113fe1e11e47b708589d82bfa035a0 (git-svn)
   First, rewinding head to replay your work on top of it...
   symlink: 'foo.txt' => ''
   error: git-checkout-index: unable to create symlink foo.txt (Invalid argument)
   HEAD is now at 9f0bc38... add foo as symlink
   Fast-forwarded master to refs/remotes/git-svn.
   *   ok 4: rebase in git-svn
   
   * passed all 4 test(s)

If you're still with me, the curious part is what the symlink call is
trying to do.

  Linux:    symlink: 'foo.txt' => 'bar.txt'
    OSX:    symlink: 'foo.txt' => ''

So it looks like the problem is some sort of off-by-one that happens
well before the symlink call.  Perhaps this is enough for someone more
knowledgable than me to have a clue where to look next?

+ seth

Hi,

Seth Falcon <sethfalcon <at> gmail.com> writes:

Eric Wong <normalperson <at> yhbt.net> writes:

quoted

I can't reproduce it on Linux with ext3.  I translated your recipe into
a test script in the patch below.  Anybody familiar with OSX and/or HFS
know if there's a workaround or fix for this?

I've been investigating this problem too, as it keeps biting me when importing
our (OpenXPKIs) subversion tree using git-svn. I'd love to work with git and
am happy to help with debugging this further. Still, I am a pretty puzzled on why
this happens ...

And so then on Linux with -v I get (after snipping most of the
output):
   First, rewinding head to replay your work on top of it...
   symlink: 'foo.txt' => 'bar.txt'

On my OSX laptop I get:
   First, rewinding head to replay your work on top of it...
   symlink: 'foo.txt' => ''

Same here (this is a MacBook Pro, for what it's worth, BTW). As said, I've
investigated this a bit further. The empty filename in new seems to come from
trying to read the wrong SHA1 file. If one outputs ce->sha1 before
        void *new = read_sha1_file(ce->sha1, &type, size);
is called, one gets different output on Linux and Mac OS X.
For Seth's example, I get 5f34b0af07646aa529b5b005cde3a9559e606210 on Linux
and e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 on Mac OS X ...

I've tried tracking down where this comes from. Here is what I've learned:
- read_blob_entry() is called from write_entry().
  SHA1 is already incorrect at that point in time.
- write_entry() is called from checkout_entry().
  SHA1 is already incorrect at that point in time.
- checkout_entry() is called from check_updates().
  SHA1 is already incorrect at that point in time.

Unluckily I could not figure out, where it is computed in the first place.
One idea was that maybe it was cached from the old file in the Mac OS X case
and recomputed on Linux or so? Or maybe it's not git's fault but git-svn
messes up (although I doubt it)?

I'll happy try out anything that has a slight chance of solving this issue
(workarounds greatly appreciated, too).

Best regards,
  Alex

On Thu, 26 Apr 2007, Alexander Klink wrote:

Same here (this is a MacBook Pro, for what it's worth, BTW). As said, I've
investigated this a bit further. The empty filename in new seems to come from
trying to read the wrong SHA1 file. If one outputs ce->sha1 before
        void *new = read_sha1_file(ce->sha1, &type, size);
is called, one gets different output on Linux and Mac OS X.
For Seth's example, I get 5f34b0af07646aa529b5b005cde3a9559e606210 on Linux
and e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 on Mac OS X ...

Well, 5f34b0af0 is the "bar.txt" blob, while e69de29b is the empty blob

(Eg do

	[torvalds@woody git]$ echo -en "blob 7\0bar.txt" | sha1sum
	5f34b0af07646aa529b5b005cde3a9559e606210  -

	[torvalds@woody git]$ echo -en "blob 0\0" | sha1sum
	e69de29bb2d1d6434b8b29ae775ad8c2e48c5391  -

to verify: git objects not only contain the data, but embed the object 
type and size too).

So yeah, the printout matches the SHA1's, and the SHA1's are clearly not 
corrupted: they are just a sign of the fact that the data that was fed to 
whoever generated the SHA1's was simply different.

But why git-svn would act differently under OS X than under Linux I have 
no idea.

		Linus

Linus Torvalds <torvalds <at> linux-foundation.org> writes:

quoted

trying to read the wrong SHA1 file. If one outputs ce->sha1 before
        void *new = read_sha1_file(ce->sha1, &type, size);
is called, one gets different output on Linux and Mac OS X.
For Seth's example, I get 5f34b0af07646aa529b5b005cde3a9559e606210 on Linux
and e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 on Mac OS X ...

Well, 5f34b0af0 is the "bar.txt" blob, while e69de29b is the empty blob

So yeah, the printout matches the SHA1's, and the SHA1's are clearly not 
corrupted: they are just a sign of the fact that the data that was fed to 
whoever generated the SHA1's was simply different.

The SHA1 hashes are generated in the close_file() function in
git-svn.perl by forking git-hash-object -w --stdin and redirecting
STDIN to the passed filehandle. This is what goes wrong (for some
perl-internal reason?) on Mac OS X. Here is a patch that works
around that by putting the data to be hashed in a temporary file
and calling git-hash-object with a filename.

Best regards,
    Alex

From 504892b882d05bdf1fbf2325e7544f52115555d1 Mon Sep 17 00:00:00 2001

From: Alexander Klink <redacted>
Date: Sat, 28 Apr 2007 14:46:27 +0200
Subject: [PATCH] Workaround for git-svn symlink problem on Mac OS X

git-svn had a problem with creating a symlink for a file which existed
as a "real" file beforehand. See the report from Seth Falcon:
http://permalink.gmane.org/gmane.comp.version-control.git/44445
and the test patch by Eric Wong:
http://permalink.gmane.org/gmane.comp.version-control.git/44469

Apparently, the reason for this is that in this case, perl on Mac OS X
does not like the STDIN redirect in close_file() (which forks to
git-hash-object -w --stdin to create the SHA1 hash).
The workaround now creates a temporary file for the git-hash-object input
using File::Temp, calls git-hash-object -w with the filename and safely
unlinks the file afterwards.
---
 git-svn.perl |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 077d6b3..7af45aa 100755
--- a/git-svn.perl
+++ b/git-svn.perl

@@ -2266,6 +2266,7 @@ use warnings;
 use Carp qw/croak/;
 use IO::File qw//;
 use Digest::MD5;
+use File::Temp;
 
 # file baton members: path, mode_a, mode_b, pool, fh, blob, base
 sub new {

@@ -2448,13 +2449,16 @@ sub close_file {
 			$buf eq 'link ' or die "$path has mode 120000",
 			                       "but is not a link\n";
 		}
-		defined(my $pid = open my $out,'-|') or die "Can't fork: $!\n";
-		if (!$pid) {
-			open STDIN, '<&', $fh or croak $!;
-			exec qw/git-hash-object -w --stdin/ or croak $!;
-		}
-		chomp($hash = do { local $/; <$out> });
-		close $out or croak $!;
+
+        # put the data for git-hash-object in a temporary file,
+        # as redirecting STDIN does not always work for some reason on
+        # Mac OS X
+        my ($temp_fh, $temp_filename) = mkstemp("git-hash-input-XXXXXX");
+        print $temp_fh do { local $/; <$fh> };
+
+        chomp($hash = qx(git-hash-object -w $temp_filename));
+        File::Temp::unlink0($temp_fh, $temp_filename)
+            or die "Error unlinking temporary file $temp_filename";
 		close $fh or croak $!;
 		$hash =~ /^[a-f\d]{40}$/ or die "not a sha1: $hash\n";
 		close $fb->{base} or croak $!;

-- 
1.5.2.rc0.34.gda94-dirty