Unable to index file

Hi everyone,

I've come across a problem that I don't believe lies in Rails.  You
needn't be familiar, I think, with Rails to see what's wrong.

I can't seem to add the files that are in
http://dev.rubyonrails.org/archive/rails_edge.zip

1) Unpack the zip
2) Initialize a git repo inside the folder that was unpacked
3) git add .

See the errors.. :o http://pastie.org/337571

Thanks,
Ramon Tayag

On Fri, 12 Dec 2008, Ramon Tayag wrote:

I've come across a problem that I don't believe lies in Rails.  You
needn't be familiar, I think, with Rails to see what's wrong.

I can't seem to add the files that are in
http://dev.rubyonrails.org/archive/rails_edge.zip

1) Unpack the zip
2) Initialize a git repo inside the folder that was unpacked
3) git add .

See the errors.. :o http://pastie.org/337571

What platform/filesystem is this?

Git is rather particular about symlinks, and it looks like your platform 
does something odd, and that makes git unhappy about your symlink.

In particular:

	ls -l vendor/rails/actionpack/test/fixtures/layout_tests/layouts/ 
	...
	lrwxrwxrwx 1 root root 48 2008-12-12 18:22 symlinked -> ../../symlink_parent

notice how the symlink content is "../../symlink_parent", but then take a 
look at the _size_ of the symlink: 48 bytes.

Git expects the lstat() information to match the return from readlink(), 
and it doesn't.

For exact details, see "index_path()" in sha1_file.c:

        case S_IFLNK:   
                len = xsize_t(st->st_size);
                target = xmalloc(len + 1);
                if (readlink(path, target, len + 1) != st->st_size) {
                        char *errstr = strerror(errno);

ie we consider it an error if we get less than st_size characters back 
from readlink().

Now, admittedly git is probably being really annoyingly anal about this 
all, and we probably should loosen the restrictions on it a bit, but I'd 
like to know why it happens. I cannot recall this having been reported 
before, so it's some specific filesystem or OS that causes this, I think.

		Linus

On Fri, 12 Dec 2008, Linus Torvalds wrote:

Now, admittedly git is probably being really annoyingly anal about this 
all, and we probably should loosen the restrictions on it a bit, but I'd 
like to know why it happens. I cannot recall this having been reported 
before, so it's some specific filesystem or OS that causes this, I think.

Anyway, the "loosen the symlink lstat() requirements" patch would likely 
look something like this. I can't really test it, though, since I only 
have filesystems that have matching lstat()/readlink() sizes.

		Linus
---
 sha1_file.c |   11 ++++++-----
 1 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 0e021c5..222c793 100644
--- a/sha1_file.c
+++ b/sha1_file.c

@@ -2522,9 +2522,9 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object,
 
 int index_path(unsigned char *sha1, const char *path, struct stat *st, int write_object)
 {
-	int fd;
+	int fd, len;
 	char *target;
-	size_t len;
+	size_t bufsize;
 
 	switch (st->st_mode & S_IFMT) {
 	case S_IFREG:

@@ -2537,9 +2537,10 @@ int index_path(unsigned char *sha1, const char *path, struct stat *st, int write
 				     path);
 		break;
 	case S_IFLNK:
-		len = xsize_t(st->st_size);
-		target = xmalloc(len + 1);
-		if (readlink(path, target, len + 1) != st->st_size) {
+		bufsize = 1+xsize_t(st->st_size);
+		target = xmalloc(bufsize);
+		len = readlink(path, target, bufsize);
+		if (len < 0) {
 			char *errstr = strerror(errno);
 			free(target);
 			return error("readlink(\"%s\"): %s", path,

Oh thank you for your very detailed explanation.  I was bothering
people on #rubyonrails and #git but got no answers.

I'm on Ubuntu 8.10.  The files I'm working on live on an NTFS drive
(my "storage" drive; yes, I still have to boot into Windows).  If it
being on NTFS makes a difference I'll try this on ext3 and let you
know what happens :)

Thanks,
Ramon Tayag



On Sat, Dec 13, 2008 at 2:07 AM, Linus Torvalds
[off-list ref] wrote:

On Fri, 12 Dec 2008, Ramon Tayag wrote:

quoted

I've come across a problem that I don't believe lies in Rails.  You
needn't be familiar, I think, with Rails to see what's wrong.

I can't seem to add the files that are in
http://dev.rubyonrails.org/archive/rails_edge.zip

1) Unpack the zip
2) Initialize a git repo inside the folder that was unpacked
3) git add .

See the errors.. :o http://pastie.org/337571

What platform/filesystem is this?

Git is rather particular about symlinks, and it looks like your platform
does something odd, and that makes git unhappy about your symlink.

In particular:

       ls -l vendor/rails/actionpack/test/fixtures/layout_tests/layouts/
       ...
       lrwxrwxrwx 1 root root 48 2008-12-12 18:22 symlinked -> ../../symlink_parent

notice how the symlink content is "../../symlink_parent", but then take a
look at the _size_ of the symlink: 48 bytes.

Git expects the lstat() information to match the return from readlink(),
and it doesn't.

For exact details, see "index_path()" in sha1_file.c:

       case S_IFLNK:
               len = xsize_t(st->st_size);
               target = xmalloc(len + 1);
               if (readlink(path, target, len + 1) != st->st_size) {
                       char *errstr = strerror(errno);

ie we consider it an error if we get less than st_size characters back
from readlink().

Now, admittedly git is probably being really annoyingly anal about this
all, and we probably should loosen the restrictions on it a bit, but I'd
like to know why it happens. I cannot recall this having been reported
before, so it's some specific filesystem or OS that causes this, I think.

               Linus

On Sat, 13 Dec 2008, Ramon Tayag wrote:

I'm on Ubuntu 8.10.  The files I'm working on live on an NTFS drive
(my "storage" drive; yes, I still have to boot into Windows).  If it
being on NTFS makes a difference I'll try this on ext3 and let you
know what happens :)

Ok, it's almost certainly the NTFS part.

And at the same time, I'm almost certain that my patch should fix it and 
is the right thing for git to do anyway. 

		Linus