Re: [PATCH] git-mv is not able to handle big directories

Alexander Litvinov [off-list ref] writes:

When moving directory with large number of files git-mv says:

quoted

git-mv jsp* .

Can't exec "git-update-index": Argument list too long at /usr/local/bin/git-mv 
line 193.
git-update-index failed to add new names with code -1

This patch fixes this by building list of files with limited len (currently 
5000) and executing git-update-index few times until all files will be 
processed. I don't know how to determinate limit of command line but 5000 
seems safe enougth to me.

Two comments.

(1) the argument limit is enforced by the operating system in
    bytes (including environment size unfortunately) so we might
    want to count bytes not number of paths.  I heard GNU xargs
    uses 131072 as the default limit.

(2) I wonder if we can detect this particular failure case and
    then fall back on splitting the arguments dynamically, maybe
    something like this:

	sub xargs_system {
        	my ($cmd, @args) = @_;
                my $rc = system(@$cmd, @args);
                if ($rc == 'argument list too long error') {
                	my (@args0) = splice(@args, 0, @args/2);
			$rc = xargs_system($cmd, @args0);
                        return $c if ($rc);
                        return xargs_system($cmd, @args);
		}
		return $rc;
	}

    and:

	$rc = xargs_system([qw(git-update-index --)], @changedfiles);
	$rc = xargs_system([qw(git-update-index --add --)], @addedfiles);
	...

I have found one error during directory movig: If I move directory with one 
file somewhere in it this script will try to add target directory instead of 
file. Commenting lines starting from 190 solve this error. But I don't 
understand what is the logic behind this case ? Why do target directory 
checked instead of target file ? Should we replace $dst my $destfiles[0] ?

at line 190 in git-mv:
    if (scalar @srcfiles == 1) {
	if ($overwritten{$dst} ==1) {
	    push @changedfiles, $dst;
	} else {
	    push @addedfiles, $dst;
	}
    }
    else {
	push @addedfiles, @dstfiles;
    }

Alexander Litvinov wrote:

I have found one error during directory movig: If I move directory with one 
file somewhere in it this script will try to add target directory instead of 
file.

Are you saying this setup
	foodir/somefile.c     <--file
	newdir/               <--directory

with this command
	git-mv foodir/ newdir

tries to create
	newdir/foodir/somefile.c  <-- directory

or does it create
	newdir/somefile.c         <-- file

?

It should create
	newdir/foodir/somefile.c  <-- file

Otherwise it's misbehaving.

Try running it with the -v switch to make it shout out loud what it's 
trying to do, and then paste the output here.

Commenting lines starting from 190 solve this error. But I don't 
understand what is the logic behind this case ? Why do target directory 
checked instead of target file ? Should we replace $dst my $destfiles[0] ?

at line 190 in git-mv:
    if (scalar @srcfiles == 1) {
	if ($overwritten{$dst} ==1) {
	    push @changedfiles, $dst;
	} else {
	    push @addedfiles, $dst;
	}
    }
    else {
	push @addedfiles, @dstfiles;
    }

This is broken. It only checks if there's just one source-file 
regardless of whether or not it resided in a subdirectory. I'm not 
exactly fluent in perl so I can't submit a patch, but the src option 
needs to be directory aware, traverse all source directories and then 
move the files axing everything but the bottom-most dirname to the 
destination directory.

Any takers?

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

On Wednesday 23 November 2005 13:57, Andreas Ericsson wrote:

Are you saying this setup
	foodir/somefile.c     <--file
	newdir/               <--directory

with this command
	git-mv foodir/ newdir

It is calling git-update-index --add newdir but actiual file structire is 
correct: 
newdir/somefile.c - is a file

On Wednesday 23 November 2005 13:57, Andreas Ericsson wrote:

This is broken. It only checks if there's just one source-file
regardless of whether or not it resided in a subdirectory. I'm not
exactly fluent in perl so I can't submit a patch, but the src option
needs to be directory aware, traverse all source directories and then
move the files axing everything but the bottom-most dirname to the
destination directory.

Any takers?

I still does not understand what this part should do. I know perl enought to 
fix it but I don't understand the logic.

On Wednesday 23 November 2005 11:21, Alexander Litvinov wrote:

On Wednesday 23 November 2005 13:57, Andreas Ericsson wrote:

quoted

This is broken. It only checks if there's just one source-file
regardless of whether or not it resided in a subdirectory.

Yes.
For git-update-index we have to use the file inside the directory.

I just sent a patch for this.

Josef

On Wednesday 23 November 2005 08:26, Alexander Litvinov wrote:

I have found one error during directory movig: If I move directory with one 
file somewhere in it this script will try to add target directory instead of 
file. Commenting lines starting from 190 solve this error. But I don't 
understand what is the logic behind this case ? Why do target directory 
checked instead of target file ?

$src/$dst pairs match the arguments given on the command line of git-mv,
And git-mv (as was git-rename) is able to move directory trees around.

It is simplier to keep the granularity at directories, as multiple moves
of directories can not overlap each other. If you do it at file level, you
have to remote and create directories yourself.

Of course, for git it is done at file level (that are the @srcfiles/@dstfiles
arrays), but this is quite trivial as git does not work with directories, but
only with files.

Should we replace $dst my $destfiles[0] ?

Yes, that was part of my patch.

Josef