How to "git add ." when some files are not accessible (permission denied)?

10 messages, 5 authors, 2016-06-15 · open the first message on its own page

How to "git add ." when some files are not accessible (permission denied)?

From: Dirk Süsserott <hidden>
Date: 2016-06-15 22:44:19

Hello --

First of all: I'm using Git with Windows. Most of the time it works 
*very* good, but now I've a problem that might appear to Unix users as 
well. I had the idea to use Git to track changes in my C:\WINDOWS 
directory. I thought the following would work:

$ cd /c/WINDOWS             (1)
$ git init                  (2)
$ git add .                 (3)
$ git commit -m "Initial"   (4)

And then issue "git status" or so to see the differences after 
installing or running some software.

However, when issueing (3) "git add ." it adds hundreds and thousands of
files and then stops with

     error: open("foo"): Permission denied: foo
     fatal: unable to index file foo

The file "foo" is not accessible for me, even though I'm administrator.
This might occur to Windows and Linux persons as well, I guess. The 
adding stops overall.

The question is: is there a way to tell "git add ." to add all files but
ignore those that cannot be added due to lack of authorization?

Or, more generally spoken: can I tell "git add" to add only those files 
it can handle and ignore the rest? The "-f" switch doesn't work and some
exclude lists on a per file basis are not applicable for my purpose as I
don't know the files in advance.

I'm aware that I could do it with some fancy shell commands, but very 
often I was surprised how many really cool commands Git offers to "do 
what I mean". Wished other software would be so usable :-).


Cheers,
  -- Dirk

Re: How to "git add ." when some files are not accessible (permission denied)?

From: Jeff King <hidden>
Date: 2016-06-15 22:44:19

On Sat, Mar 01, 2008 at 02:46:28PM +0100, Dirk Süsserott wrote:
The question is: is there a way to tell "git add ." to add all files but
ignore those that cannot be added due to lack of authorization?
No, there isn't. Under Linux, I would come up with a list of files I was
interested in and then pipe it to "xargs git-add", which is probably
unhelpful for Windows.

But I think more fundamentally, you probably _do_ want to come up with a
list of files that you can't access and add them to your .gitignore (or
your .git/info/exclude file if they are purely a local matter). That is
the official way to tell all git commands "I really don't care about
these files".  Otherwise they will keep getting brought up when you do,
e.g., a git-status.
Or, more generally spoken: can I tell "git add" to add only those files  
it can handle and ignore the rest? The "-f" switch doesn't work and some
exclude lists on a per file basis are not applicable for my purpose as I
don't know the files in advance.
The only reason I can think of to not want to generate such an ignore
list is if you are frequently and automagically doing a "git add ." to
pick up new files. For that reason, a "try to continue even if we can't
look at some files" option to git add might make some sense.

-Peff

Re: How to "git add ." when some files are not accessible (permission denied)?

From: Alex Riesen <hidden>
Date: 2016-06-15 22:44:19

Dirk Süsserott, Sat, Mar 01, 2008 14:46:28 +0100:
Or, more generally spoken: can I tell "git add" to add only those files it 
can handle and ignore the rest? The "-f" switch doesn't work and some
exclude lists on a per file basis are not applicable for my purpose as I
don't know the files in advance.
Well, "-f" means something else (include the ignored files). It is
unfortunate, because (I think) your case fits better its traditional
meaning...

You can try the following patches, which add "--ignore-errors" to
git-add. Maybe it will be enough... It is generally considered
not safe to ignore errors.

[PATCH] Make the exit code of add_file_to_index actually useful

From: Alex Riesen <hidden>
Date: 2016-06-15 22:44:19

Update the programs which used the function (as add_file_to_cache).

Signed-off-by: Alex Riesen <redacted>
---
 builtin-add.c    |    6 ++++--
 builtin-commit.c |    7 ++++---
 builtin-mv.c     |    3 ++-
 read-cache.c     |    8 ++++----
 4 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/builtin-add.c b/builtin-add.c
index 820110e..abfe473 100644
--- a/builtin-add.c
+++ b/builtin-add.c
@@ -94,7 +94,8 @@ static void update_callback(struct diff_queue_struct *q,
 		case DIFF_STATUS_UNMERGED:
 		case DIFF_STATUS_MODIFIED:
 		case DIFF_STATUS_TYPE_CHANGED:
-			add_file_to_cache(path, verbose);
+			if (add_file_to_cache(path, verbose))
+				exit(1);
 			break;
 		case DIFF_STATUS_DELETED:
 			remove_file_from_cache(path);
@@ -266,7 +267,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 	}
 
 	for (i = 0; i < dir.nr; i++)
-		add_file_to_cache(dir.entries[i]->name, verbose);
+		if (add_file_to_cache(dir.entries[i]->name, verbose))
+			exit(1);
 
  finish:
 	if (active_cache_changed) {
diff --git a/builtin-commit.c b/builtin-commit.c
index f49c22e..fb1e588 100644
--- a/builtin-commit.c
+++ b/builtin-commit.c
@@ -176,9 +176,10 @@ static void add_remove_files(struct path_list *list)
 	int i;
 	for (i = 0; i < list->nr; i++) {
 		struct path_list_item *p = &(list->items[i]);
-		if (file_exists(p->path))
-			add_file_to_cache(p->path, 0);
-		else
+		if (file_exists(p->path)) {
+			if (add_file_to_cache(p->path, 0))
+				exit(1);
+		} else
 			remove_file_from_cache(p->path);
 	}
 }
diff --git a/builtin-mv.c b/builtin-mv.c
index 68aa2a6..ec6e09d 100644
--- a/builtin-mv.c
+++ b/builtin-mv.c
@@ -260,7 +260,8 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 
 		for (i = 0; i < added.nr; i++) {
 			const char *path = added.items[i].path;
-			add_file_to_cache(path, verbose);
+			if (add_file_to_cache(path, verbose))
+				exit(1);
 		}
 
 		for (i = 0; i < deleted.nr; i++)
diff --git a/read-cache.c b/read-cache.c
index 657f0c5..4a4f511 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -461,10 +461,10 @@ int add_file_to_index(struct index_state *istate, const char *path, int verbose)
 	unsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_RACY_IS_DIRTY;
 
 	if (lstat(path, &st))
-		die("%s: unable to stat (%s)", path, strerror(errno));
+		return error("%s: unable to stat (%s)", path, strerror(errno));
 
 	if (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode) && !S_ISDIR(st.st_mode))
-		die("%s: can only add regular files, symbolic links or git-directories", path);
+		return error("%s: can only add regular files, symbolic links or git-directories", path);
 
 	namelen = strlen(path);
 	if (S_ISDIR(st.st_mode)) {
@@ -501,9 +501,9 @@ int add_file_to_index(struct index_state *istate, const char *path, int verbose)
 	}
 
 	if (index_path(ce->sha1, path, &st, 1))
-		die("unable to index file %s", path);
+		return error("unable to index file %s", path);
 	if (add_index_entry(istate, ce, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE))
-		die("unable to add %s to index",path);
+		return error("unable to add %s to index",path);
 	if (verbose)
 		printf("add '%s'\n", path);
 	return 0;
-- 
1.5.4.3.391.gf5a0c

[PATCH] Extend interface of add_files_to_cache to allow ignore indexing errors

From: Alex Riesen <hidden>
Date: 2016-06-15 22:44:19

Signed-off-by: Alex Riesen <redacted>
---
 builtin-add.c      |   37 ++++++++++++++++++++++++++++---------
 builtin-checkout.c |    2 +-
 builtin-commit.c   |    2 +-
 cache.h            |    8 +++++++-
 4 files changed, 37 insertions(+), 12 deletions(-)
diff --git a/builtin-add.c b/builtin-add.c
index abfe473..bc55a0e 100644
--- a/builtin-add.c
+++ b/builtin-add.c
@@ -79,12 +79,18 @@ static void fill_directory(struct dir_struct *dir, const char **pathspec,
 		prune_directory(dir, pathspec, baselen);
 }
 
+struct update_callback_data
+{
+	int flags;
+	int add_errors;
+};
+
 static void update_callback(struct diff_queue_struct *q,
 			    struct diff_options *opt, void *cbdata)
 {
-	int i, verbose;
+	int i;
+	struct update_callback_data *data = cbdata;
 
-	verbose = *((int *)cbdata);
 	for (i = 0; i < q->nr; i++) {
 		struct diff_filepair *p = q->queue[i];
 		const char *path = p->one->path;
@@ -94,28 +100,35 @@ static void update_callback(struct diff_queue_struct *q,
 		case DIFF_STATUS_UNMERGED:
 		case DIFF_STATUS_MODIFIED:
 		case DIFF_STATUS_TYPE_CHANGED:
-			if (add_file_to_cache(path, verbose))
-				exit(1);
+			if (add_file_to_cache(path, data->flags & ADD_FILES_VERBOSE)) {
+				if (!(data->flags & ADD_FILES_IGNORE_ERRORS))
+					exit(1);
+				data->add_errors++;
+			}
 			break;
 		case DIFF_STATUS_DELETED:
 			remove_file_from_cache(path);
-			if (verbose)
+			if (data->flags & ADD_FILES_VERBOSE)
 				printf("remove '%s'\n", path);
 			break;
 		}
 	}
 }
 
-void add_files_to_cache(int verbose, const char *prefix, const char **pathspec)
+int add_files_to_cache(const char *prefix, const char **pathspec, int flags)
 {
+	struct update_callback_data data;
 	struct rev_info rev;
 	init_revisions(&rev, prefix);
 	setup_revisions(0, NULL, &rev, NULL);
 	rev.prune_data = pathspec;
 	rev.diffopt.output_format = DIFF_FORMAT_CALLBACK;
 	rev.diffopt.format_callback = update_callback;
-	rev.diffopt.format_callback_data = &verbose;
+	data.flags = flags;
+	data.add_errors = 0;
+	rev.diffopt.format_callback_data = &data;
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
+	return !!data.add_errors;
 }
 
 static void refresh(int verbose, const char **pathspec)
@@ -193,6 +206,7 @@ static struct option builtin_add_options[] = {
 
 int cmd_add(int argc, const char **argv, const char *prefix)
 {
+	int exit_status = 0;
 	int i, newfd;
 	const char **pathspec;
 	struct dir_struct dir;
@@ -209,11 +223,16 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 	newfd = hold_locked_index(&lock_file, 1);
 
 	if (take_worktree_changes) {
+		int flags = 0;
 		const char **pathspec;
 		if (read_cache() < 0)
 			die("index file corrupt");
 		pathspec = get_pathspec(prefix, argv);
-		add_files_to_cache(verbose, prefix, pathspec);
+
+		if (verbose)
+			flags |= ADD_FILES_VERBOSE;
+
+		exit_status = add_files_to_cache(prefix, pathspec, flags);
 		goto finish;
 	}
 
@@ -277,5 +296,5 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 			die("Unable to write new index file");
 	}
 
-	return 0;
+	return exit_status;
 }
diff --git a/builtin-checkout.c b/builtin-checkout.c
index b0cd788..2def093 100644
--- a/builtin-checkout.c
+++ b/builtin-checkout.c
@@ -273,7 +273,7 @@ static int merge_working_tree(struct checkout_opts *opts,
 			 * entries in the index.
 			 */
 
-			add_files_to_cache(0, NULL, NULL);
+			add_files_to_cache(NULL, NULL, 0);
 			work = write_tree_from_memory();
 
 			ret = reset_to_new(new->commit->tree, opts->quiet);
diff --git a/builtin-commit.c b/builtin-commit.c
index fb1e588..d5e8c4c 100644
--- a/builtin-commit.c
+++ b/builtin-commit.c
@@ -242,7 +242,7 @@ static char *prepare_index(int argc, const char **argv, const char *prefix)
 	 */
 	if (all || (also && pathspec && *pathspec)) {
 		int fd = hold_locked_index(&index_lock, 1);
-		add_files_to_cache(0, also ? prefix : NULL, pathspec);
+		add_files_to_cache(also ? prefix : NULL, pathspec, 0);
 		refresh_cache(REFRESH_QUIET);
 		if (write_cache(fd, active_cache, active_nr) ||
 		    close_lock_file(&index_lock))
diff --git a/cache.h b/cache.h
index f16d341..c6c1659 100644
--- a/cache.h
+++ b/cache.h
@@ -748,7 +748,13 @@ extern int convert_to_git(const char *path, const char *src, size_t len,
 extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);
 
 /* add */
-void add_files_to_cache(int verbose, const char *prefix, const char **pathspec);
+#define ADD_FILES_VERBOSE	01
+#define ADD_FILES_IGNORE_ERRORS	02
+/*
+ * return 0 if success, 1 - if addition of a file failed and
+ * ADD_FILES_IGNORE_ERRORS was specified in flags
+ */
+int add_files_to_cache(const char *prefix, const char **pathspec, int flags);
 
 /* diff.c */
 extern int diff_auto_refresh_index;
-- 
1.5.4.3.391.gf5a0c

[PATCH] Add a test for git-add --ignore-errors

From: Alex Riesen <hidden>
Date: 2016-06-15 22:44:19

Signed-off-by: Alex Riesen <redacted>
---
 t/t3700-add.sh |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/t/t3700-add.sh b/t/t3700-add.sh
index 287e058..ca3e33d 100755
--- a/t/t3700-add.sh
+++ b/t/t3700-add.sh
@@ -179,4 +179,13 @@ test_expect_success 'git add --refresh' '
 	test -z "`git diff-index HEAD -- foo`"
 '
 
+test_expect_success 'git add --ignore-errors' '
+	git reset --hard &&
+	date >foo1 &&
+	date >foo2 &&
+	chmod 0 foo2 &&
+	git add --verbose --ignore-errors .
+	git ls-files |grep foo1
+'
+
 test_done
-- 
1.5.4.3.391.gf5a0c

[PATCH] Add --ignore-errors to git-add to allow it to skip files with read errors

From: Alex Riesen <hidden>
Date: 2016-06-15 22:44:19

Signed-off-by: Alex Riesen <redacted>
---
 Documentation/git-add.txt |    7 ++++++-
 builtin-add.c             |   11 +++++++++--
 2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/Documentation/git-add.txt b/Documentation/git-add.txt
index 4779909..9360a4f 100644
--- a/Documentation/git-add.txt
+++ b/Documentation/git-add.txt
@@ -9,7 +9,7 @@ SYNOPSIS
 --------
 [verse]
 'git-add' [-n] [-v] [-f] [--interactive | -i] [--patch | -p] [-u] [--refresh]
-          [--] <filepattern>...
+	  [--ignore-errors] [--] <filepattern>...
 
 DESCRIPTION
 -----------
@@ -81,6 +81,11 @@ OPTIONS
 	Don't add the file(s), but only refresh their stat()
 	information in the index.
 
+\--ignore-errors::
+	If some files could not be added because of errors indexing
+	them, do not abort the operation, but continue adding the
+	others. The command shall still exit with non-zero status.
+
 \--::
 	This option can be used to separate command-line options from
 	the list of files, (useful when filenames might be mistaken
diff --git a/builtin-add.c b/builtin-add.c
index bc55a0e..b67ad3f 100644
--- a/builtin-add.c
+++ b/builtin-add.c
@@ -191,6 +191,7 @@ static const char ignore_error[] =
 "The following paths are ignored by one of your .gitignore files:\n";
 
 static int verbose = 0, show_only = 0, ignored_too = 0, refresh_only = 0;
+static int ignore_add_errors;
 
 static struct option builtin_add_options[] = {
 	OPT__DRY_RUN(&show_only),
@@ -201,6 +202,7 @@ static struct option builtin_add_options[] = {
 	OPT_BOOLEAN('f', NULL, &ignored_too, "allow adding otherwise ignored files"),
 	OPT_BOOLEAN('u', NULL, &take_worktree_changes, "update tracked files"),
 	OPT_BOOLEAN( 0 , "refresh", &refresh_only, "don't add, only refresh the index"),
+	OPT_BOOLEAN( 0 , "ignore-errors", &ignore_add_errors, "just skip files which cannot be added because of errors"),
 	OPT_END(),
 };
 
@@ -231,6 +233,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 
 		if (verbose)
 			flags |= ADD_FILES_VERBOSE;
+		if (ignore_add_errors)
+			flags |= ADD_FILES_IGNORE_ERRORS;
 
 		exit_status = add_files_to_cache(prefix, pathspec, flags);
 		goto finish;
@@ -286,8 +290,11 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 	}
 
 	for (i = 0; i < dir.nr; i++)
-		if (add_file_to_cache(dir.entries[i]->name, verbose))
-			exit(1);
+		if (add_file_to_cache(dir.entries[i]->name, verbose)) {
+			if (!ignore_add_errors)
+				exit(1);
+			exit_status = 1;
+		}
 
  finish:
 	if (active_cache_changed) {
-- 
1.5.4.3.391.gf5a0c

Re: [PATCH] Make the exit code of add_file_to_index actually useful

From: Johannes Schindelin <hidden>
Date: 2016-06-15 22:44:19

Hi,

On Sun, 2 Mar 2008, Alex Riesen wrote:
-			add_file_to_cache(path, verbose);
+			if (add_file_to_cache(path, verbose))
+				exit(1);
Does it really, really _have_ to be exit(1)?  I mean, now you block even 
the faintest chance that we can libify libgit.a by overriding die_routine.

A "return -1" might make _much_ more sense, too.

Ciao,
Dscho

Re: [PATCH] Make the exit code of add_file_to_index actually useful

From: Daniel Barkalow <hidden>
Date: 2016-06-15 22:44:19

On Sun, 2 Mar 2008, Johannes Schindelin wrote:
Hi,

On Sun, 2 Mar 2008, Alex Riesen wrote:
quoted
-			add_file_to_cache(path, verbose);
+			if (add_file_to_cache(path, verbose))
+				exit(1);
Does it really, really _have_ to be exit(1)?  I mean, now you block even 
the faintest chance that we can libify libgit.a by overriding die_routine.
It would be handy to have a die_no_message(), for cases like this where a 
function wants to print an error message but it's up to the caller whether 
to abort (in the contextually reasonable way).

	-Daniel
*This .sig left intentionally blank*

Re: How to "git add ." when some files are not accessible (permission denied)?

From: Dirk Süsserott <hidden>
Date: 2016-06-15 22:44:19

Jeff King schrieb:
On Sat, Mar 01, 2008 at 02:46:28PM +0100, Dirk Süsserott wrote:

  
quoted
The question is: is there a way to tell "git add ." to add all files but
ignore those that cannot be added due to lack of authorization?
    
No, there isn't. Under Linux, I would come up with a list of files I was
interested in and then pipe it to "xargs git-add", which is probably
unhelpful for Windows.

  
Not quite. I'm using the msysGit package from 
http://code.google.com/p/msysgit/downloads/list and that comes with some 
fundamental unix tools like a sound shell, find, xargs, and many more. 
Very good!
This way prepared, I used "git ls-files -o | xargs git add -v" until 
most of my files were added.
For the rest I did "xargs -l" (ell) so that the files got added one by one.
The files that still refused to be added are finally ignored by "git 
ls-files -o >> .gitignore".

Caveat: filenames containing blanks are not handled properly as they are 
not surrounded by quotes. "git add" thinks of them as two or more files 
and fails.
I figure xargs has some cool switches to sourround the parameters with 
quotes, but I didn't find them. An option was to write a script or shell 
function that does it and pipe the filenames through that function or -- 
as filenames with blanks aren't so numerous -- to add them manually with 
"git gui".

Eventually, I solved the problem. Thanks for and to your pointers. :-)

  -- Dirk
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help