Worktree silently deleted on git fetch/gc/log

Hello,

with git 2.16.2 I have two repositories:

A (branch master) and
B (branch g) which is a worktree of the first.

This has always forked fine until yesterday, when I did:

/git/B g>$ git fetch
remote: Counting objects: 29, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 29 (delta 19), reused 25 (delta 19), pack-reused 4
Unpacking objects: 100% (29/29), done.
 From https://...
    13e4c55a0..02655d5fb  g -> origin/g
    c37a3ca25..bc7888511  master -> origin/master
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
/git/B g<>$ git log -p origin/g
fatal: Not a git repository: /git/A/.git/worktrees/B
/git/B$

Please note that on the second last prompt there is <>, so that git-prompt has found the neccessary information and this was this was later deleted - by 'gc' or 'log'.

I guess somebody who knows the code in detail can easily fix this.

What would be the procedure to restore the /git/A/.git/worktrees/B structure?

Kind regards
   Dilyan

On Wed, Feb 28, 2018 at 7:44 AM, Дилян Палаузов
[off-list ref] wrote:

A (branch master) and
B (branch g) which is a worktree of the first.

/git/B g>$ git fetch
[...]
From https://...
   13e4c55a0..02655d5fb  g -> origin/g
   c37a3ca25..bc7888511  master -> origin/master
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
/git/B g<>$ git log -p origin/g
fatal: Not a git repository: /git/A/.git/worktrees/B
/git/B$

Please note that on the second last prompt there is <>, so that git-prompt
has found the neccessary information and this was this was later deleted -
by 'gc' or 'log'.

What would be the procedure to restore the /git/A/.git/worktrees/B
structure?

Can you show us (via 'cat') the content of the following files?

/git/B/.git
/git/A/.git/worktrees/b/HEAD
/git/A/.git/worktrees/b/commondir
/git/A/.git/worktrees/b/gitdir

Hello,

/git/A/.git/worktrees/b/ is missing - that is the point.

/git/B/,git wasn't modified since the worktree was created, cat:
gitdir: /git/A/.git/worktrees/b

Regards
   Дилян



On 03/01/18 19:09, Eric Sunshine wrote:

On Wed, Feb 28, 2018 at 7:44 AM, Дилян Палаузов
[off-list ref] wrote:

quoted

A (branch master) and
B (branch g) which is a worktree of the first.

/git/B g>$ git fetch
[...]
 From https://...
    13e4c55a0..02655d5fb  g -> origin/g
    c37a3ca25..bc7888511  master -> origin/master
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
/git/B g<>$ git log -p origin/g
fatal: Not a git repository: /git/A/.git/worktrees/B
/git/B$

Please note that on the second last prompt there is <>, so that git-prompt
has found the neccessary information and this was later deleted -
by 'gc' or 'log'.

What would be the procedure to restore the /git/A/.git/worktrees/B
structure?

Can you show us (via 'cat') the content of the following files?

/git/B/.git
/git/A/.git/worktrees/b/HEAD
/git/A/.git/worktrees/b/commondir
/git/A/.git/worktrees/b/gitdir

On Thu, Mar 1, 2018 at 2:24 PM, Дилян Палаузов
[off-list ref] wrote:

/git/A/.git/worktrees/b/ is missing - that is the point.
/git/B/,git wasn't modified since the worktree was created, cat:
gitdir: /git/A/.git/worktrees/b

I'll assume that the lowercase 'b' was a typo in your email (as it was
in mine), and that the real content is "gitdir:
/git/A/.git/worktrees/B".

You should be able to get back to a working state like this:

% cd /git/A/
% mkdir -p .git/worktrees/B
% echo 'ref: refs/heads/g' >.git/worktrees/B/HEAD
% echo ../.. >.git/worktrees/B/commondir
% echo /git/B/.git >.git/worktrees/B/gitdir

% cd /git/B/
% git reset

As far as I know, there isn't any code in Git which would
automatically remove the .git/worktrees/B directory, so it's not clear
how that happened.

On Fri, Mar 2, 2018 at 3:14 AM, Eric Sunshine [off-list ref] wrote:

As far as I know, there isn't any code in Git which would
automatically remove the .git/worktrees/B directory, so it's not clear
how that happened.

"git worktree prune" does, which is called by "git gc" and that was
actually executed from the command output in the original mail.

Дилян Палаузов did you move /git/B away manually? For example, you
originally created "B" with

git worktree add /somewhere/B g

then you do this at some point

mv /somewhere/B /git/B
-- 
Duy

Hello Duy,

thanks for your answer.

Your assumption is correct: when renaming the directory apparently I have not adjusted /git/A/.git/worktrees/B/gitdir to the new location.

I fixed the situation by renaming /git/B to /git/B.bak, creating a new worktree pointing to /git/B (with branch g), then deleting /git/B and moving /git/B.bak to /git/B .

Consider adjusting the documentation as suggested below, possibly using a different wording.

Regards
   Дилян

On 03/02/18 00:56, Duy Nguyen wrote:

On Fri, Mar 2, 2018 at 3:14 AM, Eric Sunshine [off-list ref] wrote:

quoted

As far as I know, there isn't any code in Git which would
automatically remove the .git/worktrees/B directory, so it's not clear
how that happened.

"git worktree prune" does, which is called by "git gc" and that was
actually executed from the command output in the original mail.

Дилян Палаузов did you move /git/B away manually? For example, you
originally created "B" with

git worktree add /somewhere/B g

then you do this at some point

mv /somewhere/B /git/B

Signed-off-by Дилян Палаузов [off-list ref]
---

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index 571b5a7..33655c4 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt

@@ -15,8 +15,8 @@ DESCRIPTION
  -----------
  Runs a number of housekeeping tasks within the current repository,
  such as compressing file revisions (to reduce disk space and increase
-performance) and removing unreachable objects which may have been
-created from prior invocations of 'git add'.
+performance), removing unreachable objects, which may have been
+created from prior invocations of 'git add', and calling 'git worktree prune'.
  
  Users are encouraged to run this task on a regular basis within
  each repository to maintain good disk space utilization and good

diff --git a/Documentation/git-worktree.txt b/Documentation/git-worktree.txt
index 5ac3f68..4b0d32f 100644
--- a/Documentation/git-worktree.txt
+++ b/Documentation/git-worktree.txt

@@ -82,7 +82,7 @@ with `--reason`.
  
  prune::
  
-Prune working tree information in $GIT_DIR/worktrees.
+Prune working tree information in $GIT_DIR/worktrees.  Called implicitly by 'git gc'.
  
  unlock::
  
--

2.10.0.298.gc59cecb

On Fri, Mar 2, 2018 at 6:34 AM, Дилян Палаузов
[off-list ref] wrote:

Your assumption is correct: when renaming the directory apparently I have
not adjusted /git/A/.git/worktrees/B/gitdir to the new location.

I fixed the situation by renaming /git/B to /git/B.bak, creating a new
worktree pointing to /git/B (with branch g), then deleting /git/B and moving
/git/B.bak to /git/B .

Duy's patch series to add a "git worktree move" command[1] should help
avoid such problems in the future.

[1]: https://public-inbox.org/git/20180124095357.19645-1-pclouds@gmail.com/

On Fri, Mar 2, 2018 at 6:34 PM, Дилян Палаузов
[off-list ref] wrote:

Hello Duy,

thanks for your answer.

Your assumption is correct: when renaming the directory apparently I have
not adjusted /git/A/.git/worktrees/B/gitdir to the new location.

I fixed the situation by renaming /git/B to /git/B.bak, creating a new
worktree pointing to /git/B (with branch g), then deleting /git/B and moving
/git/B.bak to /git/B .

Phew.. I was worried there was some serious bug that I missed even
after the last code inspection.

The good news is "git worktree move" command is coming so you don't
have to do this manual (and risky) operation.

I'm going to improve it a bit in this case either way, I think I have
some idea: (mostly to Eric) since worktree B is alive and kicking, it
should keep at least HEAD or index updated often. We can delay
deleting a worktree until both of these are past expire time.

Also Eric (and this is getting off topic), I wonder if the decision to
store absolute path in worktrees/../gitdir and .git files is a bad one
(and is my bad).

If we stored relative path in ".git" file at least, the worktree would
immediately fail after the user moves either the linked checkout, or
the main one. This should send a loud and clear signal to the user
"something has gone horribly" and hopefully make them connect it to
the last rename and undo that. "git gc" would have near zero chance to
kick in and destroy stale worktrees.

Another bad thing about absolute paths, if you backup both main and
linked worktrees in a tar file, restoring from the backup won't work
unless you restore at the exact same place.

Hmm?

Consider adjusting the documentation as suggested below, possibly using a
different wording.

Will do.

-- 
Duy

On Fri, Mar 2, 2018 at 6:53 AM, Duy Nguyen [off-list ref] wrote:

I'm going to improve it a bit in this case either way, I think I have
some idea: (mostly to Eric) since worktree B is alive and kicking, it
should keep at least HEAD or index updated often. We can delay
deleting a worktree until both of these are past expire time.

Make sense.

Also Eric (and this is getting off topic), I wonder if the decision to
store absolute path in worktrees/../gitdir and .git files is a bad one
(and is my bad).

Your original implementation of worktrees was able to recover
automatically when a worktree was moved (via 'mv', for instance). That
feature pretty much required recording an absolute path in the
worktree's .git file. When the automatic-recovery feature was dropped,
I suppose the need for absolute path in the .git file disappeared, as
well.

I can't presently think of a reason why gitdir needed/used an absolute
or normalized path. Was it because there was some need to compare such
paths?

If we stored relative path in ".git" file at least, the worktree would
immediately fail after the user moves either the linked checkout, or
the main one. This should send a loud and clear signal to the user
"something has gone horribly" and hopefully make them connect it to
the last rename and undo that. "git gc" would have near zero chance to
kick in and destroy stale worktrees.

It would detect if the main or linked worktree moved up or down one or
more directory levels or elsewhere.

It would not detect if the worktree directory was merely renamed.

Still, detecting some cases of breakage early may be better than not
detecting breakage at all.

Another idea may be to store the worktree's own normalized/absolute
path as a second line in its .git file. It could then immediately
detect any manual movement or renaming of itself. A minor concern,
though, is if there are any external tools reading that file directly
since they could be confused by the second line. Of course, such tools
hopefully would be using "git rev-parse --git-dir", so maybe it
wouldn't be a big deal. On the other hand, older versions of Git
itself would be confused by the second line, so perhaps the idea isn't
viable.

Another bad thing about absolute paths, if you backup both main and
linked worktrees in a tar file, restoring from the backup won't work
unless you restore at the exact same place.
Hmm?

Yep, that's an issue. An even simpler case is the somewhat common
usage pattern of creating worktrees within the main worktree (Dscho
has a number of times mentioned working like this, and I've done it
myself). In such a case, even just moving or renaming the main
worktree breaks the linked ones when absolute paths are in play.

So, yes, these issue argue for use of relative paths.

On Sat, Mar 3, 2018 at 5:05 PM, Eric Sunshine [off-list ref] wrote:

I can't presently think of a reason why gitdir needed/used an absolute
or normalized path. Was it because there was some need to compare such
paths?

No, we need to re-normalize paths before comparing anyway to be safe.
I don't think I had any special reason for using absolute path either,
except that dealing with it is a bit easier than relative one.

quoted

If we stored relative path in ".git" file at least, the worktree would
immediately fail after the user moves either the linked checkout, or
the main one. This should send a loud and clear signal to the user
"something has gone horribly" and hopefully make them connect it to
the last rename and undo that. "git gc" would have near zero chance to
kick in and destroy stale worktrees.

It would detect if the main or linked worktree moved up or down one or
more directory levels or elsewhere.

It would not detect if the worktree directory was merely renamed.

True. We can be a bit more aggressive and check it anyway at command
startup (once we can peek into the main .git dir, we know where this
linked worktree is supposed to be). Or perhaps we can do this in "gc
--auto" (even though gc may end up not doing any maintenance job).
This command is usually called on heavy commands, adding one more
check should not hurt.

Still, detecting some cases of breakage early may be better than not
detecting breakage at all.

Another idea may be to store the worktree's own normalized/absolute
path as a second line in its .git file. It could then immediately
detect any manual movement or renaming of itself. A minor concern,
though, is if there are any external tools reading that file directly
since they could be confused by the second line. Of course, such tools
hopefully would be using "git rev-parse --git-dir", so maybe it
wouldn't be a big deal. On the other hand, older versions of Git
itself would be confused by the second line, so perhaps the idea isn't
viable.

Yeah I'm a bit hesitant to break that file format. Many tools ignore
our programs and peek inside anyway, I think magit is one of them.
-- 
Duy