Re: [RFC PATCH] Re: Empty directories...

Linus Torvalds [off-list ref] writes:

On Thu, 19 Jul 2007, David Kastrup wrote:

quoted

Well, kudos.  Together with the analysis from Junio, this seems like a
good start.  Would you have any recommendations about what stuff one
should really read in order to get up to scratch about git internals?

Well, you do need to understand the index. That's where all the new 
subtlety happens.

The data structures themselves are trivial, and we've supported empty 
trees (at the top level) from the beginning, so that part is not anything 
new.

However, now having a new entry type in the index (S_IFDIR) means that 
anything that interacts with the index needs to think twice. But a lot of 
that is just testing what happens, and so the first thing to do is to have 
a test-suite.

There's also the question about how to show an empty tree in a diff. We've 
never had that: the only time we had empty trees was when we compared a 
totally empty "root" tree against another tree, and then it was obvious. 
But what if the empty tree is a subdirectory of another tree - how do you 
express that in a diff? Do you care? Right now, since we always recurse 
into the tree (and then not find anything), empty trees will simply not 
show up _at_all_ in any diffs.

And what about usability issues elsewhere? With my patch, doing something 
like a

	git add directory/

still won't do anything, because the behaviour of "git add" has always 
been to recurse into directories. So to add a new empty directory, you'd 
have to do

	git update-index --add directory

and that's not exactly user-friendly.

So do you add a "-n" flag to "git add" to tell it to not recurse? Or do 
you always recurse, but then if you notice that the end result is empty, 
you add it as a directory?

Another issue I thought about was what you would do in the step
3 in the following:

 1. David says "mkdir D; git add D"; you add S_IFDIR entry in
    the index at D;

 2. David says "date >D/F; git add D/F"; presumably you drop D
    from the index (to keep the index more backward compatible)
    and add S_IFREG entry at D/F.

 3. David says "git rm D/F".

Have we stopped keeping track of the "empty directory" at this
point?

Junio C Hamano [off-list ref] wrote:

Another issue I thought about was what you would do in the step
3 in the following:

 1. David says "mkdir D; git add D"; you add S_IFDIR entry in
    the index at D;

 2. David says "date >D/F; git add D/F"; presumably you drop D
    from the index (to keep the index more backward compatible)
    and add S_IFREG entry at D/F.

 3. David says "git rm D/F".

Have we stopped keeping track of the "empty directory" at this
point?

Sadly yes.  But I don't think that's what the folks who want to
track empty directories want to have happen here.

Which is why I'm thinking we just need to track the directory, as a
node in the index, even if there are files in it, and even if we got
that directory and its contained files there by just unpacking trees.

-- 
Shawn.

Junio C Hamano [off-list ref] writes:

Another issue I thought about was what you would do in the step
3 in the following:

 1. David says "mkdir D; git add D"; you add S_IFDIR entry in
    the index at D;

 2. David says "date >D/F; git add D/F"; presumably you drop D
    from the index (to keep the index more backward compatible)
    and add S_IFREG entry at D/F.

I don't think that one should drop D here.  Operation 1 _is_ not
backward compatible, so if you want to revert it, you should
explicitly remove D.  And we can't "keep" the index backward
compatible if it isn't so after step 1.

 3. David says "git rm D/F".

Have we stopped keeping track of the "empty directory" at this
point?

The case I am worrying about is rather

mkdir D
mkdir D/E
touch D/E/file
git add D
[*]
git rm D/E/file

From a user perspective, E should be registered still.  Compare this

with

mkdir D
mkdir D/E
touch D/E/file
git add D/E/file
[*]
git rm D/E/file

Where likely both D and E should now be considered unregistered.  So
the situation is different between the first or the second [*], and
the difference might be impossible to express completely in the frame
of a backwards-compatible index, even though we don't track an empty
directory at the point [*] at all, and the only registered _file_ is
D/E/file.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

"Shawn O. Pearce" [off-list ref] writes:

Sadly yes.  But I don't think that's what the folks who want to
track empty directories want to have happen here.

Which is why I'm thinking we just need to track the directory, as a
node in the index, even if there are files in it, and even if we got
that directory and its contained files there by just unpacking
trees.

I have come to about the same conclusion.  So if
backward-compatibility is any concern, one needs to work with some
sort of extension records, and designing them in a way that

new-git add tree
old-git rm tree

will not leave empty subdirectories in the index will be tricky, to
say the least.  One will likely have to add an extension record
"directory" for each directory as well as "my containing dir takes
care of itself" to each file that has been added with new-git and has
had its parent directory entered by other means.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

"Shawn O. Pearce" [off-list ref] wrote:

Junio C Hamano [off-list ref] wrote:

quoted

Another issue I thought about was what you would do in the step
3 in the following:

 1. David says "mkdir D; git add D"; you add S_IFDIR entry in
    the index at D;

 2. David says "date >D/F; git add D/F"; presumably you drop D
    from the index (to keep the index more backward compatible)
    and add S_IFREG entry at D/F.

 3. David says "git rm D/F".

Have we stopped keeping track of the "empty directory" at this
point?

Sadly yes.  But I don't think that's what the folks who want to
track empty directories want to have happen here.

Which is why I'm thinking we just need to track the directory, as a
node in the index, even if there are files in it, and even if we got
that directory and its contained files there by just unpacking trees.

I take this back.  I really don't want that behavior.

If I do:

  mkdir -p foo/bar
  echo hello >foo/bar/world
  git add foo
  git -f rm foo/bar/world

I never asked for foo/bar or foo to stay.  In fact I want them
to disappear from Git entirely, as foo/bar is now empty and has
no content.


But we also cannot do a special --mkdir option for update-index
either, because how do we know that the user designated subtree is
a directory we must always keep in the index?

So I think the only way this works is to have a new mode that we use
in tree (04755 ?) that tells us not only is this thing a subtree,
but also that the user wants it to stay here, even if it is empty.
Those trees are always in the index as a real tree entry, even if
there are files contained in it.

And as far as getting that directory entry created/removed from
the index, well, I think a special flag to update-index would be
in order, much like --chmod=[+-]x.

Just my $0.0002 USD, which really ain't worth much at all.

-- 
Shawn.

Dear gits,

When I first started using git, I naively did

           $ mkdir NEWDIR && chmod BLAH NEWDIR
           $ git add NEWDIR

I just expected that this was content in the current directory that I
wanted tracked
together with the permissions.

It wasn't ... I spent a day or 2 thinking I was stupid, my version of git was
corrupt, my machine was busted, .... etc.  Eventually of course, I read the
documentation (when all else fails) and realised that this perfectly obvious
behaviour was not supported.  The behaviour was obviously so obvious
that eventually
an error message was added telling all the people who hadn't
read the documentation that trying to add a directory was 'fatal'.

I put up with and work around this behaviour because git is so bloody
brilliant at everything else.  But it would be nice if it worked.

Cheers,
Geoff Russell

David Kastrup [off-list ref] writes:

Junio C Hamano [off-list ref] writes:

quoted

Another issue I thought about was what you would do in the step
3 in the following:

 1. David says "mkdir D; git add D"; you add S_IFDIR entry in
    the index at D;

 2. David says "date >D/F; git add D/F"; presumably you drop D
    from the index (to keep the index more backward compatible)
    and add S_IFREG entry at D/F.

I don't think that one should drop D here.  Operation 1 _is_ not
backward compatible, so if you want to revert it, you should
explicitly remove D.  And we can't "keep" the index backward
compatible if it isn't so after step 1.

quoted

 3. David says "git rm D/F".

Have we stopped keeping track of the "empty directory" at this
point?

The case I am worrying about is rather

mkdir D
mkdir D/E
touch D/E/file
git add D
[*]
git rm D/E/file

From a user perspective, E should be registered still.  Compare this
with

mkdir D
mkdir D/E
touch D/E/file
git add D/E/file
[*]
git rm D/E/file

Let's take this through the motions with my last proposal: at the
first [*], the index now contains

D/.        [dir]
D/E/.      [dir]
D/E/file   [file]

After git rm D/E/file, it contains

D/.        [dir]
D/E/.      [dir]

Compared with the second, where we just have in the index

D/E/file   [file]

and it is gone again after the remove.

After commiting in the first case, we have in the repository
D          [tree]
D/.        [dir]
D/E        [tree]
D/E/.      [dir]
D/E/file   [file]

Now we do
git rm D/E, and the index contains

D/E/.      [remove dir]
D/E/file   [remove file]

If we commit now,
D/E        [tree]
becomes empty and is removed.  All that stays is

D          [tree]
D/.        [dir]

So we still have [tree] items only in the repository, not in the
index, and there is no such thing as an empty tree.  But directories
have a presence in index and repository.  They are not containers of
files, that role is retained by trees.  Rather they are siblings of
the files in their associated tree.

As a note aside: if one wanted to track directory permissions, one
would track them in the [dir] entries, not in the [tree] entries.
Trees remain abstract structuring entities in the repository that
don't have an outside representation.  Directories will be
auto-created and deleted as necessary in the work directory to
facilitate having a place for checking tree elements out and in.

This means that
git add D/E/file
would _not_ track permissions of D and E (nor their existence).

However, Linus is right that permissions are something to be discussed
separately.  But separating [tree] and [dir] makes for a plausible and
understandable way of treating them.

-- 
David Kastrup