Re: [RFC PATCH] Re: Empty directories...

Jakub, this mail is too long already, and it does not make sense to
tack a changed proposal to its end since then the readers will be
exhausted at the time they come there.  So I'll instead tack a
followup to the "big picture" mail instead where I outline a modified
approach which is presumably easier to understand and completely
backwards-compatible, incorporating your feedback.

There is probably little sense in wasting your time on a detailed
response: feel free to point out where you don't see myself making
sense.  I have no problem with people coming to different conclusions
that I do, but I would prefer it if it is not because they consider
myself a raving lunatic, but because they have different opinions
regarding the details.

"I can follow you, but I disagree with your conclusion" is perfectly
fine for now since I am going to propose something else, anyway.

Thanks for the feedback.  It gave me some good ideas.

Jakub Narebski [off-list ref] writes:

On Sun, 22 July 2007, David Kastrup wrote:

quoted

Jakub Narebski [off-list ref] writes:

quoted

David Kastrup wrote:

quoted

I must be really bad at explaining things, or I am losing a fight
against preconceptions fixed beyond my imagination.

Or you are wrong...

Well, there is little reason for you to take my word on it, but I
happen to have a history of designing and implementing systems where I
have been responsible for every single byte, bootloader, firmware,
applications, target compiler, assembler, whatever.  I have been
exposed to Unix and working with it several years before Linux even
existed.  I also have a track record of being not exactly stupid.

So I pretty much can rule out that I am wrong on the factual side.

But where I may be wrong is in estimating the how obvious the design
can appear to others, and how useful and maintainable for others it
may be in the long run.  Linus says "code talks", but that's actually
not half the story.  If my code says that it works and the evidence is
there, but nobody is able to understand _why_ it works, it has no
place in a project where I am not permanently around.

If smart people don't get what I am talking about, it does not matter
that the patch is surprisingly well-contained: it will be a
maintenance nightmare because people will never figure out why
something stopped working after some particular change.

quoted

I disagree here.  The object database _can_ represent an _empty_
directory that has been added explicitly, because up to now no
operations existed that actually left an empty tree.  But it can't
distinguish a _non_-empty directory that has been added explicitly
from non-empty directory that has not been added explicitly.

True. I forgot about that.

Thanks.  It is almost a revelation that anybody can agree on any point
with me at the moment.

IMHO it would be best to first provide plumbing infrastructure (as
e.g.  it was the case of submodule support), then add option to
git-update-index to change the "stickiness"/"autoremoval" status of
a directory (of a tree), and _last_ think about how to change the
porcelain (git-add and git-rm).

Sure.  It does no harm to think about reducing the amount of breaking
porcelain, though.

[...]

quoted

And a perfectly consistent way is to make those trees with an
explicitly added directory _non-empty_, by virtue of putting a file
"." in them.  This file, of course, exists in every physical
directory, but we may or may not decide to let it be tracked by
git, using the gitignore mechanism on the pattern ".".  Perfectly
expedient.

Here we disagree. I think putting "." in a tree as marker of having
it not be automatically deleted when empty, as opposed to marking
tree using filemode in the parent, is not a good idea.

Well, "not a good idea" is a far step forward from "stupid idiot
babbling nonsense", so we may make progress towards actually being
able to _weigh_ different options.  I can actually associate with "not
a good idea", not least because nobody else seems to get the idea, and
that makes it infeasible for maintenance.

So I'll address some points and then propose a different way of
implementing what will in the end amount to rather similar semantics,
but with a different view of looking at those semantics, one that
corresponds well with the implementation.

The only advantage to the "." idea is that it can use gitignore
mechanism (both in-tree .gitignore, tracked or not, and info/exclude
file). But I also think that the fact that gitignore mechanism is
recursive is more of disadvantage than advantage.

First, it is _not_ consistent. Working directory trees _always_ have
'.'  in them, while trees would have or would have not it, depending
if they would be "sticky" or "autoremoved".

Let me point out again that this inconsistency is already present in
the difference of tracked and untracked _files_: they are always in
the working directory, while trees have or not have them, depending on
whether they are "registered" or "not".

There is no inconsistency involved here, but it seems to make people
_very_ uncomfortable to factor out the "stays around even if empty"
functionality and call it "dir/." from the "can hold content"
functionality which is in effect called "dir/", and basically
associate tracked physical existence just with the former.

The recursiveness of the gitignore mechanism has the advantage that
when maintaining a large repository with actual or logical
subprojects, one does not need to pick a single policy for all
subprojects.  I think that is quite important.  It could possibly be
achieved with some other method of having per-subproject
configuration, but I see little wrong in using what is there and
documented already.

Second, the "easy implementation" is anything but easy. "git add ."
as a way to mark directory as "sticky" is not backward compatibile:
currently it mean to add _all contents_ of current directory.
Implementation is tricky: as we have seen trying to unlink '.' or
create '.' can unfortunately succeed on [some Sun OS, and UFS
filesystem] (which follows POSIX stupidly to the letter) f**king up
the filesystem.

I was not suggesting actually leaving any such calls in place: after
all, they would presumably lead to error messages.  But I agree that
this could lead to nasty surprises when somebody with a legacy version
of git worked with a repository containing "." as explicit entries of
some file type.

The alternative proposal of adding "magic mode" to mark directory as
"not remove when empty" is largely tested; it is very similar to the
subproject support.

Good.  Because it is what I converged to last night.

Third, is contrary to the git philosophy of tracking contents.
"Stickiness" is an attribute; the fact that directory is explicitely
tracked or not does not change contents of a directory. Compare to
'blob' which contains only contents of a file: not a filename, not a
pathname, not [subset of] filemode.

Fourth, is very artificial. What would you put for filemode for '.'?
040000 (i.e. directory)?

Taken already.  By something very artificial, namely a tree...  Yes,
this was a wart in my proposal.

What would you put for sha1?  Sha1 of an empty directory?

Some fixed value.  Everywhere the same.  Not really relevant.

quoted

That basically implies that no information about directories could
be tracked in the repository.  And yes, we need appropriate
information in the index.  Again, the information whether a
directory was added explicitly.

Whether directory is automatically managed by git (automatically
removed or untracked). But we need directory entry in index for
git-diff, for example to recognize if there is or there is not empty
directory, or if a directory is automanaged or not.

One conclusion that I have come to (and I think I am in agreement with
Linus here) is that the information "empty or not" is actually useless
separately: when I add files below a directory to the repository, the
directory _can't_ be empty.  And git has no way of knowing whether it
is non-empty because I wanted the directory to be there, or whether it
is non-empty because I could not have checked in the files into the
tree below it otherwise.

quoted

And the repository is a versioned and hierarchically hashed version
of the index, but its trees contain _no_ information that is not
already inherently represented by the files alone. [...]

The above sentence is nonsensical. Index is helper for repository,
and can be derived from repository. Not vice versa.

Trees do contain information which is not inherently present by the 
blobs.

Could you give examples for such information?  As long as we are not
talking about _history_, I am at a loss at what else you mean.  File
names and permissions?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum