Re: [RFC PATCH] Re: Empty directories...

Brian Gernhardt [off-list ref] writes:

My apologies for the wasted bandwidth arguing for things that had
already been decided.

Sorry, who decided what?

On Thu, 19 Jul 2007, Junio C Hamano wrote:

Brian Gernhardt [off-list ref] writes:

quoted

My apologies for the wasted bandwidth arguing for things that had
already been decided.

Sorry, who decided what?

I think people who didn't know how the world works decided that 
directories that were added manually as directories would stay as 
directories even after the last file was removed.

That's physically impossible with the git data-structures (since there is 
no way of saving "this directory was added empty" in the tree structures, 
nor any point to it), so I think it's just insane rambling.

I dunno. I think empty directories are worth supporting, mainly to be able 
to capture other SCM's notion of what _they_ track, but quite frankly, the 
level of discussion about them hasn't been exactly inspiring. It seems to 
be more about "this is what we'd like to see, without really having a 
reason for it, nor necessarily understanding what we're talking about" 
than "this is realistic and useful and here are patches".

I *do* think that it's a very valid argument that if you import something 
from SVN that has an empty directory, the git import should show that.  

That's about the only valid argument I've ever seen for them, though, and 
I think that's totally irrelevant to such issues as to whether "git rm 
file/in/directory" should remove the directory(*) from being tracked by 
git when the file goes away or not.

			Linus
 
(*) And, for anybody confused about the issue, the answer to the latter 
question is an emphatic: "Yes it should, live with it, and if you want the 
directory back, you had better add it back as an empty directory"

On Thu, 19 Jul 2007, Linus Torvalds wrote:

That's physically impossible with the git data-structures (since there is 
no way of saving "this directory was added empty" in the tree structures, 
nor any point to it), so I think it's just insane rambling.

Of course, it's physically *possible* to have a tree that contains two 
entries for the same name: first the "empty tree" and then the "real 
tree", and yeah, in theory you could track things that way.

So I guess the "physically impossible" was a bit strong. You'd have to 
have a totally insane format, and you'd have to violate deeply seated 
rules about what trees look like (and the index too, for that matter: we'd 
have to do the same for the index, and keep the S_IFDIR entry alive 
despite having other entries that are children of it), but it's 
*possible*.

It's just a really bad idea.

So to be sane, when you add files, the empty directory entry has to go 
away. Otherwise you could have two very different trees that encode the 
same *content* (just with different ways of getting there - depending on 
whether you have a history with empty trees or not), and that's very much 
against the philosophy of git, and breaks some fundamental rules (like the 
fact that "same content == same SHA1").

In fact, that may be the best way to explain why it's *not* an option to 
have "empty trees remain empty trees if we remove the last file from 
them": git fundamnetally tracks "content snapshots", and anything that 
implies the content containing any history is against the rules.

So the whole notion of "remembering" whether a directory was added 
explicitly as an empty directory or not is just not a sensible concept in 
git. 

		Linus

Linus Torvalds [off-list ref] writes:

On Thu, 19 Jul 2007, Linus Torvalds wrote:

quoted

That's physically impossible with the git data-structures (since
there is no way of saving "this directory was added empty" in the
tree structures, nor any point to it), so I think it's just insane
rambling.

Of course, it's physically *possible* to have a tree that contains
two entries for the same name: first the "empty tree" and then the
"real tree", and yeah, in theory you could track things that way.

So I guess the "physically impossible" was a bit strong. You'd have
to have a totally insane format, and you'd have to violate deeply
seated rules about what trees look like (and the index too, for that
matter: we'd have to do the same for the index, and keep the S_IFDIR
entry alive despite having other entries that are children of it),
but it's *possible*.

Excuse me?  You don't need a "totally insane format".  You need an
entry "." of a new type "directory" that can be part of the current
concept of a "tree".  This new type does _not_ have children.  It is
not a container for files.  It would be the thing that would carry
permissions or other properties if git were to store them.  It can be
put into .gitignore files like other files.

One drawback is that adding and removing it alone is not supported
with the current git-add and git-remove commands: they would require
an additional argument "-d" like "ls" does.

All of this is a straightforward extension fitting very well the
current paradigms and also existing file systems and their usage.

It's just a really bad idea.

So to be sane, when you add files, the empty directory entry has to
go away.

You really have not followed the discussion at all.  This is not
possible since otherwise you could not distinguish the cases

mkdir A
touch A/B
git-add A
git-rm A/B

where A was added and not removed and should stay and

mkdir A
touch A/B
git-add A/B
git-rm A/B

where a single file was added and removed and nothing should stay.

Otherwise you could have two very different trees that encode the
same *content* (just with different ways of getting there -
depending on whether you have a history with empty trees or not),
and that's very much against the philosophy of git, and breaks some
fundamental rules (like the fact that "same content == same SHA1").

No, the content is _different_.  One tree contains a tracked
directory, the other does not.  That means that the trees behave
_differently_ when you manipulate them, and that means that they are
_not_ the same tree.

In fact, that may be the best way to explain why it's *not* an
option to have "empty trees remain empty trees if we remove the last
file from them": git fundamnetally tracks "content snapshots", and
anything that implies the content containing any history is against
the rules.

So the whole notion of "remembering" whether a directory was added
explicitly as an empty directory or not is just not a sensible
concept in git.

Certainly.  That is why we instead remember whether or not a directory
entry "." was added or not.  It will be added (unless the defaults and
gitignore settings ask "." to be non-tracked) when git adds the
corresponding tree or subtree, and it will get removed when git
removes the corresponding tree or subtree.  Emptiness is not a special
case, and it can't be.  Currently, the main information associated
with "." is "stay around even if tree becomes empty".

Now you can do

    unlink .

in Solaris and have the name "." vanish while the directory still
works as a container by other names.

I don't propose that git be able to track this difference, though, and
I doubt that most file archivers would.

But git can or cannot ignore files, and in a similar way it can or
cannot ignore what a directory has more than being an abstract
container.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

David Kastrup wrote:

quoted

Otherwise you could have two very different trees that encode the
same *content* (just with different ways of getting there -
depending on whether you have a history with empty trees or not),
and that's very much against the philosophy of git, and breaks some
fundamental rules (like the fact that "same content == same SHA1").

No, the content is _different_.  One tree contains a tracked
directory, the other does not.  That means that the trees behave
_differently_ when you manipulate them, and that means that they are
_not_ the same tree.

You are mistaking things.  Like the executable bit on a file is not content, the fact that a directory should be kept despite being empty is also an *attribute* of the directory.  This is meta-data, not actual data (content).  So no matter how elegant tracking the "." entry might be (and I think it is, because it covers a lot of corner cases already), it puts the information at the wrong place.

That's sad, because otherwise it would be really elegant.

cheers
  simon

-- 
Serve - BSD     +++  RENT this banner advert  +++    ASCII Ribbon   /"\
Work - Mac      +++  space for low €€€ NOW!1  +++      Campaign     \ /
Party Enjoy Relax   |   http://dragonflybsd.org      Against  HTML   \
Dude 2c 2 the max   !   http://golden-apple.biz       Mail + News   / \

Simon 'corecode' Schubert [off-list ref] writes:

David Kastrup wrote:

quoted

quoted

Otherwise you could have two very different trees that encode the
same *content* (just with different ways of getting there -
depending on whether you have a history with empty trees or not),
and that's very much against the philosophy of git, and breaks some
fundamental rules (like the fact that "same content == same SHA1").

No, the content is _different_.  One tree contains a tracked
directory, the other does not.  That means that the trees behave
_differently_ when you manipulate them, and that means that they are
_not_ the same tree.

You are mistaking things.

No, I am redefining them, or rather the view on them.  Subtle
difference.

Like the executable bit on a file is not content, the fact that a
directory should be kept despite being empty is also an *attribute*
of the directory.  This is meta-data, not actual data (content).

We need to track it, anyway.  So there is little point in not using
the existing infrastructure for handling named entities.

So no matter how elegant tracking the "." entry might be (and I
think it is, because it covers a lot of corner cases already), it
puts the information at the wrong place.

I don't see that the place is wrong: after all, that is where Unix
places "." too, and for good reason.  I was arguing for _separating_
the concept of "directory" and "tree" in the repository.  The tree is
a container entity defined exclusively by its contents (which
determine its hash).  That is how git already does things.  There is
_no_ connection with the physical existence of a directory: in the
work directory, git creates and deletes directories as a _side-effect_
of storing and removing trees.  But git itself does not track
directories as a physical entity at _all_.  If you had a flat
filesystem allowing slashes in filenames, git would get along better
than it does now, without ever creating or removing a directory.
Trees are just a convenient selection and pattern matching mechanism
for files as far as git is concerned.  The correspondence to physical
directories in the work directory is a nuisance rather than an asset
as far as git is concerned.

In a recent thread here, tags with slashes were supported by
essentially doing

    mkdir -p "`dirname $TAG`"
    touch $TAG

where directory creation is just a side effect of supporting slashes.
And that, if you look closely, is git's current relation with
directories altogether.  The directories in the work file system are
created by git just as a side effect for representing slashes, which
in turn facilitate a certain manner of pattern matching.

And "." seems perfectly well suited to bring across the point that
there actually is _physical_ existence associated with a directory,
existence that remains when the rest of the tree is gone and _makes_ a
difference to what the tree is, because it has a _different_
representation in the work file system.

Storing it as an _attribute_ of the tree is a bad idea, since then the
simple rule "a tree without contents is empty" needs an exception.
And a tree stops becoming just a container of its contents and all
sort of new exceptions creep up.

There are some systems where the difference between directory as a
file and directory as a structuring method are more apparent than
under Unix (some utilities like rsync differentiate between A/B and
A/B/ to bring across that difference).

Here is an example for some Emacs function concerned with the concept:

    directory-file-name is a built-in function in `C source code'.
    (directory-file-name DIRECTORY)

    Returns the file name of the directory named DIRECTORY.
    This is the name of the file that holds the data for the directory DIRECTORY.
    This operation exists because a directory is also a file, but its name as
    a directory is different from its name as a file.
    In Unix-syntax, this function just removes the final slash.
    On VMS, given a VMS-syntax directory name such as "[X.Y]",
    it returns a file name such as "[X]Y.DIR.1".

    [back]

That's sad, because otherwise it would be really elegant.

If something is not elegant because of the angle of view, change the
view.  And it is not like the different angle has no predecessors or
no consistency.

-- 
David Kastrup

On Thu, Jul 19, 2007 at 05:15:28PM -0700, Linus Torvalds wrote:

(*) And, for anybody confused about the issue, the answer to the latter 
question is an emphatic: "Yes it should, live with it, and if you want the 
directory back, you had better add it back as an empty directory"

Wouldn't it be perfectly reasonable for git rm to re-add emptied
directories as empty transparently if the appropriate
flag/configuration is set?  rm is porcelain after all.

  OG.

Simon 'corecode' Schubert [off-list ref] writes:

You are mistaking things.  Like the executable bit on a file
is not content, the fact that a directory should be kept
despite being empty is also an *attribute* of the directory.
This is meta-data, not actual data (content).  So no matter
how elegant tracking the "." entry might be (and I think it
is, because it covers a lot of corner cases already), it puts
the information at the wrong place.

Actually, I do not think there is absolute right or wrong here.
The difference is not that the information is at the "right" or
"wrong" place, but one approach places the information at more
efficient-to-use place than the other.  In that sense, the
attribute approach _is_ a more elegant solution between the two.

Making it an attribute has a huge practical advantage.

By treating executable bit as a piece metadata, we can compare
the "contents" quickly.  If you "chmod +x" a blob without
changing anything else, we can detect that fact, because blob
object names are equal.  At the philosophical level, you _could_
argue that the executable-ness is one bit of content and include
that in the object name computation for the blob.  There is
nothing fundamentally wrong about that approach, but that
destroys the nice "cheap comparability" between blobs that
differ only by executable-ness.

David's "." in tree is essentially the same argument as treating
the executable-ness as one extra bit of content.  The fact that
a particular tree wants to stay even after emptied can be
treated as part of contents (thereby reflected in its object
name).  There is nothing fundamentally wrong there, either.  But
that means two trees that contain otherwise identical set of
blobs and subtrees, but differ only in the behaviour of when
they are emptied, would get different object names, hence you
need to descend into them to see if they are different.

Using attribute that is detached from the content itself allows
you to hoist that one bit one level up.  By treating
executable-ness not as part of content, we can compare two blobs
with different executable bits cheaply.  You can avoid
descending into such a tree when comparing it with another tree
that is different only by the "will-stay-when-emptied"-ness the
same way.

Junio C Hamano [off-list ref] writes:

Actually, I do not think there is absolute right or wrong here.  The
difference is not that the information is at the "right" or "wrong"
place, but one approach places the information at more
efficient-to-use place than the other.

Agreed.

In that sense, the attribute approach _is_ a more elegant solution
between the two.

Disagreed.  See below.

Making it an attribute has a huge practical advantage.

By treating executable bit as a piece metadata, we can compare the
"contents" quickly.  If you "chmod +x" a blob without changing
anything else, we can detect that fact, because blob object names
are equal.  At the philosophical level, you _could_ argue that the
executable-ness is one bit of content and include that in the object
name computation for the blob.  There is nothing fundamentally wrong
about that approach, but that destroys the nice "cheap
comparability" between blobs that differ only by executable-ness.

David's "." in tree is essentially the same argument as treating the
executable-ness as one extra bit of content.  The fact that a
particular tree wants to stay even after emptied can be treated as
part of contents (thereby reflected in its object name).

Small nit here: the tree does not want to stay after emptied, since it
is not empty as long as it contains ".".

There is nothing fundamentally wrong there, either.  But that means
two trees that contain otherwise identical set of blobs and
subtrees, but differ only in the behaviour of when they are emptied,
would get different object names, hence you need to descend into
them to see if they are different.

And here we disagree in our assessment, and where I find the example
of the execute bit unfitting.  We are talking about _trees_ here, not
files.  So this is only relevant if we have a _huge_, _flat_ tree with
_lots_ of entries at _bottom_ level.

How often does it occur in practice that a _large_ tree has "."  added
or removed and nothing else changes?  Never, because the normal use
case is that a directory is either tracked from the start, or not
tracked at all.  And even if you change the tracking for a whole
project at once (which is a one-time job): the cost difference is
looking at all _tree_ leaf entries, not at all the involved files.

Using attribute that is detached from the content itself allows you
to hoist that one bit one level up.  By treating executable-ness not
as part of content, we can compare two blobs with different
executable bits cheaply.  You can avoid descending into such a tree
when comparing it with another tree that is different only by the
"will-stay-when-emptied"-ness the same way.

But changing the executable bit of a file will happen often during
development.  Adding or removing "." will never usually be done _ever_
except when the tree is first created or removed, and then the cost is
negligible.

So "performance" is not an issue for making this an attribute or a
flat entry.  While the user level abstraction need not match the
actual representation, I think that it will make for lot less special
cases and problematic behavior to pull through with "." as a directory
entry that mostly behaves like other files and, like other files,
requires git to create a directory to contain it.  All the logic for
creating and deleting directories and creating and adding and ignoring
files can _perfectly_ stay the same.

There are just two differences:

a) git always sees "." as a file in every directory in the work tree
   and considers it a file.
b) when it comes to actually creating or modifying or reading the
   actual file in the work directory, it silently skips the
   operation.

It would not even be necessary to give the directory entry any special
attributes or permissions to make this scheme work: declaring it a
normal file and just special-casing the name "." on those operations
would lead to consistent and working behavior, with no change of
format in index and repository at all.

Possibly even a) alone would suffice, at the cost of letting git
complain and continue at every operation (or making a _really_ royal
mess for Solaris root users).

I might be tempted to make a proof-of-concept patch for that.

But for backward-compatibility, it will be better to use an entry type
which old versions of git will be able to ignore when checking out or
in.  And for user-friendliness, one does not really want to list such
entries as regular files.

-- 
David Kastrup

On Fri, 20 Jul 2007, Junio C Hamano wrote:

Using attribute that is detached from the content itself allows
you to hoist that one bit one level up.  By treating
executable-ness not as part of content, we can compare two blobs
with different executable bits cheaply.  You can avoid
descending into such a tree when comparing it with another tree
that is different only by the "will-stay-when-emptied"-ness the
same way.

Having thought about it a bit more, I would absolutely *detest* any kind 
of "executable bit" like behaviour.

Why? 

Merging. I think one of the fundamental issues in merging is that you do 
it "in the working tree". This is something that pretty much *everybody* 
else gets wrong, and it's somethign where git absolutely shines.

But git shines here exactly because git never tracks "history" or the 
state in the tree, and only ever tracks things that are indubitably real 
content. Which is why you never *ever* have to tell git about "I moved 
file X to file Y" - because git only tracks things that it can see right 
in front of it, in the tree.

The "sticky directory" bit simply would not be something like that. It 
simply isn't "content", and as such, it should not be tracked. It's as 
easy as that. We don't want a merge of two branches to have to specify any 
extra data "outside" the tree as to how it should be merged.

So the issue about whether a directory *exists* or not can be merged (just 
look at the tree), but the issue about whether the directory is supposed 
to be sticky is something that you'd have to tell git about *outside* of 
the tree, and that violates the whole point of working tree merges.

I do realize that if you use inferior operating systems, we already have 
these kinds of "outside the tree" data entries, thanks to issues like 
symlinks and normal file executable bits that you would have to explicitly 
tell git about when you're working in a broken environment. So in that 
sense, it wouldn't be anything technically new for git. 

But that doesn't change the fundamental issue: the limitation with 
executable bits and symlinks is a limitation of the broken environment, 
not of git. But "directories stay around after the last file is gone" is 
not that, it would simply be a design mistake in git itself.

There are other reasons to not do it. What about file renames? Maybe the 
directory got *renamed*. From a pure content angle, this is "all the files 
in that directory went away". If you have stupid rules like "directories 
stay around even though all the files went away", you would again have 
problems with this common case.

In other words: I don't care one whit about the whiners. What's MUCH more 
important than some random whiny person saying "Daddy, daddy, I want a 
pony" is whether you can afford to maintain that pony in the future. And 
this pony is just stupid.

So here:

	No, you cannot have a pony. NOT YOURS.

but I still think we should support the concept of importing things from 
other systems, and thus eventually support empty directories. Just not any 
crazy semantics with sticky histories.

			Linus

PS. As usual, per-user or per-repository *local* attributes are something 
else. They aren't "sticky history", they are just purely behavioural 
defaults. Those kinds of things may make sense. But that's not a "tracking 
content" issue.

On Fri, 20 Jul 2007, Johan Herland wrote:

Does this mean that you are firmly opposed to the concept of storing 
directories in the index/tree as such, or that you are only opposed to 
(some of) the implementation ideas that have been discussed so far?

I've already sent out a *patch* to do so, for chissake. It handled all 
these cases perfectly fine, as far as I know, but I didn't test it all 
that deeply (and made it clear when I sent that patch out).

In fact, in this whole pointless discussion, I think I'm so far the only 
one to have done anything constructive at all. Sad.

So here's my standpoint:

 - people who use git natively might as well use the ".gitignore" trick. 
   It really *does* work, and there really aren't any downsides. Those 
   directories will stay around forever, until you decide that you don't 
   want them any more. Problem solved.

   Sure, if you export the git archive into some other format, you might 
   well want to do something about the ".gitignore" files (like just 
   delete them, since they won't be meaningful in an SVN environment, for 
   example, but you might also just convert them into SVN's "attributes" 
   or whatever it is that SVN uses to ignore files).

 - If you don't use git natively, but just to track another thing, you 
   could easily use the patches that I already sent out. Yes, they need 
   more testing. Yes, you'd also probably like some user interface updates 
   (notably "git add/rm" should be taught about directories).

   And yes, I probably (almost certainly) didn't handle all cases, but the 
   patch I sent out was actually a working one. It really *did* pass my 
   trivial tests.

But once you start tracking empty directories *without* a .gitignore file, 
some things fall out of that:

 - git really *really* is designed to track "snapshots in time". You 
   generate history from these snapshots. This is a very fundmanetal 
   issue, and a lot of people seem to have trouble understanding the 
   deeper implications.

   For example, git and hg may look similar, but git tracks "snapshots in 
   time", and hg tracks "file histories tied together in snapshots". That 
   really is a fundamentally different thing. 

   And one of the fundamental results of git's approach is that content is 
   content. There is *never* any notion of "history".  A snapshot really 
   is just that: it's a standalone thing. It *has* no history. The history 
   comes entirely from outside.

   This means that the whole notion of "this directory will not go away 
   because I added it explicitly" is a totally broken notion in git. It 
   has a notion of "history" - something that simply DOES NOT EXIST, 
   unless you seriously break the whole notion of "snapshots in time".

   In other words, when I say that git is a "content tracker", I'm 
   serious. It tracks nothing *but* content. If some concept doesn't exist 
   in the working tree, git doesn't track it. If it cannot be seen in the 
   filesystem, it doesn't exist.

 - Contrast this with a lot of totally broken SCM's, that track "history" 
   of files. As a result, they have absolutely *horrid* merge problems, 
   because you can no longer just merge things in the working directory, 
   and "the result" is the result. No, if you track history, you now have 
   to tell the SCM about how the *history* moved, not just the content.

So this is why git MUST NOT make the difference between

 - a directory was was created explicitly and then had a few files added 
   to it, and then had those files deleted from it

and

 - we added a few files, we removed them

The end result MUST BE the same, because the  state IN THE WORKING TREE is 
the same!

If the contents are the same, the end result must be the same. It's that 
simple. And it all comes down to: "git tracks contents".

Now, having said that, it doesn't matter *what* the end result is, as long 
as it's the same for both cases. What we do now is that when the files go 
away, the directory is no longer tracked.

But we *could* say that when we remove files, we always add back the 
directory they were in if that directory still exists in the filesystem.

See? Both are consistent with the "git tracks contents" notion. The only 
thing that is *not* consistent with that notion is to have a flag that we 
carry along that says "keep this directory". That's no longer content, and 
now you'd be tracking some internal SCM history instead. And that is a 
mistake. It may sound like a small mistake (and it is), but down that path 
lies madness. It's much better to teach people _why_ git doesn't do it, 
than to say "ok, git tracks content, but we have this special case where 
we also track something else, namely a git internal "stickiness" notion".

SCM is too important to play games with. Git gets things right, and I 
doubt people really _realize_ that the "tracks content" is why git is so 
much better, and why git can do merges so much faster and more reliably 
than anybody else.

So the rule really *must* be:

 - if two trees look the same in the filesystem, they *must* have the same 
   git SHA1, because by definition, they have the same content.

Anything that breaks that very simple statement is fundamentally broken.

			Linus

PS. I realize that nobody actually seems to be writing code, and that this 
is a "paint the bike shed" discussion for everybody else, but just in case 
there are people who don't just masturbate about the color of the shed, 
I'd like to point out that we really *do* need to enhance the "diff" rules 
too, so that you can express the changes in a tree as a diff too. Because 
if we track empty directories, then we need to be able to also *show* the 
difference between a tree that has an empty directory, and one that does 
not.

On Friday 20 July 2007, Linus Torvalds wrote:

[...]

But that doesn't change the fundamental issue: the limitation with 
executable bits and symlinks is a limitation of the broken environment, 
not of git. But "directories stay around after the last file is gone" is 
not that, it would simply be a design mistake in git itself.

There are other reasons to not do it. What about file renames? Maybe the 
directory got *renamed*. From a pure content angle, this is "all the files 
in that directory went away". If you have stupid rules like "directories 
stay around even though all the files went away", you would again have 
problems with this common case.

In other words: I don't care one whit about the whiners. What's MUCH more 
important than some random whiny person saying "Daddy, daddy, I want a 
pony" is whether you can afford to maintain that pony in the future. And 
this pony is just stupid.

So here:

	No, you cannot have a pony. NOT YOURS.

but I still think we should support the concept of importing things from 
other systems, and thus eventually support empty directories. Just not any 
crazy semantics with sticky histories.

Does this mean that you are firmly opposed to the concept of storing 
directories in the index/tree as such, or that you are only opposed to 
(some of) the implementation ideas that have been discussed so far?

If the former is the case, does this mean that there will be no support for 
empty directories in git, alternatively that such support is limited to 
incorporating e.g. Dscho's .gitignore workaround into porcelain commands 
(i.e. "git add --directory some_dir" will be mangled/transformed 
into "touch some_dir/.gitignore && git add some_dir/.gitignore")?

(Granted, Dscho's .gitignore workaround is fairly elegant as workarounds go, 
but it still reeks of inheriting a CVS misfeature.)


Have fun!

...Johan

-- 
Johan Herland, [off-list ref]
www.herland.net

On Fri, 20 Jul 2007, Linus Torvalds wrote:

On Fri, 20 Jul 2007, Johan Herland wrote:

quoted

Does this mean that you are firmly opposed to the concept of storing
directories in the index/tree as such, or that you are only opposed to
(some of) the implementation ideas that have been discussed so far?

I've already sent out a *patch* to do so, for chissake. It handled all
these cases perfectly fine, as far as I know, but I didn't test it all
that deeply (and made it clear when I sent that patch out).

In fact, in this whole pointless discussion, I think I'm so far the only
one to have done anything constructive at all. Sad.

There was Dscho's .gitignore based patch too ...

So here's my standpoint:

- people who use git natively might as well use the ".gitignore" trick.
  It really *does* work, and there really aren't any downsides. Those
  directories will stay around forever, until you decide that you don't
  want them any more. Problem solved.

  Sure, if you export the git archive into some other format, you might
  well want to do something about the ".gitignore" files (like just
  delete them, since they won't be meaningful in an SVN environment, for
  example, but you might also just convert them into SVN's "attributes"
  or whatever it is that SVN uses to ignore files).

Personally I quite like this approach - I'm going to use it to keep all 
the empty directories from Subversion in my importer.  It seems to address 
everthing quite neatly.

I don't really understand the objections ... especially since I can't see 
why you want an empty directory if you're not going to put _something_ in 
it - in which case, presumably you want to ignore it (so maybe a 
.gitignore containing * would be better than an empty one)?  However, I'm 
sure that if people want it, they have a reason.

SCM is too important to play games with. Git gets things right, and I
doubt people really _realize_ that the "tracks content" is why git is so
much better, and why git can do merges so much faster and more reliably
than anybody else.

This is the thing that made me interested in git back in April '05.  I 
couldn't see what we were going to end up with at that point - but I was 
_convinced_ that due to the underlying design it was worth watching. 
Being a python type (sorry ... :$) hg looked interesting when it sprang up 
- but they threw away what I considered to be one of the most compelling 
features of git (at the time there wasn't the wealth of really nice tools 
that we now have).

In fact, I really should say "Thank you Linus", since I came that close to 
writing an SCM from scratch myself - having been using Subversion with 
branches for quite some time (and CVS before that - and yes I do mean 
branches + CVS).  Now I no longer feel the need to write an SCM - just a 
longing to use git.  git is probably better than anything I would have 
come up with too. :D

-- 
Julian

  ---
She is descended from a long line that her mother listened to.
 		-- Gypsy Rose Lee

On Fri, 20 Jul 2007, Julian Phillips wrote:

On Fri, 20 Jul 2007, Linus Torvalds wrote:

quoted

So here's my standpoint:

- people who use git natively might as well use the ".gitignore" trick.
  It really *does* work, and there really aren't any downsides. Those
  directories will stay around forever, until you decide that you don't
  want them any more. Problem solved.

Personally I quite like this approach - I'm going to use it to keep all the
empty directories from Subversion in my importer.  It seems to address
everthing quite neatly.

The really sad part about this discussion is that the ".gitignore trick" 
is really technically no different at all from the one that David Kastrup 
has been advocating a few times, except he calls his ".gitignore" just 
".", and seems to think that it's somehow different.

It is true that ".gitignore" and "." _are_ different.

But they are actually different in the sense that the ".gitignore" thing 
is something you can control, while the "." thing is something that is in 
all directories on UNIX, which is exactly why it _must_not_ be used by git 
to mark existence. Exactly because it has thus lost its ability to be 
something you can tune per-directory in the working tree!

That said, I actually like my patch, because the git tree structures 
actually lend themselves very naturally to the "empty tree", and I know 
people have even built up those kinds of trees on purpose, even if the 
index doesn't support that notion.

So in that sense, teaching the index about an empty tree is in some ways 
the "right thing" to do, if only because it means that the index can 
finally express something that the tree objects themselves have always 
been able to validly encode.

			Linus

Linus Torvalds [off-list ref] writes:

The really sad part about this discussion is that the ".gitignore
trick" is really technically no different at all from the one that
David Kastrup has been advocating a few times, except he calls his
".gitignore" just ".", and seems to think that it's somehow
different.

Oh no, I don't think at all that it is somehow different: actually
this is _exactly_ the reason why I think that the implementation will
be doable even by an idiot like myself, and that is because at least
in my first iteration, "."  will appear as an empty regular file to
git, just like ".gitignore".  The main worry I had was that putting
"." inside of a gitignore entry might stop "git add ." from working
like previously.  But I tried it, and it works just like it would with
".gitignore".  Or rather like it would with ".notignore" since
".gitignore" _is_ specially treated by git, after all.

It is true that ".gitignore" and "." _are_ different.

But they are actually different in the sense that the ".gitignore"
thing is something you can control, while the "." thing is something
that is in all directories on UNIX, which is exactly why it
_must_not_ be used by git to mark existence.

But I don't plan to have it used by git to mark existence.  The
_existence_ can be taken for granted.  But what can't be taken for
granted, like with any other file, is that the file is actually being
tracked by git.  To have it tracked, you need to add it, and it must
not be covered by gitignore.

Exactly because it has thus lost its ability to be something you can
tune per-directory in the working tree!

But it should not let the user lose his ability to let or let not git
track the file.

That said, I actually like my patch, because the git tree structures
actually lend themselves very naturally to the "empty tree", and I
know people have even built up those kinds of trees on purpose, even
if the index doesn't support that notion.

And that is the reason I will be working with the "empty file ."
metaphor: it would be way above my head to make the index support new
file types or even structures, and change the evaporate-when-empty
semantics of trees and so on, while catching all special cases.

I have no chance in hell to implement a new feature with a reasonable
amount of time and work.  That's a task for people with a larger brain
than mine who have my full admiration and respect.  The best I can
hope to achieve is a clever hack.

And if that works, people can still pile exceptions on it and redo it
as a "proper feature".

You are _perfectly_ correct that my proposal is _not_ a jot different
from registering a regular empty file ".notignore", and it is on
_purpose_, because I could not handle the complications if it were.

The only difference is that I am calling the file ".".  Which is in
_all_ respects nothing more than a naming convention.

However, this convention has distinct advantages over ".notignore":

a) I don't have to depart as far from reality.  Whenever I try
registering ".", I can rely on the work directory actually _having_
"." as a _real_, not a pseudofile.  It will not actually be a
_regular_ file as I'll tell git: that's a wart of my prototype
implementation which will, no doubt, eventually be fixed by others
_if_ the code does its job fine apart from being ugly to look at.  It
may not be even necessary internally to think of "." other than as an
empty regular file, but git should probably not talk too loud about it
lest people laugh at it.

b) it already means something to people.  Now this is a two-edged
sword, since "almost, but not quite, entirely unlike" concepts are not
necessarily helpful in computing.  In this case, however, I think the
match is close enough to help people understand what is going on
rather than the other way round.  "." was introduced because people
wanted to have a good way to refer to a directory as an element of
itself.  So using "." as a self-reference for a directory is quite in
the spirit of that name.

So in that sense, teaching the index about an empty tree is in some
ways the "right thing" to do, if only because it means that the
index can finally express something that the tree objects themselves
have always been able to validly encode.

If you define the tree objects by the physical in-memory or
in-repository data structures encoding them, then you are correct.  I
am somewhat reluctant to parade around another red cape, but in this
particular case, the size of the wet spot in my pants does not as much
relate to the physical layout of the data structure (big deal,
probably 30 lines of code all around), but rather to the extent and
assumptions of functions accessing it.  Namely, data layout and
accessor functions _together_ constitute a tree object.  So for me the
"evaporate-when-empty" property, while not inherent in the physical
layout of the object, is still an inherent part of its structure which
I would not want to touch: finding and fixing and debugging all code
elements which explicitly or implicitly rely on that assumption is
something I would not entrust myself with.

I might have been more inclined to dabble with that approach if the
tree stuff were written in something more object-oriented, say, clean
and concise C++, except that clean and concise C++ code in the wild is
even more of a mythical beast than clean and concise TeX code, and C++
itself is such a mindboggingly complex contraption...  I digress.

All the best,

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

David Kastrup [off-list ref] writes:

The only difference is that I am calling the file ".".  Which is in
_all_ respects nothing more than a naming convention.

However, this convention has distinct advantages over ".notignore":

a) I don't have to depart as far from reality.  Whenever I try
registering ".", I can rely on the work directory actually _having_
"." as a _real_, not a pseudofile.  It will not actually be a
_regular_ file as I'll tell git: that's a wart of my prototype
implementation which will, no doubt, eventually be fixed by others
_if_ the code does its job fine apart from being ugly to look at.

Update: well, I am still digging through the code, but this is all so
well factored that it might be perfectly feasible to have S_ISDIR
entries after all without too much of a hassle.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum