Jakub Narebski [off-list ref] writes:
David Kastrup wrote:
quoted
I must be really bad at explaining things, or I am losing a fight
against preconceptions fixed beyond my imagination.
I don't understand you, or you don't understand git. "Tree" object
in object database (in repository) represents a directory in the
working area. There was never any problem with having empty trees in
object database, or having links to empty directory in the superdir.
We don't have to change anything about object database.
I disagree here. The object database _can_ represent an _empty_
directory that has been added explicitly, because up to now no
operations existed that actually left an empty tree. But it can't
distinguish a _non_-empty directory that has been added explicitly
from non-empty directory that has not been added explicitly.
To wit: after the sequence
mkdir a
touch a/b
git-add a
git-commit -m x
git-rm a/b
git-commit -m x
I expect git to retain an empty directory a. But the _tree_ now can't
be different from the tree in the situation
mkdir a
touch a/b
git-add a/b
git-commit -m x
git-rm a/b
git-commit -m x
because after step 1, the trees have identical contents, and so there
is nothing at the _identical_ step 2 that could cause different
behavior.
But in the second case, git must _not_ retain a. So we need to record
the information that in the first case, a was added explicitly. And
this can't be done with the current repository layout. It doesn't buy
us anything that we _have_ a representation available for an _empty_
tree added explicitly. We need this "added explicitly" information
for _every_ tree, not just empty ones.
And a perfectly consistent way is to make those trees with an
explicitly added directory _non-empty_, by virtue of putting a file
"." in them. This file, of course, exists in every physical
directory, but we may or may not decide to let it be tracked by git,
using the gitignore mechanism on the pattern ".". Perfectly
expedient.
The problems with git problems with empty directories stems from the
fact that index didn't have directories.
That basically implies that no information about directories could be
tracked in the repository. And yes, we need appropriate information
in the index. Again, the information whether a directory was added
explicitly.
Index is flattened version of root tree, and before subproject
support it contained _only_ info about blobs (file contents).
And the repository is a versioned and hierarchically hashed version of
the index, but its trees contain _no_ information that is not already
inherently represented by the files alone. Permitting empty trees
would change that fundamental property, and it would not buy us the
ability to actually track directories: see above. So it is not worth
the trouble to assign any meaningful concept to persisting empty trees
rather than make them a case for git-fsck.
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
On Sun, 22 July 2007, David Kastrup wrote:
Jakub Narebski [off-list ref] writes:
quoted
David Kastrup wrote:
quoted
I must be really bad at explaining things, or I am losing a fight
against preconceptions fixed beyond my imagination.
Or you are wrong...
quoted
I don't understand you, or you don't understand git. "Tree" object
in object database (in repository) represents a directory in the
working area. There was never any problem with having empty trees in
object database, or having links to empty directory in the superdir.
We don't have to change anything about object database.
I disagree here. The object database _can_ represent an _empty_
directory that has been added explicitly, because up to now no
operations existed that actually left an empty tree. But it can't
distinguish a _non_-empty directory that has been added explicitly
from non-empty directory that has not been added explicitly.
True. I forgot about that.
Although I'd rather say that we want distinguish between automatically
cleaned up directory (directory which will be deleted if all files in
it would be deleted, and would be untracked if all tracked files in it
would be deleted), and "sticky" directory, which is explicitely tracked
and have to be explicitely deleted.
The fact that it was added explicitely or non explicitely is orthogonal
to that.
IMHO it would be best to first provide plumbing infrastructure (as e.g.
it was the case of submodule support), then add option to
git-update-index to change the "stickiness"/"autoremoval" status of a
directory (of a tree), and _last_ think about how to change the
porcelain (git-add and git-rm).
[...]
But in the second case, git must _not_ retain a. So we need to record
the information that in the first case, a was added explicitly. And
this can't be done with the current repository layout. It doesn't buy
us anything that we _have_ a representation available for an _empty_
tree added explicitly. We need this "added explicitly" information
for _every_ tree, not just empty ones.
And a perfectly consistent way is to make those trees with an
explicitly added directory _non-empty_, by virtue of putting a file
"." in them. This file, of course, exists in every physical
directory, but we may or may not decide to let it be tracked by git,
using the gitignore mechanism on the pattern ".". Perfectly
expedient.
Here we disagree. I think putting "." in a tree as marker of having it
not be automatically deleted when empty, as opposed to marking tree
using filemode in the parent, is not a good idea.
The only advantage to the "." idea is that it can use gitignore
mechanism (both in-tree .gitignore, tracked or not, and info/exclude
file). But I also think that the fact that gitignore mechanism is
recursive is more of disadvantage than advantage.
First, it is _not_ consistent. Working directory trees _always_ have '.'
in them, while trees would have or would have not it, depending if they
would be "sticky" or "autoremoved".
Second, the "easy implementation" is anything but easy. "git add ." as
a way to mark directory as "sticky" is not backward compatibile:
currently it mean to add _all contents_ of current directory.
Implementation is tricky: as we have seen trying to unlink '.' or
create '.' can unfortunately succeed on [some Sun OS, and UFS
filesystem] (which follows POSIX stupidly to the letter) f**king
up the filesystem. The alternative proposal of adding "magic mode" to
mark directory as "not remove when empty" is largely tested; it is very
similar to the subproject support.
Third, is contrary to the git philosophy of tracking contents.
"Stickiness" is an attribute; the fact that directory is explicitely
tracked or not does not change contents of a directory. Compare to
'blob' which contains only contents of a file: not a filename, not a
pathname, not [subset of] filemode.
Fourth, is very artificial. What would you put for filemode for '.'?
040000 (i.e. directory)? What would you put for sha1? Sha1 of an empty
directory? Of an empty blob? 0{40} (which is bad idea because
git-diff-tree uses 0{40} to represent 'not existance')?
quoted
The problems with git problems with empty directories stems from the
fact that index didn't have directories.
That basically implies that no information about directories could be
tracked in the repository. And yes, we need appropriate information
in the index. Again, the information whether a directory was added
explicitly.
Whether directory is automatically managed by git (automatically removed
or untracked). But we need directory entry in index for git-diff, for
example to recognize if there is or there is not empty directory, or if
a directory is automanaged or not.
quoted
Index is flattened version of root tree, and before subproject
support it contained _only_ info about blobs (file contents).
And the repository is a versioned and hierarchically hashed version of
the index, but its trees contain _no_ information that is not already
inherently represented by the files alone. [...]
The above sentence is nonsensical. Index is helper for repository,
and can be derived from repository. Not vice versa.
Trees do contain information which is not inherently present by the
blobs.
--
Jakub Narebski
Poland
Jakub, this mail is too long already, and it does not make sense to
tack a changed proposal to its end since then the readers will be
exhausted at the time they come there. So I'll instead tack a
followup to the "big picture" mail instead where I outline a modified
approach which is presumably easier to understand and completely
backwards-compatible, incorporating your feedback.
There is probably little sense in wasting your time on a detailed
response: feel free to point out where you don't see myself making
sense. I have no problem with people coming to different conclusions
that I do, but I would prefer it if it is not because they consider
myself a raving lunatic, but because they have different opinions
regarding the details.
"I can follow you, but I disagree with your conclusion" is perfectly
fine for now since I am going to propose something else, anyway.
Thanks for the feedback. It gave me some good ideas.
Jakub Narebski [off-list ref] writes:
On Sun, 22 July 2007, David Kastrup wrote:
quoted
Jakub Narebski [off-list ref] writes:
quoted
David Kastrup wrote:
quoted
I must be really bad at explaining things, or I am losing a fight
against preconceptions fixed beyond my imagination.
Or you are wrong...
Well, there is little reason for you to take my word on it, but I
happen to have a history of designing and implementing systems where I
have been responsible for every single byte, bootloader, firmware,
applications, target compiler, assembler, whatever. I have been
exposed to Unix and working with it several years before Linux even
existed. I also have a track record of being not exactly stupid.
So I pretty much can rule out that I am wrong on the factual side.
But where I may be wrong is in estimating the how obvious the design
can appear to others, and how useful and maintainable for others it
may be in the long run. Linus says "code talks", but that's actually
not half the story. If my code says that it works and the evidence is
there, but nobody is able to understand _why_ it works, it has no
place in a project where I am not permanently around.
If smart people don't get what I am talking about, it does not matter
that the patch is surprisingly well-contained: it will be a
maintenance nightmare because people will never figure out why
something stopped working after some particular change.
quoted
I disagree here. The object database _can_ represent an _empty_
directory that has been added explicitly, because up to now no
operations existed that actually left an empty tree. But it can't
distinguish a _non_-empty directory that has been added explicitly
from non-empty directory that has not been added explicitly.
True. I forgot about that.
Thanks. It is almost a revelation that anybody can agree on any point
with me at the moment.
IMHO it would be best to first provide plumbing infrastructure (as
e.g. it was the case of submodule support), then add option to
git-update-index to change the "stickiness"/"autoremoval" status of
a directory (of a tree), and _last_ think about how to change the
porcelain (git-add and git-rm).
Sure. It does no harm to think about reducing the amount of breaking
porcelain, though.
[...]
quoted
And a perfectly consistent way is to make those trees with an
explicitly added directory _non-empty_, by virtue of putting a file
"." in them. This file, of course, exists in every physical
directory, but we may or may not decide to let it be tracked by
git, using the gitignore mechanism on the pattern ".". Perfectly
expedient.
Here we disagree. I think putting "." in a tree as marker of having
it not be automatically deleted when empty, as opposed to marking
tree using filemode in the parent, is not a good idea.
Well, "not a good idea" is a far step forward from "stupid idiot
babbling nonsense", so we may make progress towards actually being
able to _weigh_ different options. I can actually associate with "not
a good idea", not least because nobody else seems to get the idea, and
that makes it infeasible for maintenance.
So I'll address some points and then propose a different way of
implementing what will in the end amount to rather similar semantics,
but with a different view of looking at those semantics, one that
corresponds well with the implementation.
The only advantage to the "." idea is that it can use gitignore
mechanism (both in-tree .gitignore, tracked or not, and info/exclude
file). But I also think that the fact that gitignore mechanism is
recursive is more of disadvantage than advantage.
First, it is _not_ consistent. Working directory trees _always_ have
'.' in them, while trees would have or would have not it, depending
if they would be "sticky" or "autoremoved".
Let me point out again that this inconsistency is already present in
the difference of tracked and untracked _files_: they are always in
the working directory, while trees have or not have them, depending on
whether they are "registered" or "not".
There is no inconsistency involved here, but it seems to make people
_very_ uncomfortable to factor out the "stays around even if empty"
functionality and call it "dir/." from the "can hold content"
functionality which is in effect called "dir/", and basically
associate tracked physical existence just with the former.
The recursiveness of the gitignore mechanism has the advantage that
when maintaining a large repository with actual or logical
subprojects, one does not need to pick a single policy for all
subprojects. I think that is quite important. It could possibly be
achieved with some other method of having per-subproject
configuration, but I see little wrong in using what is there and
documented already.
Second, the "easy implementation" is anything but easy. "git add ."
as a way to mark directory as "sticky" is not backward compatibile:
currently it mean to add _all contents_ of current directory.
Implementation is tricky: as we have seen trying to unlink '.' or
create '.' can unfortunately succeed on [some Sun OS, and UFS
filesystem] (which follows POSIX stupidly to the letter) f**king up
the filesystem.
I was not suggesting actually leaving any such calls in place: after
all, they would presumably lead to error messages. But I agree that
this could lead to nasty surprises when somebody with a legacy version
of git worked with a repository containing "." as explicit entries of
some file type.
The alternative proposal of adding "magic mode" to mark directory as
"not remove when empty" is largely tested; it is very similar to the
subproject support.
Good. Because it is what I converged to last night.
Third, is contrary to the git philosophy of tracking contents.
"Stickiness" is an attribute; the fact that directory is explicitely
tracked or not does not change contents of a directory. Compare to
'blob' which contains only contents of a file: not a filename, not a
pathname, not [subset of] filemode.
Fourth, is very artificial. What would you put for filemode for '.'?
040000 (i.e. directory)?
Taken already. By something very artificial, namely a tree... Yes,
this was a wart in my proposal.
What would you put for sha1? Sha1 of an empty directory?
Some fixed value. Everywhere the same. Not really relevant.
quoted
That basically implies that no information about directories could
be tracked in the repository. And yes, we need appropriate
information in the index. Again, the information whether a
directory was added explicitly.
Whether directory is automatically managed by git (automatically
removed or untracked). But we need directory entry in index for
git-diff, for example to recognize if there is or there is not empty
directory, or if a directory is automanaged or not.
One conclusion that I have come to (and I think I am in agreement with
Linus here) is that the information "empty or not" is actually useless
separately: when I add files below a directory to the repository, the
directory _can't_ be empty. And git has no way of knowing whether it
is non-empty because I wanted the directory to be there, or whether it
is non-empty because I could not have checked in the files into the
tree below it otherwise.
quoted
And the repository is a versioned and hierarchically hashed version
of the index, but its trees contain _no_ information that is not
already inherently represented by the files alone. [...]
The above sentence is nonsensical. Index is helper for repository,
and can be derived from repository. Not vice versa.
Trees do contain information which is not inherently present by the
blobs.
Could you give examples for such information? As long as we are not
talking about _history_, I am at a loss at what else you mean. File
names and permissions?
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
David Kastrup wrote:
"I can follow you, but I disagree with your conclusion" is perfectly
fine for now since I am going to propose something else, anyway.
Thanks for the feedback. It gave me some good ideas.
You are welcome.
Jakub Narebski [off-list ref] writes:
quoted
On Sun, 22 July 2007, David Kastrup wrote:
quoted
Jakub Narebski [off-list ref] writes:
quoted
David Kastrup wrote:
quoted
I must be really bad at explaining things, or I am losing a fight
against preconceptions fixed beyond my imagination.
Or you are wrong...
Well, there is little reason for you to take my word on it, but I
happen to have a history of designing and implementing systems where I
have been responsible for every single byte, bootloader, firmware,
applications, target compiler, assembler, whatever. I have been
exposed to Unix and working with it several years before Linux even
existed. I also have a track record of being not exactly stupid.
So I pretty much can rule out that I am wrong on the factual side.
Big words.
First, there is little matter of something like area of competence.
You might be systems master, but your idea about snapshot based
distributed revision control systems can be wrong because DSCM are
outside the area you know most about.
Second, even if you are a master at given topic, you can still be wrong.
Mind you, I was not saying you are wrong. I was saying you could be.
[...]
quoted
The only advantage to the "." idea is that it can use gitignore
mechanism (both in-tree .gitignore, tracked or not, and info/exclude
file). But I also think that the fact that gitignore mechanism is
recursive is more of disadvantage than advantage.
[...]
The recursiveness of the gitignore mechanism has the advantage that
when maintaining a large repository with actual or logical
subprojects, one does not need to pick a single policy for all
subprojects. I think that is quite important. It could possibly be
achieved with some other method of having per-subproject
configuration, but I see little wrong in using what is there and
documented already.
I think it would be best implemented by repository config, e.g.
core.dirManagement or something like that, which could be set to
1. "autoremove" or something like that, which gives old behavior
of untracking directory if it doesn't have any tracked files
in it, and removing directory if it doesn't have any files
in it.
2. "noremove" or something like that, which changes the behaviour
to _never_ untrack directory automatically. This can be done
without any changes to 'tree' object nor index. It could be useful
for git-svn repositories.
3. "marked" or something like that, for which you have to explicitely
mark directories which are not to be removed when empty.
4. "recursive" or something like that, which would automatically mark
as "sticky" all subdirectories added in a "sticky" repository.
OR directory is not removed when empty if it is marked as such,
or one of its parents is marked as such.
quoted
Second, the "easy implementation" is anything but easy. "git add ."
as a way to mark directory as "sticky" is not backward compatibile:
currently it mean to add _all contents_ of current directory.
Implementation is tricky: as we have seen trying to unlink '.' or
create '.' can unfortunately succeed on [some Sun OS, and UFS
filesystem] (which follows POSIX stupidly to the letter) f**king up
the filesystem.
I was not suggesting actually leaving any such calls in place: after
all, they would presumably lead to error messages. But I agree that
this could lead to nasty surprises when somebody with a legacy version
of git worked with a repository containing "." as explicit entries of
some file type.
The "magic mode" solution _should_ work also with older git, I think.
quoted
Fourth, is very artificial. What would you put for filemode for '.'?
040000 (i.e. directory)?
[...]
quoted
What would you put for sha1? Sha1 of an empty directory?
Some fixed value. Everywhere the same. Not really relevant.
Relevant because it has to work with legacy git on strange operating
systems. Because git has to fsck it (and adding special casing this
"some fixed value" to git-fsck is bad, bad idea).
Note that sha1 cannot be sha1 of the tree. In working area '.' is self
link. You cannot create self link in git repository object.
[...]
quoted
quoted
And the repository is a versioned and hierarchically hashed version
of the index, but its trees contain _no_ information that is not
already inherently represented by the files alone. [...]
[...]
quoted
Trees do contain information which is not inherently present by the
blobs.
Could you give examples for such information? As long as we are not
talking about _history_, I am at a loss at what else you mean. File
names and permissions?
File names and permissions. And they bind blobs and trees together.
Trees do not contain any info about history.
--
Jakub Narebski
Poland