From: Junio C Hamano <hidden> Date: 2016-06-15 22:43:22
David Kastrup [off-list ref] writes:
Junio C Hamano [off-list ref] writes:
quoted
No objections as long as a patch is cleanly made without
regression. It's just nobody agreed that it is "quite serious"
yet so far, and no fundamental reason against it.
Thanks. It certainly is not serious for the Linux kernel source, but
seems awkward for quite a few situations. Anyway, what is your take
on the situation I described?
Didn't I say I do not have an objection for somebody who wants
to track empty directories, already? I probably would not do
that myself but I do not see a reason to forbid it, either.
The right approach to take probably would be to allow entries of
mode 040000 in the index. Traditionally, we allowed only 100644
(blobs as regular files) and 120000 (blobs as symlinks). We
recently added 160000 (commit from outer space, aka subproject).
And we do that for all directories, not just empty ones. So if
you have fileA, empty/, sub/fileB tracked, your index would
probably have these four entries, immediately after read-tree
of an existing tree object:
100644 15db6f1f27ef7a... 0 fileA
040000 4b825dc642cb6e... 0 empty
040000 e125e11d3b63e3... 0 sub
100644 52054201c2a872... 0 sub/fileB
Making sure that empty/ directory exists in the working tree is
probably done in entry.c; we have been touching that area in an
unrelated thread in the past few days.
If you add sub/fileC, with "update-index" (and "add"), you
invalidate the SHA-1 object name you stored for "sub" (because
there is no point recomputing the tree object until you know you
need a subtree for "sub" part, which does not happen until the
next "write-tree"), and end up with something like:
100644 15db6f1f27ef7a... 0 fileA
040000 4b825dc642cb6e... 0 empty
040000 00000000000000... 0 sub
100644 52054201c2a872... 0 sub/fileB
100644 705bf16c546f32... 0 sub/fileC
These "missing" SHA-1 would need to be recomputed on-demand.
We have had necessary infrastructure to do this "keeping
untouched tree object names in the index" for quite some time,
but it is not a part of the index proper (it is stored in an
extension section in the index file, to keep the index
compatible with older versions of git).
Having made it sound so easy, here are the issues I would expect
to be nontrivial (but probably not rocket surgery either).
* unpack-trees, which is the workhorse for twoway merge (aka
"switching branches") and threeway merge, has a convoluted
logic to avoid D/F conflicts; it can probably be cleaned up
once we do the above conversion so that the index starts
saying "Hey, I have a directory here" more explicitly. The
end result would probably be a code easier to follow.
* status, update-index --refresh, and diff-files cares about
the information cached in the index from the last time
lstat(2) is run on each entry. What we should store there
for "tree" entries is very unclear to me, but probably we
should teach them to ignore the stat-matching logic for
these entries.
* diff-index walks the index and a tree in parallel but does
not currently expect to see a tree object in the index. It
needs to be taught to ignore these "tree" entries.
* merge-recursive and merge-index walk the index, coming up
with the merge results one path at a time. They also need to
be taught to ignore these "tree" entries.
* diff-index and "read-tree -m" should be taught to take
advantage of the "tree" entries in the index. For example,
if diff-index finds the "tree" entry in the index and the
subtree found from the tree object exactly match, it does not
even have to descend into the tree, which would be a huge
performance win (because you do not have to open the subtree
and its subtrees from the tree side; you already have read
everything on the index side, and still have to skip the
entries in the directory). "read-tree -m" also should be
able to optimize two identical subtrees in the 2 or 3 trees
involved.
Even if we follow the "lazy invalidate" strategy to maintain
the "tree" entries in the normal codepath, we could have a
special operation that says "now update all the tree entries
by recomputing the tree object names as needed". Perhaps we
might want to initiate such an operation before "read-tree
-m" automatically.
From: Johan Herland <hidden> Date: 2016-06-15 22:43:22
On Wednesday 18 July 2007, Junio C Hamano wrote:
Didn't I say I do not have an objection for somebody who wants
to track empty directories, already? I probably would not do
that myself but I do not see a reason to forbid it, either.
The right approach to take probably would be to allow entries of
mode 040000 in the index. Traditionally, we allowed only 100644
(blobs as regular files) and 120000 (blobs as symlinks). We
recently added 160000 (commit from outer space, aka subproject).
And we do that for all directories, not just empty ones. So if
you have fileA, empty/, sub/fileB tracked, your index would
probably have these four entries, immediately after read-tree
of an existing tree object:
Sorry for jumping in late...
Why do you want to add _all_ directories, and not just the ones we want to
explicitly track (independent of whether they're empty or not).
Basically, add a "--dir" flag to git-add, git-rm and friends, to tell them
you're acting on the directory itself (rather than its (recursive)
contents). "git-add --dir foo" will add the "040000 123abc... 0 foo" to the
index/tree whether or not foo is an empty directory. "git-rm --dir foo" will
remove that entry (or fail if it doesn't exist), but _not_ the contents of
foo.
Since we're making directory tracking _explicit_, this should all be trivially
backward-compatible.
...Johan
--
Johan Herland, [off-list ref]
www.herland.net
From: David Kastrup <hidden> Date: 2016-06-15 22:43:22
Johan Herland [off-list ref] writes:
On Wednesday 18 July 2007, Junio C Hamano wrote:
quoted
Didn't I say I do not have an objection for somebody who wants
to track empty directories, already? I probably would not do
that myself but I do not see a reason to forbid it, either.
The right approach to take probably would be to allow entries of
mode 040000 in the index. Traditionally, we allowed only 100644
(blobs as regular files) and 120000 (blobs as symlinks). We
recently added 160000 (commit from outer space, aka subproject).
And we do that for all directories, not just empty ones. So if
you have fileA, empty/, sub/fileB tracked, your index would
probably have these four entries, immediately after read-tree
of an existing tree object:
Sorry for jumping in late...
It could have given you a chance to read up on what has already been
discussed.
Why do you want to add _all_ directories, and not just the ones we
want to explicitly track (independent of whether they're empty or
not).
Because the problematic cases are more often than not the _implicit_
cases. Do you check a directory tree for empty directories before you
archive it? In order to archive every empty directory explicitly?
If you did that, you could equally maintain a script that manually
does mkdir/rmdir.
Basically, add a "--dir" flag to git-add, git-rm and friends, to
tell them you're acting on the directory itself (rather than its
(recursive) contents). "git-add --dir foo" will add the "040000
123abc... 0 foo" to the index/tree whether or not foo is an empty
directory. "git-rm --dir foo" will remove that entry (or fail if it
doesn't exist), but _not_ the contents of foo.
There is nothing wrong with implementing something like this in
_addition_ to treating directory entries implicitly. For example, ls
has an option -d which does just that, and even git-ls-files has an
option --directory. Heck, I even have
rm --help
Usage: rm [OPTION]... FILE...
Remove (unlink) the FILE(s).
-d, --directory unlink FILE, even if it is a non-empty directory
(super-user only; this works only if your system
supports `unlink' for nonempty directories)
[...]
which works on just the directory and not on the contents.
So a --directory option for appropriate commands would be natural for
_explicit_ manipulation of such entries.
But the important, the _really_ important thing are the implicit
behaviors. If I have to hassle with every directory myself, I don't
need a content tracking system.
The --directory stuff, in contrast, are things nice to have when the
framework is in place (and may be even necessary for some direct
manual maintenance tasks), but they don't really concern the
framework.
--
David Kastrup
From: Johan Herland <hidden> Date: 2016-06-15 22:43:22
On Friday 20 July 2007, David Kastrup wrote:
Johan Herland [off-list ref] writes:
quoted
Sorry for jumping in late...
It could have given you a chance to read up on what has already been
discussed.
I have tried to keep on top of the discussion so far.
quoted
Why do you want to add _all_ directories, and not just the ones we
want to explicitly track (independent of whether they're empty or
not).
Because the problematic cases are more often than not the _implicit_
cases. Do you check a directory tree for empty directories before you
archive it? In order to archive every empty directory explicitly?
No, of course I don't. But then archiving (as in tar) is intended to recreate
the "working copy" exactly as it was. Git (and other SCMs), however, is only
interested in recreating the part of the working copy it explicitly tracks.
Given the following working copy:
/
/tracked/
/tracked/file
/tracked/dir/
/untracked/
/untracked/file
/untracked/dir/
and the following commands:
$ git add tracked
$ git clone
The cloned result could be any of the following:
(1)
/
/tracked/
/tracked/file
This is the current behaviour; directories are not tracked at all, but only
added as necessary to support files.
(2)
/
/tracked/
/tracked/file
/tracked/dir/
/untracked/
/untracked/dir/
i.e. implicitly tracking _all_ directories. This is what you literally ask
for, but I think most would find this unreasonable.
(3)
/
/tracked/
/tracked/file
/tracked/dir/
i.e. recursively tracking directories (and files). This seems useful, but
there is nothing _implicit_ about this.
I have a feeling that you're actually arguing for doing (3) by default. What I
am arguing is to do (1) by default, and (3) if given a suitable command-line
option (i.e. "git add --with-dirs tracked").
Note that this is really an interface question. How these entries are actually
stored in the repo is a different discussion.
Finally, let's look at the case of "git add tracked/file" followed by "git rm
tracked/file". I'm arguing that "tracked/" should be automatically removed,
since I never asked for it to be tracked by git. On the other
hand, "git-add --non-recursive tracked" followed by the above two commands,
should of course leave "tracked/" in place, since I now actually asked
explicitly for the directory to be tracked.
My point is fundamentally that selectively tracking directories is a more
powerful concept than just tracking _all_ directories by default. Note that
if we support selectively tracking directories, tracking _everything_ (like
you seem to want) is trivially implemented by _always_ supplying the
appropriate option to git-add. If we track everything by design, we don't
have the option of selectively tracking some directories.
quoted
Basically, add a "--dir" flag to git-add, git-rm and friends, to
tell them you're acting on the directory itself (rather than its
(recursive) contents). "git-add --dir foo" will add the "040000
123abc... 0 foo" to the index/tree whether or not foo is an empty
directory. "git-rm --dir foo" will remove that entry (or fail if it
doesn't exist), but _not_ the contents of foo.
There is nothing wrong with implementing something like this in
_addition_ to treating directory entries implicitly.
I don't agree. By _selectively_ tracking directories you can implement any
policy you want on top of it.
For example, ls
has an option -d which does just that, and even git-ls-files has an
option --directory. Heck, I even have
Yes, having commandline options for explicitly specifying directories (and not
their contents) is _exactly_ what I want.
But the important, the _really_ important thing are the implicit
behaviors. If I have to hassle with every directory myself, I don't
need a content tracking system.
I disagree. Just as you have to decide which files to track, you similarly
should have to decide which directories to track. Of course, the tools make
this easier for you by being able to recursively handle files. In the same
way they should be able to do the same thing for directories.
Have fun!
...Johan
--
Johan Herland, [off-list ref]
www.herland.net
From: David Kastrup <hidden> Date: 2016-06-15 22:43:22
Johan Herland [off-list ref] writes:
On Friday 20 July 2007, David Kastrup wrote:
quoted
Johan Herland [off-list ref] writes:
quoted
Sorry for jumping in late...
It could have given you a chance to read up on what has already been
discussed.
I have tried to keep on top of the discussion so far.
quoted
quoted
Why do you want to add _all_ directories, and not just the ones we
want to explicitly track (independent of whether they're empty or
not).
Because the problematic cases are more often than not the
_implicit_ cases. Do you check a directory tree for empty
directories before you archive it? In order to archive every empty
directory explicitly?
No, of course I don't. But then archiving (as in tar) is intended to
recreate the "working copy" exactly as it was. Git (and other SCMs),
however, is only interested in recreating the part of the working
copy it explicitly tracks.
Yes, and
git-add some-dir
tells it to track _everything_ inside some-dir. Which means that the
included files are tracked _implicitly_. The included directories
(including some-dir itself) are not.
Given the following working copy:
/
/tracked/
/tracked/file
/tracked/dir/
/untracked/
/untracked/file
/untracked/dir/
and the following commands:
$ git add tracked
$ git clone
The cloned result could be any of the following:
(1)
/
/tracked/
/tracked/file
This is the current behaviour; directories are not tracked at all, but only
added as necessary to support files.
And so your case (1) actually rather is a single line:
/tracked/file
Everything else is just part of representing /tracked/file and
disappears as soon as /tracked/file disappears.
(2)
/
/tracked/
/tracked/file
/tracked/dir/
/untracked/
/untracked/dir/
i.e. implicitly tracking _all_ directories. This is what you literally ask
for,
I don't see how you can possibly conclude that from what I have been
writing.
but I think most would find this unreasonable.
And it is. So please _don't_ put words into my mouth. In my
proposal, the following (and nothing else) would get tracked:
/tracked/.
/tracked/file
and that's it. That is what was requested, and that is what is
tracked. There will be, incidentally, a tree "/tracked/" and a tree
"/" in the _repository_, but those collapse as soon as they are empty.
They are just an _abstract_ data structuring tool in the repository
that is _mapped_ to directories on checkout.
/
/tracked/
/tracked/file
/tracked/dir/
i.e. recursively tracking directories (and files). This seems useful, but
there is nothing _implicit_ about this.
You did not ask for "/tracked/file" and you did not ask for
"/tracked/dir/" (whatever they may be). That you wanted to track them
was _implied_ by your request of "/tracked/".
I have a feeling that you're actually arguing for doing (3) by
default. What I am arguing is to do (1) by default, and (3) if
given a suitable command-line option (i.e. "git add --with-dirs
tracked").
Note that this is really an interface question.
Not at all. It is a _conceptual_ question: in order for this to work
at _all_ (instead of being an inconsistent heap of ugly surprises),
directories need a representation in the repo. This representation,
as opposed to in the work file system, is _optional_: the repository
got perfectly well along without it up to now, and the fallback is
already implemented when there is a tree without corresponding
directory.
How these entries are actually stored in the repo is a different
discussion.
Sure. But anything that requires four dozens of special cases instead
of four because one wanted to keep "things that are under some
specialized view separate separate" is not something I am going to
implement. I am too old to juggle with complexity for the sake of
complexity. I can make much more use of the existing infrastructure
by actually making file and directory entries quite similar.
ls -la
also has no special cases for "." and ".." because they are, at a very
fundamental level, very special in achieving a special purpose
_without_ being special-cased.
Finally, let's look at the case of "git add tracked/file" followed
by "git rm tracked/file". I'm arguing that "tracked/" should be
automatically removed, since I never asked for it to be tracked by
git.
Sure. And nobody ever said otherwise. In fact, I gave about a dozen
examples in that line and more special in the thread up to now.
On the other hand, "git-add --non-recursive tracked" followed by the
above two commands, should of course leave "tracked/" in place,
since I now actually asked explicitly for the directory to be
tracked.
Sure. Use "--directory" instead of "--non-recursive" and you have a
somewhat more special option for that.
My point is fundamentally that selectively tracking directories is a
more powerful concept than just tracking _all_ directories by
default.
Perhaps you might read up on some of the past discussion before
beating dead horses. This has been covered already, and more than
once. I never asked for "all directories" to be tracked. I outlined
cases where they are tracked and where not, and I tested that the
mechanisms in "man gitignore" already work _perfectly_ with the
pattern "." for configuring the _implied_ tracking at directory,
repository, project, and user preference level.
Note that if we support selectively tracking directories, tracking
_everything_ (like you seem to want) is trivially implemented by
_always_ supplying the appropriate option to git-add. If we track
everything by design, we don't have the option of selectively
tracking some directories.
But that means manual intervention all of the time. It is fine when a
tool provides an option to shoot you in the arm instead of in the foot
as usual, but that's not really a fix, but an acerbation of the
problem.
quoted
quoted
Basically, add a "--dir" flag to git-add, git-rm and friends, to
tell them you're acting on the directory itself (rather than its
(recursive) contents). "git-add --dir foo" will add the "040000
123abc... 0 foo" to the index/tree whether or not foo is an empty
directory. "git-rm --dir foo" will remove that entry (or fail if
it doesn't exist), but _not_ the contents of foo.
There is nothing wrong with implementing something like this in
_addition_ to treating directory entries implicitly.
I don't agree. By _selectively_ tracking directories you can
implement any policy you want on top of it.
No, you can't. Because a "policy" means that things are _implied_.
Being able to do everything manually is not a policy. It may be a
lifesaver at times, but then you have little business drifting in the
river in the first place.
quoted
But the important, the _really_ important thing are the implicit
behaviors. If I have to hassle with every directory myself, I
don't need a content tracking system.
I disagree. Just as you have to decide which files to track, you
similarly should have to decide which directories to track. Of
course, the tools make this easier for you by being able to
recursively handle files. In the same way they should be able to do
the same thing for directories.
--directory _explicitly_ is not working recursively, so it does not
solve that problem.
--
David Kastrup
From: Johan Herland <hidden> Date: 2016-06-15 22:43:22
On Friday 20 July 2007, David Kastrup wrote:
Johan Herland [off-list ref] writes:
quoted
My point is fundamentally that selectively tracking directories is a
more powerful concept than just tracking _all_ directories by
default.
Perhaps you might read up on some of the past discussion before
beating dead horses. This has been covered already, and more than
once. I never asked for "all directories" to be tracked. I outlined
cases where they are tracked and where not, and I tested that the
mechanisms in "man gitignore" already work _perfectly_ with the
pattern "." for configuring the _implied_ tracking at directory,
repository, project, and user preference level.
It seems our discussion is based on so many misunderstandings of each other
that it's not very useful to reply to specific parts of it.
AFAICS, from a high-level POV, we're pretty much in agreement on the following
points:
1. Git should be able to track directories.
2. Tracked directories should be kept alive, even if empty.
3. Git must not necessarily track _all_ directories.
Conversely, we seem to disagree on these points:
4. Whether or not git should track directories by default. You say yes, I say
no.
5. How the tracking of directories should be implemented in git's object
database. I want to keep the index/tree as-is except for adding directory
entries (w/mode 040000) for the tracked directories only. You seem to want to
add directory entries for _all_ directories and then additional "." entries
for directories you don't want deleted if/when empty.
Am I making sense, or have I misunderstood our misunderstandings?
...Johan
--
Johan Herland, [off-list ref]
www.herland.net
From: David Kastrup <hidden> Date: 2016-06-15 22:43:23
Coming full circle...
Junio C Hamano [off-list ref] writes:
The right approach to take probably would be to allow entries of
mode 040000 in the index. Traditionally, we allowed only 100644
(blobs as regular files) and 120000 (blobs as symlinks). We
recently added 160000 (commit from outer space, aka subproject).
And we do that for all directories, not just empty ones. So if
you have fileA, empty/, sub/fileB tracked, your index would
probably have these four entries, immediately after read-tree
of an existing tree object:
100644 15db6f1f27ef7a... 0 fileA
040000 4b825dc642cb6e... 0 empty
040000 e125e11d3b63e3... 0 sub
100644 52054201c2a872... 0 sub/fileB
This would be very much what I am proposing now, except that instead
of 040000 we would have 040755 usually, so that when the index makes
it into the repository where 040000 already has a meaning (a
disappear-when-empty tree) we get the right information. Also note
that the above comes about when doing
git-add *
but not when doing
git-add fileA empty sub/fileB (in the latter case, the entry for sub
would be missing)
If you add sub/fileC, with "update-index" (and "add"), you
invalidate the SHA-1 object name you stored for "sub" (because
there is no point recomputing the tree object until you know you
need a subtree for "sub" part, which does not happen until the
next "write-tree"), and end up with something like:
100644 15db6f1f27ef7a... 0 fileA
040000 4b825dc642cb6e... 0 empty
040000 00000000000000... 0 sub
100644 52054201c2a872... 0 sub/fileB
100644 705bf16c546f32... 0 sub/fileC
These "missing" SHA-1 would need to be recomputed on-demand.
Ah, ok. Does it even make sense to compute the SHA-1 values in the
index in advance? What would they be useful for?
We have had necessary infrastructure to do this "keeping
untouched tree object names in the index" for quite some time,
but it is not a part of the index proper (it is stored in an
extension section in the index file, to keep the index
compatible with older versions of git).
What is the application for which this is being used?
Having made it sound so easy, here are the issues I would expect
to be nontrivial (but probably not rocket surgery either).
* unpack-trees, which is the workhorse for twoway merge (aka
"switching branches") and threeway merge, has a convoluted
logic to avoid D/F conflicts; it can probably be cleaned up
once we do the above conversion so that the index starts
saying "Hey, I have a directory here" more explicitly. The
end result would probably be a code easier to follow.
I am afraid that this is unlikely to happen, and that is because
directory tracking remains optional at a fundamental level as long as
we want to support the current behavior as an option. However, one
could conceivably add 040000 entries (rather than 040755) for
directories that have not been passed into tracking but are required
by git, if this simplifies matters. But it sounds like something that
might complicate working with several different git versions on the
same index.
* status, update-index --refresh, and diff-files cares about
the information cached in the index from the last time
lstat(2) is run on each entry. What we should store there
for "tree" entries is very unclear to me, but probably we
should teach them to ignore the stat-matching logic for
these entries.
At the current point of time, git tracks just the u+x bit for normal
files, and for directories, there is really nothing worth tracking as
long as no attempt of restoring more mode bits is done. Modification
times are probably a bit too risky to pay attention to.
* diff-index walks the index and a tree in parallel but does
not currently expect to see a tree object in the index. It
needs to be taught to ignore these "tree" entries.
Or do something sensible when comparing. Understood.
* merge-recursive and merge-index walk the index, coming up
with the merge results one path at a time. They also need to
be taught to ignore these "tree" entries.
Same here.
* diff-index and "read-tree -m" should be taught to take
advantage of the "tree" entries in the index. For example,
if diff-index finds the "tree" entry in the index and the
subtree found from the tree object exactly match, it does not
even have to descend into the tree, which would be a huge
performance win (because you do not have to open the subtree
and its subtrees from the tree side; you already have read
everything on the index side, and still have to skip the
entries in the directory). "read-tree -m" also should be
able to optimize two identical subtrees in the 2 or 3 trees
involved.
Even if we follow the "lazy invalidate" strategy to maintain
the "tree" entries in the normal codepath, we could have a
special operation that says "now update all the tree entries
by recomputing the tree object names as needed". Perhaps we
might want to initiate such an operation before "read-tree
-m" automatically.
Over my head, but it would appear that it can safely left for later.
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum