From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Hello,
I narrowed it down. It is in "git checkout" if I copy the ".git" dir via
"cp -a" and type in "git checkout" it breaks. And it _only_ breaks for
subdirectories:
(faui04a) [/var/tmp/shit] git checkout
D profiles/icpc
D profiles/sithglan
D sources/boot.msg
D sources/busybox-config
D sources/config
D sources/easix.logo.16
D sources/easixdm
D sources/f2
D sources/home-faumachine.tar.gz
D sources/home-icpc.tar.gz
D sources/home-simigern.tar.gz
D sources/home-sithglan.tar.gz
D sources/home-sitowert.tar.gz
D sources/linuxrc
D sources/rc.local
and I also see that _one_ file in each subdirectory is checked out but the
others are not.
Thomas
From: David Kastrup <hidden> Date: 2016-06-15 22:43:21
Thomas Glanzmann [off-list ref] writes:
Hello,
I narrowed it down. It is in "git checkout" if I copy the ".git" dir via
"cp -a" and type in "git checkout" it breaks. And it _only_ breaks for
subdirectories:
(faui04a) [/var/tmp/shit] git checkout
D profiles/icpc
D profiles/sithglan
D sources/boot.msg
D sources/busybox-config
D sources/config
D sources/easix.logo.16
D sources/easixdm
D sources/f2
D sources/home-faumachine.tar.gz
D sources/home-icpc.tar.gz
D sources/home-simigern.tar.gz
D sources/home-sithglan.tar.gz
D sources/home-sitowert.tar.gz
D sources/linuxrc
D sources/rc.local
and I also see that _one_ file in each subdirectory is checked out but the
others are not.
Ok, stupidly trying one of our most frequently encountered problems:
Does it help to say
export CDPATH=""
in your root shell?
--
David Kastrup
From: David Kastrup <hidden> Date: 2016-06-15 22:43:21
Thomas Glanzmann [off-list ref] writes:
Hello,
I narrowed it down. It is in "git checkout" if I copy the ".git" dir via
"cp -a" and type in "git checkout" it breaks. And it _only_ breaks for
subdirectories:
Another guess: cp -a copies files preserving all permissions and
ownerships (which works only as root). Nominally, the files now don't
belong to you, and some scripts might take you by the word even if
they _could_ just overwrite things if they actually tried. Try
making
chown -R root.root .git
on your tree and see whether this makes git more comfortable.
--
David Kastrup
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Hello David,
Does it help to say export CDPATH="" in your root shell?
no. The problem is triggered by "git-checkout-index -a -f" so my CDPATH has
absolutly _nothing_ todo with it:
(faui04a) [/var/tmp] mkdir -p test-4/.git
(faui04a) [/var/tmp] cd test-4
(faui04a) [/var/tmp/test-4] gcp -a ~/work/repositories/public/easix.git/* .git/
(faui04a) [/var/tmp/test-4] git-read-tree HEAD
(faui04a) [/var/tmp/test-4] git-checkout-index -a -f
(faui04a) [/var/tmp/test-4] git status
fatal: /usr/bin/git-status cannot be used without a working tree.
(faui04a) [/var/tmp/test-4] rm .git/config
(faui04a) [/var/tmp/test-4] git status
# On branch master
# Changed but not updated:
# (use "git add/rm <file>..." to update what will be committed)
#
# deleted: profiles/icpc
# deleted: profiles/sithglan
# deleted: sources/boot.msg
# deleted: sources/busybox-config
# deleted: sources/config
# deleted: sources/easix.logo.16
# deleted: sources/easixdm
# deleted: sources/f2
# deleted: sources/home-faumachine.tar.gz
# deleted: sources/home-icpc.tar.gz
# deleted: sources/home-simigern.tar.gz
# deleted: sources/home-sithglan.tar.gz
# deleted: sources/home-sitowert.tar.gz
# deleted: sources/linuxrc
# deleted: sources/rc.local
#
no changes added to commit (use "git add" and/or "git commit -a")
(faui04a) [/var/tmp/test-4] cd ..
(faui04a) [/var/tmp] rm -rf test-4
rm: Unable to remove directory test-4: File exists
(faui04a) [/var/tmp]
Thomas
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Hello David,
as I mentioned and showed in my original e-mail:
git checkout /path/to/repo
triggers the problem also. So no, "cp -a" has nothing todo with it
because root is able to read _all_ files no matter who the files are
owned by and what the permissions of these files are[1]. This happens on
a _local_ fileystem. And what worries me is that git does something that
breaks the _hard link counter_ of a directory.
Thomas
[1] Notes
Of course that doesn't apply to all filesystems. For example AFS
and NFS. But as I said before it happens on a _UFS_ filesystem.
From: David Kastrup <hidden> Date: 2016-06-15 22:43:21
Thomas Glanzmann [off-list ref] writes:
Hello David,
as I mentioned and showed in my original e-mail:
git checkout /path/to/repo
triggers the problem also. So no, "cp -a" has nothing todo with it
because root is able to read _all_ files no matter who the files are
owned by and what the permissions of these files are[1]. This happens on
a _local_ fileystem. And what worries me is that git does something that
breaks the _hard link counter_ of a directory.
Hm? The hard link counter is "broken" by creating or not creating
subdirectories, since their ".." is a hard link.
--
David Kastrup
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Hello David,
Hm? The hard link counter is "broken" by creating or not creating
subdirectories, since their ".." is a hard link.
exactly. The question is: Is it a Solaris bug or is it something that is
supposed a user is able to do (it doesn't make sense for me)? I posted
this problem to comp.unix.solaris and also contacted the UFS Maintainer
of Solaris (it is not the first UFS bug I original found). If I don't
receive feedback, I am going to open a call with Sun. Whatever is going
on (hopefully it isn't PEBKAC - but I don't think so) and this time it
isn't broken hardware either (like the last time I reported a serious
git bug) because it happens on two machines.
Thomas
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Hello,
might that be related to the problem:
lstat64("profiles/icpc", 0xFFBFF350) Err#2 ENOENT
=> mkdir("profiles", 0777) = 0
stat64(".git/objects/66/6197b02f46c92f0273f16ac77d34d76b28f4f0",
0xFFBFF088) = 0
open64(".git/objects/66/6197b02f46c92f0273f16ac77d34d76b28f4f0",
O_RDONLY) = 4
mmap64(0x00000000, 284, PROT_READ, MAP_PRIVATE, 4, 0) = 0xFF230000
close(4) = 0
munmap(0xFF230000, 284) = 0
open64("profiles/icpc", O_WRONLY|O_CREAT|O_EXCL, 0666) = 4
open64("profiles/.gitattributes", O_RDONLY) Err#2 ENOENT
write(4, " # I C P C P r o f i".., 420) = 420
close(4) = 0
lstat64("profiles/sithglan", 0xFFBFF350) Err#2 ENOENT
=> mkdir("profiles", 0777) Err#17 EEXIST
=> unlink("profiles") = 0
=> mkdir("profiles", 0777) = 0
I think it is. Damn it. What seems to hapen here is that git does:
- create a subdirectory
- puts a file in
- deletes a subdirectory (by call unlink - that would normally fail,
but with solaris as root it does not fail)
=> here comes the dangling hard link counter
- created the directory again
- puts the file in
That is why I only see one file in each subdirectory (the one that got
checkedout last). So the fix for git should be straight forward. But I still
think that Solaris is obviously broken. Because if you ask me it should not be
possible to unlink a directory that has files in it?!
Thomas
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Hello,
entry.c line 17:
static void create_directories(const char *path, const struct checkout *state)
{
...
if (mkdir(buf, 0777)) {
=> if (errno == EEXIST) {
struct stat st;
=> if (len > state->base_dir_len && state->force && !unlink(buf) && !mkdir(buf, 0777))
continue;
if (!stat(buf, &st) && S_ISDIR(st.st_mode))
continue; /* ok */
}
die("cannot create directory at %s", buf);
}
...
I think here is the problem. I don't understand what the code should do
actually. Or why the directory is deleted and re-created (maybe something todo
with permissions)?
Thomas
From: Johannes Sixt <hidden> Date: 2016-06-15 22:43:21
Thomas Glanzmann wrote:
static void create_directories(const char *path, const struct checkout *state)
{
...
if (mkdir(buf, 0777)) {
=> if (errno == EEXIST) {
struct stat st;
=> if (len > state->base_dir_len && state->force && !unlink(buf) && !mkdir(buf, 0777))
continue;
if (!stat(buf, &st) && S_ISDIR(st.st_mode))
continue; /* ok */
}
die("cannot create directory at %s", buf);
}
...
I think here is the problem. I don't understand what the code should do
actually. Or why the directory is deleted and re-created (maybe something todo
with permissions)?
It tries to remove a *file* that is in the way and create the directory
in its place. But since your unlink() behaves incorrectly (it is
supposed to *fail* for directories), the logic does not quite work as
expected - it mistakes the directory for a file.
Try swapping the two inner-most if-blocks.
-- Hannes
From: Brian Downing <hidden> Date: 2016-06-15 22:43:21
On Mon, Jul 16, 2007 at 04:35:10PM +0200, Johannes Sixt wrote:
It tries to remove a *file* that is in the way and create the directory
in its place. But since your unlink() behaves incorrectly (it is
supposed to *fail* for directories), the logic does not quite work as
expected - it mistakes the directory for a file.
http://www.opengroup.org/onlinepubs/007908799/xsh/unlink.html
| The path argument must not name a directory unless the process has
| appropriate privileges and the implementation supports using unlink() on
| directories.
The above seems to imply that implementations are free to do something
with directories passed to unlink, and that it is the responsibility
if the user not to do that unless they know what is going to happen.
-bcd
From: Brian Downing <hidden> Date: 2016-06-15 22:43:21
On Mon, Jul 16, 2007 at 10:45:59AM -0500, Brian Downing wrote:
http://www.opengroup.org/onlinepubs/007908799/xsh/unlink.html
| The path argument must not name a directory unless the process has
| appropriate privileges and the implementation supports using unlink() on
| directories.
The above seems to imply that implementations are free to do something
with directories passed to unlink, and that it is the responsibility
if the user not to do that unless they know what is going to happen.
Also, the HISTORY section of the FreeBSD manpages states:
| The unlink system call traditionally allows the super-user to unlink
| directories which can damage the filesystem integrity. This
| implementation no longer permits it.
Great.
-bcd
From: David Kastrup <hidden> Date: 2016-06-15 22:43:21
Thomas Glanzmann [off-list ref] writes:
Hello,
might that be related to the problem:
lstat64("profiles/icpc", 0xFFBFF350) Err#2 ENOENT
=> mkdir("profiles", 0777) = 0
stat64(".git/objects/66/6197b02f46c92f0273f16ac77d34d76b28f4f0",
0xFFBFF088) = 0
open64(".git/objects/66/6197b02f46c92f0273f16ac77d34d76b28f4f0",
O_RDONLY) = 4
mmap64(0x00000000, 284, PROT_READ, MAP_PRIVATE, 4, 0) = 0xFF230000
close(4) = 0
munmap(0xFF230000, 284) = 0
open64("profiles/icpc", O_WRONLY|O_CREAT|O_EXCL, 0666) = 4
open64("profiles/.gitattributes", O_RDONLY) Err#2 ENOENT
write(4, " # I C P C P r o f i".., 420) = 420
close(4) = 0
lstat64("profiles/sithglan", 0xFFBFF350) Err#2 ENOENT
=> mkdir("profiles", 0777) Err#17 EEXIST
=> unlink("profiles") = 0
=> mkdir("profiles", 0777) = 0
I think it is. Damn it. What seems to hapen here is that git does:
- create a subdirectory
- puts a file in
- deletes a subdirectory (by call unlink - that would normally fail,
but with solaris as root it does not fail)
=> here comes the dangling hard link counter
- created the directory again
- puts the file in
That is why I only see one file in each subdirectory (the one that got
checkedout last). So the fix for git should be straight forward. But I still
think that Solaris is obviously broken. Because if you ask me it should not be
possible to unlink a directory that has files in it?!
<URL:http://www.opengroup.org/onlinepubs/000095399/functions/unlink.html>
The path argument shall not name a directory unless the process has
appropriate privileges and the implementation supports using unlink()
on directories.
So Solaris has the right to do this.
APPLICATION USAGE
Applications should use rmdir() to remove a directory.
RATIONALE
Unlinking a directory is restricted to the superuser in many
historical implementations for reasons given in link() (see
also rename()).
In short: git should not call remove, ever. It may succeed, and is a
badly low-level call. If something is known to be a directory, then
it needs to be removed using rmdir, and if it is a nondirectory, with
unlink.
Hm, browsing through Posix indicates that unlink is probably the same
as remove. Pity. I thought that just "remove" was the potential
evildoer.
--
David Kastrup
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Hello,
exactly. The question is: Is it a Solaris bug or is it something that is
supposed a user is able to do (it doesn't make sense for me)? I posted
this problem to comp.unix.solaris and also contacted the UFS Maintainer
of Solaris (it is not the first UFS bug I original found). If I don't
receive feedback, I am going to open a call with Sun. Whatever is going
on (hopefully it isn't PEBKAC - but I don't think so) and this time it
isn't broken hardware either (like the last time I reported a serious
git bug) because it happens on two machines.
below I quote the answer of the UFS maintainer:
Thomas,
some notes on the unlink(directory) part:
In the case of calling unlink(2) on a directory as a privilidged user, things
like ".." (and the link count in the parent) would not be cleaned up.
The man page for unlink(1M) and unlink(2) is very clear about using rmdir(1/2) to
remove a directory. It's also very clear that this does exactly what
it's supposed to do: Remove a link to a file/directory. Cleaning up
the ".." entry (which would clean up the link to the parent) could
be considered "extra".
First the standards issues:
The link(BA_OS) and unlink(BA_OS) descriptions in SVID3 both specify that
a process with appropriate privileges is allowed to operate on a directory.
We have claimed to conform to SVID3 since Solaris 2.0 and have not announced
that we ever plan to EOL SVID3 conformance.
UFS does support link(2)/unlink(2) with appropriate privilidges of
directories while ZFS does not.
The change that would provide ``Cleaning up the ".." entry ...'' would
violate both SVID3 and POSIX and SUS requirements. (The unlink(path)
system call is supposed to remove the directory entry named by path; not
the directory entry named by path and an arbitrary number of other
directory entries.)
Second some history:
Back before there were mkdir() and rmdir() system calls, applications
wanting to create a directory invoked the setuid root mkdir utility
which performed a series of mknod() and link() operations to create the
directory and create dot and dot-dot entries in the new directory.
Similarly, applications wanting to remove a directory invoked the
setuid root rmdir uility which performed a series of unlink()
operations to destroy the directory if it was empty.
Not quite so far back in ancient history, there is also the point
that before symlinks were available it was common practice to make
hard links to directories. Privileged applications can still do this
today (as mandated by SVID) using the link() and unlink() system calls
and the link and unlink commands.
UFS and ZFS are actually both correct, the standards wording permits either way kind of:
<snip>
If path1 names a directory, link() shall fail unless the process has
appropriate privileges and the implementation supports using link() on directories.
The path argument shall not name a directory unless the process has
appropriate privileges and the implementation supports using unlink() on directories.
<snip end>
This seems to implicitely allow an escape route; we can declare that no process
has sufficient privileges to link(2)/unlink(2) directories or like ZFS did,
have the underlaying implementation not supporting it at all.
The proper way is for the application to use the rmdir(2) system call,
it's there since ages!
my bottom line: GIT should not call unlink on a directory.
Thomas
I think it is. Damn it. What seems to hapen here is that git does:
- create a subdirectory
- puts a file in
- deletes a subdirectory (by call unlink - that would normally fail,
but with solaris as root it does not fail)
Solaris is apparently a buggy piece of crap.
I'd suggest making a bug-report. ENOTEMPTY is not an optional error. Posix
says "Shall fail".
Linus
Solaris is apparently a buggy piece of crap.
I'd suggest making a bug-report. ENOTEMPTY is not an optional error. Posix
says "Shall fail".
Oh, it was apparetly "unlink()" that corrupts filesystems. Which isn't
against Posix, it's just stupid.
The "unlink directories" is so old-fashioned that it's not even funny. The
whole reason "rmdir()" was created was to *not* have that behaviour in the
OS.
So I'll have to correct my statement. Please remove the "buggy" from
"buggy piece of crap". The rest stands.
Linus
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Calling unlink on a directory on a Solaris UFS filesystem as root makes it
inconsistent. Thanks to Johannes Sixt for the obvious fix.
---
entry.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
From: Thomas Glanzmann <hidden> Date: 2016-06-15 22:43:21
Calling unlink on a directory on a Solaris UFS filesystem as root makes it
inconsistent. Thanks to Johannes Sixt for the obvious fix.
Signed-off-by: Thomas Glanzmann <redacted>
---
entry.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
On Mon, 2007-07-16 19:38:41 +0200, Thomas Glanzmann [off-list ref] wrote:
quoted hunk
Calling unlink on a directory on a Solaris UFS filesystem as root makes it
inconsistent. Thanks to Johannes Sixt for the obvious fix.
Signed-off-by: Thomas Glanzmann <redacted>
---
entry.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
while ((slash = strchr(slash+1, '/')) != NULL) {
len = slash - path;
memcpy(buf, path, len);
buf[len] = 0;
+ if (!stat(buf, &st) && S_ISDIR(st.st_mode))
+ continue; /* ok */
Dito.
if (mkdir(buf, 0777)) {
if (errno == EEXIST) {
- struct stat st;
if (len > state->base_dir_len && state->force && !unlink(buf) && !mkdir(buf, 0777))
continue;
- if (!stat(buf, &st) && S_ISDIR(st.st_mode))
- continue; /* ok */
}
die("cannot create directory at %s", buf);
}
MfG, JBG
--
Jan-Benedict Glaw jbglaw@lug-owl.de +49-172-7608481
Signature of: "really soon now": an unspecified period of time, likly to
the second : be greater than any reasonable definition
of "soon".
From: David Kastrup <hidden> Date: 2016-06-15 22:43:21
bdowning@lavos.net (Brian Downing) writes:
On Mon, Jul 16, 2007 at 04:35:10PM +0200, Johannes Sixt wrote:
quoted
It tries to remove a *file* that is in the way and create the directory
in its place. But since your unlink() behaves incorrectly (it is
supposed to *fail* for directories), the logic does not quite work as
expected - it mistakes the directory for a file.
Isn't it funny? The problem with the git code is that the path
argument must not name a directory _if_ the process has appropriate
privileges and the implementation supports using unlink() on
directories.
Failure is not an option, it is a requirement.
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
Calling unlink on a directory on a Solaris UFS filesystem as root makes it
inconsistent. Thanks to Johannes Sixt for the obvious fix.
Ack, I think this is the right thing to do.
As pointed out, it doesn't _guarantee_ that git won't call "unlink()" on a
directory (race conditions etc), but that's fundamentally true (there is
no "funlink()" like there is "fstat()"), and besides, that is in no way
git-specific (ie it's true of *any* application that gets run as root).
The theoretical race would only happen if somebody on purpose tries to
screw things over, it would never happen under any reasonable usage.
The old ordering of those tests was designed for sane operating systems,
so that you could basically do the unlink() without bothering, but
switching the order around is certainly not a disaster either, and if it
avoids the nasty bug in Solaris it's worth doing.
I have to say that I'm still a bit shocked that Solaris would have that
kind of behaviour. And they call that pile of sh*t "enterprise class"..
Linus
Apparently "enterprise class" in this case really means "fully
compatibility with all those wonderful userland implementations of
rmdir." :-)
I see the smiley, but it's actually not possible even for that.
The broken "unlink()" behaviour doesn't work on any other filesystem (eg
NFS) at all anyway, and even on UFS would only work for root (or
setuid-root) binaries. So any user-land that depended on it literally
wouldn't work _anyway_, even on Solaris itself.
And we're talking about the same company that was *famous* for screwing
people over when they converted from SunOS to Solaris and broke binaries
_and_ source code in the process.
So no, "compatibility" can't realistically be the reason.
Linus
From: David Kastrup <hidden> Date: 2016-06-15 22:43:21
Linus Torvalds [off-list ref] writes:
On Mon, 16 Jul 2007, Thomas Glanzmann wrote:
quoted
Calling unlink on a directory on a Solaris UFS filesystem as root makes it
inconsistent. Thanks to Johannes Sixt for the obvious fix.
Ack, I think this is the right thing to do.
As pointed out, it doesn't _guarantee_ that git won't call
"unlink()" on a directory (race conditions etc), but that's
fundamentally true (there is no "funlink()" like there is
"fstat()"), and besides, that is in no way git-specific (ie it's
true of *any* application that gets run as root).
Please note that doing "remove" before "mkdir" without checking for
directoriness still offers a race window where one can slip in a new
non-directory file.
--
David Kastrup