Junio, Brian,
it seems that the stability of the "git tar" output is broken.
On Mon, Oct 20, 2014 at 4:59 AM, Konstantin Ryabitsev
[off-list ref] wrote:
Looks like 3.18-rc1 upload didn't work:
This is why the front page still lists 3.17 as the latest mainline. Want
to try again?
Ok, tried again, and failed again.
If that still doesn't work, you may have to use version 1.7 of git when
generating the tarball and signature -- I recall Greg having a similar
problem in the past.
Ugh, yes, that seems to be it. Current git generates different
tar-files than older releases do:
tar-1.7.9.7 tar-cur differ: byte 107, line 1
and a quick bisection shows that it is due to commit 10f343ea814f
("archive: honor tar.umask even for pax headers") in the current git
development version.
Junio, quite frankly, I don't think that that fix was a good idea. I'd
suggest having a *separate* umask for the pax headers, so that we do
not break this long-lasting stability of "git archive" output in ways
that are unfixable and not compatible. kernel.org has relied (for a
*long* time) on being able to just upload the signature of the
resulting tar-file, because both sides can generate the same tar-fiel
bit-for-bit.
So instead of using "tar_umask", please make it use "tar_pax_umask",
and have that default to 000. Ok?
Something like the attached patch.
Or just revert 10f343ea814f entirely.
Linus
From: brian m. carlson <hidden> Date: 2016-06-15 23:02:45
On Mon, Oct 20, 2014 at 08:25:59AM -0700, Linus Torvalds wrote:
Junio, Brian,
it seems that the stability of the "git tar" output is broken.
It doesn't appear that the stability of git archive --format=tar is
documented anywhere. Given that, it doesn't seem reasonable to expect
that any tar implementation produces bit-for-bit compatible output
between versions. After all, look at all the contortions that Debian
has had to go through to keep pristine-tar working.
Junio, quite frankly, I don't think that that fix was a good idea. I'd
suggest having a *separate* umask for the pax headers, so that we do
not break this long-lasting stability of "git archive" output in ways
that are unfixable and not compatible. kernel.org has relied (for a
*long* time) on being able to just upload the signature of the
resulting tar-file, because both sides can generate the same tar-fiel
bit-for-bit.
It sounds like kernel.org has a bug, then. Perhaps that's the
appropriate place to fix the issue.
The issue I fixed is that leaving world-writable files around on disk is
a great way for people to cause mischief (for example, by filling up
other users' quotas), and some tar implementations and all Linux pax
implementations extract the pax headers into the working directory, and
that's often /tmp.
--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187
On Mon, Oct 20, 2014 at 3:28 PM, brian m. carlson
[off-list ref] wrote:
It doesn't appear that the stability of git archive --format=tar is
documented anywhere. Given that, it doesn't seem reasonable to expect
that any tar implementation produces bit-for-bit compatible output
between versions.
The kernel has simple stability rules: if it breaks users, it gets
fixed or reverted. That is a damn good rule.
I realize that some other projects are crap, and don't care about
their users. I hope and believe that git is not in that sad group.
The whole "it's not documented" excuse is pure and utter bollocks.
Users don't care. And stability of data should be *expected*, not need
some random documentation entry to make it explicit.
Linus
From: Konstantin Ryabitsev <hidden> Date: 2016-06-15 23:02:45
On 20/10/14 06:28 PM, brian m. carlson wrote:
quoted
Junio, quite frankly, I don't think that that fix was a good idea. I'd
quoted
suggest having a *separate* umask for the pax headers, so that we do
not break this long-lasting stability of "git archive" output in ways
that are unfixable and not compatible. kernel.org has relied (for a
*long* time) on being able to just upload the signature of the
resulting tar-file, because both sides can generate the same tar-fiel
bit-for-bit.
It sounds like kernel.org has a bug, then. Perhaps that's the
appropriate place to fix the issue.
It's not a bug, it's a feature (TM). KUP relies on git-archive's ability
to create identical tar archives across platforms and versions. The
benefit is that Linus or Greg can create a detached PGP signature
against a tarball created from "git archive [tag]" on their system, and
just tell kup to create the same archive remotely, thus saving them the
trouble of uploading 80Mb each time they cut a release.
With their frequent travel to places where upload bandwidth is both slow
and unreliable, this ability to not have to upload hundreds of Mbs each
time they cut a release is very handy and certainly helps keep kernel
releases on schedule.
So, while it's fair to point out that git-archive was never intended to
always create bit-for-bit identical outputs, it would be *very nice* if
this remained in place, as at least one large-ish deployment (us) finds
it really handy.
-K
From: Michael J Gruber <hidden> Date: 2016-06-15 23:02:45
Linus Torvalds schrieb am 21.10.2014 um 01:17:
On Mon, Oct 20, 2014 at 3:28 PM, brian m. carlson
[off-list ref] wrote:
quoted
It doesn't appear that the stability of git archive --format=tar is
documented anywhere. Given that, it doesn't seem reasonable to expect
that any tar implementation produces bit-for-bit compatible output
between versions.
The kernel has simple stability rules: if it breaks users, it gets
fixed or reverted. That is a damn good rule.
I realize that some other projects are crap, and don't care about
their users. I hope and believe that git is not in that sad group.
The whole "it's not documented" excuse is pure and utter bollocks.
Users don't care. And stability of data should be *expected*, not need
some random documentation entry to make it explicit.
Linus
Linus, with all due respect, this is not the LKML, so please watch your
tone over here on the git list (and keep ranting on LKML however you want).
Brian made a very valid point about what his patch was trying to fix -
after all that is why it was applied. Konstantin made a very valid point
about why the existing behavior is useful for KUP. Interestingly, both
cared about the users of git, just different kinds users.
Git is probably one of the most conservative projects regarding
backwards compatibility and heeding users' expectations (sometimes to my
own dismay). That being said, we distinguish between justified
expectations and those without a solid base - which is why we have
porcelain vs. plumbing, for example, to make clear which part of the ui
is stable. (Yeah, I know you know, but you didn't argue as if you did.)
"data" in git is stable. "data exports" by git are as stable as the
output format is intrinsically or due to the (hopefully documented) way
git produces it.
Unfortunately, the git archive doc clearly says that the umask is
applied to all archive entries. And that clearly wasn't the case (for
extended metadata headers) before Brian's fix.
Brian: How old is the newest tar that get's the extended metadata
headers wrong? If those tars are a "real concern" then we should
probably do the extra pax_umask as suggested by Linus, but have the
default protect the "unknowing users" and give the "knowing users" that
config knob to twitch (sorry, Linus). Otherwise a revert is in order.
Michael
On Tue, Oct 21, 2014 at 1:08 AM, Michael J Gruber
[off-list ref] wrote:
Unfortunately, the git archive doc clearly says that the umask is
applied to all archive entries. And that clearly wasn't the case (for
extended metadata headers) before Brian's fix.
Hey, it's time for another round of the world-famous "Captain Obvious
Quiz Game"! Yay!
The questions these week are:
(1) "If reality and documentation do not match, where is the bug?"
(a) Documentation is buggy
(b) Reality is buggy
(2) "Where would you put the horse in relationship to a horse-drawn carriage?"
(a) in front
(b) in the carriage
Now, if you answered (a) to both these questions, and had this been a
real quiz show, you might have been a winner and the happy new owner
of a remote-controlled four-slice toaster with a fancy digital timer.
Sadly, this was just a dry-run for the real thing, to give people a
quick taste of the world-famous "Captain Obvious Quiz Game". I hope
you tune in next week for our exciting all-new questions.
Linus
From: Junio C Hamano <hidden> Date: 2016-06-15 23:02:45
Michael J Gruber [off-list ref] writes:
Unfortunately, the git archive doc clearly says that the umask is
applied to all archive entries.
Is an extended pax header "an archive entry"? I doubt it, and the
above is not relevant. The mode bits for the archive entry that it
applies to does not come from there.
See my other message for my final judgement on this one. I wouldn't
have minded if the original used the same umask for those ignored
mode bits, but changing the bits to be ignored after the fact is not
helping any real use case and only hurts existing users.
That is not to say that we cannot later fix bigger issues in the
output. I just do not see that otherwise-unused mode bits in the
extended pax header big enough an issue to spend brain cycles to
carefully lay and execute transition plans to avoid breaking
existing users.
From: Michael J Gruber <hidden> Date: 2016-06-15 23:02:46
Junio C Hamano schrieb am 21.10.2014 um 20:14:
Michael J Gruber [off-list ref] writes:
quoted
Unfortunately, the git archive doc clearly says that the umask is
applied to all archive entries.
Is an extended pax header "an archive entry"? I doubt it, and the
above is not relevant. The mode bits for the archive entry that it
applies to does not come from there.
The problem seem to be old tar versions which mis-take the extensions
for archive entries, aren't they?
See my other message for my final judgement on this one. I wouldn't
have minded if the original used the same umask for those ignored
mode bits, but changing the bits to be ignored after the fact is not
helping any real use case and only hurts existing users.
That is not to say that we cannot later fix bigger issues in the
output. I just do not see that otherwise-unused mode bits in the
extended pax header big enough an issue to spend brain cycles to
carefully lay and execute transition plans to avoid breaking
existing users.
My question to Brian still stands which existing users he was trying to
cater for with his patch. If there indeed are no existing affected users
besides the KUP users (as you seem to assume) it's a clear case. Pun
intended ;)
As I pointed out (and you cut out), I don't mind doing the revert. I
just want us to do the right things for the right reasons (the ones you
ponted out, Junio).
Michael
From: brian m. carlson <hidden> Date: 2016-06-15 23:02:46
On Wed, Oct 22, 2014 at 11:42:48AM +0200, Michael J Gruber wrote:
Junio C Hamano schrieb am 21.10.2014 um 20:14:
quoted
Michael J Gruber [off-list ref] writes:
quoted
Unfortunately, the git archive doc clearly says that the umask is
applied to all archive entries.
Is an extended pax header "an archive entry"? I doubt it, and the
above is not relevant. The mode bits for the archive entry that it
applies to does not come from there.
The problem seem to be old tar versions which mis-take the extensions
for archive entries, aren't they?
Yes. POSIX isn't clear on how unknown entries are to be handled. I've
seen some Windows tar implementations extract GNU longlink extensions as
files, which leads to a lot of pain.
My question to Brian still stands which existing users he was trying to
cater for with his patch. If there indeed are no existing affected users
besides the KUP users (as you seem to assume) it's a clear case. Pun
intended ;)
The pax format is an extension of the tar format. All of the pax
implementations I've seen on Linux (OpenBSD's and MirBSD's) don't
actually understand the pax headers and emit them as files. 7zip does
as well. I expect there are other Unix systems where tar itself doesn't
understand pax headers, although I don't have access to anything other
than Linux and FreeBSD.
Since it's very common to extract tar archives in /tmp, I didn't want to
leave world-writable files in /tmp (or anywhere else someone might get
to them). While the contents probably aren't sensitive, a malicious
user might fill someone's quota by "helpfully" appending /dev/zero to
the file. And yes, users do these things.
--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187
From: René Scharfe <hidden> Date: 2016-06-15 23:02:47
Am 23.10.2014 um 03:09 schrieb brian m. carlson:
On Wed, Oct 22, 2014 at 11:42:48AM +0200, Michael J Gruber wrote:
quoted
Junio C Hamano schrieb am 21.10.2014 um 20:14:
quoted
Michael J Gruber [off-list ref] writes:
quoted
Unfortunately, the git archive doc clearly says that the umask is
applied to all archive entries.
Is an extended pax header "an archive entry"? I doubt it, and the
above is not relevant. The mode bits for the archive entry that it
applies to does not come from there.
The problem seem to be old tar versions which mis-take the extensions
for archive entries, aren't they?
Yes. POSIX isn't clear on how unknown entries are to be handled. I've
seen some Windows tar implementations extract GNU longlink extensions as
files, which leads to a lot of pain.
That's by design -- extended headers are meant to be extracted as plain
files by implementations that do not understand them.
http://pubs.opengroup.org/onlinepubs/009695399/utilities/pax.html says:
"If a particular implementation does not recognize the type, or the user
does not have appropriate privilege to create that type, the file shall
be extracted as if it were a regular file if the file type is defined to
have a meaning for the size field that could cause data logical records
to be written on the medium [...]."
quoted
My question to Brian still stands which existing users he was trying to
cater for with his patch. If there indeed are no existing affected users
besides the KUP users (as you seem to assume) it's a clear case. Pun
intended ;)
The pax format is an extension of the tar format. All of the pax
implementations I've seen on Linux (OpenBSD's and MirBSD's) don't
actually understand the pax headers and emit them as files. 7zip does
as well. I expect there are other Unix systems where tar itself doesn't
understand pax headers, although I don't have access to anything other
than Linux and FreeBSD.
NetBSD's tar does as well.
It's surprising and sad to see *pax* implementations not supporting pax
extended headers in 2014, though. It seems long file names etc. are not
common enough. Or perhaps pax is simply not used that much.
Since it's very common to extract tar archives in /tmp, I didn't want to
leave world-writable files in /tmp (or anywhere else someone might get
to them). While the contents probably aren't sensitive, a malicious
user might fill someone's quota by "helpfully" appending /dev/zero to
the file. And yes, users do these things.
The extracted files are only world-writable if umask & 2 == 0 or if -p
(preserve permissions) has been used, no?
René
From: brian m. carlson <hidden> Date: 2016-06-15 23:02:47
On Sun, Oct 26, 2014 at 07:59:55PM +0100, René Scharfe wrote:
Am 23.10.2014 um 03:09 schrieb brian m. carlson:
quoted
The pax format is an extension of the tar format. All of the pax
implementations I've seen on Linux (OpenBSD's and MirBSD's) don't
actually understand the pax headers and emit them as files. 7zip does
as well. I expect there are other Unix systems where tar itself doesn't
understand pax headers, although I don't have access to anything other
than Linux and FreeBSD.
NetBSD's tar does as well.
It's surprising and sad to see *pax* implementations not supporting pax
extended headers in 2014, though. It seems long file names etc. are not
common enough. Or perhaps pax is simply not used that much.
The original pax utility didn't specify the pax format, only cpio and
ustar. The pax format was first release in POSIX 1003.1-2001.
quoted
Since it's very common to extract tar archives in /tmp, I didn't want to
leave world-writable files in /tmp (or anywhere else someone might get
to them). While the contents probably aren't sensitive, a malicious
user might fill someone's quota by "helpfully" appending /dev/zero to
the file. And yes, users do these things.
The extracted files are only world-writable if umask & 2 == 0 or if -p
(preserve permissions) has been used, no?
Yes, unless you're the superuser, in which case that's the default.
--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187