Jeff King [off-list ref] writes:
The ustar format only has room for 11 (or 12, depending on
some implementations) octal digits for the size and mtime of
each file. After this, we have to add pax extended headers
to specify the real data, and git does not yet know how to
do so.
I am not a native speaker but "After" above made me hiccup. I think
I am correct to understand that it means "after passing this limit",
aka "to represent files bigger or newer than these", but still it
felt somewhat strange.
So as a prerequisite, we can feed the system tar a reference
tarball to make sure it can handle these features. The
reference tar here was created with:
dd if=/dev/zero seek=64G bs=1 count=1 of=huge
touch -d @68719476737 huge
tar cf - --format=pax |
head -c 2048
using GNU tar. Note that this is not a complete tarfile, but
it's enough to contain the headers we want to examine.
Cute. I didn't remember they had @<seconds-since-epoch> format,
even though I must have seen what they do while working on 2c733fb2
(parse_date(): '@' prefix forces git-timestamp, 2012-02-02).
+# See if our system tar can handle a tar file with huge sizes and dates far in
+# the future, and that we can actually parse its output.
+#
+# The reference file was generated by GNU tar, and the magic time and size are
+# both octal 01000000000001, which overflows normal ustar fields.
+#
+# When parsing, we'll pull out only the year from the date; that
+# avoids any question of timezones impacting the result.
... as long as the month-day part is not close to the year boundary.
So this explanation is insuffucient to convince the reader that
"that avoids any question" is correct, without saying that it is in
August of year 4147.
+tar_info () {
+ "$TAR" tvf "$1" | awk '{print $3 " " $4}' | cut -d- -f1
+}
A blank after the shell function to make it easier to see the
boundary.
Seeing an awk piped into cut always makes me want to suggest a
single sed/awk/perl invocation.
+# We expect git to die with SIGPIPE here (otherwise we
+# would generate the whole 64GB).
+test_expect_failure BUNZIP 'generate tar with huge size' '
+ {
+ git archive HEAD
+ echo $? >exit-code
+ } | head -c 4096 >huge.tar &&
+ echo 141 >expect &&
+ test_cmp expect exit-code
+'
"head -c" is GNU-ism, isn't it?
"dd bs=1 count=4096" is hopefully more portable.
ksh signal death you already know about. I wonder if we want to
expose something like list_contains as a friend of test_cmp.
list_contains 141,269 $(cat exit-code)
Thanks.
On Fri, Jun 24, 2016 at 11:56:19AM -0700, Junio C Hamano wrote:
Jeff King [off-list ref] writes:
quoted
The ustar format only has room for 11 (or 12, depending on
some implementations) octal digits for the size and mtime of
each file. After this, we have to add pax extended headers
to specify the real data, and git does not yet know how to
do so.
I am not a native speaker but "After" above made me hiccup. I think
I am correct to understand that it means "after passing this limit",
aka "to represent files bigger or newer than these", but still it
felt somewhat strange.
Yeah, I agree that it reads badly. I'm not sure what I was thinking.
I'll tweak it in the re-roll.
quoted
+# See if our system tar can handle a tar file with huge sizes and dates far in
+# the future, and that we can actually parse its output.
+#
+# The reference file was generated by GNU tar, and the magic time and size are
+# both octal 01000000000001, which overflows normal ustar fields.
+#
+# When parsing, we'll pull out only the year from the date; that
+# avoids any question of timezones impacting the result.
... as long as the month-day part is not close to the year boundary.
So this explanation is insuffucient to convince the reader that
"that avoids any question" is correct, without saying that it is in
August of year 4147.
I thought that part didn't need to be said, but I can say it
(technically we can include the month, too, but I don't think that level
of accuracy is really important for these tests).
quoted
+tar_info () {
+ "$TAR" tvf "$1" | awk '{print $3 " " $4}' | cut -d- -f1
+}
A blank after the shell function to make it easier to see the
boundary.
I was intentionally trying to couple it with prereq below, as the
comment describes both of them.
Seeing an awk piped into cut always makes me want to suggest a
single sed/awk/perl invocation.
I want the auto-splitting of awk, but then to auto-split the result
using a different delimiter. Is there a not-painful way to do that in
awk?
I could certainly come up with a regex to do it in sed, but I wanted to
keep the parsing as liberal and generic as possible.
Certainly I could do it in perl, but I had the general impression that
we prefer to keep the dependency on perl to a minimum. Maybe it doesn't
matter.
quoted
+# We expect git to die with SIGPIPE here (otherwise we
+# would generate the whole 64GB).
+test_expect_failure BUNZIP 'generate tar with huge size' '
+ {
+ git archive HEAD
+ echo $? >exit-code
+ } | head -c 4096 >huge.tar &&
+ echo 141 >expect &&
+ test_cmp expect exit-code
+'
"head -c" is GNU-ism, isn't it?
You're right; for some reason I thought it was in POSIX.
We do have a couple instances of it, but they are all in the valgrind
setup code (which I guess most people don't ever run).
"dd bs=1 count=4096" is hopefully more portable.
Hmm. I always wonder whether dd is actually very portable, but we do use
it already, at least.
Perhaps the perl monstrosity in t9300 could be replaced with that, too.
ksh signal death you already know about. I wonder if we want to
expose something like list_contains as a friend of test_cmp.
list_contains 141,269 $(cat exit-code)
I think we would want something more like:
test_signal_match 13 $(cat exit-code)
Each call site should not have to know about every signal convention
(and in your example, the magic "3" of Windows is left out).
-Peff
On Fri, Jun 24, 2016 at 3:07 PM, Jeff King [off-list ref] wrote:
On Fri, Jun 24, 2016 at 11:56:19AM -0700, Junio C Hamano wrote:
quoted
Jeff King [off-list ref] writes:
quoted
+tar_info () {
+ "$TAR" tvf "$1" | awk '{print $3 " " $4}' | cut -d- -f1
+}
quoted
Seeing an awk piped into cut always makes me want to suggest a
single sed/awk/perl invocation.
I want the auto-splitting of awk, but then to auto-split the result
using a different delimiter. Is there a not-painful way to do that in
awk?
The awk split() function is POSIX and accepts an optional separator argument.
On Fri, Jun 24, 2016 at 03:07:44PM -0400, Jeff King wrote:
quoted
"dd bs=1 count=4096" is hopefully more portable.
Hmm. I always wonder whether dd is actually very portable, but we do use
it already, at least.
Perhaps the perl monstrosity in t9300 could be replaced with that, too.
Hrm. So I wrote a patch for t9300 for this. But I wanted to flip the
order to:
dd bs=4096 count=1
because otherwise, dd will call read() 4096 times, for 1 byte each.
But it's not safe to do that on a pipe. For example:
{
echo 1
sleep 1
echo 2
} | dd bs=4 count=1
will copy only 2 bytes. So it's racily wrong, depending on how the
writer feeds the data to write().
The 1-byte reads do work (assuming blocking descriptors and that dd
restarts a read after a signal, which mine seems to). But yuck.
The difference in time between the two is measurable on my system, but
it's only a few milliseconds (for 4096 bytes). So maybe it's not worth
worrying about (though as a general technique, it does make me worry
that it's easy to get wrong in a way that will fail racily).
-Peff
On Fri, Jun 24, 2016 at 04:58:58PM -0400, Eric Sunshine wrote:
On Fri, Jun 24, 2016 at 3:07 PM, Jeff King [off-list ref] wrote:
quoted
On Fri, Jun 24, 2016 at 11:56:19AM -0700, Junio C Hamano wrote:
quoted
Jeff King [off-list ref] writes:
quoted
+tar_info () {
+ "$TAR" tvf "$1" | awk '{print $3 " " $4}' | cut -d- -f1
+}
quoted
Seeing an awk piped into cut always makes me want to suggest a
single sed/awk/perl invocation.
I want the auto-splitting of awk, but then to auto-split the result
using a different delimiter. Is there a not-painful way to do that in
awk?
The awk split() function is POSIX and accepts an optional separator argument.
Thanks. I'm not that familiar with awk functions, simply because I came
of age after perl existed, and using perl tends to be more portable and
powerful (if you can assume it's available). But this is simple enough
that it should be OK.
Replacing it with:
"$TAR" tvf "$1" |
awk '{
split($4, date, "-")
print $3 " " date[1]
}'
seems to work.
-Peff