testsuite problems

hello,

i was running 'make test' on v1.5.3.1-109-gacd6917 and i hit the
following issue:

*   ok 6: validate file modification time
*   ok 7: git get-tar-commit-id
*   ok 8: extract tar archive
*   ok 9: validate filenames
*   ok 10: validate file contents
*   ok 11: git tar-tree with prefix
*   ok 12: extract tar archive with prefix
*   ok 13: validate filenames with prefix
*   ok 14: validate file contents with prefix
*   ok 15: create an archive with a substfile
*   ok 16: extract substfile
*   ok 17: validate substfile contents
*   ok 18: git archive --format=zip
*   ok 19: extract ZIP archive
*   ok 20: validate filenames
* FAIL 21: validate file contents
        diff -r a d/a
*   ok 22: git archive --format=zip with prefix
*   ok 23: extract ZIP archive with prefix
*   ok 24: validate filenames with prefix
* FAIL 25: validate file contents with prefix
        diff -r a e/prefix/a
*   ok 26: git archive --list outside of a git repo
* failed 2 among 26 test(s)
make[1]: *** [t5000-tar-tree.sh] Error 1
make[1]: Leaving directory `/home/vmiklos/git/git/t'
make: *** [test] Error 2

$ tar --version
tar (GNU tar) 1.18

$ diff --version
diff (GNU diffutils) 2.8.1

maybe these versions are too new or old? if i missed any needed info,
please let me know.

thanks,
- VMiklos

On Mon, Sep 17, 2007 at 09:17:42PM +0000, Miklos Vajna wrote:

hello,

i was running 'make test' on v1.5.3.1-109-gacd6917 and i hit the
following issue:

[...]

* FAIL 21: validate file contents
        diff -r a d/a
*   ok 22: git archive --format=zip with prefix
*   ok 23: extract ZIP archive with prefix
*   ok 24: validate filenames with prefix
* FAIL 25: validate file contents with prefix
        diff -r a e/prefix/a
*   ok 26: git archive --list outside of a git repo
* failed 2 among 26 test(s)
make[1]: *** [t5000-tar-tree.sh] Error 1
make[1]: Leaving directory `/home/vmiklos/git/git/t'
make: *** [test] Error 2

  Well, it works for me:

$ git describe && git st
v1.5.3.1-109-gacd6917
# Not currently on any branch.
nothing to commit (working directory clean)

$ cd t && ./t5000-tar-tree.sh
*   ok 1: populate workdir
*   ok 2: add files to repository
*   ok 3: git archive
*   ok 4: git tar-tree
*   ok 5: git archive vs. git tar-tree
*   ok 6: validate file modification time
*   ok 7: git get-tar-commit-id
*   ok 8: extract tar archive
*   ok 9: validate filenames
*   ok 10: validate file contents
*   ok 11: git tar-tree with prefix
*   ok 12: extract tar archive with prefix
*   ok 13: validate filenames with prefix
*   ok 14: validate file contents with prefix
*   ok 15: create an archive with a substfile
*   ok 16: extract substfile
*   ok 17: validate substfile contents
*   ok 18: git archive --format=zip
*   ok 19: extract ZIP archive
*   ok 20: validate filenames
*   ok 21: validate file contents
*   ok 22: git archive --format=zip with prefix
*   ok 23: extract ZIP archive with prefix
*   ok 24: validate filenames with prefix
*   ok 25: validate file contents with prefix
*   ok 26: git archive --list outside of a git repo
* passed all 26 test(s)

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

Miklos Vajna schrieb:

*   ok 18: git archive --format=zip
*   ok 19: extract ZIP archive
*   ok 20: validate filenames
* FAIL 21: validate file contents
        diff -r a d/a
*   ok 22: git archive --format=zip with prefix
*   ok 23: extract ZIP archive with prefix
*   ok 24: validate filenames with prefix
* FAIL 25: validate file contents with prefix
        diff -r a e/prefix/a
*   ok 26: git archive --list outside of a git repo
* failed 2 among 26 test(s)
make[1]: *** [t5000-tar-tree.sh] Error 1
make[1]: Leaving directory `/home/vmiklos/git/git/t'
make: *** [test] Error 2

$ tar --version
tar (GNU tar) 1.18

$ diff --version
diff (GNU diffutils) 2.8.1

maybe these versions are too new or old? if i missed any needed info,
please let me know.

The failing tests check ZIP file creation, not tar file creation.
Perhaps your unzip command works a bit differently from Info-ZIP's?

Also, what is the difference between t/trash/a and t/trash/d/a after
running the test script (that's what test 21 is comparing)?

René

On Mon, Sep 17, 2007 at 11:50:32PM +0200, René Scharfe [off-list ref] wrote:

The failing tests check ZIP file creation, not tar file creation.
Perhaps your unzip command works a bit differently from Info-ZIP's?

$ zip -v
Copyright (c) 1990-2006 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 2.32 (June 19th 2006), by Info-ZIP.

should this version be ok?

Also, what is the difference between t/trash/a and t/trash/d/a after
running the test script (that's what test 21 is comparing)?

$ diff -Naur t/trash/a t/trash/d/a
diff -Naur t/trash/a/l1 t/trash/d/a/l1

--- t/trash/a/l1        2007-09-17 23:10:03.000000000 +0200
+++ t/trash/d/a/l1      2007-09-17 23:10:03.000000000 +0200

@@ -1 +1 @@
-simple textfile
+a

\ No newline at end of file

thanks,
- VMiklos

Hi,

On Tue, 18 Sep 2007, Miklos Vajna wrote:

Ah, that's my friend GNU diff again... Try running the test with "-i -v", 
and it will pass, if I am right.

I actually made a patch for GNU diff back when this hit me, only to find 
out that it was fixed in CVS.  Apparently, for a short period, during 
which 2.8.1 was released, there was a bug that prevented GNU diff from 
working properly when the output was redirected to a non-tty.

If I'm right, and it is that bug, it might be worthwhile to convert the 
calls to "diff -u" in that test script with calls to "git diff" (possibly 
with --no-index).

Ciao,
Dscho

On Mon, Sep 17, 2007 at 11:40:27PM +0100, Johannes Schindelin [off-list ref] wrote:

Ah, that's my friend GNU diff again... Try running the test with "-i -v", 
and it will pass, if I am right.

* expecting success: diff -r a d/a
diff -r a/l1 d/a/l1
1c1
< simple textfile
---

a

\ No newline at end of file
* FAIL 21: validate file contents
        diff -r a d/a

it still fails, so i don't think this is that issue

- VMiklos

Hi,

On Tue, 18 Sep 2007, Miklos Vajna wrote:

On Mon, Sep 17, 2007 at 11:40:27PM +0100, Johannes Schindelin [off-list ref] wrote:

quoted

Ah, that's my friend GNU diff again... Try running the test with "-i -v", 
and it will pass, if I am right.

* expecting success: diff -r a d/a
diff -r a/l1 d/a/l1
1c1
< simple textfile
---

quoted

a

\ No newline at end of file
* FAIL 21: validate file contents
        diff -r a d/a

it still fails, so i don't think this is that issue

Oops.  I blame it on tiredness...

Sorry for the noise,
Dscho

Miklos Vajna schrieb:

On Mon, Sep 17, 2007 at 11:50:32PM +0200, René Scharfe [off-list ref] wrote:

quoted

The failing tests check ZIP file creation, not tar file creation.
Perhaps your unzip command works a bit differently from Info-ZIP's?

$ zip -v
Copyright (c) 1990-2006 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 2.32 (June 19th 2006), by Info-ZIP.

should this version be ok?

Info-ZIP zip and unzip are two different programs, each with their
own version numbers.

Ah!  l1 is a symlink to a.  The target of a symlink is stored as file
content in a ZIP archive and marked with a special flag.  An unzipper
that doesn't understand this flag would create a file containing the
link target's name instead of a symlink pointing to the target.

The unzip on my system says:

  $ unzip -v
  UnZip 5.52 of 28 February 2005, by Ubuntu. Original by Info-ZIP.

  Latest sources and executables are at  ftp://ftp.info-zip.org/pub/infozip/ ;
  see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

  Compiled with gcc 4.1.2 (Ubuntu 4.1.2-0ubuntu4) for Unix (Linux ELF) on Mar 31 2007.

  UnZip special compilation options:
          ACORN_FTYPE_NFS
          COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
          SET_DIR_ATTRIB
          TIMESTAMP
          USE_EF_UT_TIME
          USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
          USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
          VMS_TEXT_CONV
          WILD_STOP_AT_DIR
          [decryption, version 2.9 of 05 May 2000]

I suspect you need SET_DIR_ATTRIB in order to extract symlinks.

René

On Tue, Sep 18, 2007 at 04:57:33PM +0200, René Scharfe [off-list ref] wrote:

Info-ZIP zip and unzip are two different programs, each with their
own version numbers.

ah, my bad.

quoted

quoted

Also, what is the difference between t/trash/a and t/trash/d/a after
running the test script (that's what test 21 is comparing)?

$ diff -Naur t/trash/a t/trash/d/a
diff -Naur t/trash/a/l1 t/trash/d/a/l1

--- t/trash/a/l1        2007-09-17 23:10:03.000000000 +0200
+++ t/trash/d/a/l1      2007-09-17 23:10:03.000000000 +0200

@@ -1 +1 @@
-simple textfile
+a

\ No newline at end of file

Ah!  l1 is a symlink to a.  The target of a symlink is stored as file
content in a ZIP archive and marked with a special flag.  An unzipper
that doesn't understand this flag would create a file containing the
link target's name instead of a symlink pointing to the target.

The unzip on my system says:

  $ unzip -v
  UnZip 5.52 of 28 February 2005, by Ubuntu. Original by Info-ZIP.

  Latest sources and executables are at  ftp://ftp.info-zip.org/pub/infozip/ ;
  see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

  Compiled with gcc 4.1.2 (Ubuntu 4.1.2-0ubuntu4) for Unix (Linux ELF) on Mar 31 2007.

  UnZip special compilation options:
          ACORN_FTYPE_NFS
          COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
          SET_DIR_ATTRIB
          TIMESTAMP
          USE_EF_UT_TIME
          USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
          USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
          VMS_TEXT_CONV
          WILD_STOP_AT_DIR
          [decryption, version 2.9 of 05 May 2000]

I suspect you need SET_DIR_ATTRIB in order to extract symlinks.

$ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP.  Maintained by C. Spieler.
Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for
details.

Latest sources and executables are at
ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.1.2 for Unix (Linux ELF) on May  9 2007.

UnZip special compilation options:
        ASM_CRC
        COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
        SET_DIR_ATTRIB
        TIMESTAMP
        USE_EF_UT_TIME
        USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
        USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
        VMS_TEXT_CONV
        [decryption, version 2.9 of 05 May 2000]

if i'm not wrong then these options should be ok. :S

thanks,
- VMiklos

Miklos Vajna schrieb:

$ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP.  Maintained by C. Spieler.
Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for
details.

Latest sources and executables are at
ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.1.2 for Unix (Linux ELF) on May  9 2007.

UnZip special compilation options:
        ASM_CRC
        COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
        SET_DIR_ATTRIB
        TIMESTAMP
        USE_EF_UT_TIME
        USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
        USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
        VMS_TEXT_CONV
        [decryption, version 2.9 of 05 May 2000]

if i'm not wrong then these options should be ok. :S

Well, looks OK to me at least.  The Debian package (and thus Ubuntu's
too) has a symlink related fix, from Christian Spieler, no less:

  http://packages.debian.org/changelogs/pool/main/u/unzip/unzip_5.52-9/changelog
  http://ftp.de.debian.org/debian/pool/main/u/unzip/unzip_5.52-9.diff.gz

Perhaps this is missing from your version?

Let's check if the ZIP file on our machines match.  In order for
this to succeed we need to first take out any randomness.  Please
apply the patch at the end of the mail, which sets GIT_AUTHOR_DATE
to a fixed value instead of using the current time, run the test
and then this:

  $ md5sum t/trash/d.zip	# run just after t5000-tar-tree.sh
  35f6183b7816960cadfc6cac74530640  t/trash/d.zip

The test passes just fine here, so if the checksums match then your
unzip is most likely to blame.  In this case you can work around
your problem by commenting out the "ln -s" in t5000-tar-tree.sh.

René

--- snip! ---

archive: make test repeatable

Set GIT_AUTHOR_DATE to a fixed value, just like it's already done
for GIT_COMMITTER_DATE.  This makes the test repeatable, and
resulting archive files can by compared on mailing lists simply by
comparing their checksums.

Signed-off-by: Rene Scharfe <redacted>

diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh
index 42e28ab..f341451 100755
--- a/t/t5000-tar-tree.sh
+++ b/t/t5000-tar-tree.sh

@@ -50,7 +50,7 @@ test_expect_success \
      treeid=`git write-tree` &&
      echo $treeid >treeid &&
      git update-ref HEAD $(TZ=GMT GIT_COMMITTER_DATE="2005-05-27 22:00:00" \
-     git commit-tree $treeid </dev/null)'
+     GIT_AUTHOR_DATE="2007-09-18 19:00:00" git commit-tree $treeid </dev/null)
 
 test_expect_success \
     'git archive' \

On Tue, Sep 18, 2007 at 07:05:19PM +0200, René Scharfe [off-list ref] wrote:

Well, looks OK to me at least.  The Debian package (and thus Ubuntu's
too) has a symlink related fix, from Christian Spieler, no less:

  http://packages.debian.org/changelogs/pool/main/u/unzip/unzip_5.52-9/changelog
  http://ftp.de.debian.org/debian/pool/main/u/unzip/unzip_5.52-9.diff.gz

Perhaps this is missing from your version?

that was the problem. using vanilla unzip works fine, or after applying
the two security fixes a small patch is needed.

for the archives, here is the small patch that fixed the problem:

http://git.frugalware.org/gitweb/gitweb.cgi?p=frugalware-current.git;a=commit;h=c27277b2354b9ed4d6b0fbdb988643ad203f73c8

so the problem is not git-related

sorry for the noise and thanks for your help :)

- VMiklos