Re: [Question] Unicode weirdness breaking tests on ZFS?

On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote:

On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote:

quoted

On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote:

quoted

I recently had to pave my Linux machine, so I updated it to Ubuntu
21.10 and had the choice to start using the ZFS filesystem. I thought,
"Why not?" but now I maybe see why.

Running the Git test suite at the v2.34.0 tag on my machine results in
these failures:

t0050-filesystem.sh                   (Wstat: 0 Tests: 11 Failed: 0)
  TODO passed:   9-10
t0021-conversion.sh                   (Wstat: 256 Tests: 41 Failed: 1)
  Failed test:  31
  Non-zero exit status: 1
t3910-mac-os-precompose.sh            (Wstat: 256 Tests: 25 Failed: 10)
  Failed tests:  1, 4, 6, 8, 11-16
  TODO passed:   23
  Non-zero exit status: 1

These are all related to the UTF8_NFD_TO_NFC prereq.

Zooming in on t0050, these tests are marked as "test_expect_failure" due
to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq:


$test_unicode 'rename (silent unicode normalization)' '
	git mv "$aumlcdiar" "$auml" &&
	git commit -m rename
'

$test_unicode 'merge (silent unicode normalization)' '
	git reset --hard initial &&
	git merge topic
'


The prereq creates two files using unicode characters that could
collapse to equivalent meanings:


test_lazy_prereq UTF8_NFD_TO_NFC '
	# check whether FS converts nfd unicode to nfc
	auml=$(printf "\303\244")
	aumlcdiar=$(printf "\141\314\210")
	>"$auml" &&
	test -f "$aumlcdiar"
'


What I see in that first test, the 'git mv' does change the
index, but the filesystem thinks the files are the same. This
may mean that our 'git add "$aumlcdiar"' from an earlier test
is providing a non-equivalence in the index, and the 'git mv'
changes the index without causing any issues in the filesystem.

It reminds me as if we used 'git mv README readme' on a case-
insensitive filesystem. Is this not a similar situation?

What I'm trying to gather is that maybe this test is flawed?
Or maybe something broke (or never worked?) in how we use
'git add' to not get the canonical unicode from the filesystem?

The other tests all have similar interactions with 'git add'.
I'm hoping that these are just test bugs, and not actually a
functionality issue in Git. Yes, it is confusing that we can
change the unicode of a file in the index without the filesystem
understanding the difference, but that is very similar to how
case-insensitive filesystems work and I don't know what else we
would do here.

These filesystem/unicode things are out of my expertise, so
hopefully someone else has a clearer idea of what is going on.
I'm happy to be a test bed, or even attempt producing patches
to fix the issue once we have that clarity.

Thanks,
-Stolee

Interesting.
The tests have always been working on HFS+, then we got
APFS (and needed a small fix) and now ZFS.

I'll can have a look - just installing in a virtual machine.

So, the virtual machine is up-and-running.

I got 2 messages:

ok 9 - rename (silent unicode normalization) # TODO known breakage vanished
ok 10 - merge (silent unicode normalization) # TODO known breakage vanished

Do you get the same ?

Now I am even more puzzled.
running t0050 with -x gives this:

 Author: A U Thor [off-list ref]
  1 file changed, 0 insertions(+), 0 deletions(-)
   rename "a\314\210" => "\303\244" (100%)
   ok 9 - rename (silent unicode normalization) # TODO known breakage vanished


----------------
When I create a test Git, with one file in ä-decomposed,
and rename into ä-precomposed, Git gives me:

tb@Ubuntu2021:~/ttt$ git mv "$aumlcdiar" "$auml"
fatal: destination exists, source=ä, destination=ä

and in hex form:

tb@Ubuntu2021:~/ttt$ git mv "$aumlcdiar" "$auml" 2>&1 | xxd
00000000: 6661 7461 6c3a 2064 6573 7469 6e61 7469  fatal: destinati
00000010: 6f6e 2065 7869 7374 732c 2073 6f75 7263  on exists, sourc
00000020: 653d 61cc 882c 2064 6573 7469 6e61 7469  e=a.., destinati
00000030: 6f6e 3dc3 a40a                           on=...