Re: git on MacOSX and files with decomposed utf-8 file names

7 messages, 5 authors, 2016-06-15 · open the first message on its own page

Re: git on MacOSX and files with decomposed utf-8 file names

From: Junio C Hamano <hidden>
Date: 2016-06-15 22:44:08

Kevin Ballard [off-list ref] writes:
As for dropping this conversation now, I'd love to. If you really want
to drop it, I urge you to do just that - don't respond to this
message. Read it, digest it, and then just let it sit. If this is the
last message on the subject, that would be *wonderful*. But if you
respond to this message then you have absolutely no ground to accuse
me of refusing to drop it. So please, don't.
I would not have said that if I were you.  That makes you look
very bad.  The impression I get after reading the above is that
the only thing you care about is to have the last word in the
thread.

People with opinions different from you could tone their message
down and stick to a more neutral sounding statement, "This patch
works around the issue X on HFS+", but not everybody is always
nice-and-calm.  But _you_ do not have to counter fire with fire,
especially if your goal isn't to flame but is to resolve
technical issues with cool head.  As long as you do not get
upset and start the flamewar every time whenever somebody says
"This patch works around the issue only that broken crap HFS+
has due to its stupid filename corruption choice it made", when
he could just have said it in a more neutral way, we can keep
the conversation constructive and civilized.

Let me suggest an alternative, as I think this thread raged on
long enough.  When you read somebody says "HFS+ corrupts", "HFS+
is broken", "this works around the stupidity of HFS+", just take
a deep breath, pretend that you did not hear these words that
make you feel insulted.  Instead pretend that you heard "HFS+
normalizes", "HFS+ is different", and "fixes problem on HFS+".
Do not respond with "No it is not a corruption", "No, HFS+ is
not broken" and "No, that is not a work around, but is a fix"
with another long thread.

I can imagine a civilized conversation to go this way:

	Linus: This patch would hopefully work around the stupid
	and broken normalization choice HFS+ people made years ago.

	You: Ok, I tested that patch, and it does fix the issue
	for me on HFS+ for most cases, but I still have issues
	if I use character X, Y and Z.

	Linus: Yeah, that is another direct consequence of the
	stupidity of HFS+.  At this point I think the previous
	patch bends git backwards enough and I do not know if it
	is worth addressing by bending further...

	You: How about introducing this new structure so that
	these cases can be handled in a way more friendly to
	HFS+, like this patch?

	Linus: Yeah, I can buy that, it looks ugly but it would
	not hurt people on other systems.

Hmm?

Re: git on MacOSX and files with decomposed utf-8 file names

From: Martin Langhoff <hidden>
Date: 2016-06-15 22:44:08

On Jan 24, 2008 4:11 PM, Junio C Hamano [off-list ref] wrote:
make you feel insulted.  Instead pretend that you heard "HFS+
normalizes", "HFS+ is different", and "fixes problem on HFS+".
Do not respond with "No it is not a corruption", "No, HFS+ is
not broken" and "No, that is not a work around, but is a fix"
with another long thread.
Indeed. And it'd be good if Kevin could consider that this forum is
for technical discussion - not democracy but meritocracy,
best-solution-cracy and perhaps "fix-patch-ocracy". And that people
that have written good code in the past, posted amazing patches, and
wondrous test cases can sometimes get a bit more opinionated. But
newcomers needs to earn a bit of respect before lecturing people.

Kevin, other people have already started posting nice nuggets of test
cases. Where are *your* test cases? That would be a nice way to "have
the last word" on this ;-)

cheers,



martin

Re: git on MacOSX and files with decomposed utf-8 file names

From: Kevin Ballard <hidden>
Date: 2016-06-15 22:44:08

On Jan 23, 2008, at 11:37 PM, Martin Langhoff wrote:
Kevin, other people have already started posting nice nuggets of test
cases. Where are *your* test cases? That would be a nice way to "have
the last word" on this ;-)

I'm planning on devoting time this weekend to learning enough about  
git to be able to start hacking. I'm just too busy during the week to  
be able to devote the dedicated time necessary to this stuff.  
Hopefully I'll actually be able to start producing stuff this weekend.

-Kevin Ballard

-- 
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com

Re: git on MacOSX and files with decomposed utf-8 file names

From: Steffen Prohaska <hidden>
Date: 2016-06-15 22:44:08

On Jan 24, 2008, at 6:30 AM, Kevin Ballard wrote:
On Jan 23, 2008, at 11:37 PM, Martin Langhoff wrote:
quoted
Kevin, other people have already started posting nice nuggets of test
cases. Where are *your* test cases? That would be a nice way to "have
the last word" on this ;-)

I'm planning on devoting time this weekend to learning enough about  
git to be able to start hacking. I'm just too busy during the week  
to be able to devote the dedicated time necessary to this stuff.  
Hopefully I'll actually be able to start producing stuff this weekend.
You do not need to learn much about git to post a test case.
Only a few lines of shell code that demonstrate how git fails to
handle a specific situation is needed.  To do this, knowledge
about the internals of git does not necessary help.  It should be
sufficient to know how to use git.

You may start with a simple shell script and send it to the list.
Though, a real patch would be the preferred way.  For this, you
should have a quick look into the t/ subdirectory.  Just open any
of the tNUMBER*.sh files.  It should be quite obvious how your
sequence of shell commands could be cast into a git test script.

	Steffen

Re: git on MacOSX and files with decomposed utf-8 file names

From: Mitch Tishmack <hidden>
Date: 2016-06-15 22:44:08

Here is a start maybe, I was just testing all of the HFS variants for  
fun. I will write up a test case later tonight when I am out  done  
with work.

#!/bin/sh
#
# Test git behavior on OSX's multitudes of HFS types
# So far UFS, HFS, HFS+, HFSX, HFS+J, UFS is the only sane FS  
available...
#

#cloneurl="git://git.kernel.org/pub/scm/git/git.git"
cloneurl="/Volumes/gitufs/git"
ramdiskdir="/private/tmp/gitramdisk"
results="/private/tmp/fsresults"

echo "Creating 100M UFS ramdisk for clone operation."
rawdev=`hdid -nomount ram://102400`
newfs $rawdev > /dev/null 2>&1
mkdir $ramdiskdir > /dev/null 2>&1
mount -t ufs $rawdev $ramdiskdir > /dev/null 2>&1
cd $ramdiskdir && git clone $cloneurl > /dev/null 2>&1
cd $ramdiskdir/git
if [ -f $results ] ; then
   echo "Removing old results."
   rm $results
fi

echo "Creating HFS image"
hdiutil create -size 50m -fs HFS -attach -volname "hfs" /tmp/hfs.dmg  
 > /dev/null 2>&1
echo "Creating HFS+ image"
hdiutil create -size 50m -fs HFS+ -attach -volname "hfsplus" /tmp/ 
hfsplus.d > /dev/null 2>&1
echo "Creating HFS+J image"
hdiutil create -size 50m -fs HFS+J -attach -volname "hfsplusJ" /tmp/ 
hfsplusjournal.dmg > /dev/null 2>&1
echo "Creating HFSX image"
hdiutil create -size 50m -fs HFSX -attach -volname "hfsx" /tmp/ 
hfsx.dmg > /dev/null 2>&1
echo "Creating UFS image"
hdiutil create -size 50m -fs UFS -attach -volname "hfsu" /tmp/hfsu.dmg  
 > /dev/null 2>&1

for x in `ls -d /Volumes/hfs*`;do
   echo "Testing $x clone."
   echo "Results for $x:" >> $results
   (echo "-- git clone results --" && cd ${x} && /usr/bin/time git  
clone $ramdiskdir/git) >> $results 2>&1
   (echo "-- git status results --" && cd ${x}/git && /usr/bin/time  
git status) >> $results 2>&1
   cd ${x} && perl -CO -e 'print pack("U",0x00E4)."\n"' | xargs touch  
# umlauted a
   cd ${x} && perl -CO -e 'print pack("U",0x0061).pack("U", 
0x0308)."\n"' | xargs touch # umlauted a by combining diareses
   ls -d ${x}/* | xxd >> $results
   cd && hdiutil eject ${x} > /dev/null 2>&1
done

# cleanup
cd $HOME
umount -f $ramdiskdir > /dev/null 2>&1
hdiutil detach $rawdev > /dev/null 2>&1
rm -Rf $ramdiskdir /tmp/hfs*.dmg
more $results

My results on leopard:
$ cat /tmp/fsresults
Results for /Volumes/hfs:
-- git clone results --
Initialized empty Git repository in /Volumes/hfs/git/.git/
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-06100ef5fbd98d07358505696e2e0c5600a9b279.pack: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-06100ef5fbd98d07358505696e2e0c5600a9b279.idx: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-401a5ae571eb23ec896d7e441deae4e313d0de9c.pack: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/pack- 
ab8844b63fcb4fc5896e9d75b0d10c566d5ce5bb.pack: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/pack- 
ab8844b63fcb4fc5896e9d75b0d10c566d5ce5bb.idx: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-9ffbf58084280a496aef6849fe3effe742a99d77.pack: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-9ffbf58084280a496aef6849fe3effe742a99d77.idx: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-401a5ae571eb23ec896d7e441deae4e313d0de9c.idx: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/ 
03/4ee24912da0a700ba27109825710fd84d64591: Invalid argument
         0.20 real         0.02 user         0.05 sys
-- git status results --
git_fs.sh: line 39: cd: /Volumes/hfs/git: No such file or directory
0000000: 2f56 6f6c 756d 6573 2f68 6673 2f61 cc88  /Volumes/hfs/a..
0000010: 0a                                       .
Results for /Volumes/hfsplus:
-- git clone results --
Initialized empty Git repository in /Volumes/hfsplus/git/.git/
         2.56 real         0.47 user         0.94 sys
-- git status results --
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to  
track)
         0.46 real         0.27 user         0.08 sys
0000000: 2f56 6f6c 756d 6573 2f68 6673 706c 7573  /Volumes/hfsplus
0000010: 2f61 cc88 0a2f 566f 6c75 6d65 732f 6866  /a.../Volumes/hf
0000020: 7370 6c75 732f 6769 740a                 splus/git.
Results for /Volumes/hfsplusJ:
-- git clone results --
Initialized empty Git repository in /Volumes/hfsplusJ/git/.git/
         2.29 real         0.45 user         0.91 sys
-- git status results --
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to  
track)
         0.57 real         0.31 user         0.10 sys
0000000: 2f56 6f6c 756d 6573 2f68 6673 706c 7573  /Volumes/hfsplus
0000010: 4a2f 61cc 880a 2f56 6f6c 756d 6573 2f68  J/a.../Volumes/h
0000020: 6673 706c 7573 4a2f 6769 740a            fsplusJ/git.
Results for /Volumes/hfsu:
-- git clone results --
Initialized empty Git repository in /Volumes/hfsu/git/.git/
         5.08 real         0.48 user         0.94 sys
-- git status results --
# On branch master
nothing to commit (working directory clean)
         0.26 real         0.20 user         0.05 sys
0000000: 2f56 6f6c 756d 6573 2f68 6673 752f 61cc  /Volumes/hfsu/a.
0000010: 880a 2f56 6f6c 756d 6573 2f68 6673 752f  ../Volumes/hfsu/
0000020: 6769 740a 2f56 6f6c 756d 6573 2f68 6673  git./Volumes/hfs
0000030: 752f c3a4 0a                             u/...
Results for /Volumes/hfsx:
-- git clone results --
Initialized empty Git repository in /Volumes/hfsx/git/.git/
         2.49 real         0.46 user         0.88 sys
-- git status results --
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to  
track)
         0.25 real         0.20 user         0.04 sys
0000000: 2f56 6f6c 756d 6573 2f68 6673 782f 61cc  /Volumes/hfsx/a.
0000010: 880a 2f56 6f6c 756d 6573 2f68 6673 782f  ../Volumes/hfsx/
0000020: 6769 740a                                git.


On Jan 24, 2008, at 12:39 AM, Steffen Prohaska wrote:
On Jan 24, 2008, at 6:30 AM, Kevin Ballard wrote:
quoted
On Jan 23, 2008, at 11:37 PM, Martin Langhoff wrote:
quoted
Kevin, other people have already started posting nice nuggets of  
test
cases. Where are *your* test cases? That would be a nice way to  
"have
the last word" on this ;-)

I'm planning on devoting time this weekend to learning enough about  
git to be able to start hacking. I'm just too busy during the week  
to be able to devote the dedicated time necessary to this stuff.  
Hopefully I'll actually be able to start producing stuff this  
weekend.
You do not need to learn much about git to post a test case.
Only a few lines of shell code that demonstrate how git fails to
handle a specific situation is needed.  To do this, knowledge
about the internals of git does not necessary help.  It should be
sufficient to know how to use git.

You may start with a simple shell script and send it to the list.
Though, a real patch would be the preferred way.  For this, you
should have a quick look into the t/ subdirectory.  Just open any
of the tNUMBER*.sh files.  It should be quite obvious how your
sequence of shell commands could be cast into a git test script.

	Steffen
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git on MacOSX and files with decomposed utf-8 file names

From: Mitch Tishmack <hidden>
Date: 2016-06-15 22:44:08

Apologies Steffen, I grabbed your CamelCase test and did a search/ 
replace, wasn't sure what to call it though... But I am on lunch and  
wanted to be useful. Rip it apart all you want.

Fails on hfs* on OSX, works on ufs. I will bother with zfs when it can  
be used again.

On UFS:
$ /bin/sh ./t0060-normalization.sh
*   ok 1: setup
*   ok 2: rename (silent normalization)
*   ok 3: merge (silent normalization)
* passed all 3 test(s)


On HFS:
$ /bin/sh t0060-normalization.sh
*   ok 1: setup
* FAIL 2: rename (silent normalization)
	
	
	 git mv ä ä &&
	 git commit -m "rename"
	
	
* FAIL 3: merge (silent normalization)
	
	
	 git reset --hard initial &&
	 git merge topic
	
	
* failed 2 among 3 test(s)

The test case, it uses perl, assuming only 5.6.1+ will work with this:
diff --git a/t/t0060-normalization.sh b/t/t0060-normalization.sh
new file mode 100755
index 0000000..e012c02
--- /dev/null
+++ b/t/t0060-normalization.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description='Test for silent normalization issues'
+
+. ./test-lib.sh
+
+auml=`perl -CO -e 'print pack("U",0x00E4)'`
+aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
+test_expect_success setup "
+  touch $aumlcdiar &&
+  git add $aumlcdiar &&
+  git commit -m \"initial\"
+  git tag initial &&
+  git checkout -b topic &&
+  git mv $aumlcdiar tmp &&
+  git mv tmp $auml &&
+  git commit -m \"rename\" &&
+  git checkout -f master
+
+"
+
+test_expect_success 'rename (silent normalization)' "
+
+ git mv $aumlcdiar $auml &&
+ git commit -m \"rename\"
+
+"
+
+test_expect_success 'merge (silent normalization)' '
+
+ git reset --hard initial &&
+ git merge topic
+
+'
+
+test_done
-- 
1.5.3




On Jan 24, 2008, at 12:39 AM, Steffen Prohaska wrote:

>
> On Jan 24, 2008, at 6:30 AM, Kevin Ballard wrote:
>
>> On Jan 23, 2008, at 11:37 PM, Martin Langhoff wrote:
>>
>>> Kevin, other people have already started posting nice nuggets of  
>>> test
>>> cases. Where are *your* test cases? That would be a nice way to  
>>> "have
>>> the last word" on this ;-)
>>
>>
>> I'm planning on devoting time this weekend to learning enough about  
>> git to be able to start hacking. I'm just too busy during the week  
>> to be able to devote the dedicated time necessary to this stuff.  
>> Hopefully I'll actually be able to start producing stuff this  
>> weekend.
>
> You do not need to learn much about git to post a test case.
> Only a few lines of shell code that demonstrate how git fails to
> handle a specific situation is needed.  To do this, knowledge
> about the internals of git does not necessary help.  It should be
> sufficient to know how to use git.
>
> You may start with a simple shell script and send it to the list.
> Though, a real patch would be the preferred way.  For this, you
> should have a quick look into the t/ subdirectory.  Just open any
> of the tNUMBER*.sh files.  It should be quite obvious how your
> sequence of shell commands could be cast into a git test script.
>
> 	Steffen
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git on MacOSX and files with decomposed utf-8 file names

From: Kevin Ballard <hidden>
Date: 2016-06-15 22:44:08

On Jan 24, 2008, at 1:52 PM, Mitch Tishmack wrote:
Apologies Steffen, I grabbed your CamelCase test and did a search/ 
replace, wasn't sure what to call it though... But I am on lunch and  
wanted to be useful. Rip it apart all you want.

[snip]
Well, I was planning on writing my own test case today, but you seem  
to have beaten me to the punch. I just tested your script and it does  
indeed fail as expected on HFS+. Thank you for producing this test case.

-Kevin Ballard

-- 
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com

Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help