Schrödinger's diff

git version 1.6.3.2.1299.gee46c (msysgit)

In trying to track down some annoying crlf corruption in a repo
I have found a Schrödinger's diff.  In other words it's unknown
whether the diff will produce output or not on any particular run
of the following script.

Sometimes it does, and sometimes it doesn't (seems to be about
50/50).  But either way in any given repo rerunning the git-diff will
always give the same result.

Doing an "git ls-tree HEAD" gives an identical tree in both cases.

Can anyone explain why the output to this is not deterministic?
I'm at a complete loss.

	# Clean up from last run and start over
	rm -rf .git has-crlf
	git init
	git config core.autocrlf false

	# Add a "bad" file
	perl -e 'printf( "12%c%c", 0xd, 0xa )' > has-crlf
	git add has-crlf
	git commit -m"add crlf"

	# I realize that switching is ill-advised, but I'm
	# trying to track down a possibly related problem...
	git config core.autocrlf true

	# This sometimes produces output and sometimes it doesn't.
	# Either way rerunning just git-diff always gives the same result
	# as the first run in this repo.
	git diff

- Eric

Eric Raible schrieb:

Sometimes it does, and sometimes it doesn't (seems to be about
50/50).  But either way in any given repo rerunning the git-diff will
always give the same result.

Doing an "git ls-tree HEAD" gives an identical tree in both cases.

Can anyone explain why the output to this is not deterministic?
I'm at a complete loss.

	# Clean up from last run and start over
	rm -rf .git has-crlf
	git init
	git config core.autocrlf false

	# Add a "bad" file
	perl -e 'printf( "12%c%c", 0xd, 0xa )' > has-crlf
	git add has-crlf
	git commit -m"add crlf"

	# I realize that switching is ill-advised, but I'm
	# trying to track down a possibly related problem...
	git config core.autocrlf true

	# This sometimes produces output and sometimes it doesn't.
	# Either way rerunning just git-diff always gives the same result
	# as the first run in this repo.
	git diff

If I put this in a script, I get a diff in 9 out of 10 runs. If I insert
'sleep 1' right before the 'git add', I never get a diff.

I'm handing this off to people who care about core.autocrlf and who know
how racily-clean index entries (not) work ;)

-- Hannes

Eric Raible wrote:

git version 1.6.3.2.1299.gee46c (msysgit)

In trying to track down some annoying crlf corruption in a repo
I have found a Schrödinger's diff.  In other words it's unknown
whether the diff will produce output or not on any particular run
of the following script.

Sometimes it does, and sometimes it doesn't (seems to be about
50/50).  But either way in any given repo rerunning the git-diff will
always give the same result.

I don't get the same result in the same repo, although it only
differs in 1-1.5% of the tests.

Doing an "git ls-tree HEAD" gives an identical tree in both cases.

Can anyone explain why the output to this is not deterministic?

On Linux with git version 1.6.3.3.354.g3b4cc

Pasting your commands into "repro.sh", but redirecting output from
git commit to /dev/null, and then running the following commands
has yielded 9 to 15 sample.$i files over 5 tries of the following:

sh repro.sh > correct
for i in $(seq 1 1000); do
  sh repro.sh > sample && cmp sample correct >/dev/null || \
     { echo "fail $i" && cp sample sample.$i; };
done

I'm at a complete loss.

Inserting "sync" between calls as shown below doesn't fix the issue
(although it drops from 9-15 to 4-10 fails on Linux; Not a very good
improvement and only two test-runs). I have no idea how it can behave
so strangely, and I refuse to believe that the ext3 fs driver allows
dirty reads.

	# Clean up from last run and start over
	rm -rf .git has-crlf
	git init
	git config core.autocrlf false

	# Add a "bad" file
	perl -e 'printf( "12%c%c", 0xd, 0xa )' > has-crlf
	git add has-crlf
	git commit -m"add crlf"

sync

	# I realize that switching is ill-advised, but I'm
	# trying to track down a possibly related problem...
	git config core.autocrlf true

sync

	# This sometimes produces output and sometimes it doesn't.
	# Either way rerunning just git-diff always gives the same result
	# as the first run in this repo.
	git diff

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

On Mon, 6 Jul 2009, Eric Raible wrote:

git version 1.6.3.2.1299.gee46c (msysgit)

In trying to track down some annoying crlf corruption in a repo
I have found a Schrödinger's diff.  In other words it's unknown
whether the diff will produce output or not on any particular run
of the following script.

Sometimes it does, and sometimes it doesn't (seems to be about
50/50).  But either way in any given repo rerunning the git-diff will
always give the same result.

Doing an "git ls-tree HEAD" gives an identical tree in both cases.

Can anyone explain why the output to this is not deterministic?
I'm at a complete loss.

	# Clean up from last run and start over
	rm -rf .git has-crlf
	git init
	git config core.autocrlf false

	# Add a "bad" file
	perl -e 'printf( "12%c%c", 0xd, 0xa )' > has-crlf
	git add has-crlf

If has-crlf and .git/index have the same timestamp, git does not know 
whether the file has been modified afterwards or not. If they have 
different timestamps, git knows the file hasn't been modified after the 
add. (More precisely, the index contains the mtime of the file, and it 
will agree with the file system. However, if the timestamp on the index 
matches a timestamp *in* the index, that means that, when the index was 
written, the time period represented by that timestamp was not yet over 
when git looked at the file. Therefore, the file could have changed 
again after that time and still gotten the same timestamp it already had. 
This means that git can't be sure that there's nothing new to see in the 
filesystem.)

	git commit -m"add crlf"

	# I realize that switching is ill-advised, but I'm
	# trying to track down a possibly related problem...
	git config core.autocrlf true

	# This sometimes produces output and sometimes it doesn't.
	# Either way rerunning just git-diff always gives the same result
	# as the first run in this repo.
	git diff

If git knows the file hasn't been modified, it doesn't produce a diff. 

If it doesn't know the file hasn't been modified, it looks at the actual 
contents and it find that the result of reading the disk applying autocrlf 
now doesn't match the contents of the index.

	-Daniel
*This .sig left intentionally blank*

On Tue, Jul 07, 2009 at 01:36:08PM -0400, Daniel Barkalow wrote:

quoted

	# I realize that switching is ill-advised, but I'm
	# trying to track down a possibly related problem...
	git config core.autocrlf true

	# This sometimes produces output and sometimes it doesn't.
	# Either way rerunning just git-diff always gives the same result
	# as the first run in this repo.
	git diff

If git knows the file hasn't been modified, it doesn't produce a diff. 

If it doesn't know the file hasn't been modified, it looks at the actual 
contents and it find that the result of reading the disk applying autocrlf 
now doesn't match the contents of the index.

Yes, that was my analysis upon reading the original mail, as well (and I
have been bitten by this before while testing crlf stuff). The same
thing can happen with clean/smudge, I think.

When you set up config that changes how we view worktree files (like
crlf or clean/smudge) and there is already cached stat information in
the index, you really need to invalidate the matching stat information
in the index to get sane results[1]. It might be nice for "git config"
to do this for you, but:

    1. You could just as easily be hand-editing the config.

    2. It feels wrong from a modularity standpoint. Right now "git
       config" doesn't actually care about the semantics of config,
       just the syntax. Which makes it exactly equivalent to
       hand-editing.

    3. It doesn't cover every situation. Files can also be "changed" in
       this way by editing .gitattributes, which can be changed manually
       or by any number of git commands (like checkout, reset, etc).

So I think automatically detecting this situation would require flags in
the index to say "this stat information is valid only over these
particular settings". And you would want it per-file to avoid having to
re-hash every file when you change the .gitattributes for one file. The
command using the index would check it. But even that might have holes,
I'm afraid -- we don't always look at all of the config in every
command, though perhaps we do for such core functionality.

-Peff

[1] Is there an easy way to do this with update-index? I didn't see one,
and had to resort to "git read-tree HEAD".