Serious bug with pretty format strings & empty bodies?

I'm struggling to come up with a minimal test case that confirms this,
but I seem to be hitting a bug in the pretty string formatter when
trying to print bodies of commits that don't have bodies.

For example, on a private repository, I'm trying to print the subject
& body of a commit.  This particular commit doesn't actually have a
body, so I'm expecting to see this output :

=================
commit 18d2480ab689b483ef1ebbdb3f7420904049ba0b
Try to flush log files before terminating the app

=================

However, when I actually run the command, I get gibberish in place of %b :
=================
$ git rev-list -1- --pretty=format:"%s%n%b"
18d2480ab689b483ef1ebbdb3f7420904049ba0b
commit 18d2480ab689b483ef1ebbdb3f7420904049ba0b
Try to flush log files before terminating the app
tree 57bc7cf30a10aee96251852125cf30fd2c81d7aa
parent 04c833865828538315fcdf6e187da077869ce444
author Jonathan del Strother [off-list ref] 1197901755 +0000
committer Jonathan del Strother [off-list ref] 1197901755 +0000

Check that ThreadWorker's work method actually returns a value with
method signatures
=================

So here you can see that the full details of a different commit is
appearing in place of the requested commit's (empty) body.  Rerunning
the rev-list produces the same output - the same string appears in
place of %b every time, but it's different for each commit.  Often you
only get a small fragment of another commit's message, rather than the
full commit as seen above.

When I use --pretty=full, it works fine.

This appears in quite a few places in my repository's history.
Repacking the repo semi-fixes it - I still get junk bodies, but not
quite so many.

It doesn't always happen when a commit message body is empty, but I've
never seen it happen where the message body has something in.

This happens with both git 1.5.3.7 and 1.5.4.rc0.1162.g3bfea, though
the output is slightly different - the former has a number of
<unknown> markers in it.  This is on OS X 10.5.1


Any suggestions?
Cheers,
Jon

Jonathan del Strother schrieb:

I'm struggling to come up with a minimal test case that confirms this,
but I seem to be hitting a bug in the pretty string formatter when
trying to print bodies of commits that don't have bodies.

For example, on a private repository, I'm trying to print the subject
& body of a commit.  This particular commit doesn't actually have a
body, so I'm expecting to see this output :

=================
commit 18d2480ab689b483ef1ebbdb3f7420904049ba0b
Try to flush log files before terminating the app

=================

However, when I actually run the command, I get gibberish in place of %b :
=================
$ git rev-list -1- --pretty=format:"%s%n%b"
18d2480ab689b483ef1ebbdb3f7420904049ba0b
commit 18d2480ab689b483ef1ebbdb3f7420904049ba0b
Try to flush log files before terminating the app
tree 57bc7cf30a10aee96251852125cf30fd2c81d7aa
parent 04c833865828538315fcdf6e187da077869ce444
author Jonathan del Strother [off-list ref] 1197901755 +0000
committer Jonathan del Strother [off-list ref] 1197901755 +0000

Check that ThreadWorker's work method actually returns a value with
method signatures
=================

So here you can see that the full details of a different commit is
appearing in place of the requested commit's (empty) body.  Rerunning
the rev-list produces the same output - the same string appears in
place of %b every time, but it's different for each commit.  Often you
only get a small fragment of another commit's message, rather than the
full commit as seen above.

When I use --pretty=full, it works fine.

This appears in quite a few places in my repository's history.
Repacking the repo semi-fixes it - I still get junk bodies, but not
quite so many.

It doesn't always happen when a commit message body is empty, but I've
never seen it happen where the message body has something in.

This happens with both git 1.5.3.7 and 1.5.4.rc0.1162.g3bfea, though
the output is slightly different - the former has a number of
<unknown> markers in it.  This is on OS X 10.5.1

That's strange.  Could you check if this happens with e52a5de, too?
That's the commit that introduced --pretty=format.  If it doesn't, could
you then please try to bisect the bug?

How many '<unknown>' markers are there in the output of version 1.5.3.7
(and e52a5de)?  One per %b?  Are they the only output or are they
combined with parts of unrelated commits, too?

Thanks,
René

Jonathan del Strother, Wed, Dec 19, 2007 15:32:20 +0100:

I'm struggling to come up with a minimal test case that confirms this,
but I seem to be hitting a bug in the pretty string formatter when
trying to print bodies of commits that don't have bodies.

I tried

    $ git init
    $ git rev-list --pretty=format:"%s%n%b" \
	$(echo -n "Test" |git commit-tree $(git write-tree))

but it works. It is possible that the commit in question is created
with an older commit-tree code. Could you try

    git cat-file 18d2480ab689b483ef1ebbdb3f7420904049ba0b

(or any other problematic commit) and post its output here?
Or is the repo publised somewhere?

On Dec 19, 2007 6:44 PM, Alex Riesen [off-list ref] wrote:

Could you try

    git cat-file 18d2480ab689b483ef1ebbdb3f7420904049ba0b

(or any other problematic commit) and post its output here?

You mean git cat-file commit ... ?
I get the normal output, but the problematic commits don't show a
newline character at the end of the cat-file output.

Or is the repo publised somewhere?

Afraid not - it's private code.


I'll try bisecting e52a5de tomorrow, see if that gets me anywhere.

Jonathan del Strother schrieb:

On Dec 19, 2007 6:44 PM, Alex Riesen [off-list ref] wrote:

quoted

Could you try

    git cat-file 18d2480ab689b483ef1ebbdb3f7420904049ba0b

(or any other problematic commit) and post its output here?

You mean git cat-file commit ... ?
I get the normal output, but the problematic commits don't show a
newline character at the end of the cat-file output.

Just a shot in the dark: does this patch on top of master make a difference?

diff --git a/pretty.c b/pretty.c
index 9db75b4..5f95a59 100644
--- a/pretty.c
+++ b/pretty.c

@@ -412,7 +412,7 @@ static void parse_commit_header(struct format_commit_context *context)
 		if (i == eol) {
 			state++;
 			/* strip empty lines */
-			while (msg[eol + 1] == '\n')
+			while (msg[eol] == '\n' && msg[eol + 1] == '\n')
 				eol++;
 		} else if (!prefixcmp(msg + i, "author ")) {
 			context->author.off = i + 7;

On Dec 19, 2007 6:28 PM, René Scharfe [off-list ref] wrote:

That's strange.  Could you check if this happens with e52a5de, too?
That's the commit that introduced --pretty=format.  If it doesn't, could
you then please try to bisect the bug?

As far as I can tell, e52a... never compiled - it's missing prefixcmp
from git-compat-util.h, which doesn't get merged in till 8ab3e18.
And yes, I still get junk output from that build.  (Shame, I was
looking forward to trying out the bisect tool...)

How many '<unknown>' markers are there in the output of version 1.5.3.7
(and e52a5de)?  One per %b?  Are they the only output or are they
combined with parts of unrelated commits, too?

Only one per %b, and where they appear, they're the only output.
However, even with those builds, I get junk output in addition to the
<unknown> markers.
That is, with a format string of "**%b**", there are three possible outputs :
The correct one -  eg **Added test for NSDictionary generator**
The unknown one - eg  **<unknown>**
The complete junk one -  eg **n del
Strothereb0d794bacc1761e8a0681e9b144c514702d9b7a**

The latter's 'body' is almost always terminated by an unprintable
character, which always seems to be 03.


Hmmm.   I wonder if it's coincidence that I can only seem to spot this
problem in my repos that have been set up via submodule init...

On Dec 19, 2007 11:23 PM, René Scharfe [off-list ref] wrote:

No luck with that, I'm afraid.

Hmmm.   I wonder if it's coincidence that I can only seem to spot this
problem in my repos that have been set up via submodule init...

Ignore that last bit, red herring.

I ended up trying to filter-branch my repository, see if I could come
up with a version stripped of all our private code, suitable for
making public.  Disappointingly, filter-branch magically fixes all the
commits, even when simply run as "git filter-branch HEAD".

However, looking at the rewritten repository vs the original, they
share a lot of commits, then diverge halfway through their history.  I
can't see anything notable about the commit where they diverge, with
the exception that the rewritten commit has a newline after the
subject and the original doesn't.  Neither has a commit message body.

Jonathan del Strother schrieb:

On Dec 19, 2007 11:23 PM, René Scharfe [off-list ref] wrote:

quoted

Just a shot in the dark: does this patch on top of master make a difference?

No luck with that, I'm afraid.

And how is this one?

The first chunk is the same as in the last try -- it guards against commit
messages ending with a NUL without a prior \n _and_ being followed by a \n
(in memory which shouldn't be accessed by us at all as it doesn't belong to
the commit message).  I guess that's quite rare.

The second chunk keeps the body offset from being incremented by the for
loop if we've already found a terminating NUL.

diff --git a/pretty.c b/pretty.c
index 9db75b4..5b1078b 100644
--- a/pretty.c
+++ b/pretty.c

@@ -412,7 +412,7 @@ static void parse_commit_header(struct format_commit_context *context)
 		if (i == eol) {
 			state++;
 			/* strip empty lines */
-			while (msg[eol + 1] == '\n')
+			while (msg[eol] == '\n' && msg[eol + 1] == '\n')
 				eol++;
 		} else if (!prefixcmp(msg + i, "author ")) {
 			context->author.off = i + 7;

@@ -425,6 +425,8 @@ static void parse_commit_header(struct format_commit_context *context)
 			context->encoding.len = eol - i - 9;
 		}
 		i = eol;
+		if (!msg[i])
+			break;
 	}
 	context->body_off = i;
 	context->commit_header_parsed = 1;

Hi,

On Thu, 20 Dec 2007, Jonathan del Strother wrote:

I ended up trying to filter-branch my repository, see if I could come
up with a version stripped of all our private code, suitable for
making public.  Disappointingly, filter-branch magically fixes all the
commits, even when simply run as "git filter-branch HEAD".

However, looking at the rewritten repository vs the original, they
share a lot of commits, then diverge halfway through their history.  I
can't see anything notable about the commit where they diverge, with
the exception that the rewritten commit has a newline after the
subject and the original doesn't.  Neither has a commit message body.

I do not see how Rene's patch could _not_ have fixed that problem.

As it is, I cannot even reproduce without that patch.  This is what I was 
trying:

-- snip --

diff --git a/t/t4025-pretty-format.sh b/t/t4025-pretty-format.sh
new file mode 100644
index 0000000..2826bcb
--- /dev/null
+++ b/t/t4025-pretty-format.sh

@@ -0,0 +1,23 @@
+#!/bin/sh
+
+test_description='git pretty formats'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+
+	: > file &&
+	git add file &&
+	tree=$(git write-tree) &&
+	commit=$(echo No newlin | tr "\\012" e | git commit-tree $tree) &&
+	git update-ref HEAD $commit
+
+'
+
+test_expect_success 'format:%b' '
+
+	test -z "$(git log -1 --pretty=format:%b)"
+
+'
+
+test_done
-- snap --

Seems I cannot help any more, but at least this is a starting point to 
try to reproduce.

Ciao,
Dscho

Hi,

On Thu, 20 Dec 2007, Ren? Scharfe wrote:

The second chunk keeps the body offset from being incremented by the for 
loop if we've already found a terminating NUL.

Ah!  I missed that one *writethisdownintothebook*

Thanks,
Dscho

On Dec 20, 2007 12:20 PM, René Scharfe [off-list ref] wrote:

Winnar!

Yep, that seems to have fixed things.  Wonder how I managed to get the
null into the commit message...

Has anyone actually managed to reproduce my problem?  I've got
multiple repos here that show the problem in several commits, made by
different people.  However, I can't actually come up with a way to
reproduce it at will...

I'd really like to see René's patch accepted as it seems to magically
fix all my problems, but it's a bit hard to justify if I'm the only
one that's seeing these broken commit messages.