git log doesn't allow %x00 in custom format anymore?

Hello,

I've noticed that at least since Git 1.7.2.3 custom pretty format behaves strangely if there is a null-byte in the format:

Without null byte:

git log --pretty=format:%H-%ct

Outputs something like:

ee5d714b95d133ff555bc8c7933dc752b5b277f5-1285954314
cb1b9dd688d9cf155257c94e749172820b56d87a-1285954240
b4c75be10b14d021003853e527e47ad88dc5a55b-1285833610
1963187da6a45f898e62e4e922faac6b9382b4e4-1285807494

With a null byte

git log --pretty=format:%H%x00%ct

Ignores anything appearing after the null byte:

ee5d714b95d133ff555bc8c7933dc752b5b277f5
cb1b9dd688d9cf155257c94e749172820b56d87a
b4c75be10b14d021003853e527e47ad88dc5a55b
1963187da6a45f898e62e4e922faac6b9382b4e4

Other bytes behave fine. 

Using %x00 worked in Git 1.7.1 and stopped working in 1.7.2.3 (or even earlier).

Is it a bug or a feature (not allowing null bytes in the custom format anymore)?

If it is a feature, the documentation should have note about it. Now it's even proposing to use %x00:
            %x00: print a byte from a hex code

Thanks a lot.

----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

On Thu, Oct 07, 2010 at 03:25:29PM +0400, Kirill Likhodedov wrote:

With a null byte

quoted

git log --pretty=format:%H%x00%ct

Ignores anything appearing after the null byte:

ee5d714b95d133ff555bc8c7933dc752b5b277f5
cb1b9dd688d9cf155257c94e749172820b56d87a
b4c75be10b14d021003853e527e47ad88dc5a55b
1963187da6a45f898e62e4e922faac6b9382b4e4

Other bytes behave fine. 

Using %x00 worked in Git 1.7.1 and stopped working in 1.7.2.3 (or even earlier).

I can't reproduce using v1.7.2.3. Are you sure your pager or terminal
isn't hiding everything after the NUL? Have you tried piping it through
"xxd" or "cat -A"? I get:

  $ git log -1 --pretty=foo%x00bar | xxd
  0000000: 666f 6f00 6261 720a                      foo.bar.

  $ git log -1 --pretty=foo%x00bar | cat -A
  foo^@bar$

Is it a bug or a feature (not allowing null bytes in the custom format anymore)?

If it is not working, it is most definitely a bug.

-Peff

My bad, this happens only on Windows (at least not on Mac OS X).

So it more likely is an msys porting problem or something like that, not a bug of Git itself.
I've tested it on Windows XP SP3 with official installer for Git 1.7.2.3 and 1.7.3.1

I don't think it's a terminal problem. I've tried saving the output in a file:
git log --pretty=format:%H%x00%ct  >a.txt

I've also tried invoking the git process from a program and catching its output.

How could I report this bug? Or this message is enough? :)

07.10.2010, в 18:10, Jeff King написал(а):

On Thu, Oct 07, 2010 at 03:25:29PM +0400, Kirill Likhodedov wrote:

quoted

With a null byte

quoted

git log --pretty=format:%H%x00%ct

Ignores anything appearing after the null byte:

ee5d714b95d133ff555bc8c7933dc752b5b277f5
cb1b9dd688d9cf155257c94e749172820b56d87a
b4c75be10b14d021003853e527e47ad88dc5a55b
1963187da6a45f898e62e4e922faac6b9382b4e4

Other bytes behave fine. 

Using %x00 worked in Git 1.7.1 and stopped working in 1.7.2.3 (or even earlier).

I can't reproduce using v1.7.2.3. Are you sure your pager or terminal
isn't hiding everything after the NUL? Have you tried piping it through
"xxd" or "cat -A"? I get:

 $ git log -1 --pretty=foo%x00bar | xxd
 0000000: 666f 6f00 6261 720a                      foo.bar.

 $ git log -1 --pretty=foo%x00bar | cat -A
 foo^@bar$

quoted

Is it a bug or a feature (not allowing null bytes in the custom format anymore)?

If it is not working, it is most definitely a bug.

-Peff

----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

On Thu, Oct 07, 2010 at 06:35:07PM +0400, Kirill Likhodedov wrote:

My bad, this happens only on Windows (at least not on Mac OS X).

So it more likely is an msys porting problem or something like that, not a bug of Git itself.
I've tested it on Windows XP SP3 with official installer for Git 1.7.2.3 and 1.7.3.1

I don't think it's a terminal problem. I've tried saving the output in a file:
git log --pretty=format:%H%x00%ct  >a.txt

I've also tried invoking the git process from a program and catching its output.

How could I report this bug? Or this message is enough? :)

It might get seen here, but there is also an msysgit list:

  msysgit@googlegroups.com

-Peff

Am 10/7/2010 16:35, schrieb Kirill Likhodedov:

My bad, this happens only on Windows (at least not on Mac OS X).

With git rev-list, it also happens on Linux:

$ git log -1 --pretty=foo%x00bar HEAD | od -c
0000000   f   o   o  \0   b   a   r  \n
0000010
$ git rev-list -1 --pretty=foo%x00bar HEAD | od -c
0000000   c   o   m   m   i   t       8   7   f   a   f   1   0   9   0
0000020   5   d   1   f   8   a   b   1   e   9   a   c   2   4   c   c
0000040   3   e   f   e   d   8   1   f   a   7   e   2   9   6   7  \n
0000060   f   o   o  \n
0000064
$ git version
git version 1.7.3.67.gcc234

-- Hannes

With git rev-list, it also happens on Linux:

$ git log -1 --pretty=foo%x00bar HEAD | od -c
0000000   f   o   o  \0   b   a   r  \n
0000010
$ git rev-list -1 --pretty=foo%x00bar HEAD | od -c
0000000   c   o   m   m   i   t       8   7   f   a   f   1   0   9   0
0000020   5   d   1   f   8   a   b   1   e   9   a   c   2   4   c   c
0000040   3   e   f   e   d   8   1   f   a   7   e   2   9   6   7  \n
0000060   f   o   o  \n
0000064
$ git version
git version 1.7.3.67.gcc234

Hannes, 

Thanks for pointing that out.
I confirm that on Mac OS X that happens for rev-list as well. 

# git log --pretty=format:foo%x00bar HEAD -1 | od -c
0000000   f   o   o  \0   b   a   r
0000007

# git rev-list --pretty=format:foo%x00bar HEAD -1 | od -c
0000000   c   o   m   m   i   t       2   3   6   0   1   a   2   c   3
0000020   e   4   6   4   a   4   4   7   9   f   1   7   7   4   e   3
0000040   6   e   a   5   b   9   5   8   b   4   6   0   5   2   1  \n
0000060   f   o   o  \n
0000064

And it's not a new issue:

# git version
git version 1.7.1

So here's what we have now about this bug:
1. It happens since 1.7.1 for rev-list on all platforms (at least since 1.7.1 on Mac OS X and 1.7.3 on Linux).
2. It also happens at least since 1.7.2.3 for log on Windows.

Sorry for giving some confusion in my previous messages.
I probably tested 'rev-list' at the time when I've recovered the problem, but I reported it as a problem with 'log'.

----------------------------------
Kirill Likhodedov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

On Thu, Oct 07, 2010 at 07:18:18PM +0400, Kirill Likhodedov wrote:

Thanks for pointing that out.
I confirm that on Mac OS X that happens for rev-list as well. 

# git log --pretty=format:foo%x00bar HEAD -1 | od -c
0000000   f   o   o  \0   b   a   r
0000007

# git rev-list --pretty=format:foo%x00bar HEAD -1 | od -c
0000000   c   o   m   m   i   t       2   3   6   0   1   a   2   c   3
0000020   e   4   6   4   a   4   4   7   9   f   1   7   7   4   e   3
0000040   6   e   a   5   b   9   5   8   b   4   6   0   5   2   1  \n
0000060   f   o   o  \n
0000064

Ugh. Even worse, it does print with --graph, which uses a slightly
different code path.

  $ git rev-list --graph -1 --format=foo%x00bar HEAD | cat -A
  *   commit 81d866a6a213d5524ce389369377ba3529461e1b$
  |\  foo^@bar$

I am inclined to call the rev-list behavior a bug, and the fix is
probably:

diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index efe9360..3b2dca0 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c

@@ -147,8 +147,10 @@ static void show_commit(struct commit *commit, void *data)
 			}
 		} else {
 			if (revs->commit_format != CMIT_FMT_USERFORMAT ||
-			    buf.len)
-				printf("%s%c", buf.buf, info->hdr_termination);
+			    buf.len) {
+				fwrite(buf.buf, 1, buf.len, stdout);
+				putchar(info->hdr_termination);
+			}
 		}
 		strbuf_release(&buf);
 	} else {

On Thu, Oct 7, 2010 at 7:29 PM, Jeff King [off-list ref] wrote:

This gives me a bit of a deja-vu: 1fb5fdd

Also, fwriting like that to stdout might be a bit troublesome on
Windows because the string won't end up going through our
ANSI-emulation.

On Thu, Oct 7, 2010 at 7:41 PM, Erik Faye-Lund [off-list ref] wrote:

On Thu, Oct 7, 2010 at 7:29 PM, Jeff King [off-list ref] wrote:

quoted

On Thu, Oct 07, 2010 at 07:18:18PM +0400, Kirill Likhodedov wrote:

quoted

Thanks for pointing that out.
I confirm that on Mac OS X that happens for rev-list as well.

# git log --pretty=format:foo%x00bar HEAD -1 | od -c
0000000   f   o   o  \0   b   a   r
0000007

# git rev-list --pretty=format:foo%x00bar HEAD -1 | od -c
0000000   c   o   m   m   i   t       2   3   6   0   1   a   2   c   3
0000020   e   4   6   4   a   4   4   7   9   f   1   7   7   4   e   3
0000040   6   e   a   5   b   9   5   8   b   4   6   0   5   2   1  \n
0000060   f   o   o  \n
0000064

Ugh. Even worse, it does print with --graph, which uses a slightly
different code path.

 $ git rev-list --graph -1 --format=foo%x00bar HEAD | cat -A
 *   commit 81d866a6a213d5524ce389369377ba3529461e1b$
 |\  foo^@bar$

I am inclined to call the rev-list behavior a bug, and the fix is
probably:

diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index efe9360..3b2dca0 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c

@@ -147,8 +147,10 @@ static void show_commit(struct commit *commit, void *data)

                       }
               } else {
                       if (revs->commit_format != CMIT_FMT_USERFORMAT ||
-                           buf.len)
-                               printf("%s%c", buf.buf, info->hdr_termination);
+                           buf.len) {
+                               fwrite(buf.buf, 1, buf.len, stdout);
+                               putchar(info->hdr_termination);
+                       }
               }
               strbuf_release(&buf);
       } else {

This gives me a bit of a deja-vu: 1fb5fdd

Never mind that, I'm a bit too tired and got struck with a bit of paranoia :P

On Thu, Oct 07, 2010 at 07:41:01PM +0200, Erik Faye-Lund wrote:

quoted

diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index efe9360..3b2dca0 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c

@@ -147,8 +147,10 @@ static void show_commit(struct commit *commit, void *data)

                       }
               } else {
                       if (revs->commit_format != CMIT_FMT_USERFORMAT ||
-                           buf.len)
-                               printf("%s%c", buf.buf, info->hdr_termination);
+                           buf.len) {
+                               fwrite(buf.buf, 1, buf.len, stdout);
+                               putchar(info->hdr_termination);
+                       }
               }
               strbuf_release(&buf);
       } else {

This gives me a bit of a deja-vu: 1fb5fdd

Also, fwriting like that to stdout might be a bit troublesome on
Windows because the string won't end up going through our
ANSI-emulation.

I don't know enough about the Windows output code paths to comment on
that. But this is almost identical to the fwrite in graph_show_commit
that we use for the "rev-list --graph" show path. If there is a fix, it
needs to be made to both places.

-Peff