Jan-Philip Gehrcke [off-list ref] writes:
I was surprised to see that the output of
git log --encoding=utf-8 "--format=format:%b"
can contain byte sequences that are invalid in UTF-8. Note: I am using
git 2.1.4 and the %b format specifier represents the commit message
body.
Yeah, if the original was bad and cannot be sanely expressed in
UTF-8, you have two options. You can show the contents as raw bytes
recorded in the object with a warning so that the user can use it as
such (e.g. perhaps the original was indeed an iso8859-2 but was
incorrectly marked as UTF-8, or something like that, and a human
that is more intelligent than a tool _could_ guess and attempt to
recover). Or you can error out and refuse to produce output.
We deliberately made a design choice to take the former option.
On 17.06.2015 18:42, Junio C Hamano wrote:
Jan-Philip Gehrcke [off-list ref] writes:
quoted
I was surprised to see that the output of
git log --encoding=utf-8 "--format=format:%b"
can contain byte sequences that are invalid in UTF-8. Note: I am using
git 2.1.4 and the %b format specifier represents the commit message
body.
Yeah, if the original was bad and cannot be sanely expressed in
UTF-8, you have two options. You can show the contents as raw bytes
recorded in the object with a warning so that the user can use it as
such (e.g. perhaps the original was indeed an iso8859-2 but was
incorrectly marked as UTF-8, or something like that, and a human
that is more intelligent than a tool _could_ guess and attempt to
recover). Or you can error out and refuse to produce output.
The two-option scenario is totally clear. Although one must stress that
the "error-out" option can, as discussed, be kept minimally invasive: it
is sufficient (and common) to just skip those byte sequences (and
replace them with a replacement symbol) that would be invalid in the
requested output encoding. This would retain as much information as
possible while guaranteeing a subsequent decoder to retrieve valid input.
We deliberately made a design choice to take the former option.
I totally support this design choice in general, especially when
invoking `git whatever` without options. This here is, I think, mainly
about documentation and the semantics of "--encoding". From my point of
view, `--encoding=utf-8` semantically suggests that the output *is*
valid UTF-8. But it is not, not always. May initial question was: what
do you think about this? Should we
* just make this more clear in the docs and/or
* should we adjust the behavior of --encoding or
* should we do something entirely different, like adding a new command
line option or
* should we just leave things as they are?
Thanks and cheers,
Jan-Philip
On Wed, Jun 17, 2015 at 07:07:48PM +0200, Jan-Philip Gehrcke wrote:
The two-option scenario is totally clear. Although one must stress that the
"error-out" option can, as discussed, be kept minimally invasive: it is
sufficient (and common) to just skip those byte sequences (and replace them
with a replacement symbol) that would be invalid in the requested output
encoding. This would retain as much information as possible while
guaranteeing a subsequent decoder to retrieve valid input.
I think "munge into valid UTF-8, even if it means losing data" is a
totally valid and useful option. I'm not completely sure that git should
do that, though. E.g., you could just as easily do:
git log --encoding=utf8 | drop_invalid_utf8 | your_script
Or quite possibly, your_script could do the munging itself while reading
the data. I do not know much about Python's input handling, but in Perl,
it is easy to say "the input is utf8, and replace anything bogus with a
substitution character"[1].
Should we
* just make this more clear in the docs and/or
* should we adjust the behavior of --encoding or
* should we do something entirely different, like adding a new command line
option or
* should we just leave things as they are?
I would vote for a documentation change, perhaps like:
Subject: docs: clarify that --encoding can produce invalid sequences
In the common case that the commit encoding matches the
output encoding, we do not touch the buffer at all, which
makes things much more efficient. But it might be unclear to
a consumer that we will pass through bogus sequences.
Signed-off-by: Jeff King <redacted>
---
Documentation/pretty-options.txt | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/Documentation/pretty-options.txt b/Documentation/pretty-options.txt
index 74aa01a..642af6e 100644
--- a/Documentation/pretty-options.txt
+++ b/Documentation/pretty-options.txt
@@ -37,7 +37,10 @@ people using 80-column terminals.
in their encoding header; this option can be used to tell the
command to re-code the commit log message in the encoding
preferred by the user. For non plumbing commands this
- defaults to UTF-8.
+ defaults to UTF-8. Note that if an object claims to be encoded
+ in `X` and we are outputting in `X`, we will output the object
+ verbatim; this means that invalid sequences in the original
+ commit may be copied to the output.
--notes[=<ref>]::
Show the notes (see linkgit:git-notes[1]) that annotate the
--
2.4.4.719.g3984bc6
* just make this more clear in the docs and/or
* should we adjust the behavior of --encoding or
* should we do something entirely different, like adding a new command line
option or
The general spirit is to keep things backwards compatible, so that users which
expect the "raw" (and possible corrupted UTF-8) data still get the same results,
when they updata their Git installation.
A new command line option will allow users to get clean UTF-8.
One suggestion could be
--fixbroken=ISO-8859-1 (a)
--fixbroken=octalescape (b)
--fixbroken=hexescape (c)
(a) would replace "0xf6" with "0xc3 0xb6"
(b) could write "\366"
(c) could write "<F6>"
The exact form of the syntax can be discussed of course.
However, I would probably start with (a), and add other options
if needed.
* should we just leave things as they are?
.... not the ideal thing.