Re: Should the --encoding argument to log/show commands make any guarantees about their output?

On 17.06.2015 18:42, Junio C Hamano wrote:

Jan-Philip Gehrcke [off-list ref] writes:

quoted

I was surprised to see that the output of

     git log --encoding=utf-8 "--format=format:%b"

can contain byte sequences that are invalid in UTF-8. Note: I am using
git 2.1.4 and the %b format specifier represents the commit message
body.

Yeah, if the original was bad and cannot be sanely expressed in
UTF-8, you have two options.  You can show the contents as raw bytes
recorded in the object with a warning so that the user can use it as
such (e.g. perhaps the original was indeed an iso8859-2 but was
incorrectly marked as UTF-8, or something like that, and a human
that is more intelligent than a tool _could_ guess and attempt to
recover).  Or you can error out and refuse to produce output.

The two-option scenario is totally clear. Although one must stress that 
the "error-out" option can, as discussed, be kept minimally invasive: it 
is sufficient (and common) to just skip those byte sequences (and 
replace them with a replacement symbol) that would be invalid in the 
requested output encoding. This would retain as much information as 
possible while guaranteeing a subsequent decoder to retrieve valid input.

We deliberately made a design choice to take the former option.

I totally support this design choice in general, especially when 
invoking `git whatever` without options. This here is, I think, mainly 
about documentation and the semantics of "--encoding". From my point of 
view, `--encoding=utf-8` semantically suggests that the output *is* 
valid UTF-8. But it is not, not always. May initial question was: what 
do you think about this? Should we

* just make this more clear in the docs and/or
* should we adjust the behavior of --encoding or
* should we do something entirely different, like adding a new command 
line option or
* should we just leave things as they are?

Thanks and cheers,


Jan-Philip