Linus Torvalds [off-list ref] writes:
So I repeat:
- escape as little as possible
- make the _viewer_ decide how to view it.
Under my most recent proposal, the only bytes one must escape are ",
\, and LF. Doesn't that satisfy these two main criteria?
If GNU emacs does locale translations rather than just do a binary
transfer of the data, then that's a sign that GNU emacs is being
really stupid.
Perhaps so, but it has a lot of company. I have even worse problems
with Mozilla Thunderbird. And as we observed, Pine also has problems
sending properly-formatted email containing arbitrary binary data.
I suspect the vast majority of email clients will screw up in
relatively common cases involving unusual characters in file names.
Using attachments avoids many of the problems, but lots of patches are
emailed inline and I'd rather not force people to use attachments to
send diffs.
I find that email is very robust - it's basically 8-bit clean. No
character encoding, no crap. Just a byte stream. It really _is_ the most
reliable format.
Hmm. To test that theory, I just now sent plain-text email to myself,
containing a carriage-return (CR) byte in the middle of a line.
The CR byte was transliterated into a LF. Ooops.
This was the very first (and only) test I tried, which isn't a good
sign for reliability. If you're curious, I tracked the problem down
to Exim, a popular mail transfer agent that is running on my personal
Debian GNU/Linux (stable) box. As to why Exim munges email, please see
<http://www.exim.org/exim-html-4.40/doc/html/spec_44.html#SECT44.1>.
(And I didn't know about the Exim glitch before trying my test.
I'm normally a Sendmail man myself.)
More generally, I suspect inline patches with weird bytes will suffer
greatly from encoding and recoding by mail agents.
What matters is not what it looks like, but what it _saves_ as. If
you save the email message, it should come out as the same reliable
8-bit byte stream
Unfortunately this isn't true for Emacs, and I suspect other mailers
will have similar problems. For example, with Emacs I can easily save
either the exact byte-for-byte message body that my mail transfer
agent gave me; or I can have Emacs decode the message into its
constituent characters, reencode the result as UTF-8, and put that
into a file. In neither case, though, am I saving the original byte
stream that you presented to your mail user agent. Even if I save the
byte-for-byte message body, it is often in quoted-printable format so
I'll have to decode strings like "=EF" to recover the original bytes.
This is doable, yes, but it's inconvenient in practice, at least with
the mail user agents I'm familiar with. And even if I do it, I don't
necessarily have the same byte stream you gave your mail user agent; I
merely have the byte stream that your MUA gave to your MTA, and these
may not be the same thing (they certainly aren't always the same thing
with Emacs).
The simplest fix for git may be to say "Don't use inline patches; use
attachments if you must email anything with strange characters in it."
That's fine. But I prefer a format that also allows GNU diff, if it
chooses, to generate output that resists common inline-email botches.
On Thu, 13 Oct 2005, Paul Eggert wrote:
Perhaps so, but it has a lot of company. I have even worse problems
with Mozilla Thunderbird. And as we observed, Pine also has problems
sending properly-formatted email containing arbitrary binary data.
No, pine does it right. Exactly because it sends _arbitraty_ binary data.
The fact that I turned the terminal into utf-8 mode in order to generate
the bytes (that end up being a garbage string in latin1) is not pine's
fault.
The point being that because the transport was 8-bit clean, I could do
that. I could mix a latin1-encoding with a UTF-8 encoding, and the other
side could see the mixed setting. Now, the other side had no way of
knowing that I mixed things (unless it was a smart human and could read
and understand what I wrote), so any email client would have trouble
showing it.
But it got _transferred_ right, and you could have saved the email, and
turned the terminal into latin1 or utf-8 mode, and done a "cat" both ways,
and you'd have seen both versions.
I suspect the vast majority of email clients will screw up in
relatively common cases involving unusual characters in file names.
Not if they just save it.
Oh, sure, they can't _display_ it, since they don't know what it is, but
when they save it, they'd _better_ save it bit-for-bit.
Which is the right thing to do. Then you apply it with "patch", and you
get the right answer.
Using attachments avoids many of the problems, but lots of patches are
emailed inline and I'd rather not force people to use attachments to
send diffs.
inline or attachment should not matter to any sane email client. If it
does, then the email client isn't sane.
The point is, when you save it, it _has_ to be saved bit-for-bit.
The only difference between a binary attachment and a text thing is that
an email client will _try_ to show the text thing to you as text. It has
no other meaning.
And trying is better than not trying. Attachments are _inferior_ to inline
for that reason.
quoted
I find that email is very robust - it's basically 8-bit clean. No
character encoding, no crap. Just a byte stream. It really _is_ the most
reliable format.
Hmm. To test that theory, I just now sent plain-text email to myself,
containing a carriage-return (CR) byte in the middle of a line.
The CR byte was transliterated into a LF. Ooops.
I'm not surprised, since CR/LF is special for a lot of (sad) reasons. Oh,
well.
I agree that it makes sense to escape \r, and obviously you _have_ to
escape \n. In general, escaping pretty much everything in the 0-31 range
is likely the right approach, since those are never printable anyway.
That, btw, is probably true of the patch contents too, not just the
filename. The exception being \t (and in patch contents, \n is obviously
part of the stream).
More generally, I suspect inline patches with weird bytes will suffer
greatly from encoding and recoding by mail agents.
I've had pretty good luck. We do have 8-bit stuff occasionally, but it
almost always makes it through.
Spaces and tabs are much worse (yes, they're more common too). That's
clearly just crap mailers.
Unfortunately this isn't true for Emacs, and I suspect other mailers
will have similar problems. For example, with Emacs I can easily save
either the exact byte-for-byte message body that my mail transfer
agent gave me; or I can have Emacs decode the message into its
constituent characters, reencode the result as UTF-8, and put that
into a file.
Well, as long as there's a choice.
In neither case, though, am I saving the original byte
stream that you presented to your mail user agent. Even if I save the
byte-for-byte message body, it is often in quoted-printable format so
I'll have to decode strings like "=EF" to recover the original bytes.
You have a broken mail client. Now, I'm not a big fan of QP (I think it
was making a stupid excuse for bad transport), but QP is a _mail_ level
quoting protocol, and the same way a MUA uses QP to encode, the MUA should
have de-coded the QP. It shouldn't leave it to somebody else.
I think GNU emacs is a horrible mistake ("do everything - badly"), but you
may be able to fix it by letting your mail transport agent do the un-QP
for you. A lot of them do, which makes it easier to then use weak MUA's.
Anyway, it sounds like GNU emacs made the wrong choices (hey, I'm not
surprised). It should have decoded QP, not the character set. There are
lots of tools that do charset conversions, that's not very email-specific.
Linus