Re: [PATCH] Try URI quoting for embedded TAB and LF in pathnames

Linus Torvalds [off-list ref] writes:

the simplest question to ask is "what are we protecting against?"

I'd like to protect against:

  1.  File names that cannot be handled correctly with the current
      formats.  Newline is the obvious problem here, along with
      (arguably) tab and space.

  2.  Common transliterations of patches.  Many programs (and mailers,
      alas) expand tabs to spaces, append CR to lines, prepend spaces
      to lines, break lines at spaces, etc.  'patch' already deals
      with this to some extent, but it'd be nice if the format
      resisted these transliterations better.

  3.  Humans misreading patches.  The patch format is intended to be
      human-readable, after all.

  4.  Reencoded patches.  Programs like Emacs can and will convert
      patches from UTF-8 to EUC-JP, for example.

You convinced me that (4) is not worth the hassle, but I'd still like
to address (1)-(3) when it's easy.

invalid UTF-8 [is] invalid UTF-8

Yes, but (2) and (3) can lose information about invalid UTF-8 if we
don't suitably protect the encoding errors.  I daresay that many
mailers will mishandle invalid UTF-8, for example.

There _is_ something you may want to quote, namely the standard CSI
terminal escapes.

If I understand you aright, we could do that by modifying my previous
proposal to escape all bytes in the UTF-8 representation of a control
character.  In Unicode, the characters 0080 through 009F are control
characters, so that should suffice to quote the terminal escapes you
mentioned.  (Perhaps we should also escape unassigned Unicode
characters too, on the theory that they might become control
characters in the future.)

For any UTF-8 quoting scheme you come up with, I'll point out
something that it does wrong or looks horrible for a Latin1 filename
;)

Yes, quite true.  But we don't have to come up with something that's
perfect in all cases, just something that's good enough to handle
cases that we expect will be common in practice, in a world where
UTF-8 is the preferred encoding for non-ASCII characters.

On Tue, 11 Oct 2005, Paul Eggert wrote:

Yes, quite true.  But we don't have to come up with something that's
perfect in all cases, just something that's good enough to handle
cases that we expect will be common in practice, in a world where
UTF-8 is the preferred encoding for non-ASCII characters.

The thing is, I can almost guarantee you that any quoting in the high 
characters is going to be _worse_ than no quoting at all.

Exactly because quoting as UTF-8 is the wrong thing when it isn't actually 
UTF-8, and quoting as non-UTF-8 is the wrong thing when it _is_.

Not quoting at all, on the other hand, is unambigious. If you have a 
mailer that corrupts your text stream (which-ever type it is), then it's 
clearly the mailers problem. The _mailer_ at least has a chance in hell to 
know what character set it is getting mailed as.

The other alternative is to quote _everything_ non-ASCII. That's 
definitely reliable, but it's also unquestionably ugly as hell, especially 
in the long run.

Yes, there are some complex quoting approaches you can do, which quote 
things "correctly" (ie at a byte stream level) _and_ keep it valid UTF-8 
at the same time.

For example, you can read it as a UTF-8 stream, but then quote things at a 
byte level (ie if you quote one "character", you quote _all_ bytes in that 
character). And you quote if:

 - the UTF-8 _character_ is in the 0x80-0x9f control range
 - any _raw_byte_ is in the 0x80-0x9f range (it might not be UTF-8)
 - any _raw_byte_ is 0xfe-0xff (illegal UTF-8 character)
 - misformed UTF-8 (non-shortest sequence, or just generally invalid 
   sequences with missing or wrong high bits)

but quite frankly, that's a pretty painful thing to write. The upside is 
that it's easy to decode: you can _unquote_ it just as a byte stream.

			Linus