More git status --porcelain lossage

After I posted my last, I noticed another crash landing...

A format properly designed for script parseability should use even use
whitespace as a field separator.

Why?

Because if you do that, front ends *will* do field analysis using a
naive split-on-whitespace operation.  And then...someday...someone
will try to run one of these of these on a volume from a system where
filenames contain embedded whitespace.  Like Mac OS X or Windows.

Hilarity will ensue.

Conclusion: As it is presently, git status --porcelain format is
irretrievably botched.  You need a field separator that's musch less
likely to land in a filename, like '|' - and to warn in the documentation
that careful front ends must check for and ignore '\|'. 
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

The right of the citizens to keep and bear arms has justly been considered as
the palladium of the liberties of a republic; since it offers a strong moral
check against usurpation and arbitrary power of rulers; and will generally,
even if these are successful in the first instance, enable the people to resist
and triumph over them."
        -- Supreme Court Justice Joseph Story of the John Marshall Court

Eric Raymond [off-list ref]:

A format properly designed for script parseability should use even use
whitespace as a field separator.

should *not* even use... 
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Eric Raymond [off-list ref] writes:

After I posted my last, I noticed another crash landing...

A format properly designed for script parseability should use even use
whitespace as a field separator.

Why?

Because if you do that, front ends *will* do field analysis using a
naive split-on-whitespace operation.  And then...someday...someone
will try to run one of these of these on a volume from a system where
filenames contain embedded whitespace.  Like Mac OS X or Windows.

Hilarity will ensue.

Conclusion: As it is presently, git status --porcelain format is
irretrievably botched.  You need a field separator that's musch less
likely to land in a filename, like '|' - and to warn in the documentation
that careful front ends must check for and ignore '\|'. 

Or follow what other porcelain does, like git-diff-tree raw output
format, where all fields except final filename are space separated,
filename is separated by tab character (or NUL when '-z' options is
used).  If there are two names (in the case of copy or renames),
they are separated by a tab (or NUL).  Record ends with LF (or NUL).

When '-z' option is not used, TAB, LF, " and backslash characters
are represented by '\t', '\n', '\"' and \\, and the filename is
enclosed in '"' doublequotes.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Jakub Narebski [off-list ref]:

quoted

Conclusion: As it is presently, git status --porcelain format is
irretrievably botched.  You need a field separator that's musch less
likely to land in a filename, like '|' - and to warn in the documentation
that careful front ends must check for and ignore '\|'. 

Or follow what other porcelain does, like git-diff-tree raw output
format, where all fields except final filename are space separated,
filename is separated by tab character (or NUL when '-z' options is
used).  If there are two names (in the case of copy or renames),
they are separated by a tab (or NUL).  Record ends with LF (or NUL).

When '-z' option is not used, TAB, LF, " and backslash characters
are represented by '\t', '\n', '\"' and \\, and the filename is
enclosed in '"' doublequotes.

That would be a bit trickier to parse, but acceptable.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

On Fri, Apr 09, 2010 at 03:06:01PM -0400, Eric Raymond wrote:

A format properly designed for script parseability should use even use
whitespace as a field separator.

Why?

Because if you do that, front ends *will* do field analysis using a
naive split-on-whitespace operation.  And then...someday...someone
will try to run one of these of these on a volume from a system where
filenames contain embedded whitespace.  Like Mac OS X or Windows.

Yes, that is why almost every scriptable git interface supports a "-z"
variant with NUL termination.

Conclusion: As it is presently, git status --porcelain format is
irretrievably botched.  You need a field separator that's musch less
likely to land in a filename, like '|' - and to warn in the documentation
that careful front ends must check for and ignore '\|'.

We already quote correctly, so it is only sloppy parsers that will be in
trouble. Yes, space is more common than "|", but sloppy is sloppy. Parse
it right, or use "-z".

-Peff

On Sat, Apr 10, 2010 at 12:12:48AM -0400, Jeff King wrote:

quoted

Conclusion: As it is presently, git status --porcelain format is
irretrievably botched.  You need a field separator that's musch less
likely to land in a filename, like '|' - and to warn in the documentation
that careful front ends must check for and ignore '\|'.

We already quote correctly, so it is only sloppy parsers that will be in
trouble. Yes, space is more common than "|", but sloppy is sloppy. Parse
it right, or use "-z".

BTW, this should go on your "git status --porcelain documentation
failures" list. We really need to note that the output paths may be
quoted.

-Peff

A format properly designed for script parseability should use even use
whitespace as a field separator.

Why?

Because if you do that, front ends *will* do field analysis using a
naive split-on-whitespace operation.  And then...someday...someone
will try to run one of these of these on a volume from a system where
filenames contain embedded whitespace.  Like Mac OS X or Windows.

Why not use an XML output?
Plain text is easier to parse, but XML may give this extra durability
you are looking for?

Simon

Simon [off-list ref] writes:

quoted

A format properly designed for script parseability should use even use
whitespace as a field separator.

Why?

Because if you do that, front ends *will* do field analysis using a
naive split-on-whitespace operation.  And then...someday...someone
will try to run one of these of these on a volume from a system where
filenames contain embedded whitespace.  Like Mac OS X or Windows.

Why not use an XML output?
Plain text is easier to parse, but XML may give this extra durability
you are looking for?

Are out of your f**g mind?  XML, really?  XML might be good choice to
*define* _document_ formats, but is really poor data exchange /
serialization format (being overly verbose, among others).  Also, XML
is not language but meta-language.

I could understand providing JSON format, specified using --json
option.  I think there is some GPLv2 compatibile JSON generating code
in C (MIT licensed code is GPLv2 compatibilie, isn't it?); we can
always borrow compact JSON generation code from GPSD project (if
license allows it) from ESR.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Simon [off-list ref]:

Why not use an XML output?
Plain text is easier to parse, but XML may give this extra durability
you are looking for?

Because XML is awfully heavyewight, and XML parsers tend to be slow.

If we were going to buld on a metaprotocol, JSON would be better.  IMHO.  
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Jakub Narebski [off-list ref]:

Are out of your f**g mind?  XML, really?  XML might be good choice to
*define* _document_ formats, but is really poor data exchange /
serialization format (being overly verbose, among others).  Also, XML
is not language but meta-language.

Agreed.
 

I could understand providing JSON format, specified using --json
option.

You know, that's actually an interesting idea.  I mentioned it
previously as the not-XML if we want to build on a metaprotocol;
I wasn't considering it seriously then.  But I am now, and it is
not without attractions.  JSON would certainly solve all the delimiter
and empty-object edge cases, and it has excellent extensibility.

   I think there is some GPLv2 compatibile JSON generating code
in C (MIT licensed code is GPLv2 compatibilie, isn't it?); we can
always borrow compact JSON generation code from GPSD project (if
license allows it) from ESR.

My license would allow it, but there's not really a lot of win in 
trying to reuse JSON generator code - writing your own printfs for
it by hand is easy and fast.

Emacs Lisp has a JSON parser, so it would meet my needs.

Alternatively, a cleaned-up --porcelain -Z along the lines
previously suggested would be good.

Supplying both might not be a bad idea.  The volume of code involved
would be low.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

On Sat, Apr 10, 2010 at 19:30, Eric Raymond [off-list ref] wrote:

Simon [off-list ref]:

quoted

Why not use an XML output?
Plain text is easier to parse, but XML may give this extra durability
you are looking for?

Because XML is awfully heavyewight, and XML parsers tend to be slow.

If we were going to buld on a metaprotocol, JSON would be better.  IMHO.

A lot of web services (like some Catalyst-based applications) support
all of these equally. If Git had machine readable output like this it
would be nice if every git-* program just had --format=* where * could
be xml, json, yaml, sexp, perl etc.

The program would just construct a native datastructure and then there
would be an output driver to generate the textual representation.

On Sat, Apr 10, 2010 at 3:41 PM, Eric Raymond [off-list ref] wrote:

quoted

I could understand providing JSON format, specified using --json
option.

You know, that's actually an interesting idea.  I mentioned it
previously as the not-XML if we want to build on a metaprotocol;

One issue is that there's no stream-parser JSON implementations that
I'm aware of.

Everthing I've seen is in-memory, therefore apt only for memory-bound
operations. Not sure if all commands with -z output options can be
assumed to produce bound-sized datasets.

cheers,


martin
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

On Sat, 10 Apr 2010, Eric Raymond wrote:

Jakub Narebski [off-list ref]:
 

quoted

I could understand providing JSON format, specified using --json
option.

You know, that's actually an interesting idea.  I mentioned it
previously as the not-XML if we want to build on a metaprotocol;
I wasn't considering it seriously then.  But I am now, and it is
not without attractions.  JSON would certainly solve all the delimiter
and empty-object edge cases, and it has excellent extensibility.

It is a bit chatty, but is to some extent self documenting.

The question is whether it should output well formed array of objects,
or just list of objects not wrapped in array...

quoted

   I think there is some GPLv2 compatibile JSON generating code
in C (MIT licensed code is GPLv2 compatibilie, isn't it?); we can
always borrow compact JSON generation code from GPSD project (if
license allows it) from ESR.

My license would allow it, but there's not really a lot of win in 
trying to reuse JSON generator code - writing your own printfs for
it by hand is easy and fast.

What I am worrying about is correct handling of escaping, quoting,
and non-ASCII characters in strings (the JSON-quoting and JSON-escapes
are different than C escape codes, IIRC).  JSON rules are simple,
but are different than C.

-- 
Jakub Narebski
Poland

A lot of web services (like some Catalyst-based applications) support
all of these equally. If Git had machine readable output like this it
would be nice if every git-* program just had --format=* where * could
be xml, json, yaml, sexp, perl etc.

The program would just construct a native datastructure and then there
would be an output driver to generate the textual representation.

I had something just like this in mind when I suggested XML...
I would personally avoid it for same reasons others have pointed out, but...
There are lots of tools out there that can parse and display XML very
well natively.  Firefox is one such example.

My intention is not to start a flame here, rather try to keep our
options flexible.  ASCII would clearly remain the default though! ;)

Simon

On 04/10/2010 10:31 PM, Martin Langhoff wrote:

On Sat, Apr 10, 2010 at 3:41 PM, Eric Raymond[off-list ref]  wrote:

quoted

quoted

I could understand providing JSON format, specified using --json
option.

You know, that's actually an interesting idea.  I mentioned it
previously as the not-XML if we want to build on a metaprotocol;

One issue is that there's no stream-parser JSON implementations that
I'm aware of.

Here is one.  It's ugly as hell, you're warned.  The only missing piece 
is making the stack state resizable.

Paolo

Jakub Narebski [off-list ref]:

[JSON] is a bit chatty, but is to some extent self documenting.

Yes. But to my mind, the big win of JSON is that you can extend it without
breaking parsers looking for older versions - they just skip the new
fields and all is happy.

Jakub, you seem to know this, but other listmermbers may not: I've
recently re-engineered GPSD, a service daemon for watching geolocation
sensors, to report JSON objects up the socket to client apps.  The
benefits in clarity and extensibility of the protocol have been
*huge*.  Like, today I'm adding a reporting type for digital
compass/gyroscope sensors.

The question is whether it should output well formed array of objects,
or just list of objects not wrapped in array...

Yes, I know this dance.  Answer: one big JSON object, tagged by the
name of the output generator, and also *containing a version-stamp
field*.  Array of file status objects is another top-level member.

The point is: later, if we want to enrich the reporting format, we add
whatever fields we want and bump the version stamp.  Self-describing
goodness.  Python, Perl, JavaScript, and Emacs LISP clients win
especially big.  Slurping this into a native data structure is one
function call.

The more I think about this, the better I like it.
 

What I am worrying about is correct handling of escaping, quoting,
and non-ASCII characters in strings (the JSON-quoting and JSON-escapes
are different than C escape codes, IIRC).  JSON rules are simple,
but are different than C.

Yes. Perhaps there's some scope for reuse here after all.  GPSD has
well-tested code for uttering the JSON quote/escape conventions. 
The git project is welcome to it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Paolo Bonzini [off-list ref]:

quoted

One issue is that there's no stream-parser JSON implementations that
I'm aware of.

Here is one.  It's ugly as hell, you're warned.  The only missing
piece is making the stack state resizable.

I wrote one in C for the GPSD project that has two interesting
properties:

(1) No use of malloc(),

(2) Unpacks to *fixed-extent* data structures.

It has one language restriction: Array subelements all have to be the same type.

It's not a stream parser, so there will be compile-time limits on the
volume of data it can handle.  This isn't a big deal in the GPSD 
context, where the objects are relatively short (< 1K) datagrams.

It's very well tested and, I think, pretty bulletproof.  I've been thinking
of spinning it out as a reusable project.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

On Sun, Apr 11, 2010 at 12:28:36AM +0200, Paolo Bonzini wrote:

On 04/10/2010 10:31 PM, Martin Langhoff wrote:

quoted

On Sat, Apr 10, 2010 at 3:41 PM, Eric Raymond[off-list ref]  wrote:

quoted

quoted

I could understand providing JSON format, specified using --json
option.

You know, that's actually an interesting idea.  I mentioned it
previously as the not-XML if we want to build on a metaprotocol;

One issue is that there's no stream-parser JSON implementations that
I'm aware of.

Here is one.  It's ugly as hell, you're warned.  The only missing piece  
is making the stack state resizable.

Paolo

Here's a fairly popular stream parser:

http://lloyd.github.com/yajl/

Yet Another JSON Library. YAJL is a small event-driven
(SAX-style) JSON parser written in ANSI C, and a small
validating JSON generator. YAJL is released under the BSD
license.

The license is BSD-with-advertising-clause.
Perhaps the author did not know about modified BSD.

-- 
		David