CRLF, LF ... CR ?

Hi everyone.

I'm quite fond of git, and have used it for a while.
Recently, I've started making printed circuit boards (PCBs) using an application called OsmondPCB (for Mac), and I'd like to use git to track changes on these.
This application was originally written for the old Mac OS (Mac OS 6 to Mac OS 9.2).
The old Mac OS does not use LF, nor CRLF for line endings, but only CR.

I've read that git supports two different line endings; either CRLF or LF, but it does not support CR.
Would it make sense to add support for CR (if so, I hereby request it as a new feature) ?
The alternative is to ask the developer if he would change the file format, so that new versions of his software would change the files to end in LF, but he'd have to be careful not to break compatibility.
If the software is to be changed, this would not fix similar issues that other people might have.


Love
Jens

On Thu, 2012-09-13 at 17:09 +0200, Jens Bauer wrote:

Hi everyone.

I'm quite fond of git, and have used it for a while.
Recently, I've started making printed circuit boards (PCBs) using an application called OsmondPCB (for Mac), and I'd like to use git to track changes on these.
This application was originally written for the old Mac OS (Mac OS 6 to Mac OS 9.2).
The old Mac OS does not use LF, nor CRLF for line endings, but only CR.

I've read that git supports two different line endings; either CRLF or LF, but it does not support CR.
Would it make sense to add support for CR (if so, I hereby request it as a new feature) ?

Jens,
Even if Git can't do CRLF/LF translation on a file it will still store
and track the content of it it just fine. In fact you probably want
translation completely disabled in this case. 

(Given that you seem to be working on a Mac I suspect that "disabled" is
likely the default setting for that configuration item.)

-- 
-Drew Northup
________________________________________________
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59

On Thu, Sep 13, 2012 at 11:34:50AM -0400, Drew Northup wrote:

quoted

I've read that git supports two different line endings; either CRLF or LF, but it does not support CR.
Would it make sense to add support for CR (if so, I hereby request it as a new feature) ?

Even if Git can't do CRLF/LF translation on a file it will still store
and track the content of it it just fine. In fact you probably want
translation completely disabled in this case. 

Yeah. If the files always should just have CR, then just don't ask git
to do any translation (by not setting the "text" attribute, or even
setting "-text" if you have something like autocrlf turned on globally),
and it will preserve the bytes exactly. I suspect diffs will look nasty
because we won't interpret CR as a line-ending, though.

Do the files actually need line-by-line diffing and merging? If not,
then you are fine.

If so, then it would probably be nice to store them with a canonical LF
in the repository, but convert to CR on checkout. Git can't do that
internally, but you could define clean/smudge filters to do so (see the
section in "git help attributes" on "Checking-out and checking-in";
specifically the "filter" subsection).

-Peff

On Thu, Sep 13, 2012 at 5:34 PM, Drew Northup [off-list ref] wrote:

On Thu, 2012-09-13 at 17:09 +0200, Jens Bauer wrote:

quoted

Hi everyone.

I'm quite fond of git, and have used it for a while.
Recently, I've started making printed circuit boards (PCBs) using an application called OsmondPCB (for Mac), and I'd like to use git to track changes on these.
This application was originally written for the old Mac OS (Mac OS 6 to Mac OS 9.2).
The old Mac OS does not use LF, nor CRLF for line endings, but only CR.

I've read that git supports two different line endings; either CRLF or LF, but it does not support CR.
Would it make sense to add support for CR (if so, I hereby request it as a new feature) ?

Jens,
Even if Git can't do CRLF/LF translation on a file it will still store
and track the content of it it just fine. In fact you probably want
translation completely disabled in this case.

(Given that you seem to be working on a Mac I suspect that "disabled" is
likely the default setting for that configuration item.)

This might be less than ideal, because diffing and merging will break.

Perhaps clean/smudge filters can be used instead? That way you can
have CR on disk and LF in the repo.

Hi Jeff and Drew.

Thank you for your quick replies! :)

The diffs look nasty yes; that's my main issue.
It can be worked around in many ways; eg a simple (but time consuming) way:
$ git diff mypcb.osm >mypcb.diff && nano mypcb.diff

-It'd be better to just pipe it into a regex, which changes CR to LF on the fly.

OsmondPCB is able to read files that has mixed LF and CR. (By mixed, I do not talk about CRLF)

The files do not need line-by-line diffing, but I think it would make it more readable.
Thank you very much for the hint on the clean/smudge filters. I'll have a look at it. =)


Love
Jens

On Thu, 13 Sep 2012 11:43:10 -0400, Jeff King wrote:

On Thu, Sep 13, 2012 at 11:34:50AM -0400, Drew Northup wrote:

quoted

quoted

I've read that git supports two different line endings; either CRLF 
or LF, but it does not support CR.
Would it make sense to add support for CR (if so, I hereby request 
it as a new feature) ?

Even if Git can't do CRLF/LF translation on a file it will still store
and track the content of it it just fine. In fact you probably want
translation completely disabled in this case. 

Yeah. If the files always should just have CR, then just don't ask git
to do any translation (by not setting the "text" attribute, or even
setting "-text" if you have something like autocrlf turned on globally),
and it will preserve the bytes exactly. I suspect diffs will look nasty
because we won't interpret CR as a line-ending, though.

Do the files actually need line-by-line diffing and merging? If not,
then you are fine.

If so, then it would probably be nice to store them with a canonical LF
in the repository, but convert to CR on checkout. Git can't do that
internally, but you could define clean/smudge filters to do so (see the
section in "git help attributes" on "Checking-out and checking-in";
specifically the "filter" subsection).

-Peff

Hi Jeff and Drew.

I've been messing a little with clean/smudge filters; I think I understand them partly.

Let's call the file I have on the server that have <cr> line endings, "mypcb.osm".
If I clone the project, and do the following...
$ cat mypcb.osm | tr '\r' '\n'
I can read the file in the terminal window, otherwise it's just a "one-line-file".

So far, so good.
In my home directory, I have a .gitconfig file, here's the interesting part:
[core]
        editor = nano
        excludesfile = /Users/jens/.gitexcludes
        attributesfile = /Users/jens/.gitattributes

[filter "cr"]
        clean = tr '\\r' '\\n'
        smudge = tr '\\n' '\\r'


In my home directory I added .gitattributes:
*.osm   filter=cr

I've verified that .gitattributes is read; because if I add two spaces, like "*.osm filter = cr", I get an 'invalid filter name' error.
I've also verified that the clean/smudge lines are read; if I only have '\n' for instance, I get an error.

Now, when I clone the project, make a change and then issue this command...
$ git diff mypcb.osm

...I get a strange diff. On line 3, one of the files shows a lot of control-m (<cr>) lines.
After that, I see <lf> lines, all prefixed with a '+', as if they were added.

I think I might be nearly there, just missing some obvious detail somewhere.
Any hints ?


Love
Jens

On Thu, 13 Sep 2012 17:53:00 +0200, Jens Bauer wrote:

Hi Jeff and Drew.

Thank you for your quick replies! :)

The diffs look nasty yes; that's my main issue.
It can be worked around in many ways; eg a simple (but time consuming) way:
$ git diff mypcb.osm >mypcb.diff && nano mypcb.diff

-It'd be better to just pipe it into a regex, which changes CR to LF 
on the fly.

OsmondPCB is able to read files that has mixed LF and CR. (By mixed, 
I do not talk about CRLF)

The files do not need line-by-line diffing, but I think it would make 
it more readable.
Thank you very much for the hint on the clean/smudge filters. I'll 
have a look at it. =)


Love
Jens

On Thu, 13 Sep 2012 11:43:10 -0400, Jeff King wrote:

quoted

On Thu, Sep 13, 2012 at 11:34:50AM -0400, Drew Northup wrote:

quoted

quoted

I've read that git supports two different line endings; either CRLF 
or LF, but it does not support CR.
Would it make sense to add support for CR (if so, I hereby request 
it as a new feature) ?

Even if Git can't do CRLF/LF translation on a file it will still store
and track the content of it it just fine. In fact you probably want
translation completely disabled in this case. 

Yeah. If the files always should just have CR, then just don't ask git
to do any translation (by not setting the "text" attribute, or even
setting "-text" if you have something like autocrlf turned on globally),
and it will preserve the bytes exactly. I suspect diffs will look nasty
because we won't interpret CR as a line-ending, though.

Do the files actually need line-by-line diffing and merging? If not,
then you are fine.

If so, then it would probably be nice to store them with a canonical LF
in the repository, but convert to CR on checkout. Git can't do that
internally, but you could define clean/smudge filters to do so (see the
section in "git help attributes" on "Checking-out and checking-in";
specifically the "filter" subsection).

-Peff

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Thu, Sep 13, 2012 at 08:17:20PM +0200, Jens Bauer wrote:

In my home directory, I have a .gitconfig file, here's the interesting part:
[core]
        editor = nano
        excludesfile = /Users/jens/.gitexcludes
        attributesfile = /Users/jens/.gitattributes

[filter "cr"]
        clean = tr '\\r' '\\n'
        smudge = tr '\\n' '\\r'


In my home directory I added .gitattributes:
*.osm   filter=cr

Looks right.

Now, when I clone the project, make a change and then issue this command...
$ git diff mypcb.osm

...I get a strange diff. On line 3, one of the files shows a lot of control-m (<cr>) lines.
After that, I see <lf> lines, all prefixed with a '+', as if they were added.

I think I might be nearly there, just missing some obvious detail somewhere.

Yes, that's expected.  The point of the "clean" filter is to convert
your working tree file into a canonical (lf-only) representation inside
the repository. But you've already made commits with the cr form in the
repository. So you can choose one of:

  1. Make a new commit with these settings, which will have the
     canonical format. Accept that the old history will be funny, but
     you will be OK from here on out.

  2. Rewrite the old history to pretend that it was always LF. This
     gives you a nice clean history, but if you are collaborating with
     other people, they will need to rebase their work on the new
     history. See "git help filter-branch" for details.

-Peff

Am 13.09.2012 17:53, schrieb Jens Bauer:

Hi Jeff and Drew.

Thank you for your quick replies! :)

The diffs look nasty yes; that's my main issue.
It can be worked around in many ways; eg a simple (but time consuming) way:
$ git diff mypcb.osm >mypcb.diff && nano mypcb.diff

-It'd be better to just pipe it into a regex, which changes CR to LF on the fly.

OsmondPCB is able to read files that has mixed LF and CR. (By mixed, I do not talk about CRLF)

That is good news. Just write a 'clean' filter that amounts to

   tr '\015' '\012'

You don't need a 'smudge' filter that reverts this conversion.

-- Hannes

Hi Jeff and Drew.

Excellent. I now removed the repository from the server, removed it from my gitolite.conf, added it to gitolite.conf, re-initialized and it works.
git diff shows what I wanted.

Thank you *very* much for making my dream come true. :)
-And thank you all for all the hard work you're doing. -Git is how all other open-source projects should be: Well-written and well-defined (oh - and fast!). :)


Love
Jens

On Thu, 13 Sep 2012 14:23:44 -0400, Jeff King wrote:

On Thu, Sep 13, 2012 at 08:17:20PM +0200, Jens Bauer wrote:

quoted

In my home directory, I have a .gitconfig file, here's the 
interesting part:
[core]
        editor = nano
        excludesfile = /Users/jens/.gitexcludes
        attributesfile = /Users/jens/.gitattributes

[filter "cr"]
        clean = tr '\\r' '\\n'
        smudge = tr '\\n' '\\r'


In my home directory I added .gitattributes:
*.osm   filter=cr

Looks right.

quoted

Now, when I clone the project, make a change and then issue this command...
$ git diff mypcb.osm

...I get a strange diff. On line 3, one of the files shows a lot of 
control-m (<cr>) lines.
After that, I see <lf> lines, all prefixed with a '+', as if they 
were added.

I think I might be nearly there, just missing some obvious detail 
somewhere.

Yes, that's expected.  The point of the "clean" filter is to convert
your working tree file into a canonical (lf-only) representation inside
the repository. But you've already made commits with the cr form in the
repository. So you can choose one of:

  1. Make a new commit with these settings, which will have the
     canonical format. Accept that the old history will be funny, but
     you will be OK from here on out.

  2. Rewrite the old history to pretend that it was always LF. This
     gives you a nice clean history, but if you are collaborating with
     other people, they will need to rebase their work on the new
     history. See "git help filter-branch" for details.

-Peff

Hi Johannes.

I've changed...
	tr '\\r' '\\n'
...to...
	tr '\\15' '\\12'

...As you are right in that it is more correct. (Then in theory, it would be portable).
[I once came across tftpd, tried compiling it on a Mac, but it failed to work, because \r and \n were swapped on the compiler, so I asked the author to use \15 and \12, which made it fully portable]

It now works even better. I can't and won't complain - thank you. =)


Love
Jens

On Thu, 13 Sep 2012 20:34:08 +0200, Johannes Sixt wrote:

Am 13.09.2012 17:53, schrieb Jens Bauer:

quoted

Hi Jeff and Drew.

Thank you for your quick replies! :)

The diffs look nasty yes; that's my main issue.
It can be worked around in many ways; eg a simple (but time consuming) way:
$ git diff mypcb.osm >mypcb.diff && nano mypcb.diff

-It'd be better to just pipe it into a regex, which changes CR to LF 
on the fly.

OsmondPCB is able to read files that has mixed LF and CR. (By mixed, 
I do not talk about CRLF)

That is good news. Just write a 'clean' filter that amounts to

   tr '\015' '\012'

You don't need a 'smudge' filter that reverts this conversion.

-- Hannes

On Thu, Sep 13, 2012 at 8:09 AM, Jens Bauer [off-list ref] wrote:

Hi everyone.

I'm quite fond of git, and have used it for a while.
Recently, I've started making printed circuit boards (PCBs) using an application called OsmondPCB (for Mac), and I'd like to use git to track changes on these.
This application was originally written for the old Mac OS (Mac OS 6 to Mac OS 9.2).
The old Mac OS does not use LF, nor CRLF for line endings, but only CR.

I've read that git supports two different line endings; either CRLF or LF, but it does not support CR.
Would it make sense to add support for CR (if so, I hereby request it as a new feature) ?
The alternative is to ask the developer if he would change the file format, so that new versions of his software would change the files to end in LF, but he'd have to be careful not to break compatibility.
If the software is to be changed, this would not fix similar issues that other people might have.

Do you mean that you want automatic conversion from CR to LF?

What's about just storing the files as-is,
with no conversion at all? (this is the default git behavior)

git doesn't really even support LF.  It stores content as-is which
means LF works just fine.  git prefers to not mess around with the content,
but we do have autocrlf to help folks stuck on windows.

If you need to, you can use .gitattributes to add a clean/smudge filter
that does this conversion for you.

See the "filter" section for an example:

http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

If you're serious about wanting that feature then we'll
happily review any patches you might have.  That said, I don't really
think it's a common enough case for git to natively support, so
I'd recommend going with the .gitattributes filter.

good luck,
-- 
David

Hi David.

Thank you for the information.
I am not the one to decide whether or not this should be a built-in feature; I'm only a plain user. :)
My personal opinion: If it was a built-in feature, people all over the World would not be getting into trouble with it.
I've seen on the net, that even on PC (I don't know if it's DOS or Windows), a user wrote a program, that did not output CRLF, nor LF, but only CR.

Perhaps git does not treat LF or CRLF in any special way, if that's the case, git's very beautifully programmed.
-But gitweb does not seem to make correct output unless I add a clean/smudge filter.
In addition to that, there is also the detail, that the terminal treats CR and LF differently...
echo -e "this\ris\ra\rtest"
will in bash only output...
test
...while
echo -e "this\nis\na\ntest"
in bash outputs...
this
is
a
test
...This means that 'git diff' will (to the user) only output a single line of "junk", when line-endings are CR and not LF or CRLF.

Also... git diff says that a file containing CR line endings have no end-of-line, which to me appears as that git diff thinks there is only one line.

Finally, when I want to display the file using head, cat or tail, I still see only one line of text. [That's correct behaviour!]

As you mentioned, there are good possibilites for fixing these things using clean/smudge filters.
The clean/smudge filter I ended up using is quite simple (much simpler than I first thought it would be).
I ended up with...
[filter "cr"]
	clean = tr '\\15' '\\12'
...and no smudge, because the application supports reading lines ending in LF, so I wanted the files to stay readable.
-But normally, there should ofcourse be a smudge filter, that reverses \15 and \12.

In my opinion, the benefit from having git diff support CR, would be that the user does not lose the history if he/she wants to 'clean' the projects.
Existing projects would be shown correctly. Of course, gitweb would need to support CR, so it generates correct line endings, instead of showing one long line with a lot of "\r" in them.

[When thinking about all these CR/LF problems, I am grateful that Sinclair Research did not continue to use the ZX81 characterset in the ZX Spectrum. Imagine a mess we could have had!]


Love
Jens


On Thu, 13 Sep 2012 21:06:32 -0700, David Aguilar wrote:

On Thu, Sep 13, 2012 at 8:09 AM, Jens Bauer [off-list ref] wrote:

quoted

Hi everyone.

I'm quite fond of git, and have used it for a while.
Recently, I've started making printed circuit boards (PCBs) using an 
application called OsmondPCB (for Mac), and I'd like to use git to 
track changes on these.
This application was originally written for the old Mac OS (Mac OS 6 
to Mac OS 9.2).
The old Mac OS does not use LF, nor CRLF for line endings, but only CR.

I've read that git supports two different line endings; either CRLF 
or LF, but it does not support CR.
Would it make sense to add support for CR (if so, I hereby request 
it as a new feature) ?
The alternative is to ask the developer if he would change the file 
format, so that new versions of his software would change the files 
to end in LF, but he'd have to be careful not to break compatibility.
If the software is to be changed, this would not fix similar issues 
that other people might have.

Do you mean that you want automatic conversion from CR to LF?

What's about just storing the files as-is,
with no conversion at all? (this is the default git behavior)

git doesn't really even support LF.  It stores content as-is which
means LF works just fine.  git prefers to not mess around with the content,
but we do have autocrlf to help folks stuck on windows.

If you need to, you can use .gitattributes to add a clean/smudge filter
that does this conversion for you.

See the "filter" section for an example:

http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

If you're serious about wanting that feature then we'll
happily review any patches you might have.  That said, I don't really
think it's a common enough case for git to natively support, so
I'd recommend going with the .gitattributes filter.

good luck,
-- 
David