From: Junio C Hamano <hidden> Date: 2016-06-15 22:44:34
"Ping Yin" [off-list ref] writes:
For this example,both "/if/while/ (i />/>=/ /1/0/)" and "/if/while/
(i >//=/ /1/0/)" are fine to me.
For the particular example, both are Ok, but for this other example:
-if (i > 1...
+if ((i > 1...
it probably is better to treat each non-word character as a separate
token, that is, it would be easier to read if we said "( stayed intact,
and another ( was added", instead of saying "( is changed to ((".
So "a run of punct chars" rule only sometimes produces better output but
otherwise worse output, and to make it produce better output consistently,
we would need to know the syntax of the target language for tokenization,
i.e. ">=" and ">" are comparison operators, while "(" is a token and "(("
is better split into two open-paren tokens.
So as a very longer term subproject, we may want to teach the mechanism
language specific tokenization rules, just like we can specify the hunk
header pattern via gitattributes(5) to the diff output layer.
Of course, I do not expect you to do that during this round --- and if we
choose to keep the rule simple, I think it is probably better to use
one-char-one-token rule for now.
And when designing, i think it's better to take multi-byte characters
into account. For multi-byte characters (especially CJK), every
character should be considered as a token.
If we take an idealistic view for the longer term, we should be tokenizing
even CJK sensibly, but unlike Occidental scripts, we cannot even use
inter-word spacing for tokenizing hint, so unless we are willing to learn
morphological analysis (which we are not for now), the best we can do is
to use one-char-one-token rule.
Side Note. For Japanese we could cheat and often do a slightly
better job than simple one-char-one-token without having full
morphological analysis by splicing between Kanji and Kana
boundaries, but I'd prefer not to go there and keep the rules we
would use to the minimum.
I should stress that I said "character" in the above "punct" and "CJK"
discussions, not "byte".
On Mon, May 5, 2008 at 1:00 PM, Junio C Hamano [off-list ref] wrote:
"Ping Yin" [off-list ref] writes:
> For this example,both "/if/while/ (i />/>=/ /1/0/)" and "/if/while/
> (i >//=/ /1/0/)" are fine to me.
For the particular example, both are Ok, but for this other example:
-if (i > 1...
+if ((i > 1...
it probably is better to treat each non-word character as a separate
token, that is, it would be easier to read if we said "( stayed intact,
and another ( was added", instead of saying "( is changed to ((".
So "a run of punct chars" rule only sometimes produces better output but
otherwise worse output, and to make it produce better output consistently,
we would need to know the syntax of the target language for tokenization,
i.e. ">=" and ">" are comparison operators, while "(" is a token and "(("
is better split into two open-paren tokens.
So as a very longer term subproject, we may want to teach the mechanism
language specific tokenization rules, just like we can specify the hunk
header pattern via gitattributes(5) to the diff output layer.
Of course, I do not expect you to do that during this round --- and if we
choose to keep the rule simple, I think it is probably better to use
one-char-one-token rule for now.
> And when designing, i think it's better to take multi-byte characters
> into account. For multi-byte characters (especially CJK), every
> character should be considered as a token.
If we take an idealistic view for the longer term, we should be tokenizing
even CJK sensibly, but unlike Occidental scripts, we cannot even use
inter-word spacing for tokenizing hint, so unless we are willing to learn
morphological analysis (which we are not for now), the best we can do is
to use one-char-one-token rule.
Side Note. For Japanese we could cheat and often do a slightly
better job than simple one-char-one-token without having full
morphological analysis by splicing between Kanji and Kana
boundaries, but I'd prefer not to go there and keep the rules we
would use to the minimum.
I should stress that I said "character" in the above "punct" and "CJK"
discussions, not "byte".
The one-char-one-token and multi-char-one-token rules may have
different implementation issues. I think multi-char-one-token rule may
be more representative. So for the current time, i prefer considering
both run of word characters and single non-word character as a token.
--
Ping Yin
On Mon, May 5, 2008 at 8:10 PM, Ping Yin [off-list ref] wrote:
The one-char-one-token and multi-char-one-token rules may have
different implementation issues. I think multi-char-one-token rule may
be more representative. So for the current time, i prefer considering
both run of word characters and single non-word character as a token.
If we agree on this. I will come up with an implementation still using
diff.nonwordchars few days later.
--
Ping Yin
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:34
Hi,
On Tue, 6 May 2008, Ping Yin wrote:
On Mon, May 5, 2008 at 8:10 PM, Ping Yin [off-list ref] wrote:
I will come up with an implementation still using diff.nonwordchars few
days later.
If I did not like the unnecessary negative approach "nonwordchars" (as
opposed to "wordchars"), it seems even less appropriate now, when you
actually want to discern between "spaceCharacters",
"punctuationCharacters" and "wordCharacters".
Hth,
Dscho
On Tue, May 6, 2008 at 4:55 PM, Johannes Schindelin
[off-list ref] wrote:
quoted
I will come up with an implementation still using diff.nonwordchars few
> days later.
If I did not like the unnecessary negative approach "nonwordchars" (as
opposed to "wordchars"), it seems even less appropriate now, when you
actually want to discern between "spaceCharacters",
"punctuationCharacters" and "wordCharacters".
Hmm, punctchars should be a better word than nonwordchars.
So how about this
--color-words={char,punct,word}
- char: one char one token
- punct/word: a token can be either a run of word characters or a
single punct character. diff.punctchars is used for punct, and
diff.wordchars is used for word.
We leave the choice to the user.
--
Ping Yin
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:34
Hi,
On Wed, 7 May 2008, Ping Yin wrote:
On Tue, May 6, 2008 at 4:55 PM, Johannes Schindelin
[off-list ref] wrote:
quoted
quoted
I will come up with an implementation still using diff.nonwordchars few
> days later.
If I did not like the unnecessary negative approach "nonwordchars" (as
opposed to "wordchars"), it seems even less appropriate now, when you
actually want to discern between "spaceCharacters",
"punctuationCharacters" and "wordCharacters".
Hmm, punctchars should be a better word than nonwordchars.
So how about this
--color-words={char,punct,word}
- char: one char one token
- punct/word: a token can be either a run of word characters or a
single punct character. diff.punctchars is used for punct, and
diff.wordchars is used for word.
I am rather interested in the semantics, i.e. if you can punch holes into
this 3-class approach.
Bikeshedding comes later ;-)
Ciao,
Dscho
On Wed, May 7, 2008 at 7:24 PM, Johannes Schindelin
[off-list ref] wrote:
> So how about this
>
> --color-words={char,punct,word}
>
> - char: one char one token
> - punct/word: a token can be either a run of word characters or a
> single punct character. diff.punctchars is used for punct, and
> diff.wordchars is used for word.
I am rather interested in the semantics, i.e. if you can punch holes into
this 3-class approach.
Bikeshedding comes later ;-)
Sorry, but i can't parse both sentences, especially Bikeshedding and
"punch holes into".
--
Ping Yin
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:44:34
Hi,
On Wed, 7 May 2008, Ping Yin wrote:
On Wed, May 7, 2008 at 7:24 PM, Johannes Schindelin
[off-list ref] wrote:
quoted
> So how about this
>
> --color-words={char,punct,word}
>
> - char: one char one token
> - punct/word: a token can be either a run of word characters or a
> single punct character. diff.punctchars is used for punct, and
> diff.wordchars is used for word.
I am rather interested in the semantics, i.e. if you can punch holes into
this 3-class approach.
Bikeshedding comes later ;-)
Sorry, but i can't parse both sentences, especially Bikeshedding and
"punch holes into".
"punch holes into": find cases where Junio's proposed algorithm breaks
down.
"bikeshedding": discussing minor implementation details that are not
really interesting at this stage. See also
http://en.wikipedia.org/wiki/Bikeshedding.
Ciao,
Dscho
On Wed, May 7, 2008 at 9:10 PM, Johannes Schindelin
[off-list ref] wrote:
> Sorry, but i can't parse both sentences, especially Bikeshedding and
> "punch holes into".
"punch holes into": find cases where Junio's proposed algorithm breaks
down.
I think the --color-words={char,punct,word} covers most of the cases
that junio mentioned, except the case that a token can be a run of
word chars, a run of non-word chars or a run of whitespaces.
If anyone is interested in this case, he can extend --color-words with
a fourth option or tokenizer (i havn't yet found a good name for this
tokenizer).
Of course, if neccessary, one can implement other language-specific tokenizers.
However, i think the 3 tokenizers i mentioned is enough for most cases.