On Mon, May 5, 2008 at 8:10 PM, Ping Yin [off-list ref] wrote:
The one-char-one-token and multi-char-one-token rules may have
different implementation issues. I think multi-char-one-token rule may
be more representative. So for the current time, i prefer considering
both run of word characters and single non-word character as a token.
If we agree on this. I will come up with an implementation still using
diff.nonwordchars few days later.
--
Ping Yin