Ævar Arnfjörð Bjarmason [off-list ref] writes:
The PCRE2_NO_UTF_CHECK flag means "I have checked that this is a valid
UTF-8 string so you, PCRE, don't need to re-check it".
OK, in short, barfing and stopping is a problem, but that flag is
not the right knob to tweak. And the right knob ...
1) We're oversupplying PCRE2_UTF now, and one such case is what's being
reported here. I.e. there's no reason I can think of for why a
fixed-string pattern should need PCRE2_UTF set when not combined
with --ignore-case. We can just not do that, but maybe I'm missing
something there.
2) We can do "try utf8, and fallback". A more advanced version of this
is what the new PCRE2_MATCH_INVALID_UTF flag (mentioned upthread)
does. I was thinking something closer to just carrying two compiled
patterns, and falling back on the ~PCRE2_UTF one if we get a
PCRE2_ERROR_UTF8_* error.
... lies somewhere along that line. I think that is very sensible.
Let's make sure this gets sorted out soonish.
Thanks.
Hi,
On Wed, 24 Jul 2019, Junio C Hamano wrote:
Ævar Arnfjörð Bjarmason [off-list ref] writes:
quoted
The PCRE2_NO_UTF_CHECK flag means "I have checked that this is a valid
UTF-8 string so you, PCRE, don't need to re-check it".
OK, in short, barfing and stopping is a problem, but that flag is
not the right knob to tweak. And the right knob ...
quoted
1) We're oversupplying PCRE2_UTF now, and one such case is what's being
reported here. I.e. there's no reason I can think of for why a
fixed-string pattern should need PCRE2_UTF set when not combined
with --ignore-case. We can just not do that, but maybe I'm missing
something there.
2) We can do "try utf8, and fallback". A more advanced version of this
is what the new PCRE2_MATCH_INVALID_UTF flag (mentioned upthread)
does. I was thinking something closer to just carrying two compiled
patterns, and falling back on the ~PCRE2_UTF one if we get a
PCRE2_ERROR_UTF8_* error.
... lies somewhere along that line. I think that is very sensible.
I am glad that everybody agrees with my original comment on ab/no-kwset
where I suggested that we should use our knowledge of the encoding of
the haystack and convert it to UTF-8 if we detect that the pattern is
UTF-8 encoded, and then pass the PCRE2_UTF flag only when applicable
(i.e. when we know that either needle or haystack is non-ASCII, and then
making sure that we convert to UTF-8 whenever necessary).
Okay, that came over a bit more sarcastic than I originally intended,
but if you try to filter that out, I think that is still the better
solution than to paper over the issue.
After all, PCRE2_MATCH_INVALID_UTF is only marginally better than what
Carlo suggested. _Marginally_. Not really worth considering, in my mind,
even.
Let's make sure this gets sorted out soonish.
I agree,
Dscho