Re: Git-mediawiki : Encoding problems in perl

Jérémie NIKAES [off-list ref] writes:

2011/6/8 Matthieu Moy [off-list ref]:

quoted

open(my $git, "-|:encoding(UTF-8)", "git cat-file -p $sha1");

there should probably have been a $/ = 1; or some other perl magic to
make sure we don't read only the first line there:

quoted

my $file_content = <$git>;
close($git);

Yes, that did it ! Thank you boss :-)

Then, make it a helper function to call like

my $file_content = run_git("cat-file -p $sha1");

and use it where needed.

Although, I'm still encountering issues regarding encoding file names
though, I am going to look deeper now that i have this solution.

My advice, at least in the short-term (already discussed offline): use
urlencode ( http://php.net/manual/en/function.urlencode.php ) on pull,
and don't bother with encoding on push. Non-ascii characters in
filenames are a nightmare ...

If you go for utf8 filenames, you should test that your script works in
various environments, like

LANG=fr_FR.ISO-8859-1 xterm
(launch a terminal with latin-1 encoding inside)

and Mac OS X (which does some weird utf-8-normalization on filenames),
and probably windows (no idea how filename encoding works there).

If it doesn't work in one of them, you'll have to provide a fall-back to
plain ascii for these users, which will most likely be the short-term
solution I'm proposing.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

2011/6/8 Matthieu Moy [off-list ref]:

there should probably have been a $/ = 1; or some other perl magic to
make sure we don't read only the first line there:

Yes, it indeed currently reads only the first line. I'm going to see
what kind of magic I need to use.

Then, make it a helper function to call like

my $file_content = run_git("cat-file -p $sha1");

and use it where needed.

Good idea, doing it right now

My advice, at least in the short-term (already discussed offline): use
urlencode ( http://php.net/manual/en/function.urlencode.php ) on pull,
and don't bother with encoding on push. Non-ascii characters in
filenames are a nightmare ...

Yes I tried uri_escape, but that only works in the direction mediawiki -> git.
A page named "Eté" on mediawiki comes as a Et%C3%A9.mw file on the repo.
However, when I try to send that file "Et%C3%A9" with the mediawiki
API, I get this error

"Can't use an undefined value as a HASH reference at
/usr/local/share/perl/5.10.1/MediaWiki/API.pm line 554."

So I tried to backslash the '%' but it does not do it...
Any idea ? Thanks

--
Jérémie Nikaes

On Wed, Jun 08, 2011 at 06:15:15PM +0200, Jérémie NIKAES wrote:

2011/6/8 Matthieu Moy [off-list ref]:

quoted

there should probably have been a $/ = 1; or some other perl magic to
make sure we don't read only the first line there:

Yes, it indeed currently reads only the first line. I'm going to see
what kind of magic I need to use.

You need to set $/ to undef. Use "local" to prevent it from polluting
other parts of the code, like:

  my $var = do { local $/; <$handle> };

-Peff

2011/6/8 Jeff King [off-list ref]:

You need to set $/ to undef. Use "local" to prevent it from polluting
other parts of the code, like:

 my $var = do { local $/; <$handle> };

-Peff

Worked as a charm. Thanks a lot.
-- 
Jérémie Nikaes

Jérémie NIKAES [off-list ref] writes:

2011/6/8 Matthieu Moy [off-list ref]:

[...]

quoted

My advice, at least in the short-term (already discussed offline): use
urlencode ( http://php.net/manual/en/function.urlencode.php ) on pull,
and don't bother with encoding on push. Non-ascii characters in
filenames are a nightmare ...

Yes I tried uri_escape, but that only works in the direction mediawiki -> git.
A page named "Eté" on mediawiki comes as a Et%C3%A9.mw file on the repo.
However, when I try to send that file "Et%C3%A9" with the mediawiki
API, I get this error

"Can't use an undefined value as a HASH reference at
/usr/local/share/perl/5.10.1/MediaWiki/API.pm line 554."

Can you show us this line end its neighourhood?
 
It might be bug in MediaWiki::API...

So I tried to backslash the '%' but it does not do it...

Decode if from URI encoding to UTF-8 and mark as UTF-8 before sending
to mediawiki API.

-- 
Jakub Narebski
Poland
ShadeHawk on #git