Git-mediawiki : Encoding problems in perl

Hi,
While working on the git-mediawiki project[1], we ran into some
problems regarding utf8 encoding of files. Most of them have been
solved, however, one is still pretty annoying.
Let me illustrate it :

I want to edit a page on mediawiki using the API, with a very simple example :

my $mw = MediaWiki::API->new();
$mw->edit( {
        action => 'edit',
        title => 'Main_page',
        text => 'été',
} ) ;

But, when I look at the page on mediawiki, I see weird characters : été.

I tried text => encode_utf8('été') with no success.

This makes pushing changes from git to mediawiki buggy since pulling a
file with accentuated characters and pushing it right after changes
things on the wiki.

While googling (a lot), I found that utf8 was pretty tricky in perl...
The only thing that seems to solve things is a simple addition of 'use
encoding utf8' at the top of our script.
However
A) Adding this line requires that I remove 'use strict;'
B) I found some information about this pragma encoding and it seems to
be unadvised to use it

Do you have any information regarding this issue ?

Thanks,
-- 
Jérémie Nikaes
[1] https://github.com/Bibzball/Git-Mediawiki

@ Jérémie NIKAES [off-list ref] wrote (2011-06-08 15:45+0200):

But, when I look at the page on mediawiki, I see weird characters : été.

I tried text => encode_utf8('été') with no success.

Do you have any information regarding this issue ?

I'm not a Perl guru, but i ran into the very same problem when
writing a disc-ripper/CDDB lookup/DB writer, and the following
snippet helped me out:

    $CDDB{TITLES} = $dinf->{ttitles};
    foreach (@{$dinf->{ttitles}}) {
        s/^\s*(.*?)\s*$/$1/;
        my $save = $_;
        eval { Encode::from_to($_, 'iso-8859-1', 'utf-8'); };
        $_ = $save if $@;
        Encode::_utf8_off($_);
    }

I forget the exact circumstances, but as far as i remember you
need to trigger the is-UTF-8 bit on the string object in an
discouraged (acc. to manual) way to make it work the way it
should.
--
Ciao, Steffen
sdaoden(*)(gmail.com)
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments

On Wed, Jun 08, 2011 at 03:45:43PM +0200, Jérémie NIKAES wrote:

my $mw = MediaWiki::API->new();
$mw->edit( {
        action => 'edit',
        title => 'Main_page',
        text => 'été',
} ) ;
[...]
While googling (a lot), I found that utf8 was pretty tricky in perl...
The only thing that seems to solve things is a simple addition of 'use
encoding utf8' at the top of our script.
However
A) Adding this line requires that I remove 'use strict;'
B) I found some information about this pragma encoding and it seems to
be unadvised to use it

From the "utf8" man page:

  Do not use this pragma for anything else than telling Perl that your
  script is written in UTF-8.

which is what you are doing here, since you are telling perl that the
string constant is in utf8. So from my understanding, "use utf8" is the
right solution.

That being said, this is probably just a small test case, and you are
more likely to be reading the data from a file.

For file contents, you can use:

  binmode($handle, ":utf8");

to read everything in as utf8.

For file names themselves, I think it depends where you get them.
Presumably from readdir() or from a glob. I think you can use
utf8::upgrade($string) on the result to make sure they are interpreted
as utf8 (if you already know that is how the bytes in the filename
should be interpreted).

But I admit I am not an expert on such matters, and every time I do utf8
things in perl, I end up with a lot of trial and error.

-Peff

Jérémie NIKAES [off-list ref] writes:

While working on the git-mediawiki project[1], we ran into some
problems regarding utf8 encoding of files. Most of them have been
solved, however, one is still pretty annoying.
Let me illustrate it :

I want to edit a page on mediawiki using the API, with a very simple example :

my $mw = MediaWiki::API->new();
$mw->edit( {
        action => 'edit',
        title => 'Main_page',
        text => 'été',
} ) ;

But, when I look at the page on mediawiki, I see weird characters : été.

Take a look at

  http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default

especially accepted answer.


In short (I don't agree with everything there, and not everything is
needed for all but extremal Unicode usage): if your script is written
using UTF-8 like in above examples, use

  use utf8;

If this is simplification, and this text comes from other file or is
result of output of some command, use

  use utf8::all;

or take a look what it does and put relevant parts in your script.
 

I tried text => encode_utf8('été') with no success.

This makes pushing changes from git to mediawiki buggy since pulling a
file with accentuated characters and pushing it right after changes
things on the wiki.

While googling (a lot), I found that utf8 was pretty tricky in perl...
The only thing that seems to solve things is a simple addition of 'use
encoding utf8' at the top of our script.
However
A) Adding this line requires that I remove 'use strict;'

  use encoding ':utf8';

or

  use encoding 'utf8';

B) I found some information about this pragma encoding and it seems to
be unadvised to use it

-- 
Jakub Narebski
Poland
ShadeHawk on #git

2011/6/8 Jakub Narebski [off-list ref]:

 http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default

especially accepted answer.

Thanks for the link, do have some interesting things in here.

 use encoding ':utf8';

or

 use encoding 'utf8';

That did it while keeping the use strict; intact.
Thanks a lot

-- 
Jérémie Nikaes