135

I am trying to write a Perl script using the utf8 pragma, and I'm getting unexpected results. I'm using Mac OS X 10.5 (Leopard), and I'm editing with TextMate. All of my settings for both my editor and operating system are defaulted to writing files in utf-8 format.

However, when I enter the following into a text file, save it as a ".pl", and execute it, I get the friendly "diamond with a question mark" in place of the non-ASCII characters.

#!/usr/bin/env perl -w

use strict;
use utf8;

my $str = 'Çirçös';
print( "$str\n" );

Any idea what I'm doing wrong? I expect to get 'Çirçös' in the output, but I get '�ir��s' instead.

3
  • 1
    Maybe its not the program .. i think its your shell oder your editor which does the output
    – n00ki3
    Commented Mar 9, 2009 at 19:33
  • All answers correctly answer your question how to set it explicitly to UTF8. I think you should be adjust to the locale settings of your terminal as shown in stackoverflow.com/a/14405949/498634. The terminal might not be set to UTF8 and then data written to STDOUT in UTF8 will be encoded incorrectly! Commented Jan 31, 2019 at 12:20
  • Great answer how to work with utf8: Commented Feb 27, 2020 at 7:58

6 Answers 6

185

use utf8; does not enable Unicode output - it enables you to type Unicode in your program. Add this to the program, before your print() statement:

binmode(STDOUT, ":utf8");

See if that helps. That should make STDOUT output in UTF-8 instead of ordinary ASCII.

8
  • I didn't know about this (I've only been putting UTF8 in a database, never printing it). +1. Commented Mar 9, 2009 at 19:37
  • 1
    You're welcome. See also another correct answer: stackoverflow.com/questions/627661/writing-perl-code-in-utf8/… and remember, TMTOWTDI. And @Paul - if you're writing UTF-8 to a file, you should probably use binmode() on that filehandle and make it "proper" UTF-8, but if it works..
    – Chris Lutz
    Commented Mar 9, 2009 at 20:59
  • 3
    other ways: the open pragma ( search.cpan.org/perldoc/open ), the -C switch ( perldoc.perl.org/perlrun.html#-C )
    – ysth
    Commented Mar 10, 2009 at 2:22
  • 1
    FWIW here is the reason: strings that contains only latin1 (ISO-8859-1) characters, despite being stored more or less in utf8, will be output as latin1 by default. This way scripts from a pre-unicode era still work the same, even with a unicode-aware perl.
    – mirod
    Commented Mar 10, 2009 at 10:00
  • 4
    The utf8 pragma does not let you write your source in UNICODE, it forces understand of your source in the UTF-8 (or UTF-EBCDIC) encoding of UNICODE, an important distinction. Commented Apr 21, 2009 at 13:06
91

You can use the open pragma.

For eg. below sets STDOUT, STDIN & STDERR to use UTF-8....

use open qw/:std :utf8/;
1
  • 2
    BTW... I gave u +1. I think binmode(STDOUT, ':utf8') is probably more correct in this situation. "use open" has other good uses but I can't seem to find how u can set it to just encode STDOUT only?
    – draegtun
    Commented Mar 9, 2009 at 23:16
82

TMTOWTDI, chose the method that best fits how you work. I use the environment method so I don't have to think about it.

In the environment:

export PERL_UNICODE=SDL

on the command line:

perl -CSDL -le 'print "\x{1815}"';

or with binmode:

binmode(STDOUT, ":utf8");          #treat as if it is UTF-8
binmode(STDIN, ":encoding(utf8)"); #actually check if it is UTF-8

or with PerlIO:

open my $fh, ">:utf8", $filename
    or die "could not open $filename: $!\n";

open my $fh, "<:encoding(utf-8)", $filename
    or die "could not open $filename: $!\n";

or with the open pragma:

use open ":encoding(utf8)";
use open IN => ":encoding(utf8)", OUT => ":utf8";
2
  • 2
    +1 for a comprehensive answer; note that SDL is implied both with -C and PERL_UNICODE. The use open ':locale' pragma is also worth mentioning, because it is the in-script equivalent of -C and export PER_UNICODE=. Any of these 3 will give you UTF8 support for all input and output streams (whether files or stdin/stdout/stderr), assuming your environment's locale is UTF8-based. Finally, to also treat source code as UTF8, use the use utf8; pragma.
    – mklement0
    Commented May 18, 2014 at 5:40
  • 1
    perl -Mutf8 -CSDL -e '...' allows to consume/output UTF-8 as well as use UTF-8 literals inside -e e.g. for a poor man's case folder: perl -Mutf8 -CASDL -pe 'y/āáǎàēéěèīíǐìōóǒòūúǔùǖǘǚǜĀÁǍÀĒÉĚÈĪÍǏÌŌÓǑÒŪÚǓÙǕǗǙǛ/aaaaeeeeiiiioooouuuuüüüüAAAAEEEEIIIIOOOOUUUUÜÜÜÜ/'
    – vladr
    Commented May 8, 2018 at 21:32
2

You also want to say, that strings in your code are utf-8. See Why does modern Perl avoid UTF-8 by default?. So set not only PERL_UNICODE=SDAL but also PERL5OPT=-Mutf8.

1

Thanks, finally got an solution to not put utf8::encode all over code. To synthesize and complete for other cases, like write and read files in utf8 and also works with LoadFile of an YAML file in utf8

use utf8;
use open ':encoding(utf8)';
binmode(STDOUT, ":utf8");

open(FH, ">test.txt"); 
print FH "something éá";

use YAML qw(LoadFile Dump);
my $PUBS = LoadFile("cache.yaml");
my $f = "2917";
my $ref = $PUBS->{$f};
print "$f \"".$ref->{name}."\" ". $ref->{primary_uri}." ";

where cache.yaml is:

---
2917:
  id: 2917
  name: Semanário
  primary_uri: 2917.xml
-3

do in your shell: $ env |grep LANG

This will probably show that your shell is not using a utf-8 locale.

2
  • Actually, it was set to utf-8. The problem was that I was outputting to STDOUT without setting binmode to utf-8;
    – Peter Conrey
    Commented Mar 9, 2009 at 19:47
  • 2
    This would be an orthogonal concern. You need your Perl script to output correct data before you can worry about how your terminal emulator interprets it.
    – jrockway
    Commented Mar 10, 2009 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.