6

Here's a simple perl script that is supposed to write a utf-8 encoded file:

use warnings;
use strict;

open (my $out, '>:encoding(utf-8)', 'tree.out') or die;

print $out readpipe ('tree ~');

close $out;

I have expected readpipe to return a utf-8 encoded string since LANG is set toen_US.UTF-8. However, looking at tree.out (while making sure the editor recognizes it a as utf-8 encoded) shows me all garbled text.

If I change the >:encoding(utf-8) in the open statement to >:encoding(latin-1), the script creates a utf-8 file with the expected text.

This is all a bit strange to me. What is the explanation for this behavior?

1
  • See Enocde::Locale for how to incorporate locale information.. For your current problem: readpipe returns bytes (which will be already encoded as UTF-8). PerlIO layer >:encoding(utf-8) will encoded it once more when you print it to file. Solution: Convert the byte string to a Perl string before printing to the file. For example, use Encode::decode() May 2, 2016 at 17:42

1 Answer 1

2

readpipe is returning to perl a string of undecoded bytes. We know that that string is UTF-8 encoded, but you've not told Perl.

The IO layer on your output handle is taking that string, assuming it is Unicode code-points and re-encoding them as UTF-8 bytes.

The reason that the latin-1 IO layer appears to be functioning correctly is that it is writing out each undecoded byte unmolested because the 1st 256 unicode code-points correspond nicely with latin-1.

The proper thing to do would be to decode the byte-string returned by readpipe into a code-point-string, before feeding it to an IO-layer. The statement use open ':utf8', as mentioned by Borodin, should be a viable solution as readpipe is specifically mentioned in the open manual page.

5
  • " readpipe returns to perl a string of bytes" Please would you link to the documentation that says so?
    – Borodin
    May 2, 2016 at 15:09
  • @Borodin, I'd love to, unfortunately I'm having trouble finding any docs for it that reflect it's been modified for the post IO-layers world. I wonder if perl's -C command line argument can modify it's behavior, but I've not checked.
    – tjd
    May 2, 2016 at 15:21
  • 1
  • 2
    @ThisSuitIsBlackNot: Thank you very much. Since opendir is a PerlIO operation I suspect it will be affected by use open ':utf8' which is a nice tidy solution
    – Borodin
    May 2, 2016 at 16:36
  • 1
    @Borodin after your use open ':utf8' suggestion, I looked at the docs for it. Finally, a mention of readpipe in a modern, IO layered context. I've updated my answer to reflect this information.
    – tjd
    May 2, 2016 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.