3

Does this work correctly? Some error messages are already decode and some need do be decoded do get a correct output.

#!/usr/bin/env perl
use warnings;
use strict;
use utf8;
use open qw(:utf8 :std);
use Encode qw(decode_utf8);

# ...

if ( not eval{
    # some error-messages (utf8) are decoded some are not
    1 }
) {
    if ( utf8::is_utf8 $@ ) {
        print $@;
    }
    else {
        print decode_utf8( $@ );
    }
}
1
  • Do the messages come out correct? If yes, it is probably working.
    – choroba
    Commented Jan 29, 2013 at 9:43

1 Answer 1

9

Am I using utf8::is_utf8 correctly?

No. Any use of utf8::is_utf8 is incorrect as you should never use it! Using utf8::is_utf8 to guess at semantics of a string is what's known as an instance of The Unicode Bug. Except for inspecting the internal state of variables when debugging Perl or XS module, utf8::is_utf8 has no use.

It does not indicate whether the value in a variable is encoded using UTF-8 or not. In fact, that's impossible to know reliably. For example, does "\xC3\xA9" produce a string that's encoded using UTF-8 or not? Well, there's no way to know! It depends on whether I meant "é", "é" or something entirely different.

If the variable may contain both encoded and decoded strings, it's up to you to track that using a second variable. I strongly advise against this, though. Just decode everything as it comes in from the outside.

If you really can't, your best bet it to try to decode $@ and ignore errors. It's very unlikely that something readable that isn't UTF-8 would be valid UTF-8.

# $@ is sometimes encoded. If it's not,
# the following will leave it unchanged.
utf8::decode($@);

print $@;
11
  • I agree, except that utf8::is_utf8 can be useful for working around The Unicode Bug on older Perls. Sadly we still have 5.8.8 machines in production...
    – rjh
    Commented Aug 6, 2016 at 12:02
  • @rjh, No, you just need utf8::upgrade or utf8::downgrade.
    – ikegami
    Commented Aug 6, 2016 at 20:01
  • @Thorsten Schöning. Re "So yes, the flag DOES distinguish exactly that.", The passage you quoted doesn't contradict what I said at all. The docs are talking about how the string is stored internally, which is irrelevant here. My answer only talks about the value of the variable, not how it's stored internally.
    – ikegami
    Commented Jun 12, 2018 at 17:38
  • @ikegami Your tests are simply wrong, just print the result of is_utf8 directly after inet_aton and you will see the result is 0, because you have a byte array. Passing that to encode is wrong, because that only works on character strings. Passing again to decode is wrong as well, because you don't have a UTF-8 encoded byte array, but something arbitrary different. decode only doesn't fail, but the problem is you making wrong assumptions, not the flag. Garbage in, garbage out. Your 2. example is dealing with a byte array as well, so 0 is expected output, regardless of Unicode. Commented Jun 13, 2018 at 6:19
  • @Thorsten Schöning, That's not how Perl works. Just like Perl considers zero to be zero whether stored as a IV, UV, NV, PV[UTF8=0] or PV[UTF8=1], Perl doesn't distinguish between string stored using PV[UTF8=0] or PV[UTF8=1]. is_utf8 says nothing about the string except how it's stored. You can verify this by using print or eq on the output of inet_aton before and after the encode-decode. $_ still contains the exact same bytes. How the value is stored doesn't affect the value.
    – ikegami
    Commented Jun 13, 2018 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.