6

I was somewhat surprised to observe that the following code

# comment 
say 1;
# comment 
say 2;
# comment say 3;
# comment say 4;

prints 1, 2, 3, and 4.

Here are the relevant characters after "# comment":

say "

".uninames.raku;
# OUTPUT: «("PARAGRAPH SEPARATOR", "LINE SEPARATOR", "<control-000B>", "<control-000C>").Seq»

Note that many/all of these characters are invisible in most fonts. At least with my editor, none cause the following text to be printed on a new line. And at least one (<control-000C>, aka Form Feed, sometimes printed as ^L) is in fairly wide use in Vim/Emacs as a section separator.

This raises a few questions:

  1. Is this intentional, or a bug?
  2. If intentional, what's the use-case (other than winning obfuscated code contests!)
  3. Is it just these 4 characters, or are there others? (I found these because they share the mandatory break Unicode property. Does that property (or some other Unicode property?) govern what Raku considers as a newline?)
  4. Just, really, wow.

(I realize #4 is not technically a question, but I feel it needed to be said).

1
  • The real problem here is a lack of support for Unicode in other languages and thus other software. Jan 16, 2022 at 13:56

1 Answer 1

7

Raku's syntax is defined as a Raku grammar. The rule for parsing such a comment is:

token comment:sym<#> {
   '#' {} \N*
}

That is, it eats everything after the # that is not a newline character. As with all built-in character classes in Raku, \n and its negation are Unicode-aware. The language design docs state:

\n matches a logical (platform independent) newline, not just \x0a. See TR18 section 1.6 for a list of logical newlines.

Which is a reference to the Unicode standard for regular expressions.

I somewhat doubt there was ever a specific language design discussion along the lines of "let's enable all the kinds of newlines in Unicode, it'll be cool!" Rather, the decisions were that Raku should follow the Unicode regex technical report, and that Raku syntax would be defined in terms of a Raku grammar and thus make use of the Unicode-aware character classes. That a range of different newline characters are supported is a consequence of consistently following those principles.

2
  • Interesting. I suppose that makes sense; there's certainly value in consistently supporting Unicode. On the other hand, having invisible characters that terminate comments makes Raku trivially vulnerable to the sort of "trojan code" attacks that usually require more complex manipulations of Bidi characters or something. (e.g., krebsonsecurity.com/2021/11/…) Jan 15, 2022 at 15:25
  • 2
    I guess we just need everyone else to be as grumpy as me in code reviews about left-behind commented out code. :-) Jan 15, 2022 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.