2

I am working on validating the Rust parser's handwritten stuff against a model written in antlr. I am hitting a problem with antlr escaping strings for me:

[15:48:50]~/src/rust2/src/grammar> grun RustLexer tokens -tokens                                                                               
"\n"
[@0,0:3='"\n"',<46>,1:0]

and

[15:51:15]~/src/rust2/src/grammar> grun RustLexer tokens -tokens
"
"
[@0,0:2='"\n"',<46>,1:0]

Create the same string content. Is there a way for antlr to behave in any other way here? In particular, it would be acceptable if it escaped literal \ to \\, I could then collapse those down in my tool. As it stands, I am losing information about the input.

4
  • I suppose it's just an issue with the debug print where probably the `\` is not escaped itself. It should be fine in the code behind, e.g. Java or C#. Have you tried this?
    – Onur
    Jul 15, 2014 at 6:46
  • As you can see, the range is longer for the first case (0:3) as opposed to (0:2), so it should really only be a debug print issue. But you could raise an issue at github anyway.
    – Onur
    Jul 15, 2014 at 6:48
  • I don't know what to try, all I've used is the grun tool. If there is something I can do manually to print in a different way, that is fine as well. Jul 15, 2014 at 6:50
  • I don't think you can fix the grun output, but the problem should not occur in a "real" application that uses the antlr parser in a e.g. Java or C# project. If you plan to just use grun, you'll have to raise an issue at github and wait for the fix.
    – Onur
    Jul 15, 2014 at 7:08

1 Answer 1

1

grun is probably doing the expanding of "\n"to a line break, because the lexer surely won't do this (luckily).

Given the grammar Test:

grammar Test;

parse
 : .*? EOF
 ;

LINE_BREAK
 : '\n'
 ;

OTHER
 : .
 ;

that parses "\n\\n":

TestLexer lexer = new TestLexer(new ANTLRInputStream("\n\\n"));

for (Token token : lexer.getAllTokens()) {
  System.out.printf("%s -> <%s>%n", TestLexer.ruleNames[token.getType() - 1], token.getText());
}

which will print the following:

LINE_BREAK -> <
>
OTHER -> <\>
OTHER -> <n>

B.t.w., I presume you are aware of this repository?

1
  • As to rust-antlr, yes, I'm working on updating that, but I need to verify the lexer separately to handle macro expansion correctly later. In short, it's complicated :). No, my shell is not expanding "\n" to a line break, the shell isn't even involved here, that's just stdin to grun. I'll take a closer look at the runtime API and how I can use that to get what I want, thanks for the code snippet! Jul 15, 2014 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.