1

This is my tree grammar:

grammar t;
options{
  output = AST;
}
type
  :
  'NVARCHAR' -> "VARCHAR"
  ;

ANTLR3 3.1.3 says:

syntax error: antlr: t.g:12:5: unexpected token: 'NVARCHAR'

What's wrong here? I took it from this article.

ps. I'm using this grammar later in order to get AST out of it. Once the AST is retrieved I'm walking through it and add every token's text to some string buffer. The idea of the rewriting above is to replace certain tokens. I'm doing language-to-language mapping (SQL to SQL dialect, to be more specific).

2 Answers 2

4

Note the first sentence Terence starts with: "just had some cool ideas about a semantic rule specification language...". That's what the first example is: an idea. It's not valid syntax.

There are (at least) two options for you:


1. rewrite the text in the token immediately

grammar T;

options{
  output=AST;
}

@parser::members {
  public static void main(String[] args) throws Exception {
    TLexer lexer = new TLexer(new ANTLRStringStream("NVARCHAR"));
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.type();
  }
}

type
  :  NVARCHAR {System.out.println("token=" + $NVARCHAR.text);}
  ;

NVARCHAR
  :  'NVARCHAR' {setText("VARCHAR");}
  ;

But this only adjusts the text, not the type of the token, which remains a NVARCHAR type.


2. use an imaginary token:

grammar T;

options{
  output=AST;
}

tokens {
  VARCHAR='VARCHAR';
}

@parser::members {
  public static void main(String[] args) throws Exception {
    TLexer lexer = new TLexer(new ANTLRStringStream("NVARCHAR"));
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.type();
  }
}

type
  :  NVARCHAR -> VARCHAR
  ;

NVARCHAR
  :  'NVARCHAR'
  ;

which changes the text and type of the token.


As you can see, with both demos, token=VARCHAR is being printed to the console:

bart@hades:~/Programming/ANTLR/Demos/T$ java -cp antlr-3.3.jar org.antlr.Tool T.g
bart@hades:~/Programming/ANTLR/Demos/T$ javac -cp antlr-3.3.jar *.java
bart@hades:~/Programming/ANTLR/Demos/T$ java -cp .:antlr-3.3.jar TParser 
token=VARCHAR
7
  • @Bart I like the first option (with setText()), but it doesn't work. ANTLR generates .java class which is not compilable: cannot find symbol method setText(java.lang.String) in TParser. Any ideas?
    – yegor256
    May 30, 2011 at 22:06
  • @yegor256, it does work. See the demo I added. My guess is you're trying to call this method inside a parser rule (my suggestion shows this in a lexer rule).
    – Bart Kiers
    May 30, 2011 at 22:35
  • @Bart I have a tree grammar, as in the question above. Could you please make corrections to my example?
    – yegor256
    May 30, 2011 at 22:57
  • @yegor256, no, you have posted a combined grammar that outputs an AST. A tree grammar starts with tree grammar T; instead of grammar T;. Or did you not include your tree grammar? You said you wanted to change the text of the token. The type of the token will just stay NVARCHAR (with the text: "VARCHAR"). Perhaps it's time to edit your question and add some more information.
    – Bart Kiers
    May 30, 2011 at 23:01
  • @Bart I added some details to my question, hope that they explain the situation better. You're right, it's a grammar that outputs an AST. I need to get an AST with "already replaced" tokens. Makes sense?
    – yegor256
    May 30, 2011 at 23:11
1

in antlr4 replacing text and type can be achieved with the type action:

OldTokenType: 
    ('Token1' | 'Token2' | 'Token3' ) {setText("New Token");} 
    -> type(NewTokenType); 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.