22

Let's say that I want to match "beer", but don't care about case sensitivity.

Currently I am defining a token to be ('b'|'B' 'e'|'E' 'e'|'E' 'r'|'R') but I have a lot of such and don't really want to handle 'verilythisisaverylongtokenindeedomyyesitis'.

The antlr wiki seems to suggest that it can't be done (in antlr) ... but I just wondered if anyone had some clever tricks ...

3
  • According to the page you cite, I woulnd't say it's impossible in ANTLR. There is no off-the-shelf option for lexer definition to handle tokens in case insensitive way. But it can be done by implementing custom string/file stream that normalizes characters to a definite (e.g., UPPER) case. Then you will be able to define tokens in a standard way, e.g., @tokens { BEER = 'BEER'; }.
    – dzieciou
    Feb 2, 2012 at 23:42
  • 1
    The ANTLR URL is 404. Nov 3, 2014 at 16:04
  • Thanks for pointign that out (+1). I have updated the link to point at a copy on archive.org (AKA teh Wayback Machine)
    – Mawg
    Oct 5, 2016 at 8:12

7 Answers 7

41

I would like to add to the accepted answer: a ready -made set can be found at case insensitive antlr building blocks, and the relevant portion included below for convenience

fragment A:[aA];
fragment B:[bB];
fragment C:[cC];
fragment D:[dD];
fragment E:[eE];
fragment F:[fF];
fragment G:[gG];
fragment H:[hH];
fragment I:[iI];
fragment J:[jJ];
fragment K:[kK];
fragment L:[lL];
fragment M:[mM];
fragment N:[nN];
fragment O:[oO];
fragment P:[pP];
fragment Q:[qQ];
fragment R:[rR];
fragment S:[sS];
fragment T:[tT];
fragment U:[uU];
fragment V:[vV];
fragment W:[wW];
fragment X:[xX];
fragment Y:[yY];
fragment Z:[zZ];

So an example is

   HELLOWORLD : H E L L O W O R L D;
1
  • This should be the solution...this is clean and results in the least amount of boilerplate. Mar 1, 2018 at 0:45
17

How about define a lexer token for each permissible identifier character, then construct the parser token as a series of those?

beer: B E E R;

A : 'A'|'a';
B: 'B'|'b';

etc.

1
  • 12
    If you take this approach, I think the "beer" rule should probably be a lexer rule name in all caps (BEER: B E E R;), and each of the per-letter rules should be prefixed by the fragment keyword. This way you get "BEER" as a single token, rather than four tokens that individually mean nothing.
    – Darien
    Aug 20, 2012 at 23:06
10

A case-insensitive option was just added to ANTLR

options { caseInsensitive = true; }

https://github.com/antlr/antlr4/blob/master/doc/options.md#caseinsensitive

The old links are now broken, these should continue to work.

1
  • The old links are broken because they are not actual since ANTLR 4.10 Mar 11, 2023 at 18:14
5

Define case-insensitive tokens with

BEER: [Bb] [Ee] [Ee] [Rr];
0
1

New documentation page has appeared in ANTLR GitHub repo: Case-Insensitive Lexing. You can use two approaches:

  1. The one described in @javadba's answer
  2. Or add a character stream to your code, which will transform an input stream to lower or upper case. Examples for the main languages you can find on the same doc page.

My opinion, it's better to use the first approach and have the grammar which describes all the rules. But if you use well-known grammar, for example from Grammars written for ANTLR v4, then second approach may be more appropriate.

1
1

I use antlr-4.7.1-complete.jar to generate TrinoSqlParser java code, but found "warning(83): TrinoLexer.g4:22:4: unsupported option caseInsensitive".

So, I try to use com.facebook.presto.sql.parser.CaseInsensitiveStream to wrap the charstream, then it works. Seems like that CaseInsensitiveStream just transform lowercase to upppercase.

My code is below:

@Test
public void testSqlRewriterL() {
    CaseInsensitiveStream upper = new CaseInsensitiveStream(CharStreams.fromString((scripts.get(1))));
    TrinoLexer lexer = new TrinoLexer(upper);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    TrinoParser sqlBaseParser = new TrinoParser(tokens);
    SqlRewriterL sqlRewriterL = new SqlRewriterL(tokens);
    ParseTreeWalker walker = new ParseTreeWalker();
    walker.walk(sqlRewriterL, sqlBaseParser.statement());
    System.out.println(sqlRewriterL.getResult());
}
0

A solution I used in C#: use ASCII code to shift character to smaller case.

class CaseInsensitiveStream : Antlr4.Runtime.AntlrInputStream {
  public CaseInsensitiveStream(string sExpr)
     : base(sExpr) {
  }
  public override int La(int index) {
     if(index == 0) return 0;
     if(index < 0) index++;
     int pdx = p + index - 1;
     if(pdx < 0 || pdx >= n) return TokenConstants.Eof;
     var x1 = data[pdx];
     return (x1 >= 65 && x1 <= 90) ? (97 + x1 - 65) : x1;
  }
}
1
  • Why did you hard code 65, 90 and 97? Your could would be much more readable, and maintainable, if you used A, Z and a
    – Mawg
    Jan 26, 2018 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.