0

I'm trying to create a lexer just for sake of learning, and I'm having a hard time to understand exactly what is token vs. token rule vs token type. If I understand correctly:

  • token: these are instances of lexemes that are returned to a parser when requested. They can have a value (a.k.a lexeme), type (e.g. KEYWORD_IF), literal and position.

  • token type: an enumeration of the special keywords (e.g. RIGHT_PARENTHESIS)

  • token rule: same as above

To better get my head around it, I'm creating a simple lexer using TypeScript. The user will be able to define the rules as follow:

import { lexer } from '..'

lexer.rules
  .add('<IDENTIFIER>', /<regex>/)
  .add('<IDENTIFIER>', /<regex>/)

and the signature of add:

export function add(id: string, regexp: RegExp) {
  // code here
}

What I don't fully understand is if the terminology and names are correct. Should I use rules or tokens, e.g:

lexer.tokens
  .add('<IDENTIFIER>', /<regex>/)
  .add('<IDENTIFIER>', /<regex>/)

or maybe tokenTypes:

lexer.tokenTypes
  .add('<IDENTIFIER>', /<regex>/)
  .add('<IDENTIFIER>', /<regex>/)

Also, should I call the left-side identifier or maybe name?

2
  • The rule is the part that recognizes the type. In your case it is the regular expression that identifies the token type.
    – user207421
    Mar 24, 2021 at 23:58
  • Personally I would call it tokenTypes even though it contains pairs of (token type, rule), just because "token type" is more readily understandable by someone who hasn't read a textbook about parsers. Or call it tokenRegexes or something like that.
    – kaya3
    Mar 25, 2021 at 0:07

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.