BEGIN
changes the current start condition in the lexer.
Start conditions are a way of choosing which rules are currently used by the lexer and which ones are ignored. In a way, they allow creating multiple lexers in one, which may (or may not) share some of the rules.
This is useful if you want temporarily change the lexer's behavior.
For example, you've encountered a beginning of a string and now you want to scan for escape sequences instead of normal keywords.
You can create a start condition that does that and switch to it at the beginning of a string, then switch back when you encounter the end of the string.
The macro BEGIN
chooses which starting condition will now be used.
INITIAL
is just an integer constant - the ID of the start condition that is active by default.
(Simpler scanners don't have other start conditions. In that case you don't need to worry about BEGIN
at all.)
BEGIN(INITIAL);
, it's like "forget the token we're in, we're gonna scan the next token."BEGIN
operates on start conditions, not on tokens.