Raku grammars are a form of scannerless parsing, where the lexical structure and parse structure are specified together.
While it's true that rules form a recursive descent parser, that's only half of the story. When protoregexes or alternations (the |
kind, not the ||
kind) are used, the declarative prefixes of these are gathered and a NFA is formed. It is then used to determine which of the alternation branches should be explored, if any; if there are multiple, they are ranked according to longest first, with longest literal and inheritance depth used as a tie-breaker.
Forming the declarative prefix involves looking down through the subrule calls to find lexical elements - effectively, then, the tokens. Thus, we could say that Raku grammars derive the tokenizer (actually many tokenizers) for us. These are typically generally generated at compile time, however, for things like custom operators, which are done by mixing in to the grammar, further NFAs will have to be produced at runtime too in order to account for the new tokens.
There's currently no way to hook into grammar compilation and do things differently (at least, not without playing with compiler internals). However, there probably will be in the next major language release, where the AST of a Raku program will be made available to the language user, and it will thus be possible to write modules that affect the compilation of different program constructs.
token
regex
andrule
are almost identical. Atoken
is just aregex
that doesn't have backtracking. Arule
is just atoken
that has the:sigspace
feature turned on. Note that they are all just treated as functions in Raku with a different base syntax and semantics.