3

I'm currently working on a programming language as a hobby. It would make lexing errors massively easier to debug if it was possible to have ocamllex print out the tokens it matches as it finds them, I occasionally just add print statements to my rules manually but there should be an easier way to do that.

So what I'm asking is, given a .mll file and some input, is there an automatic way to see the corresponding tokens?

1
  • 1
    why are people downvoting this?
    – hugomg
    Commented May 16, 2015 at 4:37

1 Answer 1

5

I don't think there is a built-in way to ask the lexer to print its tokens.

If you use ocamlyacc, you can set the p option in OCAMLRUNPARAM to see a trace of the parser's actions. This is described in Section 12.5 of the OCaml manual. See Section 10.2 for a description of OCAMLRUNPARAM.

If you don't mind a crude hack, I just wrote a small script lext that adds tracing to the output generated by ocamllex:

#!/bin/sh
#
echo '
    let my_engine a b lexbuf =
        let res = Lexing.engine a b lexbuf in
        Printf.printf "Saw token [%s]'\\\\'n" (Lexing.lexeme lexbuf);
        res
'
sed 's/Lexing\.engine/my_engine/g' "$@"

It works like this:

$ cat ab.mll
rule token = parse
    [' ' '\t'] { token lexbuf }
  | '\n'       { 1 }
  | '+'        { 2 }
  | _          { 3 }
{
    let lexbuf = Lexing.from_channel stdin in
    try
        while true do
            ignore (token lexbuf)
        done
    with _ -> exit 0
}
$ ocamllex ab.mll
5 states, 257 transitions, table size 1058 bytes
$ lext ab.ml > abtraced.ml
$ ocamlopt -o abtraced abtraced.ml
$ echo 'a+b' | abtraced
Saw token []
Saw token [a]
Saw token [+]
Saw token [b]
Saw token [
]
Saw token []
1
  • 2
    As always, Jeff's answer goes above and beyond expectations. Commented May 16, 2015 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.