What does the error 'Exception: Failure "lexing: empty token".' mean in OCamlex?

Learning
1

What does the error:

Exception: Failure "lexing: empty token".

mean in OCamlex?

2

I’ve never seen this error, but it stands to reason that the token matched by a rule in ocamllex must be non-empty (not the empty string). Otherwise the next call will return the same empty token, and so on forever.

3

I can reproduce this error by running coccinelle on SDL’s src/video/yuv2rgb/yuv_rgb.c.

spatch --sp-file /path/to/SDL/build-scripts/SDL_migration.cocci /path/to/SDL/src/video/yuv2rgb/yuv_rgb.c

4

I usually see this error when my lexer encounters a character that it doesn’t expect. To avoid the cryptic error, I typically use the following as the final pattern in every rule:

	| _ as c { failwith (Printf.sprintf "unexpected character: %C" c) }
5

I do something similar: install a printer to pretty-print the lexbuf contents, and then merely tracing the function that I think is failing (for example, the token function) gives me a trace of how the contents of the lexbuf evolves over time. Typically it’s enough to see what went wrong.

let lexbuf_contents lb =
  let open Lexing in
  let pos = lb.lex_curr_pos in
  let len = lb.lex_buffer_len - lb.lex_curr_pos in
  (Bytes.to_string (Bytes.sub lb.lex_buffer pos len))
;;

let pp_lexbuf lb =
  Format.print_string "#<lexbuf:<";
  Format.print_string (lexbuf_contents lb);
  Format.print_string ">>"
;;
#install_printer pp_lexbuf ;;