# Examples using compiler-libs Lexer

**URL:** <https://discuss.ocaml.org/t/examples-using-compiler-libs-lexer/17196>\
**Category:** Ecosystem\
**Tags:** lexer, compiler\
**Created:** [September 1, 2025, 1:43pm UTC](https://discuss.ocaml.org/t/examples-using-compiler-libs-lexer/17196 "2025-09-01T13:43:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rwmjones](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/rwmjones/32/818_2.png) [@rwmjones](https://discuss.ocaml.org/u/rwmjones)\
**Post date:** [September 1, 2025, 1:43pm UTC](https://discuss.ocaml.org/t/examples-using-compiler-libs-lexer/17196/1 "2025-09-01T13:43:19Z")

</div>

I wonder if anyone has any example code using compiler-libs `Lexer` module, eg. just to read in an OCaml `*.ml` file.

What I’m actually trying to do is lex some OCaml source files, to discover “long” multi-line strings that might be converted into quoted literals (ie. `{|…|}`) . So I want something that does the lexing step (but not parsing) and then lets me call my own function on each string that is found so I can decide if I want to convert it or not.

(Something like [this commit](https://github.com/libguestfs/libguestfs-common/commit/7ecf3992b997d2774bece091a93b5b86cf5cd592) but more automatically and across several large code bases, and yes I did try to write an OCaml lexer in Perl first, but that didn’t exactly go well!).

---

<div class="post-metadata">

**Author:** ![rwmjones](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/rwmjones/32/818_2.png) [@rwmjones](https://discuss.ocaml.org/u/rwmjones)\
**Post date:** [September 1, 2025, 2:16pm UTC](https://discuss.ocaml.org/t/examples-using-compiler-libs-lexer/17196/2 "2025-09-01T14:16:29Z")

</div>

```ocaml
let () =
  Lexer.init ();
  let fp = open_in Sys.argv.(1) in
  let lexbuf = Lexing.from_channel fp in
  let rec loop () =
    let token = Lexer.token lexbuf in
    (match token with
     | Parser.STRING (s, loc, sopt) ->
        Printf.printf "string = %s\n" s;
     | _ -> ()
    );
    loop ()
  in
  loop ()

(* ocamlfind opt -package compiler-libs.common find_strings.ml -o find_strings.opt -linkpkg *)

```

This is my initial attempt, but it just spins using 100% CPU.

---

<div class="post-metadata">

**Author:** ![nojb](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/nojb/32/519_2.png) [@nojb](https://discuss.ocaml.org/u/nojb)\
**Post date:** [September 1, 2025, 2:29pm UTC](https://discuss.ocaml.org/t/examples-using-compiler-libs-lexer/17196/3 "2025-09-01T14:29:57Z")

</div>

> [@rwmjones](#):
>
> This is my initial attempt, but it just spins using 100% CPU.

It is spinning because the OCaml lexer returns a dummy `EOF` token on end-of-file. You should end your recursive loop when you come across it. Otherwise, code looks OK to me.

Cheers,  
Nicolas

---

<div class="post-metadata">

**Author:** ![rwmjones](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/rwmjones/32/818_2.png) [@rwmjones](https://discuss.ocaml.org/u/rwmjones)\
**Post date:** [September 1, 2025, 3:17pm UTC](https://discuss.ocaml.org/t/examples-using-compiler-libs-lexer/17196/4 "2025-09-01T15:17:59Z")

</div>

Excellent, it works! For reference the final code is:

```ocaml
let () =
  Lexer.init ();
  let fp = open_in Sys.argv.(1) in
  let lexbuf = Lexing.from_channel fp in
  let rec loop () =
    let token = Lexer.token lexbuf in
    (match token with
     | Parser.STRING (s, loc, sopt) ->
        Printf.printf "string = %S\n" s;
        loop ();
     | EOF -> ();
     | _ ->
        loop ();
    )
  in
  loop ()

```