# Call Menhir-generated parser without a lexbuf

**URL:** <https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570>\
**Category:** Learning\
**Tags:** menhir\
**Created:** [October 2, 2022, 1:54pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570 "2022-10-02T13:54:05Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![olleharstedt](https://avatars.discourse-cdn.com/v4/letter/o/f1d935/32.png) [@olleharstedt](https://discuss.ocaml.org/u/olleharstedt)\
**Post date:** [October 2, 2022, 1:54pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/1 "2022-10-02T13:54:05Z")

</div>

OK, so I’ve setup two parsers, and one lexer. I’m getting a token list from the lexer, then call the second parser from the first parser. BUT, now the second parser expects a lexbuf, when I already _have_ a token list (and in fact no lexbuf). When I look at the generated “parser.ml” file, I only find one function that’s supposed to be used from the outside, which is the same name as the start rule in the grammar. So I’d have to hack around it? Or convert the token list to a string or something? 🤔

Generated function:

```ocaml
let docblock =
  fun _menhir_lexer _menhir_lexbuf ->
    let _menhir_stack = () in
    let MenhirBox_docblock v = _menhir_run_0 _menhir_stack _menhir_lexbuf _menhir_lexer in
    v

```

I want to be able to do this in the main parser (unrelated code removed):

```ocaml
    | d=DOCBLOCK "function" {
        Function {
            docblock = Docblockparser.docblock d;
        }
    }

```

where DOCBLOCK is defined as

```auto
%token <Docblockparser.token list> DOCBLOCK

```

Or maybe I should just hard-code the grammar of docblock in the lexer instead… It follows a pretty rigid pattern. 🤔

---

<div class="post-metadata">

**Author:** ![thierry-martinez](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/thierry-martinez/32/1481_2.png) [@thierry-martinez](https://discuss.ocaml.org/u/thierry-martinez)\
**Post date:** [October 2, 2022, 4:17pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/2 "2022-10-02T16:17:01Z")

</div>

There are as many functions in the generated interface of the parser than there are start symbols in the grammar: each of these functions has type `(Lexing.lexbuf -> token) -> Lexing.lexbuf -> t`, where `t` is the type associated to the corresponding start symbol. To call such a function, you therefore have to provide a function `Lexing.lexbuf -> token` and a `lexbuf`: as far as I know, the `lexbuf` is just used by the parser to extract position informations, and otherwise is just passed as is as argument to the provided function to get tokens. Therefore, if you have a `token list`, you may convert it to a function that dispenses tokens, ignoring the lexbuf. For instance:

```auto
let dispenser_of_token_list (l : token list) : Lexing.lexbuf -> token =
  let d = Seq.to_dispenser (List.to_seq l) in
  fun _lexbuf -> Option.get (d ())

```

and then you just have to give a dummy lexbuf to the parser function (for instance, `Lexing.of_string ""`).

However, positions are lost with this approach, and that prevents your code from reporting the errors correctly. As far as I know, the `lexbuf` fields used for error reporting are `lex_start_p` and `lex_end_p`: therefore, you can enrich your `token list` as a `(token * Lexing.position * Lexing.position) list`, saving for each token the values of `lexbuf.lex_start_p` and `lexbuf.lex_end_p` in the lexer, and restoring these values (the fields are mutable) in the dispenser.

---

<div class="post-metadata">

**Author:** ![octachron](https://avatars.discourse-cdn.com/v4/letter/o/49beb7/32.png) [@octachron](https://discuss.ocaml.org/u/octachron)\
**Post date:** [October 2, 2022, 4:41pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/3 "2022-10-02T16:41:06Z")

</div>

The MenhirLib library has few options to convert a `lexbuf` parser generated by Menhir into a dispenser-based parser, the simplest one is `MenhirLib.Convert.Simplified.traditional2revised`:

```ocaml
let revised_parser dispenser =
  MenhirLib.Convert.Simplified.traditional2revised
    menhir_generated_parser
    dispenser

```

---

<div class="post-metadata">

**Author:** ![olleharstedt](https://avatars.discourse-cdn.com/v4/letter/o/f1d935/32.png) [@olleharstedt](https://discuss.ocaml.org/u/olleharstedt)\
**Post date:** [October 2, 2022, 4:42pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/4 "2022-10-02T16:42:53Z")

</div>

> [@octachron](#):
>
> dispenser-based parser

Hm, what does this mean?

---

<div class="post-metadata">

**Author:** ![olleharstedt](https://avatars.discourse-cdn.com/v4/letter/o/f1d935/32.png) [@olleharstedt](https://discuss.ocaml.org/u/olleharstedt)\
**Post date:** [October 2, 2022, 4:58pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/5 "2022-10-02T16:58:05Z")

</div>

I just keep getting weird errors…

> Error: docblock generates the language {epsilon}.

with

```auto
%start<Ast.docblock_comment list> docblock
%%

docblock:
  | {[]}

```

😵‍💫

---

<div class="post-metadata">

**Author:** ![olleharstedt](https://avatars.discourse-cdn.com/v4/letter/o/f1d935/32.png) [@olleharstedt](https://discuss.ocaml.org/u/olleharstedt)\
**Post date:** [October 2, 2022, 4:59pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/6 "2022-10-02T16:59:05Z")

</div>

This is really cool, thanks. 🙂 Just gotta get the second parser logic to work now.

---

<div class="post-metadata">

**Author:** ![olleharstedt](https://avatars.discourse-cdn.com/v4/letter/o/f1d935/32.png) [@olleharstedt](https://discuss.ocaml.org/u/olleharstedt)\
**Post date:** [October 2, 2022, 6:43pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/7 "2022-10-02T18:43:42Z")

</div>

> [@thierry-martinez](#):
>
> ` let d = Seq.to_dispenser (List.to_seq l) in`

“A” for effort, but I this is not available on my 4.08 installation, and when switching to 4.14, expect\_test\_helpers\_kernel becomes uninstallable for some reason, with debug output just claiming “conflict!”, hehe. Reported it to their repository.

Testing with OSeq instead…

Hm, getting

> Invalid\_argument “option is None”

inside the generated parser code. The failing line looks like this:

```auto
let _tok = _menhir_lexer _menhir_lexbuf in

```

so probably related to the lexbuf somehow?

Token provider code:

```ocaml
 let linebuf = Lexing.from_string "" in
 let dispenser_of_token_list (l : Docblockparser.token list) : Lexing.lexbuf -> Docblockparser.token =
     let d = OSeq.to_gen (OSeq.of_list l) in
     fun _lexbuf -> Option.get (d ())
 in
 let disp = dispenser_of_token_list cb in
 let cb = if List.length cb > 0 then Docblockparser.docblock disp linebuf else [] in
 Function { ... }

```

OK, maybe I’d better separate the docblock parser into a completely separate lexer+parser step instead of trying to merge them together… Unless parameterized rules in Menhir would work. Dunno, never tried.

---

<div class="post-metadata">

**Author:** ![thierry-martinez](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/thierry-martinez/32/1481_2.png) [@thierry-martinez](https://discuss.ocaml.org/u/thierry-martinez)\
**Post date:** [October 2, 2022, 8:06pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/8 "2022-10-02T20:06:03Z")

</div>

> [@olleharstedt](#):
>
> Hm, getting
> 
> > Invalid\_argument “option is None”

This error means that the parser tries to get tokens whereas the end of the list has been reached (since, in this case, `d ()` returns `None`, and `Option.get` fails). Whether it is expected or not for the parser to reach the end of the list depends of how you have designed your grammar (is there an explicit end for the phrase in token list or not), but if the end of the phrase is implicit in the token list, therefore you probably want to introduce an `End_of_stream` token, and either add `End_of_stream` explicity at the end of the token list, or use something like `Option.value ~default:End_of_stream (d ())`.

---

<div class="post-metadata">

**Author:** ![thierry-martinez](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ocaml.org/thierry-martinez/32/1481_2.png) [@thierry-martinez](https://discuss.ocaml.org/u/thierry-martinez)\
**Post date:** [October 2, 2022, 8:16pm UTC](https://discuss.ocaml.org/t/call-menhir-generated-parser-without-a-lexbuf/10570/9 "2022-10-02T20:16:15Z")

</div>

@octachron, I didn’t know about these conversion functions, thanks!

> [@olleharstedt](#):
>
> > [@octachron](#):
> >
> > dispenser-based parser
> 
> Hm, what does this mean?

A parser that takes as input a “dispenser”, that is to say a function `unit -> t` for some `t`, instead of a `lexbuf` (in the case of what `MenhirLib` expects, `t` is `token * Lexing.position * Lexing.position`, that is to say tokens enriched with position informations).