Using merlin with camlp5

Community
,
#1

I created a simple dune project where I have a library preprocessed with camlp5 (more precisely, as a standalone executable built using camlp5 and which outputs serialized OCaml AST) and another library preprocessed with PPX.

Library 1 is being preprocessed into file _build/default/lib1.pp.ml and another into _build/default/libppx/lib2.pp.ml. Mimetype of both files is OCaml abstract syntax tree implementation file (Version 027)

The problem is that it’s OK to use PPX-preprocessed library in IDE but for camlp5 one I get and error. It seems it treats OCaml abstract syntax tre file as a plain text file.

image
image1096×349 44 KB

It seems the same problem appears both in just merlin and ocaml-lsp.

Any thoughts where should I diagnose this issue?

P.S. I create a test for merlin testsuite to reproduce this behaviour.

#2

If I remember correctly, merlin doesn’t have support for the camlp4/5 syntax. And for ppxes, merlin doesn’t read the preprocessed file but runs the ppxes itself.

You can probably follow an approach similar to the one taken by reason. Reason has a different frontend for merlin that is automatically transforming the reason file into an ocaml ast. And merlin is working on this ast. I don’t know if camlp5 generates good enough locations for this to be practical.

#3

I thought that merlin applies various rewriters and runs custom parser on the result. Am I wrong?

#4

Kakadu,

I just want to clarify what you’re seeing:

(1) you have a test-case where merlin or LSP is inhaling serialized AST, and not source-code (via parsing), and it’s blowing up. So if this is a camlp5 bug, it’s due to camlp5’s outputting bad AST, not a parsing issue, yes?

(2) furthermore, these same serialized ASTs are consumed by the OCaml compiler just fine: sans any sort of complaint.

(2) It would be good to -verify- that merlin/lsp is not somehow attempting to invoke camlp5, but I suspect that that’s the case, maybe not high-priority.

(2) I don’t see a standalone test-case. Perhaps a next step would be to take a unit-test from merlin/lsp and modify it to use these serialized AST files, to elicit a failure?

With a standalone unit-test involving merlin/lsp, I could try to explore whether it’s something bad put out by Camlp5.

I’m not a merlin/lsp user, so a full-stack reproduction involving and editor, multiple processes, etc, is Greek to me.

#5

(1) I’m not sure whose bug it is. If it will indeed be a bug of camlp5 I will let you know

(2) I’m sure merlin/lsp doesn’t invoke camlp5. AFAIU it was design not to handle camp5 from very beginning

(2) I don’t want prepare a unit-test right now. I hope that I will get an answer like Merlin doesn't care about contents of '*.pp.ml' files, look at another files or these files should have another mime type or something.

#6

Maybe the contents of .merlin files will help

$  cat .merlin 
EXCLUDE_QUERY_DIR
B _build/default/.lib1.objs/byte
S .
FLG -pp '/media/oldhome/kakadu/prog/asp/camlp5_pp_demo/_build/default/rewriter.exe'
FLG -w @1..3@5..28@30..39@43@46..47@49..57@61..62-40 -strict-sequence -strict-formats -short-paths -keep-locs -rectypes -warn-error -A -w -33

$ cat libppx/.merlin 
EXCLUDE_QUERY_DIR
B ../_build/default/libppx/.lib2.objs/byte
S .
FLG -ppx '/media/oldhome/kakadu/prog/asp/camlp5_pp_demo/_build/default/.ppx/c3461e88ae1503d575d582eb5874e916/ppx.exe --as-ppx --cookie '\''library-name="lib2"'\'''
FLG -w @1..3@5..28@30..39@43@46..47@49..57@61..62-40 -strict-sequence -strict-formats -short-paths -keep-locs -rectypes -warn-error -A -w -33

I also prepared a log while reproducing the same behavior in vim+merlin.