Another Q: getting proper location information into testcase code

Ecosystem
1

Since the last time I asked a Q here, I got a great solution, I thought I’d ask another question.

I’m writing a bunch of lexers/parsers, and of course I want to test the Dickens out of them. So I have code like:

assert_equal () (ignore({|Hello, <name>!|} |> Pa_st.Template.of_string))

(and later, I will change the thing being compared-against, to something more meaningful, but for now, just knowing that the parser succeeds is useful. When the parser is driven from a file, or the string is from reading a file, this is easy: the function of_string takes a (Camlp5-style) location argument, viz:

val of_string : ?startloc:Ploc.t -> string -> t

and the lexer (underneath the parser) uses that location as the starting-point for the input; so any location within the text is offset from that start-location. But what about when (as above) the string is a constant in a file? I’d like to hand start-location of the string to of_string, but that seems not-obvious. ppx_here can get you a Position, but it’s not at the start of the string, and I don’t know of some obvious way to get that. So e.g.

{|abc def<|} |> Pa_st.Template.of_string ~startloc:(Util.ploc_of_position [%here])

works (and the string is malformed) and the error-message looks like this:

(Ploc.Exc (<"stringtemplate_test.ml":15:110-110>,
   (Stream.Error "[map_expr] expected after '<' (in [expr_tag])")))

which is almost right! It’s just … offset from the string. I don’t know of an obvious way to get the location for the start of the string, short of a PPX rewriter kind of like ppx_here, which would work like this: the source phrase

[%wrap_here <expr>]

would be rewritten to

(<position of the expression>, <expr>)

So, kind of like ppx_here, but slightly different.

Does anybody have any thoughts on a way to do this without a new PPX rewriter? FTR, the problem isn’t writing the rewriter – it’s utterly trivial – but rather, I’m loath to pollute the space of PPX rewriters with yet another tiny, trivial rewriter, unless it’s actually, y’know, needed.

Your advice appreciated.

2

Not exactly what you asked, but have you looked at __POS_OF__?

"abc" |> __POS_OF__ |> ...

Cheers,
Nicolas

3

Oh wow, I had no idea! And it’s so close! I think there’s a (actually two?) bug there? I tried it thus:

(1) a file, “foo.ml” (first line is blank):


let p = __POS_OF__ "abc" ;;

(2) compile it with ocamlc -c foo.ml (version 5.5.0)

(3) load into top-level:

env TOP=.. ocamlfind camlp5-buildscripts/LAUNCH  -- ocaml -nopromptcont
OCaml version 5.5.0
Enter #help;; for help.

# #load "foo.cmo";;
# Foo.p;;
- : (string * int * int * int) * string = (("foo.ml", 2, 8, 24), "abc")
#

There seems to be two problems:

(a) the “location” is of the entire expression __POS_OF__ "abc" and not of just "abc"

(b) but -also-, the latter two offsets are from the beginning of the line, and there is no way to recover what is needed to construct a Lexing.position, which is what would be needed to drive any sort of lexer/parser.

But … so close! So close!

Should I open a bugreport at github/ocaml/ocaml for this?

ETA: re bug #a, the exhibited behaviour is different from what is documented at the page you pointed me to.

ETA2: I thought I should add that even with these changes, it wouldn’t be quite … perfect. In my description of the problem, I suggested that the position would be of the start of the expression, but really, for best results, what would would want is for this to work only for strings (quoted or raw) and for the position to be that of the first character. Hence, not generally useful for all expressions. If the position is of the beginning of the lexeme that is the string itself, then the programmer has to do the offset calculation each time they’re debugging. It’s trivial, but also easy-to-get wrong if one isn’t paying attention. And certainly would mean that one can’t blindly use automatic “jump to the location of this error” as in Emacs compilation mode.

So maybe it’s not worth filing a bugreport.

4

Hi Chet,

Your (a) certainly looks like a bug and is definitely worth to open an issue over at GitHub - ocaml/ocaml: The core OCaml system: compilers, runtime system, base libraries · GitHub. The fix is likely rather easy a good junior job (always good to have those around !).

Your (b) seems to heave been a clear choice towards not depending on Lexing in this API. It is a long-running discussion which concrete types should be used to represent locations in the user-facing parts of the compiler/standard library. A similar discussion has been happening in [RFC] Implicit source positions by OlivierNicole · Pull Request #13886 · ocaml/ocaml · GitHub.

Cheers,
Nicolas

5

You may want to have a look at the testing framework I’m using which uses __POS_OF__ for in source snapshot testing without all the ppx nonsense.

Even though as I note in my processing of __POS_OF__ the range is not aways clear, I’m not sure I’d advocate to change (a) by now. It will only make it harder for everyone who needs to process these across versions and you can easily compensate the offset.

6

@dbuenzli , @nojb thank you both. I won’t submit a bugreport, per Daniel. Also, I appreciate what you say, that this is a way of getting location information that doesn’t commit you to the … weight of the Location machinery. Which … OK, it’s lovely, but a lot more than what a casual user of “I wanna print the line/col in my source, man” would want.

7

For that you have __POS__ no ?

8

Yep, that’s what I meant. Just echoing back what you said!

9

Ah I think I misunderstood what you said :–) Just for clarity:

  1. __POS__ just get the line/col in my source.
  2. __POS_OF__ get the pair of line/col that spans a value in my source.

However for 2. given the representation returned to you, you need the actual source in your hands to properly compute the line/col of the end of the span.

10

grin no worries. In my case, I want this to drive a lexer for testing purposes, so it needs to be a Position/Location. Luckily PPX is here to help!

ETA: “lexer”, not “lever”. Android keyboards!