For my application (a dependently typed proof assistant) I need a parser with (among other things) two properties: It is aware of UTF-8. Yes, any OCaml string can contain UTF-8 data, but it seems to me (correct me if I’m wrong) that parsing operations like “consume characters until you reach a cl…

[image] viritrilbia: So if you can recommend a parser-combinator library that works on an undifferentiated stream of tokens that would be output by something like sedlex, I would be happy to try that. All the parser-combinator libraries I’ve found so far seem designed to work on a stream of chars…

I haven’t learned yet what all the different names for parser types mean. (Another nice thing about a parser-combinator library is that I don’t have to.) But if I don’t find anything better, I’ll look into that; thanks! I know of Coq libraries that do successfully use unicode, including even the s…

The Ucs_scan module in my Orsetto project has those properties. Not sure I want to recommend it to you, because having used it in anger for a similar project, I have now commenced a project to refactor the whole thing from top to bottom, and I’m a long way away from releasing that still. That said,…

I remember that Angstrom handled basic Unicode just fine - I used it in a production environment. Not sure about more complex cases. Sadly, seems the project looks abandoned and not updated for a while.

I would never use a parser combinator library or a parser generator for a compiler. The real problems for a parser (and lexer) is resilient error handling and “interactive” usage like as a LSP for editing. Everything else is relatively easy, if you have a working grammar. For handling user defined…

@viritrilbia : Have you looked into Fmlib_parse ? Fmlib_parse is a combinator library, therefore it satisfies your second requirement as any combinator library should do. The character parser in the library works on Ascii characters i.e. is not (yet) unicode aware. I am planning to make it unicode a…

@hbr That looks like it might be exactly what I want, thanks! I will check it out and maybe get back to you.

With all the discussions about Unicode, what are the obvious pitfalls I’m missing? Long long time ago, we used Menhir in a standard way where our language used Unicode characters for quantifiers and logical connectives. We tried, and it worked; we didn’t do anything Unicode-aware-specific IIRC.

What I wanted to convey is: [image] What are the biggest reasons newcomers give up on OCaml? First it should be stressed that for many programs just passing around UTF-8 encoded string values is entirely good enough, even more so that structural text properties (e.g. think of splitting on a comm…

[image] lukstafi: We tried, and it worked; we didn’t do anything Unicode-aware-specific IIRC. The main issue is how to describe tokens. Say you want to allow identifiers to contain arbitrary unicode (accented) letters, then writing the proper regular expression is possible but quite painful. …

Unicode-aware parser combinators

Learning

lukstafi September 12, 2023, 11:07am 15

What I wanted to convey is:

Topic		Replies	Views
OCaml Parser combinator library as powerful as fastparse(Scala)? Ecosystem	27	4631	November 25, 2022
High-performance lexing in OCaml Ecosystem performance , ocamllex	20	3375	April 1, 2021
[ANN] Fmlib 0.5.8 with unicode parsers Ecosystem	2	334	February 5, 2024
[ANN] Parseff: parser combinator library for OCaml 5 Ecosystem announce	33	1657	April 17, 2026
Sedlex (why not ocamllex?) Ecosystem sedlex , ocamllex	2	1562	March 26, 2021

Unicode-aware parser combinators

Related topics