I am happy to announce the initial opam release of
pyre-ast, a Python parsing library.
The library features its full-fidelity to the official Python spec. Apart from a few technical edge cases, as long as a given file can be parsed/rejected by the CPython interpreter,
pyre-ast will be able to parse/reject it as well. Furthermore, abstract syntax trees obtained from
pyre-ast is guaranteed to 100% match the results obtained by Python’s own
ast.parse API, down to every AST node and every line/column number.
Another notable feature of this library is that it represents the Python syntax using the tagless-final style. This style typically offers more flexibility and extensibility for the downstream consumers of the syntax, and allow them to build up their analysis without explicitly constructing a syntax tree. That said, for developers who are less familiar with the tagless-final approach, we also offer alternative interfaces that operates on traditional syntax tree represented as algebraic data types.
Documentation of the library can be found here.
The reason why we can can claim full-conformance with CPython is really simple: the library is, under the hood, merely an OCaml wrapper around the parsing logic in CPython source code. The project was initially motivated to replace the custom
menhir-based parser currently used in the Pyre type checker (hence the name), but I figured that it would be useful to release this as a standalone
opam package to the community so other static Python analyzers or other DSLs with Python-based syntax can leverage it as well.
The library has yet to be put into production for Pyre (I’m working on it though) so please do expect bugs/jankiness at times. Feedback and bug reports are very welcomed.