ATD for dynamic schemas

I’m curious if anyone can give any advice (for or against) about using ATD with dynamic schemas / type definitions. My basic use case will be to add custom schemas (at runtime) using ATD to customisable portions of data in our model. We currently use json schema for this purpose (the data is encoded as as json and validated with a json schema validator/lib). I’m currently looking at rewriting this module in OCaml to implement quite a few new features (it’s in Go right now, which has excellent support for json schema, but the other new features are more cumbersome to build in Go).

It seems a bit easier to do some structural validation with ATD vs. json schema (and OCaml seems to lack a good schema json schema validator - e.g. instances of valid documents and the schema definition itself). I saw that it was possible to export an ATD definition as json schema which solves the outward-facing problem (making schemas available to clients), but the internal problem of validating the dynamically supplied schemas against some constraints (e.g. if you extend a type, you must extend [inherit] a given base schema/type definition) as well as validating input json documents with the supplied schema seems less clear to me. It also doesn’t seem to be a core use case of the ATD library (it’s more suited for up front generation of types, serialisers/deserialisers vs doing these kinds of activities at runtime / dynamically).

Any help is greatly appreciated!

Thanks
Sam

In YOCaml, we use ADT to define/decode structured document : lib/core/data.mli · main · funkywork / yocaml · GitLab.
The previous implementation was based on first-class modules and was not easy to use. We do not rely on extension because we “guess” that lib/core/data.mli · main · funkywork / yocaml · GitLab is sufficient to describe every scenario.

Thank you for the suggestion. I think this idea is similar to the approach I was considering (if I understand correctly what you are doing). The high level idea would be:

  1. Define some structural constraints at the level of each type (e.g. to extend “User”, you need certain “Base User” structures to be a part of any proposed extension schema). I was thinking I would be able to rely on the AST here.
  2. Validate a proposed dynamically supplied schema against any needed structural validations (e.g. the supplied schema’s AST contains the required structures)
  3. Report any inconsistencies with the constraints (I hope in a sane manner…)

The design of ATD seemed to be straightforward from a usability standpoint (easy to write extensions that visibly include the required elements), but that would require nearly everyone (our clients) to become familiar with the ATD language whereas if I could keep the specification in json [schema] it would at least be somewhat familiar territory. I just wasn’t finding great json schema support in ocaml libs so was looking for other more native options.

Oh yes, for that kind of purpose, are you aware about GitHub - ahrefs/atd: Static types for JSON APIs

For sure, yes. That was actually my original question (about ahrefs/atd… I probably should’ve included a link in my OP). I’m trying to evaluate using it like I described, but it doesn’t seem like a typical use case for it. It seems more designed for build-time usage (generate ser/des and validator code from the schemas) and I wanted to use it at runtime / dynamically. I wasn’t sure how well this would work out, so if you have any thoughts on that specifically, I’d be interested in perspectives on this type of use case for ATD (some example usage similar to my use case, if it exists would be super as I’m new to OCaml and also just started looking at ATD last week).

Thanks again for your input here!

My bad, I was reading ADT (abstract/algebraic data type) and not ATD, hence the enormous confusion. Sorry for the noise!

1 Like