[ANN] v0.14 release of Jane Street packages

Community
1

Dear OCaml developers,

We are pleased to announce the v0.14 release of Jane Street packages!

This release comes with 15 new packages, and a number of fixes and
enhancements. The documentation for this release is available on our
website:

https://ocaml.janestreet.com/ocaml-core/v0.14/doc/

The remainder of this mail highlights the main changes since the v0.13
release; we hope it will be useful to developers in the process of
migrating to the new version. A comprehensive changelog is available
at the end.

Notable changes

  • Updated Bin_prot functors Binable.Of_sexpable and
    Binable.Of_stringable to take UUIDs.

  • Changed Base and Core partition* functions from using [ `Fst of 'a | `Snd of 'b ] to using ('a, 'b) Either.t.

  • Replaced Date functions add_weekdays and add_business_days
    with functions that clarify the behavior w.r.t. weekends.

  • Added syntax while%bind, analogous to if%bind.

  • Added to the Async.Log.create function a required unit argument,
    and an optional argument to specify a Synchronous_time_source.t.

  • Made Core_kernel deprecate the Unix module, recommending
    Core.Unix, with Caml_unix as a fallback.

  • Changed Core_kernel.failwiths (aka Error.failwiths) to require
    its ~here argument.

  • Added ppx_let syntax let%mapn and let%bindn, for n-ary map and
    bind.

  • Renamed Expect_test_helpers as Expect_test_helpers_async, and
    renamed Expect_test_helpers_kernel as Expect_test_helpers_core.
    Removed the inclusion of Expect_test_helpers_core in
    Expect_test_helpers_async.

  • Increased Async’s default maximum number of open file descriptors
    from 2^15 to 2^16.

  • Moved most of Core.Iobuf to a standalone library, Iobuf, that
    depends only on Core_kernel, and can be used in javascript. Moved
    the rest of Core.Iobuf into a standalone library, Iobuf_unix,
    that depends on Core.

New packages

Deprecations / Removals

Async:

  • Deleted functions that were deprecated in 2018 and earlier.

  • Deprecated the Deferred.choice type alias, in favor of
    Deferred.Choice.t.

  • Removed Pipe.init, which has been deprecated since 2016.

  • Deprecated Throttle.Deferred, which was a pointless alias for
    Deferred.

Core:

  • Removed the Iobuf_debug module.

Core_kernel:

  • Deprecated the Bug exception.

  • Made Core_kernel deprecate the Unix module, recommending
    Core.Unix, with Caml_unix as a fallback.

Moves

Core:

  • Moved most of Core.Iobuf to a standalone library, Iobuf, that
    depends only on Core_kernel, and can be used in javascript. Moved
    the rest of Core.Iobuf into a standalone library, Iobuf_unix,
    that depends on Core.

  • Moved the Bigstring module out of Core to a standalone library,
    Bigstring_unix.

  • Moved the Linux_ext module out of Core to a standalone library.

Core_kernel:

  • Renamed Fqueue functions to be consistent with Queue and
    Fdeque.

  • Moved Bus to a standalone library.

Changelog

Async:

  • Deleted functions that were deprecated in 2018 and earlier.

Async_kernel:

  • Added Pipe.folding_filter_map' function.

  • Added Synchronous_time_source.next_alarm_fires_at function.

  • Added Synchronous_time_source.Event.Option module, an immediate
    option with support for optional syntax.

  • Added Pipe.concat_pipe function.

  • Added Scheduler.run_every_cycle_end, analogous to
    run_every_cycle_start.

  • Exposed Scheduler.Expert.last_cycle_num_jobs function.

  • Added Async_kernel_scheduler.Expert functions run_every_cycle_{start,end} and
    deprecated set_on_{start,end}_of_cycle.

  • Added Async_kernel_scheduler.last_cycle_time accessor function.

  • Deprecated the Deferred.choice type alias, in favor of
    Deferred.Choice.t.

  • Exposed Synchronous_time_source function
    Expert.max_alarm_time_in_min_timing_wheel_interval.

  • Optimized Time_source.advance_by_alarms and
    Scheduler.yield_until_no_jobs_remain to reduce the per-alarm cost
    from ~660ns to ~140ns. Added to yield_until_no_jobs_remain an
    optional argument, ?may_return_immediately:bool to opt in to the
    faster behavior.

  • Fixed an issue with Pipe.upstream_flushed, so that its result does
    not become determined if an exception happens while processing some
    of the elements.

  • Removed Pipe.init, which has been deprecated since 2016.

  • Deprecated Throttle.Deferred, which was a pointless alias for
    Deferred.

Async_rpc_kernel:

  • Renamed Pipe_rpc.Direct_stream_writer.Group.flushed as
    flushed_or_closed, and changed it so that its result becomes
    determined if the individual pipes are flushed or closed.

Async_rpc:

  • Added Rpc.Connection.serve_inet function, for synchronously
    creating a TCP IP server.

Async_unix:

  • Added Tcp.Server.create_sock_inet function, which is synchronous
    and serves only inet addresses.

  • Added Tcp.Server.create_inet function, for synchronously creating
    a TCP IP server.

  • In Unix functions that create a File_descr.t supplied
    ~close_on_exec:true rather than calling set_close_on_exec.

  • Changed blocking Sexp.save* functions from @@deprecated to
    @@alert.

  • Made Async_unix shadow Core_kernel.eprint_s, which can block.

  • Added to the Log.create function a required unit argument, and an
    optional argument to specify a Synchronous_time_source.t.

  • Made Writer.flushed_* functions consistently flush the synchronous
    output channel if one is set.

  • Fixed a bug in Writer error handling so that it calls
    stopped_permanently when the writer is not allowed to write due to
    file-descriptor flags.

  • Added to the Log module the ability to set a transform function
    applied to each message that is logged.

  • Added Unix support for flock, paralleling the existing support
    for lockf, and extended the with_file function to support both
    mechanisms. Changed the lockf functions’ lock names from Read | Write to Shared | Exclusive.

  • Added Writer functions flushed_or_failed_*, which, unlike
    Writer.flushed, return a result when the underlying writer fails.

  • Fixed a race condition in Shutdown.don't_finish_before.

  • Added to Writer.with_file an optional ?syscall argument.

  • Improved Fd error messages for syscall* functions and
    with_file_descr_deferred_exn.

  • Improved Writer.with_file_atomic's error message when f closes
    t.

  • Increased Async’s default maximum number of open file descriptors
    from 2^15 to 2^16.

  • Fixed a race in Writer.with_file_atomic and the save* functions
    that use it that could cause the file permissions to be incorrect.

  • Fix a bug in Shutdown.shutdown_on_unhandled_exn where a process
    would never shutdown if Debug.log raised (e.g. if stderr was a
    broken pipe).

  • Added to Sys.when_file_changes an on_exn optional argument.

  • Added Process.send_signal function that is safe against pid reuse.

  • Added Unix.waitpid_prompt function, a version of waitpid that
    guarantees that it’s determined in the same async job where the
    wait system call is done.

Base:

  • Fixed a bug in Int.round_nearest, which could overflow.

  • Added Hashtbl.Merge_into_action.t type.

  • Eliminated the Base_boot library, lib/base/boot, by including
    its ppx expansions directly in the Base library, lib/base/src.

  • Changed partition* functions from using [ `Fst of 'a | `Snd of 'b ] to using ('a, 'b) Either.t.

  • Added Hashtbl functions find_and_call1 and find_and_call2,
    aimed at avoiding allocation.

  • Improved the Gc module’s deprecation message to recommend using
    Caml.Gc.

  • Added List.is_suffix function.

  • Added Map.combine_errors function.

  • Added Result functions to_either and of_either, and deprecated
    ok_fst.

  • Added Char.Caseless submodule, analogous to String.Caseless.

  • Added String.Caseless functions is_substring* and substr*, for
    case-insensitive substring matching.

  • Added the Nothing module, moved from Core_kernel.

  • Added Ref function sets_temporarily, a generalization of
    set_temporarily to multiple refs.

  • Changed Float.min and Float.max to never return Float.nan, and
    instead to always return one of the arguments. This improves
    performance by avoiding allocation.

  • Added Option.try_with_join.

  • Added Int_intf.S functions clz and ctz, for leading and
    trailing zeros in the binary representation of an int.

  • Added Nothing module (moved from Core_kernel), which defines
    type t = |, the uninhabited type.

  • Dropped support for OCaml < 4.07.

  • Added Source_code_position.of_pos, which allows you to create
    Source_code_position.ts from the OCaml built-in Caml.__POS__ special
    value.

  • Added Sequence functions to convert between Sequence.t and
    Caml.Seq.t.

  • Added integer byte-swap functions, bswap*.

  • Added Set.are_disjoint function.

  • Added String.Search_pattern accessor functions pattern and
    case_sensitive.

  • Added String functions chop_{suffix,prefix}_if_exists.

  • Added Comparable functors Infix and Polymorphic_compare, which
    implement the comparisons in their eponymous signatures, and require
    only compare as input.

  • Added Blit.S1_distinct interface, which allows the src and dst
    types to differ.

  • Added Uniform_array.iteri function.

  • Added Map.validatei function.

Bin_prot:

  • Updated functors Binable.Of_sexpable and Binable.Of_stringable
    to take UUIDs.

Core:

  • For Unix functions that create a File_descr.t, added an optional
    argument, ?close_on_exec:bool, for atomic creation of
    close-on-exec file descriptors.

  • Added Unix function flock_blocking.

  • Improved Unix.create_process_env's error message when it is called
    with ~prog_search_path:[].

  • Removed the Iobuf_debug module.

  • Added Iobuf functions copy and clone.

  • Moved most of Core.Iobuf to a standalone library, Iobuf, that
    depends only on Core_kernel, and can be used in javascript. Moved
    the rest of Core.Iobuf into a standalone library, Iobuf_unix,
    that depends on Core.

  • Added Unix.Env submodule, with type Unix.Env.t = Unix.env, and
    exposed functions for manipulating Env.t.

  • Added Filename function open_temp_file_fd, which is like
    open_temp_file but returns a file descriptor rather than an out
    channel.

  • Made Time_ns.{Span,Ofday}.Option match Quickcheck.S.

  • Moved the Bigstring module out of Core to a standalone library,
    Bigstring_unix.

  • Moved the Linux_ext module out of Core to a standalone library.

Core_kernel:

  • Added Bus.Callback_arity.Arity5.

  • Generalized Univ_map to support a user-supplied key type.

  • Added Gc.For_testing submodule, with functions for measuring
    allocation, previously in Expect_test_helpers_kernel.

  • Renamed Fqueue functions to be consistent with Queue and
    Fdeque.

  • Moved Bus to a standalone library.

  • Added to Option.Stable.V1.t @@deriving equal.

  • Added to Time_ns.Stable.V1 Map and Set modules.

  • Replaced Date functions add_weekdays and add_business_days
    with functions that clarify the behavior w.r.t. weekends.

  • Fixed a bug in Command.run that caused incorrect calls in
    let%test_unit to mistakenly succeed.

  • Added Command.Auto_complete submodule, naming the type of
    Arg_type.create's ~complete argument.

  • Added Date.Option submodule, Immediate_option.S.

  • Added submodule Percent.Option : Immediate_option.S_without_immediate.

  • Changed Command.Param.choose_one to disallow parameters which
    don’t have any CLI flags.

  • Changed Blang.or_ and Blang.and_ to make trees that
    short-circuit more quickly by traversing fewer Blang.t nodes at
    eval.

  • Added to Command.run an optional argument,
    ?when_parsing_succeeds:(unit -> unit), which runs before the
    program’s main thunk.

  • Improved Command.Param.choose_one's error messages.

  • Deprecated the Bug exception.

  • Made Core_kernel deprecate the Unix module, recommending
    Core.Unix, with Caml_unix as a fallback.

  • Added Gc function allocated_words : unit -> int.

  • Changed Core_kernel.failwiths (aka Error.failwiths) to require
    its ~here argument.

  • Changed Command argument parsing so that -- is only interpreted
    as a prefix of exactly --, an in particular not a prefix of
    --help.

  • Extended String.Stable.V1 to match Stringable.S.

  • Added Hash_queue function lookup_and_remove.

  • Add Set support for writing Set.V1.M(Elt).t in stable types.

  • Added Byte_units.arg_type.

  • Extracted from Command the Shape-related code into a new file,
    command_shape.ml.

  • Made Date.Option match Comparable.S_plain and
    Quickcheckable.S.

  • Added Quickcheckable.Of_quickcheckable* functors.

  • Added Time_ns.Ofday.every function.

  • Made [@@deriving quickcheck] work with Base-style maps and and
    sets, e.g. type t = int Map.M(String).t [@@deriving quickcheck].

  • Added to Command support for validating a command line, via the
    function Command_test_helpers.validate_command_line.

  • Added Filename.to_absolute_exn function.

  • Updated gc_stubs.c to work better with OCaml 4.10.

  • Exposed Quickcheck.Shrinker.filter* functions.

  • Exposed the Univ_map.Data module type.

Expect_test_helpers:

  • Generalized print_and_check_container_sexps so that it applies to
    stable container modules.

  • Fixed the handling of quickcheck shrinkers to print only the final
    failing value, rather than every value that was tried and failed.

  • Added Async analogs of Ref.set_temporarily and sets_temporarily:
    set_temporarily_async, sets_temporarily_async.

  • Renamed Expect_test_helpers as Expect_test_helpers_async, and
    renamed Expect_test_helpers_kernel as Expect_test_helpers_core.
    Removed the inclusion of Expect_test_helpers_core in
    Expect_test_helpers_async.

Expect_test_helpers_async:

  • Removed Expect_test_helpers_async.Expect_test_config.

Expect_test_helpers_base:

  • Added functions replace and replace_s, for rewriting strings and
    sexps in output.

  • Added function hide_temp_files_in_string, for hiding temp file
    names in expect-test output.

  • Changed require_sets_are_equal to take a module matching
    Comparator.S rather than a fragment of Set.S, which better fits
    with Base idioms.

Incremental:

  • Added function node_value, to get the current value of a node,
    even if it is stale or invalid.

  • Added support for syntaxes let%mapn and let%bindn.

  • Added Incremental.Var.replace function, analogous to
    Ref.replace.

Linux_ext:

  • Added function peer_credentials, for using the SO_PEERCRED
    socket option.

Stdio:

  • Optimized In_channel.input_all to eliminate an unnecessary copy
    between buffers.

Timezone:

  • Added Timezone support in javascript.
8 Likes
2

I noticed there is a msgpack library inside vcaml, maybe it makes sense to update http://opam.ocaml.org/packages/msgpack/ package?

3

It’s a totally unrelated package, as far as I can tell? And there’s also
msgpck on opam.

4

The message-pack library inside of vcaml is a fully functioning msgpack and msgpack-rpc implementation. We decided to write a new library because the official msgpack/msgpack-ocaml implementation doesn’t implement Ext-types, which are required for Neovim, and the “msgpck” package didn’t exist at the time.

@XVilka: what do you mean by “update [msgpack] package”? The libraries are not API compatible, so I’m unsure how you’d do that without breaking everyone currently using it.

1 Like
5

This makes total sense. In opam now, there are three implementations of message-pack:

From the opam repository perspective, it would be nice to have these merge into the upstream repository. We generally frown on contracted names that duplicate larger ones, but I understand why the situation arose in this case.

Next steps would be for the maintainers of all three to chat: who owns the repo in the upstream, is it still maintained and used, and if not what the procedure would be to replace it with one of the other two. This might take some time, but that’s ok; there’s no rush. But I’d rather not see a fourth competing implementation show up in a year :slight_smile:

2 Likes
6

What is the problem with having several implementations in opam? I use msgpck and I like its performance, stability, and relative lack of dependencies :slightly_smiling_face:

4 Likes
7

Minor correction: it’s part of Vcaml, not Base. (Though it does depend on Base).

9

Hi @xclerc

Thank you for the detailed changelog. This will be very useful in migrating from 0.13.

While looking at the new packages, I’ve noticed that some of the published libraries don’t define a public_name in their dune file, resulting in the libraries not being visible.

I opened https://github.com/janestreet/higher_kinded/pull/1, but looking further i noticed the same in the following libraries:

  • accessor
  • accessor_base
  • accessor_async
  • accessor_core
  • ppx_accessor

I stopped short of opening pull requests for those libraries as I’m not sure if you have internal tooling to update the library metadata.

Best,
Anurag

2 Likes
10

As someone who has been doing a lot with nested applicative/monadic traversals lately, seeing something like lenses appear in the OCaml ecosystem is very exciting, and I look forward to playing around with it in the medium-term future!

1 Like
11

Thanks for reporting the issue; I double checked that no other
library suffers from the same problem and opened the following
pull request:

2 Likes
12

It doesn’t look like the accessor or higher_kinded packages have any publicly accessible rendered API documentation on the Jane Street docs server yet. Is this intentional? (It’s not a massive problem, as I can always look at the .mlis, but it’d be nice for them to be published!)

13

This is now fixed, I believe. Sorry for the delay…

1 Like
14

Hi ! Sorry to dig up the post, but I’ve been bitten by the deprecation of Core_kernel.Unix, so I wanted to use Caml_unix instead. In a file with open Core_kernel at the top, I’m getting “Unbound module Caml_unix”. What am I missing ?

15

Caml_unix is part of the core_kernel.caml_unix library (installed with core_kernel), so you’ll need add that to your libraries list. E.g., in dune:

(executable
 (name foo)
 (libraries core_kernel core_kernel.caml_unix))

If you’re able to use Core (i.e., not on Windows or JavaScript), you can also consider using the Core.Unix module, which is a unix module with Core-style idioms.

1 Like
16

Will this be released on opam? I can’t see it on opam - Packages.

17

It is a separate library but it is still under the core package. To add it as a dependency you’d need the following in your dune file:

(dependencies core.linux_ext)
1 Like
18

I know all of these maintainers so I think I can make a chat opportunity. Shall I set it up?