[ANN] OCaml User Survey 2020

Community
#1

Hi everyone,

We are happy to announce the OCaml User Survey 2020. We are trying to get a better picture of the OCaml community and its needs. It would be very useful if you could fill the survey (10-15 minutes), and share it widely with other OCaml programmers!

https://forms.gle/MAT7ZE7RtxTWuNgK7

The survey is run by the OCaml Software Foundation. Thanks in particular to our sponsors OCamlPro (@MuSSF) for preparing many of the questions, Jane Street (@Yaron_Minsky) for excellent feedback, and to Kim @K_N Nguyá»…n for his technical help.

This is our first year running the survey, we hope to continue in following years. There are many things to improve; please feel free to give us feedback! (There is a feedback question at the end of the survey, or you can post here, or send me a message/email.)

The survey was inspired by programming-language surveys ran by other communities. See for example past survey results for Go, Haskell, Rust, and Scala.

33 Likes
Suggestions from the OCaml Survey result
#2
#3

On page 5, OCaml 4.04 is missing. Is that intended?

#4

Really glad to see you doing this! Just took the survey and I’m very much looking forward to seeing the results.

For next year’s survey – contact me ahead of time, and I’ll be happy to do a review. (I’m a sociologist, specializing in research methods.) It looks good overall but there are little things that can be improved (e.g., there’s almost never an “inapplicable” option, which means that you won’t know if people skipped a question b/c it doesn’t apply to them or for some other reason).

8 Likes
#5

To add to that–a couple of the questions allow you to choose one option, where they could allow choosing multiple. E.g. what features would you like to see in OCaml. There’s no reason to restrict the question to only one option, and allowing multiple choices (i.e., approval voting), allows people to express preferences in a more nuanced way so you get a more accurate overall view of what the community is interested in.

That said–thanks for conducting this much-needed survey :slight_smile:

4 Likes
#6

At “For how long have you been using OCaml?”
there is no option to say “1-2 years”.

1 Like
#7

I think the survey is well done and raises interesting questions. I do like to think that maintainers’ most burning options tend to come first among the choices. So some randomisation might avoid bias in future surveys.

1 Like
#8

As a suggestion for one more possible question. I would take a list from https://ocamlverse.github.io/content/help_needed.html and ask something “Mark 2 as the most important for you to be done”.

1 Like
#9

This might be a good addition, but note that that list contains projects that are open to help from new contributors. They are not necessarily the most important projects for the community. For example, multicore is probably high on most people’s list of projects that are important, but it’s being implemented by a small group of experts, and probably would not benefit from “too many cooks in the kitchen”, so it wouldn’t be in a list of projects looking for help.

#10

I would like to thank everyone for your feedback, in this thread and in the responses. We will incorporate it for next year’s survey run.

I wonder if some people out there would be volunteers to help with processing the survey results, in preparation of a post about them. (Right now there are about 650 responses, which I think is already significant to draw conclusions; I am planning to resend it for another round in a couple weeks.) We are planning to publish the full results in any case, so a volunteer would not have access to more information (anyone will be free to look at the result and post their own analysis/deductions/conclusions), but they could help providing the initial analysis, writing the analysis post, and (tedious but useful) clustering free-form answers into pertinent new options. (Otherwise, most likely, I will do this work myself. I’m not looking for crowd-sourcing, I think one/two people work best.)

@cjr do you think that you would have some methodological advice on how to analyze the survey results? One thing I don’t know about, for example, is what are good practices to analyze “feelings” question with answers of the form (“I (strongly) {agree, disagree}” or “Neutral”): in the analysis, how should we handle the “neutral” responses?

1 Like
#11

Regarding the Likert-type scales, what I frequently do is collapse response ranges together and drop the middle/neutral responses. Collapsing response ranges is useful because one person’s “strongly agree” might be another’s “agree.” How many categories is a judgment call but with a 1-5 range (which I think is what the survey used), I’ll often simply collapse all positive responses together and all negative responses together.

I usually drop the neutral category entirely because it’s rarely clear what it means. Does it mean that the respondent is truly ambivalent? Or that they don’t know or that they don’t care? Or that the question is irrelevant to them? Something to look for: if you’re getting a higher-than-average number of “neutral” responses for a particular question, it might be a sign that the question in bad in some way. (Incidentally, I rarely include middle categories with my Likert-type scales precisely because I don’t know what they mean. I force a positive or negative response and then provide options for “Don’t know,” “Inapplicable,” etc.)

1 Like
#12

I wouldn’t mind helping you out :slightly_smiling_face:

Re “publishing all results”: does that mean you plan to make all responses openly accessible? Should it come with some privacy-preserving techniques applied, particularly for free form questions?

1 Like
#13

for next year, force people to also vote amongst “housekeeping” buckets, where one has to choose amongst various less interesting tasks that need to be done but aren’t as exciting as new language features or new libraries.

1 Like
#14

Thanks to whoever wrote the questions. This is really well put together.

1 Like
#15

One other minor comment. someone mentioned open-collective as a way for individual community members to vote on community projects, in a related ocaml community thread. Maybe this survey can eventually feed into an Ocaml open collective project with modest funding goals, as a continuous way to get feedback on which projects people want to see realized sooner. The projects that reach their funding goal can trigger getting corporate matching funds.

#16

The OCaml survey 2020 is now closed. It attracted 745 replies. Everyone should be able to see the summary of results and to download the raw data.

I and a few others will try to write a summary of the replies, especially of the many free-form replies. Everyone is welcome to help sifting through the results! Short comments can be posted in this discussion thread, and longer analyses can be put somewhere else and mentioned here.

12 Likes
Suggestions from the OCaml Survey result
#17

Thanks a lot for this!
Some of the bar graphs look a bit weird (for example on “Which types of software do you develop with OCaml?”), with some of the labels on the left not being there (though one can still hover the bar itself to get it). I wonder if there’s something that could be done about it?

#18

Thank you for this insight!

If I may make a suggestion: to me, histograms illustrate results more clearly than pie charts, especially with arbitrary colors like that. If colors were a gradient from red to green for those [disagree..agree] range questions, it’d already be easier to interpret at a glance.

2 Likes
#19

Right from Vg’s test image database here is a demonstration that pie charts are a bad visualisation device (the eye is not good at comparing angles).

5 Likes
#20

Keep in mind that some people have no difficulty sorting pie slices by size. To us, the pie chart is more readable than the equivalent bar chart.