How do I reduce complexity of this algorithm that take first and last element?

Hello,
The algorithm I need to implement is an assignment for a website called Codewars.

The exercise is here : Exercise kata list


let rec remove_last (l: 'a list) : 'a list  = match l with 
  | [] -> []
  | [_] -> []
  | hd::tl -> hd :: (remove_last tl)

let arrange (s: 'a list): 'a list = 
  let rec arr2 (lst: 'a list) (inc: int) (lengthList: int) : 'a list = match lst with 
    | [] -> []
    | ed :: lst -> if lst <> [] then
          if inc mod 2 = 0 then
            ed :: (List.nth lst (lengthList - 1)) :: arr2(remove_last lst) (inc+1) (lengthList - 2)
          else
            (List.nth lst (lengthList - 1)) :: ed :: arr2(remove_last lst) (inc+1) (lengthList - 2)
        else
          ed :: []
  in arr2 s 0 (List.length s - 1)

The results are not terrible, I fail when it comes to big list size because it takes too much time :

Iā€™m aware that the builtin List.nth is O(n), same for the function remove_last and List.length, and I do it for each element of the original listā€¦

I ask here if there is a way to reduce complexity and optimize this or even if Iā€™m on the wrong direction.

Of course I can click on the solution button to get it but I want to search before and ask here if I did something wrong with this program.

Thanks in advance

One ā€œsimpleā€ thing which you can do is to convert the list to an array.

Than ā€œremovngā€ first and last is easy, but you need to keep track of when the list/array should be
considered reversed.

One easy way to reverse the array is to have a pair of array, one is the reversed version. Then from a (a,b) pair, if we recurse with the (b,a), it will be like a reversion of the array at no cost.

Without array, one can get the list length, the reversed list, compose a pair of both lists (original and reversed). Suppressing the first of both is O(1)ā€¦ and decreasing a counter set initially to length/2 is also O(1).

3 Likes

Reading on my phone, it looks like you can simplify.

Since it is asked that you donā€™t mutate the input, I would stick to applying functions recursively to the initial list. I think you only need 2 functions.

Also, I would try not to call `List.nthā€˜ at all.

I guess it would not be fair that we solve the puzzle for you, but the strong hints are: do not use List.length nor List.nth. But you can use List.revā€¦

Hello all,
Thanks for replying to this topic.
I wanted to get some advices, sorry for my poorly formuled question.

I will reply for each person what I tried :

@sanette I only use once List.length and I always do a -2 or -1 operation on an int.
I only manipulate integers, so It should be better than one of my old versions calling a lot List.length at each call of the function.
For List.rev, I used it and associated it with this :

List.hd (List.rev lst)

But I know that List.rev is same complexity O(n) than List nth because I have to go through all the list to get the last element, or reverse all the list to get the first element.

I tested some things about this solution : the performance is nearly the same on this website (0.03 ms difference when I did my tests, not sure if it is really impactful).

@benjamin-thomas Yes I try to do with this type of idea at the beginning.
But at one moment I have to call the nth function to get the last elementā€¦
Is it really costing a lot when I define a function into multiple little functions ?
Maybe there is some magic behind it and I donā€™t see itā€¦

@Frederic_Loyer and @vrotaru Thank you for the suggestion about the array.
I wasnā€™t aware about this type because it only specifies 'a list in the exercise.
I will try to change my solution and do it with yours with the array type.

Regards

Update,
I solved it with arrays and doing some tricks with the modulo.
It is quite dirty unfortunately even if I understood all the cases.
edit : I deleted solution answer (not sure about the codewars thing about posting solution), it would be better for a solution logic after some thoughts :

logic

I convert it in array first.
I define i incrementator.
I go through the array and get the i index and the arrayLength - i index values.
If my i is even, it is (a,b)
If my i is odd, it is (b,a)
I return my list, all with recursive.

The array solution is very good thanks for giving this hint.
I saw some people with the Array solution.

Many thanks.
Regards

The idea I have proposed is to use List.rev once, List.length once (O(n)), and call an auxiliary function with the_list, the_list_reversed, thelength/2. The auxiliary will call itself recursively, but only use O(1) operations (removing a item, composing the result, swapping lists. The end condition will be on the counter since at the end, each list contains half of the original list.

Sorry, I misunderstood it.
I got the part that I only need to check List.length/2.
I didnā€™t use at all the reverse function because it doesnā€™t need to be reversed.
When you increment a given said i variable, you can verify if i is even or odd and do this (a,b) or (b,a) when i is odd or even.
So I went with the solution of not doing it at all finally and changing a lot with converting it to an array and changing the logic of the program.
I should try your solution too because it seems interesting.

Finally, call the aux with Length of the list, and decrement it by two at each step. There are two final conditions : the counter= 0 and the_counter=1.

The List.stable_sort function from the standard library is an interesting example of tricks to make algorithms on lists faster. It is unfortunately not documented at all, but if you can figure out how it works this will likely give you some ideas for your own exercise.
You can find the code here: https://github.com/ocaml/ocaml/blob/ef62c57357fb202d0caa791f840a71af4bd07fd0/stdlib/list.ml#L368-L433

1 Like

Since you said you got it, hereā€™s my solution:

let arrange =
  let rec aux acc = function
    | [] -> List.rev acc
    | x :: y :: xs -> aux (y :: x :: acc) (List.rev xs)
    | [ x ] -> aux (x :: acc) []
  in
  function
  | [] -> []
  | x :: xs -> x :: aux [] (List.rev xs)
;;

Unfortunately, it doesnā€™t pass the ā€œBig sizeā€ test.

Since accessing the last element of a list is the most expensive thing we could do, and given the fact that we have to do this multiple times, Iā€™m not even sure we can pass the exercise without going through an array.

Since you asked for advice, I thought I should mention that I found the list exercises on OCamlā€™s website really good: Exercises

Working through them really helped me understand the bigger picture.

I have a solution that passed the ā€œBig sizeā€ test with pure lists. Itā€™s not exactly straightforward (and I found the description of the exercise misleading too) but itā€™s possible.
It uses more or less the algorithm that @Frederic_Loyer described above, manipulating two lists at the same time (normal and reversed) and swapping them after each step.

Excellent, thanks for letting me know!

I managed to get an implementation that passes the ā€œBig sizeā€ test now. Cool idea @Frederic_Loyer.

Looking at the posted solutions at codewars, your implementation is the best :exploding_head:

I have made:

let arrange (s: 'a list): 'a list =
  let rec aux l lr n =
    match n with
    | 0 -> []
    | 1 -> [List.hd l]
    | _ -> (List.hd l) :: (List.hd lr) :: 
              aux (List.tl lr) (List.tl l) (n-2)
  in
    aux s (List.rev s) (List.length s)

The point is to place List.rev outside the recursive aux function. The complexity is O(n).

1 Like

my version was essentially the same but with pattern matching (for the list):

let rec loop l t s rev =
  if l = 0 then t
  else if l = 1 then (List.hd s) :: t
  else match s, rev with
    | x1 :: rest1, x2 :: rest2 ->
      loop (l-2) (x2 :: (x1 :: t)) rest2 rest1
    | _ -> failwith "Should not happen"

let arrange s =
  List.rev (loop (List.length s) [] s (List.rev s))

In a small benchmark, this does not seem to make a difference, except for very large n, where pattern matching seems faster. Anybody knows why? Maybe itā€™s just because of tail recursivity

n time (pattern matching) time (List.hd and List.tl)
1 0 0
10 0 0
100 0,000001 0,000001
1000 0,000014 0,000011
10000 0,000253 0,000221
100000 0,009722 0,009957
1000000 0,162797 0,172738
10000000 1,609477 5,150795

Your program deconstruct the each list once (matchā€¦). My program deconstruct the lists twi (hd, tl). It can make a difference.

The idea to construct the result with an accumulator seems good. It makes the function tail recursive and avoids many ā€œreturnā€ in the assembly generated program. However I could have written

  let[@tail_mod_cons]  rec aux l lr n =

(With OCaml >=4.14)

And let the compiler optimise the function in a way similar to tail recursive function. See:

https://v2.ocaml.org/manual/tail_mod_cons.html

I havnā€™t triedā€¦

Just for fun, hereā€™s my latest (third) attempt.

On my prior (second) attempt, I had used only one partial function, once (List.tl).

I then wondered if it would be possible at all to use only total functions and I think the result is okay (a bit more complicated but still okay).

Iā€™ve been wondering for a while what it means to express a function using partial functions vs only using total functions. I understand it may not matter in practice, but Iā€™m wondering if it has meaning ā€œmathematicallyā€ so to speak.

Iā€™d be happy to read anyoneā€™s thoughts on this matter.

let arrange lst =
  let rec aux flip n acc fwd bwd =
    match (fwd, bwd) with
    | (x :: xs, y :: ys) ->
        if n = 0 then
          acc
        else if n = 1 then
          x :: acc
        else
          let new_acc =
            if flip then
              x :: y :: acc
            else
              y :: x :: acc
          in
          aux (not flip) (n - 2) new_acc xs ys
    | _ -> []
  in
  List.rev @@ aux false (List.length lst) [] lst (List.rev lst)
;;

@sanette
I have tried my program: 1.455s
Itā€™s [@tail_mod_cons] version: 1.364s
Itā€™s [@tail_mod_cons] version with pattern matching: 0.801s
And curiously, your version: 1.381s

These times are from a ocamlopt compiled version, looping 100 times arrange(r) where r is a list of 100 000 items.

Then having a tail recursion is niceā€¦ but if we have de reverse the list, the gain of the optimisation is lost. And [@tail_mod_cons] indicates a slight gain (without changing the logic). The pattern matching however has an important impact. With List.hd and List.tl, Ocaml has to check twice if the list is not nul. With a pattern matching, only once.

1 Like

Did you try n = 1_000_000 ?

1 Like