96

I have a ruby script array when each element needs processing :

threads = []
elemets.each do  |element|
    threads.push(Thread.new{process(element)}}
end
threads.each { |aThread|  aThread.join }

how ever due to resource limitations, the script works in an optimal way if no more the four elements are processed at a time.

no I know I can dump the each loop and use a variable to count 4 elements and then wait but is there a cooler ruby way to do it ?

5 Answers 5

204

You can enumerate in groups of 4 for an array:

>> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].each_slice(4) {|a| p a}
[1, 2, 3, 4]
[5, 6, 7, 8]
[9, 10, 11, 12]

So you can try something like

elements.each_slice(4) do | batch |
    batch.each do | element |
        threads.push(Thread.new{process(element)}}

    end
    (do stuff to check to see if the threads are done, otherwise wait )
end

Its may not be what you need, though - I have been up since 3 AM and I only had a couple of hours sleep. :/

4
  • 2
    @Rilindo: this is brilliant! modified two lines and I was good to go. thanks.
    – Eli
    Jan 1, 2010 at 11:17
  • 2
    (my) solution below should be more efficient when tasks take variable time to process. This solution assumes each thread will take the same time to process a list of 4 elements. Jul 22, 2012 at 19:57
  • 3
    I think I just fell in love with Ruby all over again :) Sep 12, 2012 at 11:26
  • If you're using Rails, there's the even more readable "in_groups_of"... elements.in_groups_of(4) do |group| blah end
    – Jason
    Apr 19, 2017 at 20:26
21

If I read you right, you want to have no more than 4 threads processing at a time.

Sounds to me like you should launch only 4 threads, and have them all read from a shared Queue (part of the standard thread lib) to process the elements.

You can have the threads end when the queue is empty.

Slicing the array into 4 equal arrays, and having each thread process 1/4 of the elements assumes that each element processes in the same time. If some take longer than others, some of your threads will finish early.

Using a queue, no thread stops until the shared queue is empty, so it is I think a more efficient solution.

Here is a working program based on your code to demonstrate:

require 'thread'

elements = [1,2,3,4,5,6,7,8,9,10]

def process(element)
    puts "working on #{element}"
    sleep rand * 10
end

queue = Queue.new
elements.each{|e| queue << e }

threads = []
4.times do
    threads << Thread.new do
      while (e = queue.pop(true) rescue nil)
        process(e)
      end
    end
end

threads.each {|t| t.join }
3
  • This solution is pretty close to perfect for me except that it throws an error upon completion: ArgumentError: tried to create Proc object without a block Seems that it doesn't like the while (e = queue.pop(true) rescue nil) Nov 17, 2018 at 6:38
  • I don't get that error, tried 2 versions of ruby - what version are you using? Nov 29, 2018 at 7:06
  • Version 2.3.1. I was running it inside of a rake task in Rails so it's quite possible that there is a conflict elsewhere. Nov 30, 2018 at 18:08
13

In rails(not Ruby) a more readble form can be used in_groups_of

arr= [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
arr.in_groups_of(4, false) {|a| p a}

result:

[1, 2, 3, 4]
[5, 6, 7, 8]
[9, 10, 11]

The last row has only 3 elements as we have specified false as second argument in in_group_of. If you want nil or any other value you can replace false with that value.

1
  • 3
    in_groups_of is a rails method, won't work with plain ruby
    – Subash
    Jul 24, 2019 at 4:57
2

Yes, but you need to do some method overriding. Usual approach is to override '/' for Array like so:

class Array
  def / len
    a = []
    each_with_index do |x,i|
      a << [] if i % len == 0
      a.last << x
    end
    a
  end
end 

And with that defined you can now easily do:

foo = [1,2,3,4,5,6]
foo / 2
# Result is [[1,2], [3,4], [5,6]]
1
  • 2
    I think that overriding methods on such basic classes is rather dangerous - even if (like in this case) they weren't defined before. Why is it / and not %? What if another developer (or I, who implemented this) comes by in a year or two and wants to make sense of the code, asking "what the heck does an Array divided by a number actually mean"?
    – haslo
    Jan 26, 2015 at 14:06
2

Not sure if the following variant counts as just using a "variable to count 4 elements", or could be considered cool, but it gives you an array in slices of size no greater than 4 elements:

x = (1..10).to_a
0.step(x.size - 1, 4) do |i|
    # Choose one
    p x.slice(i, 4)
    p x[i, 4]
end

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.