3

I wish to know if my understanding of the producer consumer design is correct by using the ExecutorService & ArrayBlockingQueue. I understand there are different ways to implement this design but I guess, at the end, it depends on the problem itself.

The problem I had to confront is this: I have a ONE producer who reads from a big file (6GB); it reads line by line and converts every line to an object. It places the object in an ArrayBlockingQueue.

The consumers (few) take the object from the ArrayBlockingQueue and persist this to the database.

Now, obviously the producer is much faster than the consumer; it takes fractions of seconds to convert each line to an object but for the consumers it takes longer time.

So...if I wish to speedup this process by doing this: I created 2 classed 'ProducerThread' and 'ConsumerThread' they share the ArrayBlockingQueue. The Thread that coordinate between the 2 looks like this:

@Override
public void run()
{
    try{

        ArrayBlockingQueue<Ticket> queue = new ArrayBlockingQueue<Ticket>(40);
        ExecutorService threadPool = Executors.newFixedThreadPool(8);

        threadPool.execute(new SaleConsumerThread("NEW YORK", queue)); 
        threadPool.execute(new SaleConsumerThread("PARIS", queue));
        threadPool.execute(new SaleConsumerThread("TEL AVIV", queue));
        threadPool.execute(new SaleConsumerThread("HONG KONG", queue));
        threadPool.execute(new SaleConsumerThread("LONDON", queue));
        threadPool.execute(new SaleConsumerThread("BERLIN", queue));
        threadPool.execute(new SaleConsumerThread("AMSTERDAM", queue));

        Future producerStatus = threadPool.submit(new SaleProducerThread(progressBar, file, queue)); 
        producerStatus.get(); 
        threadPool.shutdown();   

    }catch(Exception exp)
    {
        exp.printStackTrace();
    }
}

My questions are:

  1. Would the design above actually use each thread concurrently? My computer is Two 2.4GHz Quad-Core.

  2. I'm not sure what does the Future and the .get() are for?

The result, by the way, are fast (consider the first version was sequential and it took 3hr) now it takes ~40 min (but maybe there're room for improvement).

Thanks for any pointer

2 Answers 2

2

I would look at how much time is spent waiting for IO and how much time is spent in CPU. I suspect your main bottleneck is the database and you need to look at how you can make the import more efficient. You could try batching your updates as this can improve throughput.

1
  • 1
    +1 signalling a thread to add one line of text to a DB is not going to be productive. I wuld be amazed if the performance did not get worse than when using just one thread. Batching up the lines and/or BULK INSERT and/or a stored procedure would be a better optimization, May 1, 2012 at 8:04
1

Answers:

  1. I'm not sure what you mean by "use each thread concurrently". But certainly all of the threads can execute concurrently. Your performance is going to depend on how many threads you have and how your data is partitioned. You can experiment with the number of threads to try to get better results, and instead of assigning threads per city, maybe you can use a record number and just assign each thread to the modulus of the record number. Suppose you have 10 threads, records 1, 11, 21, etc will go to thread one, 2, 22, etc to thread2. This way you will get the same number of transactions per thread so you will have full utilization of the threads until you are done.
  2. The Future is to allow the code to block when an event is done. The get method returns the result of the SaleProducerThread in this case.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.