2

I have a kind of tricky problem involving multi-threading. What I do is that I use a thread pool (ExecutorService) that is tasked with opening connections and putting them in a LinkedBlockingQueue.

So far I have used:

//run method in "getter threads"
public void run() {

    try {

    URL url = new URL(url_s); //url_s is given as a constructor argument

    //if I am correct then url.openStream will wait until we have the content
    InputStream stream = url.openStream();

    Request req = new Request(); //a class with two variables:
    req.html_stream = new InputSource(stream);
    req.source = stream;

    //this is a class variable (LinkedBlockingQueue<Request>)
    blocking_queue.put(req);

    } catch  (Exception ex) {
    logger.info("Getter thread died from an exeption",ex);
    return;
    }
}

I then have consumer thread (java.lang.Thread) that takes these InputSources and InputStreams and does:

public void run() {
   while(running) {
        try {
            logger.info("waiting for data to eat");
            Request req = blocking_queue.take();
            if(req.html_stream != null)
            eat_data(req);
        } catch (Exception ex) {
            logger.error(ex);
            return;
        }
   }
}

Where eat_data calls an external library that takes InputSource. The library uses a singleton instance to do the processing so I cant put this step in the "getter" threads.

When I tested this code for small amounts of data it worked fine, but when I supplied it with several thousands of URLs I started to have real problems. Its not easy to find out exactly what is wrong, but I suspect that the connections time out before the consumer thread get to them, sometimes even causing deadlock.

I implemented it this way because it was so easy to go from url.openStream() to InputSource but I realize that I really must store the data locally for this to work.

How do I get from url.openStream() to some object I can store in my LinkedBlockingQueue (all data in memory) that I can later turn into an InputSoruce when my consumer thread has time to process it?

3
  • 3
    Personally, I wouldn't open the connections until you them. I'd pass the URL: to the Request object and have it crack open the connection as it needs it. It also means that you can control the closing of the connection (you opened it, you're responsible for closing it). IMHO Oct 1, 2012 at 3:05
  • @MadProgrammer But would this remove the need to wait for a reply from the connection? That is the reason I do it multithreaded. The "getter" threads are used so that the processor thread does not need to wait for IO (e.g. the http response). Oct 1, 2012 at 4:08
  • No, the reason for threading like this is to reduce the number of active connections (and resources) that this would otherwise consume. Only open the connection when you need it, close it when you done. On Windows, you'd run into problems pretty quick as it has a pretty tight ceiling on the number of active connections you can have open and as you've pointed out, your running into timeouts anyway. I'd personally use the Request to open, read, store and close the content from the URL. I would then place it on a queue for a processor thread to come along and process when it is free to do so... Oct 1, 2012 at 4:13

1 Answer 1

2

You can copy the contents of the URL into a ByteArrayOutputStream and close the URL Stream. Then store the ByteArrayInputStream in the queue.

Pseudo Code :

InputStream in = null;
try {
    in = url.openStream();
    ByteArrayOutputStream buffer = new ByteArrayOutputStream();
    IOUtils.copy(in, buffer);

    ByteArrayInputStream bin = new ByteArrayInputStream(buffer.toByteArray());
    queue.put(bin);
}

References :

  1. java.io.ByteArrayInputStream
  2. java.io.ByteArrayOutputStream
  3. org.apache.commons.io.IOUtils.IOUtils

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.