1

I'm building a web crawler. Having read this I understand that DNS resolution is slow and so we should separate out the DNS Resolver.

So say that you have String urlString http://google.com you can then convert that into an ip by doing

URL url = new URL(urlString)
InetAddress ip = InetAddress.getByName(url.getHost());

But then how do you download the actual website itself?

With the url, we could just dow something like this:

String htmlDocumentString = new Scanner(new url.openStream(), "UTF-8").useDelimiter("\\A").next();

But if we want to used the resolved IP, do we have to manually reconstruct the URL with an ip? There is no url.setHost() method, it just seems kinda messy?

2
  • 1
    "Separate out the DNS resolver?" I don't understand the point. Yes, DNS resolution adds overhead, but I don't see how "separating it out" makes your web crawler any faster.
    – Matt Ball
    Apr 12, 2013 at 2:51
  • 1
    The simpliest way would be to have a local DNS cache since when you do URL content loading, many servers relies on VirtualHost with domains. If you request the IP directly you won't have the right content. Apr 12, 2013 at 3:16

2 Answers 2

0

Reading from URL is simple :

public class URLReader {
public static void main(String[] args) throws Exception {

    URL oracle = new URL("http://www.oracle.com/");
    BufferedReader in = new BufferedReader(
    new InputStreamReader(oracle.openStream()));

    String inputLine;
    while ((inputLine = in.readLine()) != null)
        System.out.println(inputLine);
    in.close();
}

Taken from : http://docs.oracle.com/javase/tutorial/networking/urls/readingURL.html

0

Try this instead:

  URL oracle = new URL("http://www.oracle.com/");
  URLConnection urlc = oracle.openConnection();
  urlc.setDoInput(true);
  urlc.setRequestProperty("Accept", "text/text");
  InputStream inputStream = urlc.getInputStream();
  String myString = IOUtils.toString(inputStream, "UTF-8");

... using IOUtils from Apache Commons above:

http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString(java.io.InputStream,%20java.lang.String)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.