0

I am currently building a web crawler and want to add a delay on request so I don't hit web servers to hard.

What is the most appropriate way to add this delay?

Also, what is an appropriate amount of delay for a crawler? I was thinking a .5-1 second.

Thanks

1
  • @CuongLe How is that related to the question? The rules are not absolute, there are questions which don't need any code.
    – user2032433
    Mar 27, 2013 at 16:38

1 Answer 1

7

You can use

Thread.Sleep(TimeSpan.FromSeconds(1))

Search engines may use larger delays of 10-15 or even 60 seconds. And polite crawlers respect the "Crawl-delay" that site provided in robots.txt. For details, take a look at http://en.wikipedia.org/wiki/Web_crawler#Politeness_policy

1
  • Short and sweet. I like it :)
    – feralin
    Mar 27, 2013 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.