0

So I'm working on a webscraping script in R and because the particular website I'm scraping doesn't take too kindly to people who scrape their data in large volumes, I have broken down my loop to handle only 10 links at a time. I still want to go through all the links, however, just in a random and slow manner.

productLink # A list of all the links that I'll be scraping
 x<- length(productLink)  
 randomNum <- sample(1:x, 10)
library(rvest)

for(i in 1:10){
url <- productLink[randomNum[i]]
specs <- url %>%
  html() %>%
  html_nodes("h5") %>%
  html_text()
specs

message<- "\n                Temporarily unavailable\n            "

if(specs == message){
  print("Item unavailable")
}
else{
  print("Item available")
}


}

Now the next time I run this for-loop I want to exclude all the random numbered indices that have already been tried in the previous running of the loop. That way this for loop runs through 10 new links each time until all the links have been used. There is another aspect to this that I'd like some input on. Since I can raise alarm flags by brute force scraping the particular company's website, is there any way I can slow down this loop so that it only runs every couple of minutes? I'm thinking of a timeout function or such where the code runs the for-loop once, waits a few minutes then runs it again (with new links each time as mentioned above). Any ideas?

3
  • 2
    Sys.sleep() will make R sleep for a specified number of seconds. I would suggest you create a vector of random numbers, exclude all unique but one and iterate the loop through that. Commented Jun 23, 2015 at 19:07
  • @RomanLuštrik I'm not quite sure I understand. An example would help. It seems like a simple problem, I just can't formulate the syntax. I was originally thinking of creating a dummy array that stores all the random numbers generated and then maybe trap the random number generator in a while loop where the code will force it to print out numbers that don't appear in the dummy array. Rinse and repeat. Seems pretty inefficient though. I'd like to know if you or anyone else have a more elegant solution. Commented Jun 23, 2015 at 19:15
  • It would appear @Vlo beat me to it. :) The solution will work for sample but you may need an extra step in checking the uniqueness for say rnorm, runif and kin. Commented Jun 24, 2015 at 8:01

1 Answer 1

1

Use something like this. Loop over all the product index randomly.

for (i in sample(1:x)){
  <Your code here>
  # Sleep for 120 seconds
  Sys.sleep(120)
}

And if you want to do 10 at a time. Sleep for 120 seconds every 10 executions.

n = 1
for (i in sample(1:x)){
  # Sleep for 120 seconds every 10 runs
  if (n == 10) {Sys.sleep(120); n = 0}
  n = n+1
  <Your code here>
}
7
  • Thanks. The sys.sleep works. Do you have ideas for the first part of my query? Commented Jun 23, 2015 at 19:11
  • What is the first part of your query? You can also use a random sleep like runif(1, 100, 140)
    – Vlo
    Commented Jun 23, 2015 at 19:13
  • Excluding random numbers that have already been generated. I want to run the for loop on 10 unique links each time until I run out of all the links (but then again if the number of links is not a multiple of 10 I don't know how I'd handle that). Commented Jun 23, 2015 at 19:17
  • When you loop over sample(1:x), you will not be looping over duplicates. i will always be unique.
    – Vlo
    Commented Jun 23, 2015 at 19:20
  • Ah I see. You did address it but maybe I didn't convey my idea properly. The i still increments sequentially and since the links themselves only differ by a number or two, the scrape is still happening in a recognizable pattern. All of that is to say, I would like to access the links at random indices (randomNum[i]) and run my code on those. No obvious patterns that way. Commented Jun 23, 2015 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.