So I'm working on a webscraping script in R and because the particular website I'm scraping doesn't take too kindly to people who scrape their data in large volumes, I have broken down my loop to handle only 10 links at a time. I still want to go through all the links, however, just in a random and slow manner.
productLink # A list of all the links that I'll be scraping
x<- length(productLink)
randomNum <- sample(1:x, 10)
library(rvest)
for(i in 1:10){
url <- productLink[randomNum[i]]
specs <- url %>%
html() %>%
html_nodes("h5") %>%
html_text()
specs
message<- "\n Temporarily unavailable\n "
if(specs == message){
print("Item unavailable")
}
else{
print("Item available")
}
}
Now the next time I run this for-loop I want to exclude all the random numbered indices that have already been tried in the previous running of the loop. That way this for loop runs through 10 new links each time until all the links have been used. There is another aspect to this that I'd like some input on. Since I can raise alarm flags by brute force scraping the particular company's website, is there any way I can slow down this loop so that it only runs every couple of minutes? I'm thinking of a timeout function or such where the code runs the for-loop once, waits a few minutes then runs it again (with new links each time as mentioned above). Any ideas?
Sys.sleep()
will make R sleep for a specified number of seconds. I would suggest you create a vector of random numbers, exclude all unique but one and iterate the loop through that.sample
but you may need an extra step in checking the uniqueness for sayrnorm
,runif
and kin.