1

As part of a project, I am trying to scrape the complete reviews from Google + (in previous attempts on other websites, my reviews were truncated by a More which hides the full review unless you click on it).

I have chosen the package rvest for this. However, I do not seem to be getting the results I want.

Here are my steps

library(rvest)
library(xml2)
library(RSelenium)

queens <- read_html("https://www.google.co.uk/search?q=queen%27s+hospital+romford&oq=queen%27s+hospitql+&aqs=chrome.1.69i57j0l5.5843j0j4&sourceid=chrome&ie=UTF-8#lrd=0x47d8a4ce4aaaba81:0xf1185c71ae14d00,1,,,")

#Here I use the selectorgadget tool to identify the user review part that I wish to scrape

reviews=queens %>%
html_nodes(".review-snippet") %>%
html_text()

However this doesn't seem to be working. I do not get any output here.

I am quite new to this package and web scraping, so any inputs on this would be greatly appreciated.

1
  • 2
    This is a violation of Google's Terms of Service.
    – hrbrmstr
    May 5, 2018 at 1:31

1 Answer 1

4

Here is the workflow with RSelenium and rvest:
1. Scroll down any times to get as many contents as you want, remember to pause once a while to let the contents load.
2. Click on all "click on more" buttons and get full reviews.
3. Get pagesource and use rvest to get all reveiws in a list

What you want to scrape is not static, so you need the help of RSelenium. This should work:

library(rvest)
library(xml2)
library(RSelenium)

rmDr=rsDriver(browser=c("chrome"), chromever="73.0.3683.68")
myclient= rmDr$client
myclient$navigate("https://www.google.co.uk/search?q=queen%27s+hospital+romford&oq=queen%27s+hospitql+&aqs=chrome.1.69i57j0l5.5843j0j4&sourceid=chrome&ie=UTF-8#lrd=0x47d8a4ce4aaaba81:0xf1185c71ae14d00,1,,,")
#click on the snippet to switch focus----------
webEle <- myclient$findElement(using = "css",value = ".review-snippet")
webEle$clickElement()
#simulate scroll down for several times-------------
scroll_down_times=20
for(i in 1 :scroll_down_times){
    webEle$sendKeysToActiveElement(sendKeys = list(key="page_down"))
    #the content needs time to load,wait 1 second every 5 scroll downs
    if(i%%5==0){
        Sys.sleep(1)
    }
}
#loop and simulate clicking on all "click on more" elements-------------
webEles <- myclient$findElements(using = "css",value = ".review-more-link")
for(webEle in webEles){
    tryCatch(webEle$clickElement(),error=function(e){print(e)}) # trycatch to prevent any error from stopping the loop
}
pagesource= myclient$getPageSource()[[1]]
#this should get you the full review, including translation and original text-------------
reviews=read_html(pagesource) %>%
    html_nodes(".review-full-text") %>%
    html_text()

#number of stars
stars <- read_html(pagesource) %>%
    html_node(".review-dialog-list") %>%
    html_nodes("g-review-stars > span") %>%
    html_attr("aria-label")


#time posted
post_time <- read_html(pagesource) %>%
    html_node(".review-dialog-list") %>%
    html_nodes(".dehysf") %>%
    html_text()
16
  • On running rmDr=rsDriver(browser = "chrome") I seem to be getting an error which says 'Undefined error in httr call. httr output: Failed to connect to localhost port 4567: Connection refused'
    – Varun
    May 5, 2018 at 13:36
  • It happens sometimes, port 4567L may have been taken by other applications, you can try other ports like 4444L or 4445L.
    – Yifu Yan
    May 5, 2018 at 16:56
  • Now on running pagesource= myclient$getPageSource()[[1]] I get another error message that says chrome not reachable\n (Session info: chrome=xxxxxx). Any idea how to get around this error?
    – Varun
    May 6, 2018 at 9:56
  • I don't have this error. What version of R are you using right now? Recently, Rselenium's dependent packages were no longer available due to the 3.5 update. If you are using the newest version. You can swith to older one first.
    – Yifu Yan
    May 7, 2018 at 8:14
  • 1
    Hello Yifu Yan, I have recently posted a question, based on the question that you answered here. I was hoping you could help. Here is the link to the question. Thanks stackoverflow.com/questions/50680985/…
    – Varun
    Jun 4, 2018 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.