I am attempting to scrape a Dynamic website Morningstar.com via XHR requests.
The exact site I am scraping is: http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY®ion=USA&culture=en_US
What I am trying to scrape is the Quarterly performance number (1-month). The result should be 0.64 as of today.
try(res <- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
query = list(
t="SPY",
region="usa",
culture="en-US"
)
))
tryCatch(x <- content(res) %>%
html_nodes(xpath = '//*[@id="tab-quar-end-content"]/table/tbody/tr[1]/td[1]') %>%
html_text() %>%
trimws() %>%
as.numeric()
, error = function(e) x <-NA)
However, the result is numeric(0)
Any idea what I am doing wrong?
Sody
Update:
I was able to get the html data with the following code:
try(res <- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
query = list(
t = "SPY",
region = "usa",
culture = "en-US",
ops = "clear",
s = "0P0000J533",
ndec = "2",
ep = "true",
align = "q",
annlz = "true",
comparisonRemove = "false"
)
))
But I am still having problems pointing to the data using either the CSS selector or the xpath with rvest.
What do you guys use to find those data points? is SelectorGadget still the go to?
Cheers, Aaron