I hope you can help me? I'm trying to pull the "results" table from this URL: https://www.moneysupermarket.com/credit-cards/search/results/?goal=CC_ALLCARDS
I've been using RVEST, referencing this blog post: https://www.r-bloggers.com/a-text-mining-function-for-websites/
This worked when I adapted the code to use on uSwitch, but I think the MSM site is more complicated.
Here's my uSwitch code
## Load the libraries
library(tidyverse) # General purpose data wrangling
library(rvest) # Parsing of html/xml files
library(stringr) # String manipulation
library(rebus) # Verbose regular expressions
library(lubridate) # Eases datetime manipulation
################################################################################
## BUILD THE BT TABLE
## Define the page to scrape
bt.url <- 'https://www.uswitch.com/credit-cards/credit-card-balance-transfers/'
## Get the top ten brands
bt.brand <- read_html(bt.url) %>%
html_nodes(".us-ct-row__title--mobile strong") %>%
html_text()
bt.primary.offer <- read_html(bt.url) %>%
html_nodes(".us-ct-row__col--highlight") %>%
html_text()
# Get the offer details
bt.offer.details <- read_html(bt.url) %>%
html_nodes(".us-ct-row__key-details-col:nth-child(1)") %>%
html_text()
bt.clean.offer.details <- bt.offer.details %>%
str_replace("^Card details*", "")
## Get the cost to the customer
bt.cost.to.cust <- read_html(bt.url) %>%
html_nodes(".us-ct-row__name--fee span") %>%
html_text()
## Create a list of even numbers
even.seq <- seq(2, 20, 2)
## Extract even obs because the £ sign and the value are split into separate rows
bt.cost.to.cust <- bt.cost.to.cust[even.seq]
## Get the APR
bt.apr <- read_html(bt.url) %>%
html_nodes(".us-ct-row__col--highlight+ .us-ct-row__col--stretch .us-ct-row__name span") %>%
html_text()
## Get the offer duration
# .us-ct-row__col--highlight strong
bt.offer.duration <- read_html(bt.url) %>%
html_nodes(".us-ct-row__col--highlight strong") %>%
html_text()
## Stitch it all together
bt.table <- as.matrix(cbind(bt.brand, bt.primary.offer, bt.offer.duration, bt.cost.to.cust,
bt.apr, bt.clean.offer.details))
This all works as I want, I just would like to be able to replicate for the above web page, if possible?
Failing that, I found a post that suggested checking the DOC or XHR section in the network tab of the console.
Scrape data from flash page using rvest
I did that and I can see the results table in the console, under Network > XHR > Results > Preview, but I can't pull it back in R. This method would be best as it gives me more info than is rendered on the page, but after about a week of trial and error, I'll take anything!
results
table on this page? I do not see any table on this page? – mnm Apr 12 '19 at 7:05