3

As the following code shows, html in rvest package uses htmlParse from XML package. .

html
function (x, ..., encoding = NULL) 
{
    parse(x, XML::htmlParse, ..., encoding = encoding)
}
<environment: namespace:rvest>

htmlParse
function (file, ignoreBlanks = TRUE, handlers = NULL, replaceEntities = FALSE, 
    asText = FALSE, trim = TRUE, validate = FALSE, getDTD = TRUE, 
    isURL = FALSE, asTree = FALSE, addAttributeNamespaces = FALSE, 
    useInternalNodes = TRUE, isSchema = FALSE, fullNamespaceInfo = FALSE, 
    encoding = character(), useDotNames = length(grep("^\\.", 
        names(handlers))) > 0, xinclude = TRUE, addFinalizer = TRUE, 
    error = htmlErrorHandler, isHTML = TRUE, options = integer(), 
    parentFirst = FALSE) 
.....

So, for the following url:

 myurl<-"http://www.nepalstock.com.np/"
parse_XML<-htmlParse(myurl) #runs without error
parse_rvest<-html(myurl) # throws out the Internal Sever error
Error in parse.response(r, parser, encoding = encoding) : 
  server error: (500) Internal Server Error

Any idea?

1 Answer 1

3

Reset the default user-agent from the underlying httr::GET request, then it works:

library(httr)
library(rvest)
parse_rvest <- html(myurl, add_headers("User-Agent" = "myagent"))

or

parse_rvest <- html(myurl, user_agent("myagent"))

Note that for debugging purposes, you can add verbose() to html(...).

Add:

Using the new rvest / xml2 / curl combo, it should look like:

library(xml2)
library(rvest)
library(curl)
parse_rvest <- curl(myurl, handle = new_handle("useragent" = "myua")) %>%
               read_html()
6
  • Thanks, but it still throws an error: Error in open.connection(x, "rb") : HTTP error 500..
    – user227710
    Jun 17, 2015 at 18:48
  • Not here (Windows 7 x64, R version 3.1.3, rvest_0.2.0, httr_0.6.1, RCurl_1.95-4.6, XML_3.98-1.1)
    – lukeA
    Jun 17, 2015 at 18:51
  • Mine is (windows 8 x64 R version 3.2, rvest ‘0.2.0.9000’ httr 0.6.1 RCurl: 1.95.4.6’, XML,‘3.98.1.2’). I also noticed that html is deprecated now.
    – user227710
    Jun 17, 2015 at 18:57
  • 2
    A solution that worked for me is this (using read_html): read_html(GET(myurl,user_agent("myagent"))).
    – user227710
    Jun 17, 2015 at 19:30
  • Either this way, or see my edit. curl may become the new RCurl / httr.
    – lukeA
    Jun 17, 2015 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.