2

I am adding functionality that scrapes an XML page from a source that requires the use of an HTTPS connection with authentication. I am trying to use Ryan Bates' Railscast #190 solution but I'm running into a 401 Authentication error.

Here is my test Ruby script:

require 'rubygems'
require 'nokogiri'
require 'open-uri'

url = "https://biblesearch.americanbible.org/passages.xml?q[]=john+3:1-5&version=KJV"
doc = Nokogiri::XML(open(url, :http_basic_authentication => ['username' ,'password']))
puts doc.xpath("//text_preview")

Here is the output of the console after I run my script:

/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:799:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:799:in `block in connect'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:799:in `connect'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:744:in `start'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:306:in `open_http'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:775:in `buffer_open'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:203:in `block in open_loop'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:201:in `catch'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:201:in `open_loop'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:146:in `open_uri'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:677:in `open'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:33:in `open'
from scrape.rb:6:in `<main>'

In my research, I saw one post in which it was suggested that in 1.9.3 the following option could be used:

doc = Nokogiri::XML(open(url, :http_basic_authentication => ['username' ,'password'], :ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE))

However, this did not work either. I would appreciate some insight into addressing this challenge.

1
  • Be careful using :ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE. That disables the SSL certificate check, which is there to ensure the integrity of the connection, helping avoid man-in-the-middle attacks. Commented Dec 7, 2012 at 16:15

3 Answers 3

4

The given URL will be redirected to /v1/KJV/passages.xml?q[]=john+3%3A1-5 with HTTP status code 302 Found. OpenURI understands the redirection, but automatically deletes authentication header (maybe) for security reason. (*)

If you access "http://biblesearch.americanbible.org/v1/KJV/passages.xml?q[]=john+3%3A1-5" directly, you will get the expected result. :-)

(*) You can find in open-uri.rb:

if redirect
  ### snip ###
  if options.include? :http_basic_authentication
    # send authentication only for the URI directly specified.
    options = options.dup
    options.delete :http_basic_authentication
  end
3
  • If I remember right, security headers have to be deleted when a redirection is from a secure to a non-secure URL. It was a hole or exploit that was found several years ago and fixed in clients, not just OpenURI. It should be automatically handled by OpenURI. Commented Dec 7, 2012 at 16:12
  • Basic authentication header will be removed even if redirected from HTTPS to HTTPS as of ruby 1.9.3p286 (newest stable version). Check open-uri.rb for detail.
    – yasu
    Commented Dec 7, 2012 at 16:32
  • Thank you, @HIRATA Yasuyuki! I see now that read only does not require authentication, only RESTful writing does. Pointing to the URL you specified works fine now. I appreciate the help. Commented Dec 7, 2012 at 18:01
2

You can do this and it should work too:

open(url, :http_basic_authentication => [user, pass] ) doc = Nokogiri::HTML(open(url, :http_basic_authentication => [user, pass] ))

You can then parse the doc anyway you want. By passing the http_basic_authentication in the header again in the second request, you will make up for the deleted header in the first request. hope this works for you.

http://http-basic-authentication-nokogiri.blogspot.com/2014/08/http-basic-authentication-using-nokogiri.html

0
1

You say you need to use HTTPS, but you're using the HTTP protocol:

url = "http://biblesearch...."

OpenURI understands both HTTP and HTTPS. If you want to connect using HTTPS, change the protocol in the URL to HTTPS, then make the connection:

url = "https://biblesearch...."
1
  • Yes, sorry. I updated the code and the output. It appears that there is now an issue verifying the certificate. Researching... Commented Dec 7, 2012 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.