1

I'm trying to scrape google headlines for a given keyword (eg. Blackrock) for a given period (eg. 7-jan-2012 to 14-jan-2012). I'm trying to do this by constructing the url and then using urllib2 as shown in the code below. if I put the constructed url in a browser, it gives me the correct result. however, if I use it through python, I get news results for the right keyword but for the current period. here'e the code. Can someone tell me what I'm doing wrong and how I can correct it?

import urllib
import urllib2
import json
from bs4 import BeautifulSoup
import requests

url = 'https://www.google.com/search?q=Blackrock&hl=en&gl=uk&authuser=0&source=lnt&tbs=cdr%3A1%2Ccd_min%3A7%2F1%2F2012%2Ccd_max%3A14%2F1%2F2012&tbm=nws'


req = urllib2.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3 Gecko/2008092417 Firefox/3.0.3')
response = urllib2.urlopen(req)


html = response.read()
soup = BeautifulSoup(html)

text = soup.text

start = text.index('000 results')+11
end = text.index('NextThe selection')
text = text[start:end]
print text
3
  • You say you're using requests, but you're using urllib instead Dec 8, 2014 at 14:49
  • you're right Vincent. please disregard my comment about using requests. If there's a way to use it, i'll be happy to know of that too. I have now changed it to urllib2. Dec 8, 2014 at 15:00
  • I wrote a script that parses Google News Title, Link and supports actual pagination. Apr 15, 2021 at 6:20

1 Answer 1

1

The problem is with your user-agent, it works for me with:

req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36')

You are using a user-agent for Firefox 3, which is about 6 years old.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.