As Julien Salinas wrote, it's because there's no user-agent
specified in your request headers
.
For example, the default requests
user-agent
is python-requests thus Google blocks a request because it knows that it's a bot and not a "real" user visit and you received a different HTML with different selectors and elements, and some sort of an error. User-agent
fakes user visit by adding this information into HTTP request headers.
Pass user-agent
into request headers
using requests
library:
headers = {
'User-agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582'
}
response = requests.get('YOUR_URL', headers=headers)
Code and example in the online IDE:
from bs4 import BeautifulSoup
import requests, lxml
headers = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
params = {
"q": "best potato recipes",
"hl": "en",
"gl": "us",
"tbm": "nws",
}
html = requests.get('https://www.google.com/search', headers=headers, params=params)
soup = BeautifulSoup(html.text, 'lxml')
for result in soup.select('.WlydOe'):
title = result.select_one('.nDgy9d').text
link = result['href']
print(title, link, sep='\n')
----------
'''
Call of Duty Vanguard (PS5) Beta Impressions – A Champion Hill To Die On
https://wccftech.com/call-of-duty-vanguard-ps5-beta-impressions-a-champion-hill-to-die-on/
Warzone players call for fan-favorite MW2 map to be added to Verdansk
https://charlieintel.com/warzone-players-call-for-fan-favorite-mw2-map-to-be-added-to-verdansk/114014/
'''
Alternatively, you can achieve the same thing by using Google News Results API from SerpApi. It's a paid API with a free plan.
The difference in your case is that you don't figure out why certain things don't work, bypass blocks from search engines since it's already done for the end-user, and you only need to iterate over structured JSON and get the data you want.
Code to integrate:
import os
from serpapi import GoogleSearch
params = {
"engine": "google",
"q": "Call of duty 360 no scope",
"tbm": "nws",
"api_key": os.getenv("API_KEY"),
}
search = GoogleSearch(params)
results = search.get_dict()
for news_result in results["news_results"]:
print(f"Title: {news_result['title']}\nLink: {news_result['link']}\n")
----------
'''
Call of Duty Vanguard (PS5) Beta Impressions – A Champion Hill To Die On
https://wccftech.com/call-of-duty-vanguard-ps5-beta-impressions-a-champion-hill-to-die-on/
Warzone players call for fan-favorite MW2 map to be added to Verdansk
https://charlieintel.com/warzone-players-call-for-fan-favorite-mw2-map-to-be-added-to-verdansk/114014/
'''
P.S - I wrote a bit more in-depth blog post about how to scrape Google News Results.
Disclaimer, I work for SerpApi.