The response may be empty because the requests
request may be blocked, since the default user-agent
in the requests
library is python-requests
to tell the website that it is a bot or script that is sending the request. Check what user agent you have.
An additional step besides providing browser user-agent could be to rotate user-agent
, for example, to switch between PC, mobile, and tablet, as well as between browsers e.g. Chrome, Firefox, Safari, Edge and so on.
It is also possible to fetch all results from all pages using pagination, the solution to this would be to use an infinite while
loop and test for something (button, element) that will cause it to exit.
In our case, this is the presence of a button on the page (.pagination__next
selector).
Check code in online IDE.
from bs4 import BeautifulSoup
import requests, lxml
import pandas as pd
# https://requests.readthedocs.io/en/latest/user/quickstart/#custom-headers
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
}
params = {
'_nkw': 'oakley+sunglasses', # search query
'LH_Sold': '1', # shows sold items
'_pgn': 1 # page number
}
data = []
while True:
page = requests.get('https://www.ebay.com/sch/i.html', params=params, headers=headers, timeout=30)
soup = BeautifulSoup(page.text, 'lxml')
print(f"Extracting page: {params['_pgn']}")
print("-" * 10)
for products in soup.select(".s-item__pl-on-bottom"):
title = products.select_one(".s-item__title span").text
price = products.select_one(".s-item__price").text
try:
sold_date = products.select_one(".s-item__title--tagblock .POSITIVE").text
except:
sold_date = None
data.append({
"title" : title,
"price" : price,
"sold_date": sold_date
})
if soup.select_one(".pagination__next"):
params['_pgn'] += 1
else:
break
# save to CSV (install, import pandas as pd)
pd.DataFrame(data=data).to_csv("ebay_products.csv", index=False)
Output:
file is created: "ebay_products.csv"
As an alternative, you can use Ebay Organic Results API from SerpApi. It's a paid API with a free plan that handles blocks and parsing on their backend.
Example code:
from serpapi import EbaySearch
import os
import pandas as pd
params = {
"api_key": os.getenv("API_KEY"), # serpapi key, https://serpapi.com/manage-api-key
"engine": "ebay", # search engine
"ebay_domain": "ebay.com", # ebay domain
"_nkw": "oakley+sunglasses", # search query
"LH_Sold": "1" # shows sold items
}
search = EbaySearch(params) # where data extraction happens
page_num = 0
data = []
while True:
results = search.get_dict() # JSON -> Python dict
if "error" in results:
print(results["error"])
break
for organic_result in results.get("organic_results", []):
title = organic_result.get("title")
price = organic_result.get("price")
data.append({
"title" : title,
"price" : price
})
page_num += 1
print(page_num)
if "next" in results.get("pagination", {}):
params['_pgn'] += 1
else:
break
pd.DataFrame(data=data).to_csv("ebay_products.csv", index=False)
Output:
file is created: "ebay_products.csv"
There's a 13 ways to scrape any public data from any website blog post if you want to know more about website scraping.