Questions tagged [scrapy]
Scrapy is a fast open-source high-level screen scraping and web crawling framework written in Python used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
17,660
questions
0
votes
0
answers
15
views
How to get all the categories/subcategories from a dynamic website?
I need to extract all the products from a site (https://www.hiperlibertad.com.ar/) that has multiple branches/stores.
The main problem I have now is that I need to find a way to adapt the code to be ...
-2
votes
0
answers
21
views
Scrapy scrape hidden keywords from IMDB keywords page
I have scrapy code that downloads the list of all imdb titles and related keywords based on provided keyword.
This code was working perfectly and downloading all keywords until few days back when imdb ...
0
votes
0
answers
7
views
TCP Connection Timed Out Scrapy Error as soon as downloading files starts
So I'm using scrappy to extract data from a website.
Here's what I'm doing...
Navigating to subpages from the main pages.
--- from these subpages, I'm extracting data and downloading attachments t ...
-1
votes
1
answer
31
views
Webscraping have incorrect result
When I scrape website it was total correct but have a lot of blank and some of incorrect data.
import scrapy
class AudibleSpider(scrapy.Spider):
name = 'audible'
allowed_domains = ['www....
0
votes
1
answer
17
views
How can I navigate and extract restaurant details from zaubee.com when the href attribute is set to "#" for each restaurant link?
How can I scrape the zaubee.com website to extract business details from each restaurant's page when the href attribute is set to "#" in scrapy??
I'm presently working on a web scraping ...
0
votes
0
answers
23
views
Why Scrapy spider is closing after first request to start_urls?
I know this question was asked multiple times but the solutions to the problems are always different. I have the following scrapy crawler:
from scrapy.spiders import CrawlSpider, Rule
from scrapy....
1
vote
1
answer
28
views
How to run playwright asynchronously in Scrapy Python?
My example code works synchronously without mistakes. It parses all the start urls and prints needed text to the output.
How to make this code work asynchronously?
(I need to run playwright calling ...
-1
votes
0
answers
13
views
Get dynamic classes that end in number using scrapy [duplicate]
I want to get all elements whose classes are like panel-x, where the X is always a number
Example:
<div class="content">
<div class="panel panel-default panel-1">&...
0
votes
1
answer
11
views
How to solve 'Spider_error_processing_headers' problem while using Scrapy's CrawlSpider and LinkExtractor?
I got a massage lik ERROR: Spider error processing and line 276, in aiter_errback
yield await it.anext() in my terminal, and my code given below
Can anyone tell me where is the problem.
import scrapy
...
0
votes
0
answers
28
views
Scraping Zillow with Scrapy Playwright only retrieves 8 listings instead of 40. How can I fix this?
i use scrapy playwright to scrape zillow but instead of getting all 40 listings i just get 8 i made sure i used the css selector for all 40 listings please help
import scrapy
from ..items import ...
-1
votes
0
answers
34
views
I am trying to install Scrapy; however, this is the error I am experiencing : Failed building wheel for lxml . Please assist
Experiencing error Failed building wheel for lxml
src/lxml/etree.c:96:10: fatal error: 'Python.h' file not found
#include "Python.h"
^~~~~~~~~~
1 error generated.
ERROR: Could not build ...
0
votes
1
answer
34
views
Issues with Python Scrapy script with multiple start urls
I have a scraper written in Scrapy. It works perfectly fine as long as there is just one url in start_urls. As soon as I add 2 or more start urls, the behavior is erratic. The number of pages it ...
0
votes
1
answer
38
views
How to send mail with scraped data?
How to send email with specific scraped data after spider ends in Scrapy.
I'm trying to send some item data with multiple lines after the end of a spider, but i'm only receiving one line. I think i ...
0
votes
0
answers
38
views
Scrapy only scraping a few items
I am practicing a bit with scrapy on a website, with 60 items maximum per page. For each item, I need to extract more data, so I go to each items' url. However, I am only able to scrape a fraction (...
-1
votes
2
answers
37
views
How to get the initial URL in Scrapy
I need to get the first URL which Scrapy spider currently processes to map result to it. More spicifically. If an URL for scraping is abc.de and it's redirected to https://www.abc.de/en, I want to get ...
0
votes
0
answers
21
views
Run crawling function sequentially on multiple URLs with Scrapy
How can I run a crawler on multiple URLs sequentially? we already know that we can pass multiple URLs & allowed_domains at the same time but I need a function to handle this as follows:
import ...
1
vote
2
answers
40
views
How do I scrape multiple pages with different alphabetical labels using scrapy
I am working on a web scraping project using Scrapy and I am facing an issue with iterating through multiple pages with alphabet labels.
My goal is to scrape medicine data from the website https://www....
0
votes
1
answer
45
views
UnicodeEncodeError: 'ascii' codec can't encode character '\xe3' in position 7: ordinal not in range(128) [duplicate]
Some days ago, accepted this answer to a question of mine as correct, but after a while I noticed that in some URLs, I got the following error:
2023-05-29 19:22:20 [scrapy.core.scraper] ERROR: Spider ...
-1
votes
1
answer
22
views
Scrapy: How can I execute a CMD command?
How can I execute a cmd line from within a Scrapy script? I'm using Scrapy in a Microsoft Windows environment.
Normally, from within native Python, I would do something like this:
import os
os....
0
votes
0
answers
30
views
How to bypass the Error 403 with scrapy and selenium?
I have the example code write in scrapy with selenium.
Do you know how to bypass the 403 error?
I changed the setting
DOWNLOAD_DELAY = 3 # change delay time
DOWNLOADER_MIDDLEWARES = {
"testfx....
0
votes
0
answers
20
views
Retrieving multi-category data from REST Data API using Scrapy
With help from the developers of this community, I was able to scrape data from an online supermarket. This page shows the food category of the website. However, I have been unable to get sub-category,...
0
votes
2
answers
25
views
Scrapy not saving output to jsonline
I put together a crawler to extract URLs found on a website and save them to a jsonline file:
import scrapy
from scrapy.crawler import CrawlerProcess
class MySpider(scrapy.Spider):
name = '...
0
votes
0
answers
18
views
Error while executing lua with scrapy_splash which fails with 400 and says "Required argument is missing: lua_source"
Here's my code:
def start_requests(self):
headers = {
"method": "GET",
"path": "/",
"scheme&...
0
votes
1
answer
31
views
Need some scraping guidance regarding missing pagination elements [closed]
Noob web scraper here. I built a spider using Scrapy and Playwright to scrape auto ads for the results of a parameterized search URL on autotrader.com and it's working great to grab data from the ...
0
votes
1
answer
34
views
Using FormRequest to extract data via HTTP POST
Using FormRequest to extract data via HTTP POST
hey guys
I will crawl the details of all the products of the site https://bitsclassic.com/fa/ with scrapy
To get the url of the products, I have to ...
1
vote
1
answer
20
views
Scrapy not giving output but works in shell
So i'm new to web scraping and i've been trying to scrape this website but the problem i'm encountering is that when i run my code in shell it runs but it doesn't when i run it as a whole scrapy code.
...
1
vote
1
answer
21
views
How can I extract the amount written after the 'span' tag under the 'div' tag using css-selectors in Scrapy?
I want to get the amount(182.78) written after span tag under div tag , but I am getting only the "MRP" element content as a string, I only want to extract the amount written. By the way I'm ...
0
votes
0
answers
18
views
How to Save Scrapy Response Output to minio Pod on a Cluster
I saw a similar question in:
Scrapy files pipeline for minio storage
But I did not see a successful response. I am using scrapy to scrape various websites, and I can save the output to a .csv file in ...
-1
votes
0
answers
56
views
Yet another problem with using scrapy-splash to click the button on the website
I tried many different solutions how to make splash click the button and I am unable to understand what I am doing wrong. I tried getting the element by its type (button), tried by its class, tried ...
0
votes
1
answer
58
views
Download embedded PDF on website using Selenium/Scrapy/Python
There are some pages like this that I want to download the embedded PDF file, but researching and testing various proposed solutions on SO and other sources, I couldn't find a correct solution. When I ...
0
votes
0
answers
22
views
How to pass a variable from init to parse_item in Scrapy CrawlSpider?
Maybe it's the wrong approach alltogether. I'd be happy about an alternative approach as well, if you have an idea.
Connect to my DB, get the next batch of seed URLs (as a dict, with some other info)
...
0
votes
0
answers
24
views
Getting 'Invalid request method' error when trying to pass Scrapy data to Django using Python APIs
I have a problem, I created a scrapy project and a django project with an application, I registered the application, set the urls, set the views, and also wrote the spider, but when I go to the ...
-1
votes
1
answer
22
views
How to use an API-provided proxy with rotating proxy middleware in Scrapy?
I'm trying to scrape websites using Scrapy and want to use an API-provided proxy with the rotating proxy middleware. The proxy is provided by an API endpoint that returns a new proxy IP with each ...
1
vote
1
answer
22
views
Scrapy: select last decendant node?
I have a dict with selectors which I use to get data:
for key, selector in selectors.items():
data[key] = response.css(selector).get().strip()
One of the selectors is span::text, but sometimes ...
-2
votes
0
answers
25
views
How can I prevent duplicate entries in MongoDB using Scrapy and Python?
a solution to avoid duplicate entries in momgodb using scrapy python
i'm trying to avoid scrapy mongo db duplicate records, my code
scrapy items function some of the items
def parse_article(self, ...
0
votes
0
answers
56
views
scrapy-playwright getting null values
I am trying to scrape this page https://www.sephora.com/shop/skincare, but I am getting this bug:
{'title': None}
2023-05-22 13:56:59 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www....
-1
votes
0
answers
24
views
Why isn't Scrapy scraping anything when using FormRequest to log into a website?
I am using Scrapy FormRequest to login in this website. However, the crawler didn't scrape anything even though it didn't show any error in the result.
This is part of my code:
class ...
0
votes
0
answers
16
views
New to the Scrapy library, how do I troubleshoot an empty JSONL file while web scraping?
Spider not doing its job scrapy
i tried to run the folllowing scrapy spider to scrap and parse a website and put the results in a jsonl, which stays empty, im new to the libary
import scrapy
from ...
0
votes
0
answers
12
views
How can I return scraped information from my scrapy spider to my Flask server?
I want to call my scraper from my flask server and have the value returned to the server. At the moment I am using subprocess to call the scraper, and a POST request to send the scraped information.
...
0
votes
0
answers
19
views
Scrapy get objects from different pages
I am using scrapy in python.
I am scraping 3 pages in a website that says A page, B page, C page.
First thing Ive done is that I got link from A page to go into B page.
Secondly, I got X and Y element ...
0
votes
1
answer
68
views
Google BigQuery Update is 70x slower then Insert. How to fix?
Im using BigQuery as my DB with Scrapy spider.
Below are 2 pipelines to store data into DB. One uses Insert, another Update methods. The Update method is 70 times slower then insert (merely 20 updated ...
0
votes
0
answers
23
views
Unable to authenticate with web scraper using Python - need help resolving login issue
I'm currently working on a Python web scraper to extract data from MyFitnessPal (MFP) using the holtchesley/mfp_food_and_excercise GitHub repository as a reference (link: https://github.com/...
-1
votes
0
answers
26
views
Scraping kaggle page with scrapy
I encountered an issue while attempting to scrape data from Kaggle using Scrapy. When I executed the fetch command with the URL 'https://www.kaggle.com/datasets?search=target', it returned a 404 error,...
0
votes
0
answers
44
views
Scraping a JS heavy website on Windows - using Scrapy + (Playwright/Selenium/Puppeteer) excluding Splash
I am trying to scrape a JS heavy Website using Scrapy, which means I have to render the page first.
I have trouble following scraped links since Scrapy only does:
for link in links:
respsonse....
0
votes
0
answers
29
views
How to send python scrapy requests in order (From a list of websites->Website -> Extract Webpages -> Webpage (until done)-> Other Website -> .... )?
The problem is that I want my spider to send requests to different websites and in between I have a callback that extract all the links of the current website domain and send requests to them and ...
1
vote
1
answer
35
views
Scrapy download Images and rename the image as md5 hash
I have a Scrapy spider which is working as far as scraping is concerned but I am having issues during downloading the images. I want to download the images and rename them as md5 hash for example: c69/...
-1
votes
1
answer
83
views
Scrapy with playwright - scraping immoweb
Configuration: working on WSL with ubuntu terminal. Coding in python with vs code.
Installed modules : scrapy, scrapy-playwright, playwright
Project: extract the data from the website www.immoweb.be (...
-4
votes
0
answers
62
views
What does this line in the robots.txt mean and how should I interpret it?
I want to know whether I am allowed to scrape URLs that end with /kommentare/x where x is replaced with an integer (/kommentare/2 for example).
When trying to access this URL with scrapy without any ...
-1
votes
0
answers
29
views
Python airflow dynamic task definition
in trying to use airflow for my crawlers, I had to do something like this:
with dag:
@task
def crawl_one_site(spider_path: str):
spider_runner(spider_path)
@task
def ...
0
votes
1
answer
36
views
selenium.common.exceptions.ElementNotInteractableException:
selenium.common.exceptions.ElementNotInteractableException: Message: >Element name="Description"> is not reachable by keyboard
I developed the python source code:
# Fill out ...