As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [scrapy]

Scrapy is a fast open-source high-level screen scraping and web crawling framework written in Python used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

How to get all the categories/subcategories from a dynamic website?

I need to extract all the products from a site (https://www.hiperlibertad.com.ar/) that has multiple branches/stores. The main problem I have now is that I need to find a way to adapt the code to be ...
Kiko's user avatar
  • 1
-2 votes
0 answers
21 views

Scrapy scrape hidden keywords from IMDB keywords page

I have scrapy code that downloads the list of all imdb titles and related keywords based on provided keyword. This code was working perfectly and downloading all keywords until few days back when imdb ...
Ajmul's user avatar
  • 33
0 votes
0 answers
7 views

TCP Connection Timed Out Scrapy Error as soon as downloading files starts

So I'm using scrappy to extract data from a website. Here's what I'm doing... Navigating to subpages from the main pages. --- from these subpages, I'm extracting data and downloading attachments t ...
Mudassir Ahmed's user avatar
-1 votes
1 answer
31 views

Webscraping have incorrect result

When I scrape website it was total correct but have a lot of blank and some of incorrect data. import scrapy class AudibleSpider(scrapy.Spider): name = 'audible' allowed_domains = ['www....
user22027271's user avatar
0 votes
1 answer
17 views

How can I navigate and extract restaurant details from zaubee.com when the href attribute is set to "#" for each restaurant link?

How can I scrape the zaubee.com website to extract business details from each restaurant's page when the href attribute is set to "#" in scrapy?? I'm presently working on a web scraping ...
Licey Soremap's user avatar
0 votes
0 answers
23 views

Why Scrapy spider is closing after first request to start_urls?

I know this question was asked multiple times but the solutions to the problems are always different. I have the following scrapy crawler: from scrapy.spiders import CrawlSpider, Rule from scrapy....
PParker's user avatar
  • 1,399
1 vote
1 answer
28 views

How to run playwright asynchronously in Scrapy Python?

My example code works synchronously without mistakes. It parses all the start urls and prints needed text to the output. How to make this code work asynchronously? (I need to run playwright calling ...
Space Pilot 3000's user avatar
-1 votes
0 answers
13 views

Get dynamic classes that end in number using scrapy [duplicate]

I want to get all elements whose classes are like panel-x, where the X is always a number Example: <div class="content"> <div class="panel panel-default panel-1">&...
eternaty's user avatar
  • 125
0 votes
1 answer
11 views

How to solve 'Spider_error_processing_headers' problem while using Scrapy's CrawlSpider and LinkExtractor?

I got a massage lik ERROR: Spider error processing and line 276, in aiter_errback yield await it.anext() in my terminal, and my code given below Can anyone tell me where is the problem. import scrapy ...
Moniruzzaman Monir's user avatar
0 votes
0 answers
28 views

Scraping Zillow with Scrapy Playwright only retrieves 8 listings instead of 40. How can I fix this?

i use scrapy playwright to scrape zillow but instead of getting all 40 listings i just get 8 i made sure i used the css selector for all 40 listings please help import scrapy from ..items import ...
Sabry Belal's user avatar
-1 votes
0 answers
34 views

I am trying to install Scrapy; however, this is the error I am experiencing : Failed building wheel for lxml . Please assist

Experiencing error Failed building wheel for lxml src/lxml/etree.c:96:10: fatal error: 'Python.h' file not found #include "Python.h" ^~~~~~~~~~ 1 error generated. ERROR: Could not build ...
Ayushi jain's user avatar
0 votes
1 answer
34 views

Issues with Python Scrapy script with multiple start urls

I have a scraper written in Scrapy. It works perfectly fine as long as there is just one url in start_urls. As soon as I add 2 or more start urls, the behavior is erratic. The number of pages it ...
jay queue's user avatar
  • 275
0 votes
1 answer
38 views

How to send mail with scraped data?

How to send email with specific scraped data after spider ends in Scrapy. I'm trying to send some item data with multiple lines after the end of a spider, but i'm only receiving one line. I think i ...
Harker's user avatar
  • 5
0 votes
0 answers
38 views

Scrapy only scraping a few items

I am practicing a bit with scrapy on a website, with 60 items maximum per page. For each item, I need to extract more data, so I go to each items' url. However, I am only able to scrape a fraction (...
Rema's user avatar
  • 1
-1 votes
2 answers
37 views

How to get the initial URL in Scrapy

I need to get the first URL which Scrapy spider currently processes to map result to it. More spicifically. If an URL for scraping is abc.de and it's redirected to https://www.abc.de/en, I want to get ...
Dmitry's user avatar
  • 707
0 votes
0 answers
21 views

Run crawling function sequentially on multiple URLs with Scrapy

How can I run a crawler on multiple URLs sequentially? we already know that we can pass multiple URLs & allowed_domains at the same time but I need a function to handle this as follows: import ...
Amir's user avatar
  • 15.9k
1 vote
2 answers
40 views

How do I scrape multiple pages with different alphabetical labels using scrapy

I am working on a web scraping project using Scrapy and I am facing an issue with iterating through multiple pages with alphabet labels. My goal is to scrape medicine data from the website https://www....
032_Rushabh Kayadra's user avatar
0 votes
1 answer
45 views

UnicodeEncodeError: 'ascii' codec can't encode character '\xe3' in position 7: ordinal not in range(128) [duplicate]

Some days ago, accepted this answer to a question of mine as correct, but after a while I noticed that in some URLs, I got the following error: 2023-05-29 19:22:20 [scrapy.core.scraper] ERROR: Spider ...
Higo Felipe Silva Pires's user avatar
-1 votes
1 answer
22 views

Scrapy: How can I execute a CMD command?

How can I execute a cmd line from within a Scrapy script? I'm using Scrapy in a Microsoft Windows environment. Normally, from within native Python, I would do something like this: import os os....
Ron's user avatar
  • 1
0 votes
0 answers
30 views

How to bypass the Error 403 with scrapy and selenium?

I have the example code write in scrapy with selenium. Do you know how to bypass the 403 error? I changed the setting DOWNLOAD_DELAY = 3 # change delay time DOWNLOADER_MIDDLEWARES = { "testfx....
NovLe's user avatar
  • 13
0 votes
0 answers
20 views

Retrieving multi-category data from REST Data API using Scrapy

With help from the developers of this community, I was able to scrape data from an online supermarket. This page shows the food category of the website. However, I have been unable to get sub-category,...
WX1505's user avatar
  • 97
0 votes
2 answers
25 views

Scrapy not saving output to jsonline

I put together a crawler to extract URLs found on a website and save them to a jsonline file: import scrapy from scrapy.crawler import CrawlerProcess class MySpider(scrapy.Spider): name = '...
Enrico Shippole's user avatar
0 votes
0 answers
18 views

Error while executing lua with scrapy_splash which fails with 400 and says "Required argument is missing: lua_source"

Here's my code: def start_requests(self): headers = { "method": "GET", "path": "/", "scheme&...
Afsheen Amroliwala's user avatar
0 votes
1 answer
31 views

Need some scraping guidance regarding missing pagination elements [closed]

Noob web scraper here. I built a spider using Scrapy and Playwright to scrape auto ads for the results of a parameterized search URL on autotrader.com and it's working great to grab data from the ...
Sandman's user avatar
  • 17
0 votes
1 answer
34 views

Using FormRequest to extract data via HTTP POST

Using FormRequest to extract data via HTTP POST hey guys I will crawl the details of all the products of the site https://bitsclassic.com/fa/ with scrapy To get the url of the products, I have to ...
Amir Hamidi's user avatar
1 vote
1 answer
20 views

Scrapy not giving output but works in shell

So i'm new to web scraping and i've been trying to scrape this website but the problem i'm encountering is that when i run my code in shell it runs but it doesn't when i run it as a whole scrapy code. ...
032_Rushabh Kayadra's user avatar
1 vote
1 answer
21 views

How can I extract the amount written after the 'span' tag under the 'div' tag using css-selectors in Scrapy?

I want to get the amount(182.78) written after span tag under div tag , but I am getting only the "MRP" element content as a string, I only want to extract the amount written. By the way I'm ...
032_Rushabh Kayadra's user avatar
0 votes
0 answers
18 views

How to Save Scrapy Response Output to minio Pod on a Cluster

I saw a similar question in: Scrapy files pipeline for minio storage But I did not see a successful response. I am using scrapy to scrape various websites, and I can save the output to a .csv file in ...
Rasputin's user avatar
  • 124
-1 votes
0 answers
56 views

Yet another problem with using scrapy-splash to click the button on the website

I tried many different solutions how to make splash click the button and I am unable to understand what I am doing wrong. I tried getting the element by its type (button), tried by its class, tried ...
Joanna's user avatar
  • 160
0 votes
1 answer
58 views

Download embedded PDF on website using Selenium/Scrapy/Python

There are some pages like this that I want to download the embedded PDF file, but researching and testing various proposed solutions on SO and other sources, I couldn't find a correct solution. When I ...
Higo Felipe Silva Pires's user avatar
0 votes
0 answers
22 views

How to pass a variable from init to parse_item in Scrapy CrawlSpider?

Maybe it's the wrong approach alltogether. I'd be happy about an alternative approach as well, if you have an idea. Connect to my DB, get the next batch of seed URLs (as a dict, with some other info) ...
Chris's user avatar
  • 1,225
0 votes
0 answers
24 views

Getting 'Invalid request method' error when trying to pass Scrapy data to Django using Python APIs

I have a problem, I created a scrapy project and a django project with an application, I registered the application, set the urls, set the views, and also wrote the spider, but when I go to the ...
Roman's user avatar
  • 5
-1 votes
1 answer
22 views

How to use an API-provided proxy with rotating proxy middleware in Scrapy?

I'm trying to scrape websites using Scrapy and want to use an API-provided proxy with the rotating proxy middleware. The proxy is provided by an API endpoint that returns a new proxy IP with each ...
Waleed Farrukh's user avatar
1 vote
1 answer
22 views

Scrapy: select last decendant node?

I have a dict with selectors which I use to get data: for key, selector in selectors.items(): data[key] = response.css(selector).get().strip() One of the selectors is span::text, but sometimes ...
bur's user avatar
  • 594
-2 votes
0 answers
25 views

How can I prevent duplicate entries in MongoDB using Scrapy and Python?

a solution to avoid duplicate entries in momgodb using scrapy python i'm trying to avoid scrapy mongo db duplicate records, my code scrapy items function some of the items def parse_article(self, ...
martia road's user avatar
0 votes
0 answers
56 views

scrapy-playwright getting null values

I am trying to scrape this page https://www.sephora.com/shop/skincare, but I am getting this bug: {'title': None} 2023-05-22 13:56:59 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www....
Narsil91's user avatar
-1 votes
0 answers
24 views

Why isn't Scrapy scraping anything when using FormRequest to log into a website?

I am using Scrapy FormRequest to login in this website. However, the crawler didn't scrape anything even though it didn't show any error in the result. This is part of my code: class ...
codddder's user avatar
0 votes
0 answers
16 views

New to the Scrapy library, how do I troubleshoot an empty JSONL file while web scraping?

Spider not doing its job scrapy i tried to run the folllowing scrapy spider to scrap and parse a website and put the results in a jsonl, which stays empty, im new to the libary import scrapy from ...
מתן רוכמן's user avatar
0 votes
0 answers
12 views

How can I return scraped information from my scrapy spider to my Flask server?

I want to call my scraper from my flask server and have the value returned to the server. At the moment I am using subprocess to call the scraper, and a POST request to send the scraped information. ...
Calvinism's user avatar
0 votes
0 answers
19 views

Scrapy get objects from different pages

I am using scrapy in python. I am scraping 3 pages in a website that says A page, B page, C page. First thing Ive done is that I got link from A page to go into B page. Secondly, I got X and Y element ...
codddder's user avatar
0 votes
1 answer
68 views

Google BigQuery Update is 70x slower then Insert. How to fix?

Im using BigQuery as my DB with Scrapy spider. Below are 2 pipelines to store data into DB. One uses Insert, another Update methods. The Update method is 70 times slower then insert (merely 20 updated ...
Billy Jhon's user avatar
  • 1,035
0 votes
0 answers
23 views

Unable to authenticate with web scraper using Python - need help resolving login issue

I'm currently working on a Python web scraper to extract data from MyFitnessPal (MFP) using the holtchesley/mfp_food_and_excercise GitHub repository as a reference (link: https://github.com/...
prayag purohit's user avatar
-1 votes
0 answers
26 views

Scraping kaggle page with scrapy

I encountered an issue while attempting to scrape data from Kaggle using Scrapy. When I executed the fetch command with the URL 'https://www.kaggle.com/datasets?search=target', it returned a 404 error,...
Estiven's user avatar
  • 11
0 votes
0 answers
44 views

Scraping a JS heavy website on Windows - using Scrapy + (Playwright/Selenium/Puppeteer) excluding Splash

I am trying to scrape a JS heavy Website using Scrapy, which means I have to render the page first. I have trouble following scraped links since Scrapy only does: for link in links: respsonse....
hm2330's user avatar
  • 1
0 votes
0 answers
29 views

How to send python scrapy requests in order (From a list of websites->Website -> Extract Webpages -> Webpage (until done)-> Other Website -> .... )?

The problem is that I want my spider to send requests to different websites and in between I have a callback that extract all the links of the current website domain and send requests to them and ...
Sahil Dua's user avatar
1 vote
1 answer
35 views

Scrapy download Images and rename the image as md5 hash

I have a Scrapy spider which is working as far as scraping is concerned but I am having issues during downloading the images. I want to download the images and rename them as md5 hash for example: c69/...
X-somtheing's user avatar
-1 votes
1 answer
83 views

Scrapy with playwright - scraping immoweb

Configuration: working on WSL with ubuntu terminal. Coding in python with vs code. Installed modules : scrapy, scrapy-playwright, playwright Project: extract the data from the website www.immoweb.be (...
Sarkis T.'s user avatar
-4 votes
0 answers
62 views

What does this line in the robots.txt mean and how should I interpret it?

I want to know whether I am allowed to scrape URLs that end with /kommentare/x where x is replaced with an integer (/kommentare/2 for example). When trying to access this URL with scrapy without any ...
Mike's user avatar
  • 1
-1 votes
0 answers
29 views

Python airflow dynamic task definition

in trying to use airflow for my crawlers, I had to do something like this: with dag: @task def crawl_one_site(spider_path: str): spider_runner(spider_path) @task def ...
Hossein Gholami's user avatar
0 votes
1 answer
36 views

selenium.common.exceptions.ElementNotInteractableException:

selenium.common.exceptions.ElementNotInteractableException: Message: >Element name="Description"> is not reachable by keyboard I developed the python source code: # Fill out ...
Amir Hamidi's user avatar

1
2 3 4 5
354