Questions tagged [scrapy]

Scrapy is a multi-threaded open-source high-level screen scraping and web crawling framework written in Python used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data extraction to monitoring and automated testing.

scrapy
Filter by
Sorted by
Tagged with
0 votes
2 answers
85 views

scrapy, selenium doesn't return all the element in the page

I want to reduce the time it takes the code to finish scraping the pages, I'm using selenium. I used Scrapy for this scraping project, but JavaScript was hiding the email elements from Scrapy . Scrapy ...
mohamed agnaby's user avatar
0 votes
0 answers
54 views

python3: can't open file '/app/app.py': [Errno 2] No such file or directory - Docker

So I'm bulding Scrapy/Flask app and I'm trying to dockerize it so I could run it with docker-compose up. I already dockerized Scrapy app. But when I tried to implement Flask web app, I'll always get ...
Jaroslav H.'s user avatar
0 votes
1 answer
22 views

Scrapy: Where to init database connection, so it is available and accessible in spiders, pileines, and classes

I have a fairly standard Scrapy project, its dir structure looks like this my_project scrapy.cfg my_project __init__.py items.py itemsloaders.py middlewares.py MyStatsCollector....
user984621's user avatar
  • 47.4k
-2 votes
1 answer
87 views

Scrapy crawlspider in python found an error "'str' object has no attribute 'iter'"

I get an error message that says: AttributeError: 'str' object has no attribute 'iter' 2024-03-15 14:01:19 [scrapy.core.engine] INFO: Closing spider (finished) when I use this code: class ...
Spatdy's user avatar
  • 41
0 votes
0 answers
23 views

Navigation across the site using scrapy

So I am working on a personal project where i am learning Web Scraping. I have chose this as my website to scrape. I am using scrapy as it seemed like a good library. For this website. I need to loop ...
Sairam S's user avatar
  • 329
1 vote
1 answer
65 views

Can't Scrape a webpage whose contents are dynamically generated through JavaScript

I am trying to scrape table data from a webpage but it's not a normal webpage that can be scraped using its html tags and CSS class or ID. The contents of the webpage are dynamically generated using ...
Abhinay's user avatar
  • 11
0 votes
0 answers
21 views

Using scrapy and scrapy-playwright behind proxy

I try to run two spiders behind company proxy. One spider is scrapy spider ("quotes") and one scrapy-playwright spider(quotes-playwright): from typing import Iterable import os import scrapy ...
Luftuq's user avatar
  • 15
0 votes
1 answer
96 views

Getting 403 error with Scrapy request, though python 'get' request works fine

Trying to get content of few websites using Scrapy, however all of them returns 403 (Forbidden) response code. Though same websites works fine when i make request using 'get' function as below: import ...
Mah3sh's user avatar
  • 25
0 votes
0 answers
33 views

How to properly manage errors and logs of a spider?

I am testing Scrapy for the last couple of days and it is an amazing framework. One thing that I struggle with to fully understand is logging information after the spider is finished and managing logs....
user984621's user avatar
  • 47.4k
0 votes
0 answers
32 views

How do I emit flask_socketio events during a scrapy crawling process?

Since crawling takes time, I am trying to build a progress bar on the front-end by tracking the number of items my spider has crawled. The number of items is compared to a capped number of items, at ...
mpx's user avatar
  • 1
0 votes
1 answer
42 views

Scrapy error - str object has no attribute css

Article = response.css(‘.article-main).getall() Len(article) Till here we are getting result Art =article [0] Art.css(‘.art2-amp::text’).get() Facing error str object has no attribute css. Note ...
raju's user avatar
  • 1
1 vote
1 answer
39 views

Parsel is not able to access nested elements

I am working with Parsel. Unfortunately, I am not able to parse <a> tag, which is child of another <a> tag (I know, that <a> inside <a> isn't HTML standard). How can I handle ...
ifdef14's user avatar
  • 25
0 votes
0 answers
22 views

Scrapy-playwright scraper does not work on Javascript website

I'm trying to scrape customer stories from Knime.com. This is the website I want to scrape: https://www.knime.com/customers I'm triying to get all of the stories on the website, it's url, title, ...
Ashkan Lotfipoor's user avatar
-1 votes
1 answer
109 views

How to access child class in Scrapy?

So I am trying to scrape the following page: https://www.inmuebles24.com/departamentos-en-renta-en-orizaba.html. I want to access the children of postings-container and the information within them. I ...
Emilio Garcia's user avatar
1 vote
1 answer
98 views

Python scraper for facebook marketplace isn't working anymore

Since yesterday, I encountered an issue where my facebook marketplace scraper ceased to fetch data, I'm currently using scrapy due to his features, am I doing any mistakes? Output has been shared on ...
Rafael Ribeiro's user avatar
1 vote
1 answer
43 views

How to access particular stats ("finish_reason", "elapsed_time_seconds") in Scrapy?

I am playing with Scrapy and I am able to scrape with it data from different URLs. One thing I am trying to obtain, and so far unsuccessfully, is stats data, particularly finish_reason and ...
user984621's user avatar
  • 47.4k
-1 votes
1 answer
45 views

css selectors correctly identify items in scrapy shell but not when I run the spider

The CSV returns all empty links except the 'href' tag. I think it may have something to do with the get request. I read somewhere that you can only perform one then you have to parse, I'm just not ...
so_ko's user avatar
  • 1
-2 votes
1 answer
54 views

Issue when running Scrapy from a script in Celery: The installed reactor does not match the requested

There are many ways to start a scrapy spider from a script (docs). But when you deal with it in Celery it becomes somewhat complicated. What I want to have is a function that will start scrapy with ...
whatserface's user avatar
0 votes
1 answer
31 views

My spider doesn't enters yield scrapy.Request() at all in Scrapy

I'm having a problem in my spider in Scrapy. Based on a search key I scraped a search result page and found the links. but the next yield scrapy.Request() which scrapes the result pages doesn't enters ...
Horr Seraj's user avatar
0 votes
0 answers
90 views

Login auth not working on pages opened by scrapy-playwright

Question: How do I recreate the effect of manually opening a new tab in scrapy-playwright? Explanation: I have an issue in scrapy-playwright where a site does not recognize login credentials unless a ...
cforcomputer's user avatar
0 votes
1 answer
38 views

Scrapy Feeds not saving

I'm taking a Scrapy spider which succesfully scrapes a few thousand items (see logs below) and trying to store them into S3. However, regardless of the settings I use, no export is saved. The matter ...
GBonc's user avatar
  • 1
0 votes
0 answers
26 views

Why does the scrapy spider file run so slowly(/not run properly) after being imported in main file?

I wrote a scrapy spider which runs normally in the python script and can scrap texts. When I imported it as a module in the main.py file and tried to run it as a module there, my IDE just keeps ...
Fangyi Chen's user avatar
0 votes
0 answers
23 views

Looping JavaScript Processes in Scrapy code

very new to Scrapy in particular and somewhat new to coding in general. I'm trying to parse some data for my school project from this website: https://www.brickeconomy.com/sets/theme/sets/theme/...
puncakeman's user avatar
0 votes
0 answers
26 views

Encountering Problems After Modifying Items in Scrapy Project

I've previously developed a project that was functioning correctly. However, after changing the dictionary to "items," it began experiencing issues. While the page appears to be scraping ...
Muhammad Noman's user avatar
0 votes
1 answer
25 views

Understanding Scrapy Python: Refactoring parse method does not work

I have a spider code that works. class MySpider(BaseScrapper): name = "my_spider" def parse(self, response, **kwargs): self.logger.info(f"Parse: Processing {response....
inquilabee's user avatar
1 vote
1 answer
51 views

Python Scrapy Shell Error While Scraping Wallmart

I am scraping walmart.com using scrapy. when i am fetching https://www.walmart.com/ there is no error but when trying to fetch "https://www.walmart.com/search?q=tablets&typeahead=tabltes"...
Sadam Shah's user avatar
0 votes
0 answers
64 views

Unable to use scrapy to login into Instagram

I am trying to use scrapy to log into Instagram to scan and read comments but I cant seem to be able to login looks likr the request gets blocked by a bot. Is there a better way to attempt the scrapy ...
Leo Torres's user avatar
0 votes
2 answers
60 views

Web scraping with Scrapy and Python from one script and a javascript website

Hi I'm trying to web scrape (with Scrapy) this website https://www.vaniercollege.qc.ca/sports-recreation/weekly-schedule/ from this script below script.py import scrapy from scrapy.crawler import ...
SpicyyRiice's user avatar
1 vote
1 answer
55 views

Scraping using CSS Selectors with ::before isn't displaying text

I'm trying to scrap Monday, 9:30 AM from this ebay listing, using scrapy. From a scrapy shell scrapy shell https://www.ebay.com/itm/145599690533?: >>> response.css('span.ux-timer__time-left::...
Lord Elrond's user avatar
  • 14.5k
0 votes
0 answers
21 views

Unable to return yielded item to Scrapydo run_spider

I am calling the Scrapy spider using scrapydo run_spider method but Spider parse method is unable to return the yielded items Parse method. It basically calls the Spider's parse method dynamically ...
Arslan Haider Sherazi's user avatar
1 vote
1 answer
56 views

Scrapy spider skips pages and stops before end

I'm developing a spider with scrapy and Playwright to scrape a retail brand website. Basically, it's parsing all images of all colors of all products on all pages (page -> product -> color -> ...
paulbouu's user avatar
  • 132
0 votes
0 answers
60 views

Scrapy inside Azure functions throwing "signal only works in main thread"

I have implemented webcrawling upto certain depth. my code Skelton is below. class class SiteDownloadSpider(scrapy.Spider): # Regex pattern to match a URL HTTP_URL_PATTERN = r'^http[s]*://.+' ...
NSVR's user avatar
  • 302
0 votes
2 answers
76 views

Scrapy nameerror while running multiple functions

I'm trying to run the following code but i'm getting this error 'NameError: name 'scrapedate' is not defined' import scrapy from datetime import datetime, timedelta from dogscraper.items import ...
jw32001's user avatar
  • 43
1 vote
1 answer
44 views

Trouble following links in a page to scrape data. Code goes through the pages but returns nothing

I want to make scrapy return a specific webpage with a table so that it can then "click into" each entry in that table to go to a new webpage and scrape data from there. The webpage that I ...
TB_4240's user avatar
  • 13
2 votes
2 answers
102 views

Export into a same file items from two different spiders using CrawlerProcess

UPDATED I am using the following example code following the Scrapy documentation. # main_test.py import time from typing import Any import pandas from scrapy import Field, Item from scrapy.crawler ...
Benjamin's user avatar
0 votes
0 answers
46 views

429 Unknown status while scraping using scrapy

I got the following error after running the following code to scrape data form sofifa.com import scrapy from scrapy.linkextractors import LinkExtractor from scrapy.spiders import CrawlSpider, Rule ...
Muaaz Abu Zaid's user avatar
0 votes
0 answers
58 views

How to structure a monorepo with Flask, SQLAlchemy (SQLite) and Scrapy

I'm building a series of crawlers in Scrapy which output I'm saving to a local SQLite database via SQLAlchemy as the ORM. At the same time, I want to expose the scraped data via a web interface for ...
Jan Richter's user avatar
  • 1,996
1 vote
1 answer
23 views

How to Extract the Name 'Terence Crawford' from an HTML Segment, Excluding the Span Element?

I am currently facing difficulty retrieving the name 'Terence Crawford' from an HTML segment. The challenge lies in excluding the span element, which is present within the same parent element. <td ...
Muhammad Noman's user avatar
2 votes
2 answers
74 views

Abnormal response while scraping sofifa.com

I'm trying to scrape sofifa.com with scrapy tool. With the code below, I'm trying to scrape the full name and rating for the 60 players only exist in the first page, but I got more than 60 and the ...
Muaaz Abu Zaid's user avatar
0 votes
2 answers
56 views

How to parse an XML sitemap using python scrapy?

I'm trying to scrape the following sitemap using scrapy: https://comercialtrevino.com/sitemap/sitemaprssatom_products.xml At first I would like to use the scrapy shell and then I'll write a spider to ...
Tezcatlipoca0000's user avatar
0 votes
0 answers
21 views

Scraping "Hidden" Content in Google Group with Scrapy

I'm scraping messages on Google Groups, and in general the script works as I intend. It grabs the following and outputs them to csv or json, depending on my needs/mood: name of poster topic date of ...
simishu's user avatar
3 votes
1 answer
102 views

Persistent error deploying Docker image to Scrapy Cloud

I'm not sure what is causing the problem, but while attempting to deploy Docker image to Scrapy Cloud (using shub deploy) with my spider I'm getting following error: $ shub deploy Error: Detected ...
sebap123's user avatar
  • 2,645
0 votes
1 answer
79 views

Python Scrapy to scrap a dynamically loaded website

I am currently working on a web scraping project using Scrapy to extract course information from https://www.discoveruni.gov.uk/course-finder/results/. I've encountered a challenge due to the website'...
Jason_D's user avatar
  • 13
0 votes
0 answers
18 views

How to use scrapy and selenium to interact with the body of the scrapy response?

I've seen a lot of people making a first request with the get method of the selenium webdriver and then scrape the response with scrapy. I want to do the opposite: a first request with scrapy and then ...
David's user avatar
  • 1
0 votes
1 answer
51 views

Scrapy crawled 0 page 0 item

I create a crawler using scrapy to scrape data from Indeed, which expected to return job title, company, location, description, qualification into a csv file. But i ran into problem: it not return any ...
embiekhochiu's user avatar
0 votes
0 answers
33 views

Why is Scrapy-splash not returning expected HTML from dynamic javascript page?

I'm attempting to scrape the Market table data from the following page utilizing scrapy-splash: "manta.layerbank.finance/bank" (Put in quotes because might be causing spam issue?) So far I'm ...
Kody F's user avatar
  • 1
1 vote
0 answers
48 views

Empty Json File when Web Scraping

I'm trying to scrape the website adamchoi for specific information using Scrapy and I'm getting a json file which is empty. All the code seems to be correct so this does not make sense. I've searching ...
DevSmartz's user avatar
0 votes
2 answers
108 views

Unwanted newline characters in JSON while web scraping

I want to extract info from this website using Scrapy. But the info I need is in a JSON file; and this JSON file has unwanted literal newlines characters in only the description section. Here is an ...
esyilmaz's user avatar
0 votes
0 answers
52 views

Scrapy carousel categories not extracting

I am trying to scrap a website to get the list of categories from a carousel but it's not working. Below is my code import scrapy class CourtsmuSpider(scrapy.Spider): name = "courtsmu" ...
Mervin Hemaraju's user avatar
2 votes
0 answers
49 views

Why do I have an infinite loop when using a middleware with Scrapy?

I am implementing a program to scrape job offers on a site. However, I have a problem: on this same site links are sometimes programmed with relative href, sometimes absolute (ex: sometimes I have ...
Kevin's user avatar
  • 21