Questions tagged [scrapy]

Scrapy is a multi-threaded open-source high-level screen scraping and web crawling framework written in Python used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data extraction to monitoring and automated testing.

Filter by
Sorted by
Tagged with
0 votes
1 answer
26 views

Scrapy dont save json file on Jupyter Notebook

I have a script created in a Jupyter Notebook that scrap a url and should save the result in a json file, but it doesn't do it, even in the log it says it does. I am using Google Drive to save the ...
Regue's user avatar
  • 35
1 vote
0 answers
19 views

Scrapy with Python 2.7 on wsl/Red Hat, errors all the way

So I need to run Scrapy at work. I cannot use python3 for the moment because the installation is messed up (long story). I install scrapy using pip install scrapy. The version with scrapy --version is ...
Synthwave09's user avatar
-2 votes
0 answers
20 views

Scrapy - selector

I want to "select" the individual adress lines (Torhoutsesteenweg 109 Bus 501, 8400 — Oostende) of this link: https://www.immoweb.be/nl/zoekertje/appartement/te-koop/oostende/8400/10993990 ...
fruesto's user avatar
0 votes
0 answers
44 views

How to extract the `GST NO` and `URL` form `https://www.tradeindia.com/` which is comes in anchor tag `href` value and save as output.csv?

I want to scrap the GST NO and URL which is comes in GST NO form the https://www.tradeindia.com/ and nested url pages and save the csv file. I want to extract the all GST NO. I'm using below code, I'...
Sam's user avatar
  • 847
-2 votes
0 answers
19 views

crawl Dynamic Tabs with Scrapy [closed]

I am trying to extract the product tabs from this website and save them in a JSON file: https://www.douglas.de/de/p/5011106061 The extraction of information from Tab 1 (product details) works ...
Xemerel's user avatar
1 vote
1 answer
30 views

CURL in PHP to SCRAPYD using schedule.json not returning anything

I have set up Scrapyd on my server and everything seems to be working fine. I can use CURL to get a list of my spiders, like so curl -u super:secret http://111.111.111.111:6800/listspiders.json?...
Sebastian's user avatar
  • 841
0 votes
0 answers
16 views

Scrapy Splash: Lua script is not clicking on button

I am trying to execute a simple lua script on my localhost (http://localhost:8050) but due to some reason it is not working. I am new to lua so can you please check it? function main(splash) ...
Ashutosh Goyal's user avatar
1 vote
1 answer
36 views

Get the Element Name From Attribute Value Using Xpath

I am trying to get the element/tag name of each node where I have a particular attribute value. I have an xml: <a node='1'>This</a> <b node='2'>Is</b> <c node='23'>A</...
SC4RECROW's user avatar
  • 131
0 votes
0 answers
25 views

The spider (scrapy) does not restart [duplicate]

I am trying to restart my spider once every 30 minutes, for parsing a website, to which I get an error. The first launch is successful, the next one is an error. Spider Code: def run_spider(): ...
Danil Pet's user avatar
1 vote
1 answer
39 views

Scrapy error on redirects to PDF file: AttributeError: Response content isn't text

I've got a scrapy spider hosted on Zyte using Smart Proxies. My spider is fairly simple as it crawls starts from a list of URLs. the parse method uses a simple linkextractor to extract links on the ...
leeprevost's user avatar
0 votes
0 answers
38 views

Scrapy doesn't scrape all pages (Crawled 200: Referer None )

I'm trying to scrape the review from amazon in, I have created a script for this: It works but sometimes it returns a message and doesn't scrape the page and then goes to another one. I used Xpath ...
Lara Farron's user avatar
-2 votes
0 answers
29 views

Running scrapy spider from FastAPI [closed]

I want to be able to run scrapy spider over a FastAPI endpoint, however I have not found anything similar to my case in the scrapy documentation. I need to be able to retrieve items collected after it ...
revolex's user avatar
1 vote
2 answers
51 views

Scrapy response returns an empty array

I'm crawling this page with scrapy and I'm trying to extract all the rows of the main table. The following XPath expression should give me the wanted result: //div[@id='TableWithRules']//tbody/tr ...
Maurizio Brini's user avatar
1 vote
0 answers
52 views

Reproducible crawling behavior with Scrapy

How can I make the Scrapy crawler reproducible? By reproducibility, I mean that if we crawl the same website twice, we crawl it in exactly the same way. Therefore, we visit pages in the same order ...
Amir's user avatar
  • 16.2k
1 vote
1 answer
21 views

Scrapy: cannot reach second callback function when sending request with query parameter strings

I am scraping engineering blogs from meta. Right now I am just trying to print the title and url for each blog. Thanks for any help here is what I have done. It doesn't reach parse_loadmore function ...
Frankie's user avatar
  • 13
1 vote
0 answers
17 views

Tell scrapy to start executing some requests and not wait on creating all of them

I have a scrapy spider which downloads an input file from my private Github with around 1 million playlist IDs, which I want to search for using Chartmetric (a music data analytics tool). To this end, ...
thematthiaz's user avatar
1 vote
1 answer
43 views

get data from two different urls to the same ScrapyItem()

I'm new at scrapy and I've been trying to scrape this website: https://quotes.toscrape.com/ The datas I want is quote; author; date of birth and local of birth. To get the first 2 datas (quote ...
Gabriel Lucena's user avatar
2 votes
2 answers
55 views

Scrapy-playwright with multiple start_urls

Similar problem was discussed here but I was not able to make my code work. The aim is to scrapy-playwright generate request-response for each URL in start_urls and parse each response the same way. ...
ReCodeRa's user avatar
-5 votes
0 answers
52 views

Why isn't my Python script for scraping 4 pages working? [duplicate]

So I've been practicing scraping and came up with the theoretical practice of "what if I needed to just scrape a few pages?" so I made this script with the help of someone who answered my ...
Calvin Fermin's user avatar
0 votes
1 answer
34 views

Scrapy returning same value over iterating

I am using Scrapy to extract information from a website. My goal is to use Scrapy to pull the name of golf clubs, prices, etc and track cost over the winter and buy what I want, when the price goes ...
Brian Hamilton's user avatar
1 vote
1 answer
58 views

Trouble scraping BBC with Python Scrapy (2023)

we want to scrape articles (content + headline) to extend our dataset for text classification purposes. GOAL: scrape all articles from all pages at >> https://www.bbc.com/news/technology PROBLEM:...
Sebastian's user avatar
0 votes
0 answers
29 views

Scrapy Pagination not working after a few days

I am new to webscraping with python and scrapy and wrote my first spider last weekend (see code below). import scrapy from scrapy.exceptions import CloseSpider class MobileSpider(scrapy.Spider): ...
tek90ka's user avatar
2 votes
1 answer
50 views

scrapy.core.scraper ERROR: Error downloading -- OSError: Errno 24 Too many open files

I inherited a scrapy application which crawls through several 1000 pages on a domain and writes the final results out to a json file. The author had been running this on a mac and had encountered a ...
Mark Wojciechowicz's user avatar
0 votes
1 answer
41 views

Empty response when scraping things with scrapy (python)

I'm trying to scrap a website ('ellenmacarthurfoundation') using scrapy and its Selector.css, specifically this page: Here's the website Here's the html of the web that I'm trying to scrape I'm ...
Ignacio Perez vilaplana's user avatar
-2 votes
0 answers
20 views

While visiting a website, take screenshot with network tab open with xhr option clicked

I want to take screenshot of website on the left and it's xhr from devtools opened on the right. My approach which works but is very slow: I tried doing this using selenium and pyautogui. I open the ...
Ashutosh Singh's user avatar
1 vote
1 answer
36 views

scrapy extract text from nested div in nested classes

Really new to scrapy and webscraping in general, ive been working through the tutorials and wanted to branch out and see what I could do. Im have some content from a website I want to scrape which has ...
Jonathan's user avatar
0 votes
3 answers
44 views

Helper functions in another file, ModuleNotFoundError when trying to import

I have a simple Python project using scrapy. My file structure looks like this: top_level_folder |-scraper |--spiders |---help_functions.py |---<some more files> |--items.py |--pipelines.py |--...
Allen Y's user avatar
  • 420
0 votes
0 answers
42 views

Scrapy how to make async request from FSFilesStore/media pipeline?

What I need is to stat file based on head request, instead of downloading entire file. File is video(can be big), always have same name. Can determine update based on size and update change from head. ...
frenzy's user avatar
  • 1,664
0 votes
0 answers
28 views

Dynamic Content Loading and Interaction with Splash in Scrapy: Email Input Popup Issue

I'm working on a Scrapy spider that involves interacting with a webpage using Splash. The webpage has a dynamic email login feature where, upon clicking the "E-posta adresiyle devam et" (...
eminaruk's user avatar
-1 votes
1 answer
64 views

Scrapy Spider: Stuck with callback not firing

I am trying to scrape a github repo. I want to extract all XML file urls at level1 of every repo and in the best case also extract information from the XML files. import scrapy from scrapy.crawler ...
lupl's user avatar
  • 789
0 votes
0 answers
62 views

How to yield results from page.on('response') in Playwright?

I'm trying to yield the results from a page.on() response function. I'm using Playwright with Scrapy in Python. This is how the code looks like: def start_requests(self): yield scrapy.Request( ...
Marco Rodrigues's user avatar
-1 votes
0 answers
36 views

Scrapy 403 HTTP status code is not handled or not allowed

I'm trying to scrap articles from ScienceDirect website for a school project using scrapy and splash, I tried a lot of solutions I've found online but I keep getting the 403 forbidden error. I've ...
najlae01's user avatar
-1 votes
1 answer
44 views

Python, Scrapy, multiple items and links

I'm working on learning web scraping and doing some personal projects to get going. I've been able to learn some of the basics but having trouble with saving the scraped data to a csv file. import ...
John's user avatar
  • 1
-3 votes
0 answers
47 views

Having trouble with scraping infinite scroll website

I recently did a course in web scraping to get down the basics and as practise I wrote a program to scrape reviews from steam pages using the scrapy module. I've hit a bit of a barrier with it as it ...
Bradley Jones's user avatar
0 votes
1 answer
55 views

Scraping from IMDB with Scrapy

I'm scraping data from the IMDB Top 250 Movies page, specifically trying to retrieve user voting information. The HTML structure for user voting is like this: "&nbsp;(" "3.8M" &...
alexquilis1's user avatar
0 votes
1 answer
18 views

Splash doesn't render the full page

I'm not able to figure out how to get full page rendered with splash. I tried to set proxies and differents user-agent, without success. The main div with the content is not rendered.. If someone ...
Ivan's user avatar
  • 21
0 votes
0 answers
43 views

My Xpaths don't work in Scrapy Splash, but work in Selenium

I am trying to scrape a list all of the scholarships in the https://bigfuture.collegeboard.org/scholarships/; I was able to scrape all of the links and store it in a list using Selenium. However, ...
correa's user avatar
  • 1
0 votes
0 answers
18 views

How to use scrapy to scrape the value of a HTML element tag which is JSON with xpath or in another way?

I using Scrapy to scrape a page and so far I had success with XPATH but I'm in a bit of a struggle with that one. I am trying to get the value of dimensionsImageKey : <img id="fullViewImg"...
hanku8's user avatar
  • 77
0 votes
0 answers
22 views

wget/curl/scrapy behave differently in aws

I was trying to use scrapy to crawl a website using my own machine (ubuntu or a mac) and a server in aws ec2 instance, I was able to using my own machine but not in aws. And when I use wget or curl to ...
G Wesign's user avatar
0 votes
1 answer
31 views

Why doesn't Scrapy save it to json file?

I am new to Scrapy and I made a simple program to get the title tag from the page qoutes.toscrape.com. I tried to get the tag in the terminal and it works, but when I try to extract it as a json file, ...
Frane's user avatar
  • 15
0 votes
1 answer
58 views

Why is my scraping script printing the first item multiple times before working properly?

Beginner here, trying to learn web scraping and Python in general. I've been working on a spider after watching a few tutorials on basic web scraping to scrape the product names ("special_offers&...
Calvin Fermin's user avatar
0 votes
0 answers
11 views

Scrapy-Playwrite program only load periphery page elements

A program that I wrote using scrapy and scrapy-playwright only seems to load the periphery elements of the page. The "meat of the page" remains blank, but unfortunately that is the ...
Darwin's user avatar
  • 11
0 votes
1 answer
32 views

Scrapy FormRequest Username and password

I am trying to login to this form using Scrapy shell. But I am not sure if I am using the right name for the form field. Get token works token = response.css("form input[name=_csrf_token]::attr(...
Leo Torres's user avatar
-1 votes
1 answer
73 views

PYTHON: Scraping Researchgate.net with scrapy returns 'Just a moment' instead of the author's page

For a project, I want to gather coauthorship data from researchgate. I am completely new to webscraping, and got recommended scrapy for this project. I want to start scraping from this url (url = ...
Stu_gro's user avatar
3 votes
0 answers
95 views

Issue with Docker Image for Custom Deployment on Zyte (formerly Scrapinghub) Platform

I am currently facing an issue while trying to create a Docker image for custom deployment on the Zyte platform (formerly known as Scrapinghub). My goal is to set up a Python environment with Scrapy, ...
Kristiqn Tashev's user avatar
0 votes
0 answers
28 views

Multiple spiders in one project - scrapy runs the first spider (from the .py sorted alphabetically) on "scrapy crawl"

My project folder structure is like this: my_scrapers/ ├── scrapy.cfg ├── website1/ │ ├── __init__.py │ ├── items.py │ ├── middlewares.py │ ├── pipelines.py │ ├── settings.py │ └── ...
Rajiv Patki's user avatar
0 votes
0 answers
20 views

Inherited Scrapy spider does not store any output from super basic tutorial site

See the full Scrapy project filenames and code below. It creates 2 output files myspecificspider.json AND myspecificspider_items.json (which is weird on its own as I'd expect only the latter), but ...
Adam's user avatar
  • 6,161
-1 votes
0 answers
17 views

Where does the modules that a crawling project based on Scrapy needed come from?

First comes why I put this question: When I practice a python crawl project based on Scrapy , I meet an error called "ModuleNotFoundError: No module named ‘pymongo’" like this: enter image ...
Li World's user avatar
0 votes
1 answer
41 views

Scrapy Splash with xpath not returning any results

The page I'm trying to scrape is https://www.biggerpockets.com/forums/88/topics/895460-cap-rate-vs-interest-rate The xpath in the developer console returns the text element which corresponds to the ...
Ksyar2001's user avatar
0 votes
1 answer
41 views

How to Simulate Data Entry in an Input Field and Trigger a Button Click Using Scrapy Instead of Selenium

Having worked with Selenium, I'm in the process of transitioning my project to Scrapy, but I've encountered some challenges. I've explored the documentation, but I haven't found a solution yet. ...
pensive's user avatar
  • 300

1
2 3 4 5
356