Questions tagged [scrapinghub]

a web scraping development and services company, supplies cloud-based web crawling platforms.

Filter by
Sorted by
Tagged with
3 votes
0 answers
94 views

Issue with Docker Image for Custom Deployment on Zyte (formerly Scrapinghub) Platform

I am currently facing an issue while trying to create a Docker image for custom deployment on the Zyte platform (formerly known as Scrapinghub). My goal is to set up a Python environment with Scrapy, ...
Kristiqn Tashev's user avatar
0 votes
0 answers
19 views

Scrapy Cloud Deployment Error: "Referer Header Missing"

I'm working on a Scrapy project and I'm trying to deploy it to Scrapy Cloud (by Zyte, formerly known as Scrapinghub). However, during the deployment process, I encounter an error: [INFO] Deploying ...
Cauder's user avatar
  • 2,223
1 vote
1 answer
59 views

Webscraping yml files from Github

I am trying to scrape certain open source file from GitHub but I'm having an issue with their new format. This if an example link: https://github.com/xavierLowmiller/xcodegen-action/blob/main/action....
Artemis's user avatar
  • 15
0 votes
1 answer
73 views

I'm having issue while deploying scrapper to Zyte formerly (Scraping hub)

My spider has to read some data from input.csv file. It runs fine locally. But when I try to deploy it on Zyte by shub deploy it does not includes input.csv in build. So when I try to run it on the ...
Muhammad Ahmad's user avatar
0 votes
1 answer
318 views

Extract data from company sharepoint using Python

Can I extract data from company's sharepoint using python. Used power automate but I want to use python code
sonakshi pattnaik's user avatar
-1 votes
1 answer
81 views

Can't fetch url in scrapy shell with splash

Please help me! When I try to fetch a URL in scrapy shell with scrapy splash, I use the following statement to get a response: >>> fetch('http://localhost:8050/render.html?url=https://www....
sejteN's user avatar
  • 9
-1 votes
1 answer
131 views

Is it possible to create a proxy failover with Python Scrapy?

Is it possible to create a proxy failover within Scrapy, so that when one fails the other will take over scraping the rest of the requests? I would of thought that it would be done using the retry ...
webbie1985's user avatar
0 votes
0 answers
23 views

Are there any tools or 3rd parties - free or paid that can scrape URLs for Price?

I have a list of URLs and need auto update on the price found on website. Are there any tools or 3rd parties - free or paid that can scrape URLs for Price? I have a list of URLs and need auto update ...
Here2Grow's user avatar
1 vote
0 answers
299 views

Web scraping using Octoparse

I have been trying to use Octoparse to scrape data from a particular webpage. It has a total of 361 pages and 10 data rows on each page (total of 3610 data points). However, what I get is only 3260 ...
Anthony Nguyen's user avatar
-1 votes
1 answer
59 views

How to get the XPATH or CSS selector from dynamically loaded website to follow links?

This is a dynamically-loaded website https://www.gelbeseiten.de/suche/hotels/n%c3%bcrnberg. I'm trying to follow every link from the results. I found //article[@class='mod mod-Treffer']/a to follow ...
Raisul Islam's user avatar
0 votes
1 answer
55 views

How to save Scrapy Broad Crawl Results?

Scrapy has a built-in way of persisting results in AWS S3 using the FEEDS setting. but for a broad crawl over different domains this would create a single file, where the results from all domains are ...
NightOwl's user avatar
  • 1,079
0 votes
1 answer
236 views

Selenium Problem extracting Google business description

I seem to be struggling with this issue for a couple of days and could really use some help. I am trying to scrape Google busineses information with Python beautifulsoups and Selenium and I want to ...
Thresh Bot's user avatar
0 votes
1 answer
186 views

Why error with installing csv when its part of python core package in scrapinghub

I have 3 spiders defined. All the related requirements are mentioned in requirements.txt scrapy pandas pytest requests google-auth functions-framework shub msgpack-python Also, the scrapinghub.yml ...
Avirup Das's user avatar
0 votes
1 answer
259 views

YouTube Subscriptions List Scraping

I want to scrap my YouTube subscriptions list into one csv file. I typed this code (but I didn't finish coding yet): import requests from bs4 import BeautifulSoup import csv url = 'https://www....
Mohamed Hendy's user avatar
0 votes
3 answers
2k views

(Scrapy) How do you scrape all the external links on each website from a list of hundreds of websites (and run the whole thing on Zyte)?

I am looking for some help regarding my Scrapy project. I want to use Scrapy to code a generic Spider that would crawl multiple websites from a list. I was hoping to have the list in a separate file, ...
Alban's user avatar
  • 21
0 votes
2 answers
382 views

Scrapinghub scrapy: ModuleNotFoundError: No module named 'pandas'

I have tried deploying to Zyte via command line and GitHub but I have been stuck with the above error. I have tried different versions of Scrapy version 1.5 to 2.5 but the error still persists. I have ...
chuky pedro's user avatar
0 votes
1 answer
758 views

Scrapinghub/Zyte: Unhandled error in Deferred: No module named 'scrapy_user_agents'

I'm deploying my Scrapy spider via my local machine to Zyte Cloud (former ScrapingHub). This is successful. When I run the spider I get the output below. I already checked here. The Zyte team is not ...
Adam's user avatar
  • 6,141
-1 votes
1 answer
108 views

How can I scrape a button that does not return a value

I am trying to scrape from the website https://tonaton.com/en/ads/ghana/electronics. There is a "next" button that I want to click and scrape the contents. The problem is the xpath or css ...
Danny Stringz's user avatar
2 votes
1 answer
840 views

Scrapy crawlera authentication issue

I've been trying to use scrapy-crawlera as a proxy for scraping some data with scrapy. I've added these rows in settings.py: DOWNLOADER_MIDDLEWARES = { 'scrapy_crawlera.CrawleraMiddleware': 610, } ...
memeister's user avatar
1 vote
0 answers
63 views

print data on a sample image but the text is going out of image

ALL of the data in the picture is taken from an excel sheet, size of the image is 220×320, I cant increase the image size. But I want Line 4 data that is exiting the image to go down below row 4. ...
Abdurehman Dar's user avatar
1 vote
1 answer
570 views

Crawlera/Zyte proxy authentication using C# and Selenium

I've tried a number of ways of using Zyte (formally Crawerla) proxies with Selenium. They provide 1- API key (username) 2- Proxy url/port. No password is needed. What I have tried... ChromeOptions ...
MattHodson's user avatar
1 vote
1 answer
85 views

Not able to scrape image URLs using beautiful soup and python

So basically I am using the below code to scrape the image urls of the credit cards from the respective links in the explore_more_url variable. from urllib.request import urlopen from bs4 import ...
user avatar
0 votes
1 answer
1k views

How to scrape card details using beautiful soup and python

I am trying to scrape this link : https://www.axisbank.com/retail/cards/credit-card Using the following code from urllib.request import urlopen from bs4 import BeautifulSoup import json, requests, re ...
user avatar
0 votes
1 answer
581 views

How to specify css selector in beautiful soup and python?

I am trying to scrape the titles of the cards from this link : https://www.axisbank.com/retail/cards/credit-card Using the below code from urllib.request import urlopen from bs4 import BeautifulSoup ...
user avatar
-2 votes
2 answers
141 views

Is there any alternative for \ in f string in python?

So I am scraping this website with link : https://www.americanexpress.com/in/credit-cards/payback-card/ using beautiful soup and python. link = 'https://www.americanexpress.com/in/credit-cards/payback-...
user avatar
0 votes
2 answers
199 views

Trying to scrape apply now and learn more urls but not able to get it using beautiful soup and python

I am scraping this link : https://www.americanexpress.com/in/credit-cards/all-cards/?sourcecode=A0000FCRAA&cpid=100370494&dsparms=dc_pcrid_408453063287_kword_american%20express%20credit%...
Ali Baba's user avatar
1 vote
1 answer
305 views

How can I scrape the image using Beautiful Soup and python

I am trying to scrape the image link from the below link but I am not able to Link : https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM I have ...
Ali Baba's user avatar
1 vote
2 answers
275 views

Trying to scrape image urls but not able to get it using beautiful soup and python

I am scraping this link : https://www.americanexpress.com/in/credit-cards/all-cards/?sourcecode=A0000FCRAA&cpid=100370494&dsparms=dc_pcrid_408453063287_kword_american%20express%20credit%...
Ali Baba's user avatar
0 votes
2 answers
255 views

How to scrape the heading and description using python and beautiful soup?

Overview of the problem: Link : https://www.bobfinancial.com/eterna.jsp In the Details Section: Basically I want all Points. details: [ #This is an array of Strings... "Milestone Rewards: ...
Ali Baba's user avatar
2 votes
0 answers
57 views

Is it possible to use a monitor on a script if it fails?

I use scrapinghub to run my spiders. I have a FinishReasonMonitor that slacks me if a spider fails. Is it possible to apply this to a script? My spiders rarely fail, but my scripts occasionally do. ...
weston6142's user avatar
0 votes
0 answers
65 views

Scrapy project working fine locally but not returning anything on scrapinghub, and not even showing any error

import json from json.decoder import JSONDecodeError import requests from ..items import PlayerstatisticsItem from scrapy.spiders import CrawlSpider import scrapy class PlayerStatsSpider(CrawlSpider): ...
Rafay Javed's user avatar
1 vote
2 answers
244 views

I am using scrapy to scrape data from Yelp. I cannot see any error but data is not getting scraped from the StartURLs mentioned in the spider

Code for the items.py and other files are mentioned below. The logs are also mentioned at the end.I am not getting any error but according to the logs the scrapy has not scraped any pages. ``` import ...
sneha s's user avatar
  • 11
2 votes
1 answer
123 views

Data flow template cant be created because Scrapinghub Client Library doesn't accept ValueProvider

I'm trying to create a data flow template that can be called from a cloud function that is triggered by a pubsub message. The pubsub message sends a job id from Scrapinghub (a platform for scrapy ...
pa-nguyen's user avatar
  • 407
0 votes
0 answers
137 views

Scrapinghub (Scrapy Cloud) - Is there a limit to the number of spiders per project?

I am using scrapy cloud to host & run my scrapy project. All seems to work well but when 11th spider is added to the project, it doesn't show up in the scrapinghub dashboard. I know this should be ...
Hamza Tasneem's user avatar
6 votes
1 answer
1k views

Scrapy hidden memory leak

Background - TLDR: I have a memory leak in my project Spent a few days looking through the memory leak docs with scrapy and can't find the problem. I'm developing a medium size scrapy project, ~40k ...
Hector Haffenden's user avatar
0 votes
1 answer
288 views

Scrapy Error : Missing scheme in request url

I am facing issues with some urls while running scrappy ValueError: Missing scheme in request url: mailto:?body=https%3A%2F%2Fiview.abc.net.au%2Fshow%2Finsiders [scrapy.core.scraper:168|ERROR] ...
user1633298's user avatar
0 votes
0 answers
99 views

Where to put logs in Scrapinghub? Scrapy

I am trying to put my logs in a "logs" folder but when I try to deploy to Scrapy I get No such file or directory: '/scrapinghub/sfb/logs/random_log.log' but I think I am declaring it ...
weston6142's user avatar
2 votes
0 answers
499 views

How to use crawlera proxies in selenium

I have a selenium project. I am going to use Crawlera proxy in selenium. I have already an API Key of Crawlera. headless_proxy = "127.0.0.1:3128" proxy = Proxy({ '...
pystack-piter's user avatar
0 votes
1 answer
48 views

No module found named toplevelfolder when importing github Scrapy project into Scrapinghub

When I am trying to import my Scrapy project onto Scrapinghub using the website through GitHub I get this error: ModuleNotFoundError: No module named 'sfb' Here is my project structure: requirements....
weston6142's user avatar
1 vote
1 answer
168 views

How to iterate through a list of Beautful soup tag elements and get a particular text if found else an empty string?

Case1: <li style="padding:5px;border-bottom:1px solid #ccc"> <div itemscope="" itemtype="http://schema.org/LocalBusiness"> <h5 itemprop="name"&...
dashkandhar's user avatar
0 votes
0 answers
173 views

How to use one crawler for multiple domains?

I'm working on a project that involves crawling multiple domains in their entirety. My scraper simply crawls the whole domain and doesn't check specific parts of the html, it just gets all the html. ...
Jake 1986's user avatar
  • 592
0 votes
0 answers
158 views

Failed to Get Scrapinghub Notification on Slack. Facing Error

I want to get Scrapinghub Spiders All (e.g Spider Run,Completion,Error) Notification on Slack. I have created Monitor.py file and Action.py file and also added spidermoon slack settings in my settings....
Hannan Zafar's user avatar
0 votes
1 answer
494 views

Unable to access scrapyd interface on the server machine with public IP

I am trying to run scrapyd my ubuntu server which has a public IP using the following config file named scrapy.cfg [settings] default = web_crawler.settings [deploy:default] url = http://127.0.0.1:...
Amanda's user avatar
  • 2,053
2 votes
3 answers
1k views

How to scrape a large amount (>800) Google My Maps location data ("Details from Google Maps") using Web Scraper or other alternatives?

I tried to use Web Scraper, but it only works for a few data entries not for hundreds of data points. Is there a way to scrape a large amount of data solely using Web Scraper or is there a better ...
cgybb's user avatar
  • 59
0 votes
0 answers
166 views

How to upload file in custom format to AWS S3 bucket using Scrapinghub?

I'm currently building a Python backend that will be deployed item's list to S3 bucket in the format I need using Scrapinghub service and Scrapy module. I successfully log into the website. Made ...
IK KLX's user avatar
  • 121
1 vote
0 answers
376 views

504 Timeout Exception when using scrapy-splash with crawlera

I tried scrapy-splash with http://www.google.com and followed all the prerequisite steps given in the following Github Repo https://github.com/scrapy-plugins/scrapy-splash and i was able to render the ...
Shashikiran Neelakantaiah's user avatar
0 votes
0 answers
410 views

How can I debug this? AttributeError: 'StdoutLogger' object has no attribute 'buffer'

My scrapy code gets this error on Scraping Hub, but not locally. AFAICT I'm using Python 3.8 and Scrapy 2.1.0 in both places. Anybody know what's going on here? I can't find the source to ...
Dogweather's user avatar
  • 15.8k
2 votes
1 answer
410 views

Scrapinghub Deploy Failed

I am trying to deploy a project to scrapinghub and here's the error I am getting slackclient 1.3.2 has requirement websocket-client<0.55.0,>=0.35, but you have websocket-client 0.57.0. Warning: ...
johncsmith427's user avatar
1 vote
1 answer
304 views

ScrapingHub Deploy Fails

I am trying to deploy to ScrapingHub and here is the error I am getting... Deploy log last 30 lines: File "/app/python/lib/python3.8/site-packages/scrapy/cmdline.py", line 142, in execute cmd....
johncsmith427's user avatar
0 votes
0 answers
48 views

Scrapinghub spider finishes and closes before task is done

I am using scrapinhub cloud with a splash instance to scrape content and images from a large list of urls that are provided with the spider. There are around 50 000 urls that I wish to crawl. The ...
BradleyB19's user avatar