Questions tagged [scrapinghub]
a web scraping development and services company, supplies cloud-based web crawling platforms.
179
questions
3
votes
0
answers
94
views
Issue with Docker Image for Custom Deployment on Zyte (formerly Scrapinghub) Platform
I am currently facing an issue while trying to create a Docker image for custom deployment on the Zyte platform (formerly known as Scrapinghub). My goal is to set up a Python environment with Scrapy, ...
0
votes
0
answers
19
views
Scrapy Cloud Deployment Error: "Referer Header Missing"
I'm working on a Scrapy project and I'm trying to deploy it to Scrapy Cloud (by Zyte, formerly known as Scrapinghub). However, during the deployment process, I encounter an error:
[INFO] Deploying ...
1
vote
1
answer
59
views
Webscraping yml files from Github
I am trying to scrape certain open source file from GitHub but I'm having an issue with their new format.
This if an example link: https://github.com/xavierLowmiller/xcodegen-action/blob/main/action....
0
votes
1
answer
73
views
I'm having issue while deploying scrapper to Zyte formerly (Scraping hub)
My spider has to read some data from input.csv file. It runs fine locally. But when I try to deploy it on Zyte by shub deploy it does not includes input.csv in build.
So when I try to run it on the ...
0
votes
1
answer
318
views
Extract data from company sharepoint using Python
Can I extract data from company's sharepoint using python.
Used power automate but I want to use python code
-1
votes
1
answer
81
views
Can't fetch url in scrapy shell with splash
Please help me!
When I try to fetch a URL in scrapy shell with scrapy splash, I use the following statement to get a response:
>>> fetch('http://localhost:8050/render.html?url=https://www....
-1
votes
1
answer
131
views
Is it possible to create a proxy failover with Python Scrapy?
Is it possible to create a proxy failover within Scrapy, so that when one fails the other will take over scraping the rest of the requests? I would of thought that it would be done using the retry ...
0
votes
0
answers
23
views
Are there any tools or 3rd parties - free or paid that can scrape URLs for Price?
I have a list of URLs and need auto update on the price found on website. Are there any tools or 3rd parties - free or paid that can scrape URLs for Price?
I have a list of URLs and need auto update ...
1
vote
0
answers
299
views
Web scraping using Octoparse
I have been trying to use Octoparse to scrape data from a particular webpage.
It has a total of 361 pages and 10 data rows on each page (total of 3610 data points). However, what I get is only 3260 ...
-1
votes
1
answer
59
views
How to get the XPATH or CSS selector from dynamically loaded website to follow links?
This is a dynamically-loaded website https://www.gelbeseiten.de/suche/hotels/n%c3%bcrnberg.
I'm trying to follow every link from the results. I found //article[@class='mod mod-Treffer']/a to follow ...
0
votes
1
answer
55
views
How to save Scrapy Broad Crawl Results?
Scrapy has a built-in way of persisting results in AWS S3 using the FEEDS setting.
but for a broad crawl over different domains this would create a single file, where the results from all domains are ...
0
votes
1
answer
236
views
Selenium Problem extracting Google business description
I seem to be struggling with this issue for a couple of days and could really use some help. I am trying to scrape Google busineses information with Python beautifulsoups and Selenium and I want to ...
0
votes
1
answer
186
views
Why error with installing csv when its part of python core package in scrapinghub
I have 3 spiders defined.
All the related requirements are mentioned in requirements.txt
scrapy
pandas
pytest
requests
google-auth
functions-framework
shub
msgpack-python
Also, the scrapinghub.yml ...
0
votes
1
answer
259
views
YouTube Subscriptions List Scraping
I want to scrap my YouTube subscriptions list into one csv file. I typed this code (but I didn't finish coding yet):
import requests
from bs4 import BeautifulSoup
import csv
url = 'https://www....
0
votes
3
answers
2k
views
(Scrapy) How do you scrape all the external links on each website from a list of hundreds of websites (and run the whole thing on Zyte)?
I am looking for some help regarding my Scrapy project.
I want to use Scrapy to code a generic Spider that would crawl multiple websites from a list. I was hoping to have the list in a separate file, ...
0
votes
2
answers
382
views
Scrapinghub scrapy: ModuleNotFoundError: No module named 'pandas'
I have tried deploying to Zyte via command line and GitHub but I have been stuck with the above error.
I have tried different versions of Scrapy version 1.5 to 2.5 but the error still persists.
I have ...
0
votes
1
answer
758
views
Scrapinghub/Zyte: Unhandled error in Deferred: No module named 'scrapy_user_agents'
I'm deploying my Scrapy spider via my local machine to Zyte Cloud (former ScrapingHub). This is successful. When I run the spider I get the output below.
I already checked here. The Zyte team is not ...
-1
votes
1
answer
108
views
How can I scrape a button that does not return a value
I am trying to scrape from the website https://tonaton.com/en/ads/ghana/electronics. There is a "next" button that I want to click and scrape the contents. The problem is the xpath or css ...
2
votes
1
answer
840
views
Scrapy crawlera authentication issue
I've been trying to use scrapy-crawlera as a proxy for scraping some data with scrapy. I've added these rows in settings.py:
DOWNLOADER_MIDDLEWARES = { 'scrapy_crawlera.CrawleraMiddleware': 610, }
...
1
vote
0
answers
63
views
print data on a sample image but the text is going out of image
ALL of the data in the picture is taken from an excel sheet, size of the image is 220×320, I cant increase the image size.
But I want Line 4 data that is exiting the image to go down below row 4.
...
1
vote
1
answer
570
views
Crawlera/Zyte proxy authentication using C# and Selenium
I've tried a number of ways of using Zyte (formally Crawerla) proxies with Selenium.
They provide
1- API key (username)
2- Proxy url/port.
No password is needed.
What I have tried...
ChromeOptions ...
1
vote
1
answer
85
views
Not able to scrape image URLs using beautiful soup and python
So basically I am using the below code to scrape the image urls of the credit cards from the respective links in the explore_more_url variable.
from urllib.request import urlopen
from bs4 import ...
0
votes
1
answer
1k
views
How to scrape card details using beautiful soup and python
I am trying to scrape this link : https://www.axisbank.com/retail/cards/credit-card
Using the following code
from urllib.request import urlopen
from bs4 import BeautifulSoup
import json, requests, re
...
0
votes
1
answer
581
views
How to specify css selector in beautiful soup and python?
I am trying to scrape the titles of the cards from this link : https://www.axisbank.com/retail/cards/credit-card
Using the below code
from urllib.request import urlopen
from bs4 import BeautifulSoup
...
-2
votes
2
answers
141
views
Is there any alternative for \ in f string in python?
So I am scraping this website with link : https://www.americanexpress.com/in/credit-cards/payback-card/
using beautiful soup and python.
link = 'https://www.americanexpress.com/in/credit-cards/payback-...
0
votes
2
answers
199
views
Trying to scrape apply now and learn more urls but not able to get it using beautiful soup and python
I am scraping this link : https://www.americanexpress.com/in/credit-cards/all-cards/?sourcecode=A0000FCRAA&cpid=100370494&dsparms=dc_pcrid_408453063287_kword_american%20express%20credit%...
1
vote
1
answer
305
views
How can I scrape the image using Beautiful Soup and python
I am trying to scrape the image link from the below link but I am not able to
Link : https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM
I have ...
1
vote
2
answers
275
views
Trying to scrape image urls but not able to get it using beautiful soup and python
I am scraping this link : https://www.americanexpress.com/in/credit-cards/all-cards/?sourcecode=A0000FCRAA&cpid=100370494&dsparms=dc_pcrid_408453063287_kword_american%20express%20credit%...
0
votes
2
answers
255
views
How to scrape the heading and description using python and beautiful soup?
Overview of the problem:
Link : https://www.bobfinancial.com/eterna.jsp
In the Details Section: Basically I want all Points.
details:
[ #This is an array of Strings...
"Milestone Rewards: ...
2
votes
0
answers
57
views
Is it possible to use a monitor on a script if it fails?
I use scrapinghub to run my spiders. I have a FinishReasonMonitor that slacks me if a spider fails. Is it possible to apply this to a script? My spiders rarely fail, but my scripts occasionally do. ...
0
votes
0
answers
65
views
Scrapy project working fine locally but not returning anything on scrapinghub, and not even showing any error
import json
from json.decoder import JSONDecodeError
import requests
from ..items import PlayerstatisticsItem
from scrapy.spiders import CrawlSpider
import scrapy
class PlayerStatsSpider(CrawlSpider):
...
1
vote
2
answers
244
views
I am using scrapy to scrape data from Yelp. I cannot see any error but data is not getting scraped from the StartURLs mentioned in the spider
Code for the items.py and other files are mentioned below. The logs are also mentioned at the end.I am not getting any error but according to the logs the scrapy has not scraped any pages.
```
import ...
2
votes
1
answer
123
views
Data flow template cant be created because Scrapinghub Client Library doesn't accept ValueProvider
I'm trying to create a data flow template that can be called from a cloud function that is triggered by a pubsub message. The pubsub message sends a job id from Scrapinghub (a platform for scrapy ...
0
votes
0
answers
137
views
Scrapinghub (Scrapy Cloud) - Is there a limit to the number of spiders per project?
I am using scrapy cloud to host & run my scrapy project. All seems to work well but when 11th spider is added to the project, it doesn't show up in the scrapinghub dashboard.
I know this should be ...
6
votes
1
answer
1k
views
Scrapy hidden memory leak
Background - TLDR: I have a memory leak in my project
Spent a few days looking through the memory leak docs with scrapy and can't find the problem.
I'm developing a medium size scrapy project, ~40k ...
0
votes
1
answer
288
views
Scrapy Error : Missing scheme in request url
I am facing issues with some urls while running scrappy
ValueError: Missing scheme in request url: mailto:?body=https%3A%2F%2Fiview.abc.net.au%2Fshow%2Finsiders
[scrapy.core.scraper:168|ERROR] ...
0
votes
0
answers
99
views
Where to put logs in Scrapinghub? Scrapy
I am trying to put my logs in a "logs" folder but when I try to deploy to Scrapy I get No such file or directory: '/scrapinghub/sfb/logs/random_log.log' but I think I am declaring it ...
2
votes
0
answers
499
views
How to use crawlera proxies in selenium
I have a selenium project. I am going to use Crawlera proxy in selenium. I have already an API Key of Crawlera.
headless_proxy = "127.0.0.1:3128"
proxy = Proxy({
'...
0
votes
1
answer
48
views
No module found named toplevelfolder when importing github Scrapy project into Scrapinghub
When I am trying to import my Scrapy project onto Scrapinghub using the website through GitHub I get this error:
ModuleNotFoundError: No module named 'sfb'
Here is my project structure:
requirements....
1
vote
1
answer
168
views
How to iterate through a list of Beautful soup tag elements and get a particular text if found else an empty string?
Case1:
<li style="padding:5px;border-bottom:1px solid #ccc">
<div itemscope="" itemtype="http://schema.org/LocalBusiness">
<h5 itemprop="name"&...
0
votes
0
answers
173
views
How to use one crawler for multiple domains?
I'm working on a project that involves crawling multiple domains in their entirety. My scraper simply crawls the whole domain and doesn't check specific parts of the html, it just gets all the html.
...
0
votes
0
answers
158
views
Failed to Get Scrapinghub Notification on Slack. Facing Error
I want to get Scrapinghub Spiders All (e.g Spider Run,Completion,Error) Notification on Slack.
I have created Monitor.py file and Action.py file and also added spidermoon slack settings in my settings....
0
votes
1
answer
494
views
Unable to access scrapyd interface on the server machine with public IP
I am trying to run scrapyd my ubuntu server which has a public IP using the following config file named scrapy.cfg
[settings]
default = web_crawler.settings
[deploy:default]
url = http://127.0.0.1:...
2
votes
3
answers
1k
views
How to scrape a large amount (>800) Google My Maps location data ("Details from Google Maps") using Web Scraper or other alternatives?
I tried to use Web Scraper, but it only works for a few data entries not for hundreds of data points. Is there a way to scrape a large amount of data solely using Web Scraper or is there a better ...
0
votes
0
answers
166
views
How to upload file in custom format to AWS S3 bucket using Scrapinghub?
I'm currently building a Python backend that will be deployed item's list to S3 bucket in the format I need using Scrapinghub service and Scrapy module.
I successfully log into the website. Made ...
1
vote
0
answers
376
views
504 Timeout Exception when using scrapy-splash with crawlera
I tried scrapy-splash with http://www.google.com and followed all the prerequisite steps given in the following Github Repo https://github.com/scrapy-plugins/scrapy-splash and i was able to render the ...
0
votes
0
answers
410
views
How can I debug this? AttributeError: 'StdoutLogger' object has no attribute 'buffer'
My scrapy code gets this error on Scraping Hub, but not locally. AFAICT I'm using Python 3.8 and Scrapy 2.1.0 in both places.
Anybody know what's going on here? I can't find the source to ...
2
votes
1
answer
410
views
Scrapinghub Deploy Failed
I am trying to deploy a project to scrapinghub and here's the error I am getting
slackclient 1.3.2 has requirement websocket-client<0.55.0,>=0.35, but you have websocket-client 0.57.0.
Warning: ...
1
vote
1
answer
304
views
ScrapingHub Deploy Fails
I am trying to deploy to ScrapingHub and here is the error I am getting...
Deploy log last 30 lines:
File "/app/python/lib/python3.8/site-packages/scrapy/cmdline.py", line 142, in execute
cmd....
0
votes
0
answers
48
views
Scrapinghub spider finishes and closes before task is done
I am using scrapinhub cloud with a splash instance to scrape content and images from a large list of urls that are provided with the spider. There are around 50 000 urls that I wish to crawl.
The ...