Questions tagged [web-crawler]

A Web crawler (also known as Web spider) is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, Web spiders, Web robots, or – especially in the FOAF community – Web scutters.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Empty Content in HTTP Response despite Receiving 200 OK Status [closed]

I'm attempting to scrape data from a game website where pressing a button triggers an HTTP response that contains JSON data. I'm using Python to simulate the HTTP response headers and send a POST ...
Yu Xing's user avatar
  • 49
0 votes
1 answer
10 views

Extrapolate missing sitemap links

We are exploring the sitemaps of several websites from their robots.txt. We are seeing that often the sitemap does not contain the complete map of the website. In many cases, only few pages are ...
NedStarkOfWinterfell's user avatar
-2 votes
0 answers
21 views

How to check if instagram account is business account or normal account?

I am using Selenium in Python to scrape the followers of particular Instagram account. I want to know if the account is a business account or a normal account. In the past, Instagram used to have this ...
RudraSama's user avatar
0 votes
0 answers
19 views

Web Crawler : How to optimize processing efficiently in case each URL Page is iterated by client [closed]

I am writing a possible implementation of Web Crawler. However, the crawler functionality has been enclosed within a class library. The client of the class library will be using the Next method of the ...
Usman's user avatar
  • 2,729
0 votes
2 answers
69 views

Why BeautifulSoup find_all not returning elements with <br> in them?

Environment: Python 3.9.4 beautifulsoup4==4.12.2 Code: from bs4 import BeautifulSoup test_content = '''<html><head></head><body><p>123</p><p>123<br>...
wings's user avatar
  • 799
0 votes
0 answers
14 views

How to get the Xpath from the script tag

Eg- window.run.params={ Data: { -----------------} } The challenge is to get the Xpath from the script tag which holds all the hidden data in the webpage. I have tried to inspect the webpage and get ...
archana's user avatar
0 votes
1 answer
15 views

Node.js Promises returning undefined for some elements, what am I doing wrong or what can I improve?

Some of my promises are returning "undefined" and I can't see what I'm doing wrong. Tried to add a timeout but didn't solve it, I'm still learing and struggling to see my error. it looks ...
Guido Aguiar's user avatar
0 votes
0 answers
23 views

How can I scrape, crawl, collect results from Google search with JSoup

It already worked a few weeks before, but now I am getting a page with the choice to accept or reject cookies. My code: String useragent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, ...
Peter Graf's user avatar
0 votes
1 answer
43 views

Requests to tesla inventory api timeout with request.get in python

I am writing a python web crawler to Tesla inventory with url = "https://www.tesla.com/inventory/api/v1/inventory-results?query={%22query%22:{%22model%22:%22my%22,%22condition%22:%22new%22,%...
ivory's user avatar
  • 243
2 votes
1 answer
28 views

How to get element in MathJax when crawl data?

I'm crawling a website that has the following tags by "cheerio", how can I get the entire text of the p tag and also **span * *with attribute "data-mathml". <p><strong ...
Mai Đức Thắng's user avatar
-1 votes
0 answers
23 views

How to use Python crawler to automatically log in to WeChat public accounts with QR codes

I now have a need to crawl WeChat official account articles. Before crawling, I must log in to the WeChat official account. How to use Python crawler to automatically log in to the WeChat official ...
likai's user avatar
  • 1
0 votes
1 answer
80 views

ERR_HTTP2_PROTOCOL_ERROR error when opening a website using playwright in nestjs

Error: page.goto: net::ERR_HTTP2_PROTOCOL_ERROR at https://example.com const browser = await chromium.launch({ headless: false, args: ['--disable-web-security'], }); const page = await browser....
Richard's user avatar
0 votes
1 answer
46 views

How can I limit the number of pages scraped with Scrapy CrawlSpider?

I want to limit the number of pages scraped to 5 with below code although the website has 50 pages. I'm using Scrapy's CrawlSpider. How can I do that? from scrapy.linkextractors import LinkExtractor ...
Bilal Anees's user avatar
0 votes
0 answers
6 views

Pyppeteer on GCP VM Instance Not Detecting Network Requests

I'm using Pyppeteer on a GCP VM instance to monitor network requests from a specific website. The code works perfectly on my local machine, capturing all the network traffic as expected. However, when ...
윤태일's user avatar
  • 537
0 votes
1 answer
37 views

Webcrawler for Testing und learn

Hi I wanted to try to program a crawler. I started with a very simple code but already when I execute it I get an error message. What is wrong with the code? I geht this Error at the source point. ...
D1skanime's user avatar
-1 votes
0 answers
39 views

Passing an argument to a Scrapy spider using CrawlerRunner

I'm working on a function where I'm getting the output from the first spider and trying to pass the output to the other spider, but I'm not able figure out how to do it with CrawlerRunner. I tested it ...
kunalmehta14's user avatar
1 vote
1 answer
25 views

How to scrape data from a website with multiple pages with Python

I am trying to scrape the data table from the website Vietnamese drug bank I am trying to scrape it with Python howevere it did not respone import requests import pandas as pd params = { 'size': ...
Linh Tuấn's user avatar
1 vote
1 answer
31 views

How to select an option from fiba page using playwright

I am trying to get the fiba world cup 2023 team stats where my country (Philippines) is one of the host nations. The url of the page is: https://www.fiba.basketball/basketballworldcup/2023/teamstats ...
ferdy's user avatar
  • 4,416
0 votes
1 answer
18 views

Bot detection changing values of JSON data before returning, any ideas how to get around this?

Probably a long shot, but I am trying to scrape Hyatt's website for points availabilities using Selenium-Profiles. https://www.hyatt.com/shop/service/rooms/roomrates/dpsas?spiritCode=dpsas&rooms=1&...
burnmypoints's user avatar
1 vote
1 answer
35 views

Opening a link after manually building it works, but from code it does not

I have a website that uses 2 API calls in order to build the actual link to download a gzip file, the problem is that the headers are changing a lot I think and the cookies too, I tried finding out ...
Dolev Mitz's user avatar
0 votes
1 answer
23 views

Get `scrapy` to produce a nested datastructure

I am using scrapy to crawl this website and scrape data I want the scraped data to have a nested structure. something like this { denomination1: { date1: { bondNumbers:...
hammi's user avatar
  • 847
-1 votes
1 answer
40 views

Scrapy Spider returns only first product for every page

I uses scrapy to scrap products from amazon but It only returns first product for each page and go to next page what wrong with my code import scrapy from ..items import AmazonItem class ...
Muhammad Kamran's user avatar
8 votes
2 answers
1k views

Error when crawl data: 'EPollReactor' object has no attribute '_handleSignals'

I am trying to crawl data from a list of URLs. I have already done with the code below and succeeded yesterday without any error. But today, when I came back and ran the code again, there was an error ...
Yến Phan's user avatar
0 votes
0 answers
38 views

Why does my web crawler work on localhost but not in the docker container?

I'm trying to create a web crawler using crawlee and apify With NodeJS. This crawler works when I run on the localhost, but when I run on docker container, I receive a timeout error waiting for ...
Eduardo Queiroz's user avatar
1 vote
1 answer
38 views

`scrapy` can't get response from a website but `requests` can

I am using scrapy to crawl this page but for some reason scrapy cannot receive a response from this website. when i run the crawler I receive https 500 error here is my basic spider import scrapy ...
hammi's user avatar
  • 847
0 votes
1 answer
18 views

How to click the icon without herf link with selenium

I'm a newcomer of programming and I'm trying to crawl this site. I'm sorry that the site is only for the Korean. What i want to is move to next page in search using selenium (python). At the other ...
Sungwon Min's user avatar
0 votes
1 answer
55 views

How to Scrape PressReader Website for News with Horizontal Scrolling Pagination

I am trying to scrape news articles from the PressReader website. Each page on the site only displays a few news articles, and to see more articles, I need to scroll horizontally (left or right) ...
Shu Cheng's user avatar
-3 votes
2 answers
93 views

"all" or "index,follow" in robots meta tag?

Is this a correct way to allow the Google bot to crawl our website? <meta name="robots" content="all"/> What is better to allow the Google bot by using? <meta name="...
peter Kane's user avatar
0 votes
1 answer
18 views

Scrapy Problem - Whan I ran scrapy file, there is no output

` import scrapy class AudibleSpider(scrapy.Spider): name = 'audible' allowed_domains = ['www.audible.com'] def start_requests(self): yield scrapy.Request( url='...
Sazzad Hossain Nibir's user avatar
0 votes
1 answer
40 views

How to use rotating proxies while using google_play_scraper python library to avoid IP address getting blocked?

I am trying to scrape reviews from google play store using the gooogle_play_scraper library. When trying to retrieve reviews for multiple apps, server blocks the server communication to the play store ...
Saran Divakaran's user avatar
1 vote
1 answer
105 views

Scrapy - crawling archives of website plus all subdirectories

So I'm trying to scrape data from archived versions of a website using Scrapy. Here is my code: import scrapy from scrapy.crawler import * from scrapy.item import * from scrapy.linkextractors import * ...
El Ectric's user avatar
  • 152
0 votes
0 answers
12 views

How to trigger a route inside a react app in production mode using console

I'm developing a web crawler for a website that utilizes ReactJS. This script must navigate through multiple pages. The issue is that their app is heavy, causing each request to take a few seconds to ...
user3362882's user avatar
0 votes
1 answer
105 views

Getting error while Using ONLYOFFICE with React.js

i'm using onlyOffice to render with web while i'm hiting the right documentServerUrl getting error (Error load DocsAPI from http://confidential.com/example/) and but i don't no why it is crawing on ...
Yogesh Dhengale's user avatar
0 votes
2 answers
108 views

Element not Interactable - Selenium headless chrome=115.0.5790.170 - How to solve this?

I'm having a problem using selenium with the headless chrome=115.0.5790.170, I have a table and a td with this a tag. <a id="formSegundaViaFatura:dtListaSegundaViaFaturaDebitoPendente:0:...
Igor Brizack's user avatar
0 votes
1 answer
24 views

Problem occurred while web crawling instagram

def insta_searching(word): url = "https://www.instagram.com/explore/tags/" + str(word) return url def select_first(driver): first = driver.find_element_by_css_selector("div....
jeonghyeon's user avatar
1 vote
1 answer
35 views

Why isn't this website loading cookies properly with Selenium + Python?

I'm doing some webscraping and I actually have a problem with my code. All I wanna do is: Entering the website https://solucoes.receita.fazenda.gov.br/Servicos/cnpjreva/cnpjreva_solicitacao.asp ...
Paulo Silva's user avatar
0 votes
1 answer
58 views

Googlebot does not find ads.txt file

I'm trying to get google adsense on my website, but every time google crawls my website it says that the ads.txt status is "not found." I can view the ads.txt file on many accounts by going ...
ChaseTheCoder's user avatar
0 votes
1 answer
59 views

Unable to export in JSON file

The following code does export in CSV file format, but it does not export in JSON format. I don't know what the problem is maybe it is because I can't use crawl syntax in my scrapy spider and I use ...
umer's user avatar
  • 1
0 votes
0 answers
18 views

problem in web crawling using cheerio and axios

const axios = require('axios'); const cheerio = require('cheerio'); axios({ // URL url: 'http://nstatic.dcinside.com/company/rules/service_policy.html', method: 'GET', }) // success ...
calvin0907's user avatar
0 votes
0 answers
32 views

Chrome driver does not update data from server(content in the browser not changed) after clicking as when I click manual on Chrome

When I open manual url on Chrome, the first page is: and click manually to move the next page of stock list, I see after loading => stock list are updated but when I run my code and auto open ...
Pham Ngoc Dinh's user avatar
0 votes
1 answer
58 views

Selenium driver does not reflect the page changes after click()

I want to get stock list on etoro. But when I trying to click on selenium => After making click(), I want to get the elements already updated but the elements all the same as before click() from ...
Pham Ngoc Dinh's user avatar
0 votes
1 answer
46 views

Extract number of products from a supermarket webpage depending on postcode

I am trying to extract the number of products available on this webpage: https://www.sklavenitis.gr/eidi-artozacharoplasteioy/keik-tsoyrekia-kroyasan/ I have code using Scrapy which does manage to do ...
Starski's user avatar
0 votes
0 answers
62 views

TradingView crawler API about the account profile and post/comment content

According to this paper Misbehavior and Account Suspension in an Online Financial Communication Platform They access the data of TradingView from APIs, especially the suspended account would return ...
KKK's user avatar
  • 1
0 votes
0 answers
24 views

How to download a PDF file with Selenium?

I'm trying to download the PDF from the following website: http://esaj.tjsp.jus.br/cjsg/getArquivo.do?conversationId=&cdAcordao=16548741 So, the first thing I do is to create a time.sleep to ...
Claudio's user avatar
0 votes
0 answers
29 views

The python request was detected by server-side anti-crawlers

When I was crawling a website, my python request was detected by the server and did not return the correct data, but my python request went through the Charles local proxy and I was able to get the ...
hrhhr's user avatar
  • 31
-1 votes
3 answers
86 views

Google shows old title, Does anyone have an idea how to change it?

enter image description hereWhen typing in the name of the company, the old title is still shown by search engines. We couldnt figure out how that happens. The page has been freshly crawled however ...
Danzer's user avatar
  • 1
0 votes
0 answers
73 views

Selenium - undetected_chromedriver python: How to load a packed .crx extension file to undetected_chromedriver?

I know how to load an unpacked/extracted extension file but Is there any way to load an extension in .crx format within undetected_chromedriver? I have been trying to automate a chrome extension and ...
Baki's user avatar
  • 1
0 votes
0 answers
41 views

Crawlee no such file or directory storage/request_queues/default/[id].json

I'm trying to run a fairly simple scraper, but I keep getting the error in the title. I want to scrape around 64,000 pages, but I get the no such file error every time. Setting ...
sbrass's user avatar
  • 915
0 votes
0 answers
124 views

Disable web crawling for Nextjs app from robots.txt for particular sub-domain only

I have my website deployed on vercel, The site is a Next js application directly (not using nginx or any other web serving servers) deployed on vercel. There are two domains assigned to the same ...
Anand Yadav's user avatar
0 votes
1 answer
22 views

How can scrape "Accessibility" part of Chrome Developer Tools? (Python)

enter image description here (Please check out the image description above) Hello, I am scraping some websites with Python. It is an active web so I failed to scrape some contents with the libraries, ...
Jaemoo Hong's user avatar

1
2 3 4 5
194