Questions tagged [crawlee]
The crawlee tag has no usage guidance.
18
questions
0
votes
0
answers
5
views
Get the text content from a locator (Playwright and Crawlee) and default to an specific value if that locator is not found
I'm trying to develop a little crawling/scraping project with Crawlee and Playwright on JavaScript/TypeScript, For each URL I feed the crawler it tries to scrap some data like this:
...
0
votes
0
answers
38
views
Why does my web crawler work on localhost but not in the docker container?
I'm trying to create a web crawler using crawlee and apify With NodeJS. This crawler works when I run on the localhost, but when I run on docker container, I receive a timeout error waiting for ...
0
votes
0
answers
41
views
Crawlee no such file or directory storage/request_queues/default/[id].json
I'm trying to run a fairly simple scraper, but I keep getting the error in the title. I want to scrape around 64,000 pages, but I get the no such file error every time. Setting ...
0
votes
1
answer
41
views
How to get apify to make every request round-robin style?
Using Apify to crawl a job board, but with multiple concurrently. I have an array of proxies but my queued urls aren't using my proxies in a round robin fashion even though I use this setup. How can ...
0
votes
1
answer
82
views
How to scrape a webpage with infinite scroll using crawlee/apify?
I am trying to scrape some data from twitch, the problem I am facing is that the site uses infinite scroll and I am only able to get data from the first page.
I have tried to scroll by using the built ...
0
votes
0
answers
29
views
How to grab links only from sitemap?
I want to grab links only from the sitemap with Crawlee and not grab links from pages found in the sitemap. The main problem is that it starts from a sitemap and then follows all links on the newly ...
0
votes
0
answers
65
views
How to set data in localStorage by Crawlee Puppeteer? In my case it return Access is denied
I'm using Crawlee with Puppeteer. I want set data in localStorage of related page. So I used the following code
preNavigationHooks: [
async (crawlingContext,) => {
const { page, request }...
0
votes
0
answers
67
views
How to parallelly run several crawlers by Crawlee? I get some errors
I use Crawlee in my project.
I want run 2 crawler parallelly by this way:
await Promise.all([
crawler1.run(),
crawler2.run(),
]);
But I get this error:
ENOENT: no such file or directory, open 'D:\...
0
votes
0
answers
27
views
Compiling error importing Crawlee in react
I'm trying to integrate the crawlee library (https://crawlee.dev) inside my react app for a social scrape project, but as soon as I import the PlaywrightCrawler module I get the following compile ...
0
votes
1
answer
206
views
crawlee - How to add the same URL back to the requestQueue
How do i enqueue the same URL that i am currently handling the request for?
I have this code and want to scrape the same URL again (possibly with a delay), i added enviroment variables that cached ...
2
votes
2
answers
408
views
How to reset crawlee URL cache?
I'm running a crawler that is called via an expressjs call.
When I call the same route again, my crawler runs again but shows that all routes have already finished. I'm even removing the './storage' ...
0
votes
0
answers
105
views
How to access page.getby* functions inside crawlee
I'm using crawlee with PlaywrightCrawler. I'm getting a new url to crawl after clicking a few elements in the starting page. The way that i'm clicking those elements is using
page.getByRole().click(), ...
0
votes
0
answers
791
views
Reclaiming failed request back to the list or queue. waiting for selector `ul` failed: timeout 30000ms exceeded Puppeteer Crawler
WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. waiting for selector .show-more-less-html__markup failed: timeout 30000ms exceeded {"id":"nWhJU8Vn62Xzbr3&...
0
votes
1
answer
116
views
timeoutSecs for RequestQueue ignoring user config?
I use RequestQueue like so:
const requestQueue = await RequestQueue.open();
requestQueue.timeoutSecs = 60;
await requestQueue.addRequest(...);
but when running scraper I still see the ...
0
votes
1
answer
114
views
Number of dataset files incorrect
I am following this article. The number of files generated are sometimes < 3. I am expecting 000000001.json, 000000002.json and 000000003.json files to be generated in the storage/datasets/default
...
4
votes
1
answer
415
views
Extending Crawlee scraper requestHandler
I'm using [email protected], following the quick tutorial here to spin up a scraper.
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page,...
0
votes
1
answer
253
views
Electron Project: Crawlee Puppeteer can't find module "puppeteer" in compiled version
I recently added crawlee 3.0.2 to a project that has worked fine for months using a standard puppeteer setup (both with and without puppeteer-extra/stealth).
Everything runs fine when debugging and ...
1
vote
1
answer
1k
views
Blocking specific resources (css, images, videos, etc) using crawlee and playwright
I'm using [email protected] (not released yet, from github), and I'm trying to block specific resources from loading with playwrightUtils.blockRequests (which isn't available in previous versions). When I ...