Questions tagged [crawlee]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

Get the text content from a locator (Playwright and Crawlee) and default to an specific value if that locator is not found

I'm trying to develop a little crawling/scraping project with Crawlee and Playwright on JavaScript/TypeScript, For each URL I feed the crawler it tries to scrap some data like this: ...
Estratachuela's user avatar
0 votes
0 answers
38 views

Why does my web crawler work on localhost but not in the docker container?

I'm trying to create a web crawler using crawlee and apify With NodeJS. This crawler works when I run on the localhost, but when I run on docker container, I receive a timeout error waiting for ...
Eduardo Queiroz's user avatar
0 votes
0 answers
41 views

Crawlee no such file or directory storage/request_queues/default/[id].json

I'm trying to run a fairly simple scraper, but I keep getting the error in the title. I want to scrape around 64,000 pages, but I get the no such file error every time. Setting ...
sbrass's user avatar
  • 915
0 votes
1 answer
41 views

How to get apify to make every request round-robin style?

Using Apify to crawl a job board, but with multiple concurrently. I have an array of proxies but my queued urls aren't using my proxies in a round robin fashion even though I use this setup. How can ...
Justin Young's user avatar
  • 2,393
0 votes
1 answer
82 views

How to scrape a webpage with infinite scroll using crawlee/apify?

I am trying to scrape some data from twitch, the problem I am facing is that the site uses infinite scroll and I am only able to get data from the first page. I have tried to scroll by using the built ...
Syed Hasnain's user avatar
0 votes
0 answers
29 views

How to grab links only from sitemap?

I want to grab links only from the sitemap with Crawlee and not grab links from pages found in the sitemap. The main problem is that it starts from a sitemap and then follows all links on the newly ...
user3389's user avatar
  • 489
0 votes
0 answers
65 views

How to set data in localStorage by Crawlee Puppeteer? In my case it return Access is denied

I'm using Crawlee with Puppeteer. I want set data in localStorage of related page. So I used the following code preNavigationHooks: [ async (crawlingContext,) => { const { page, request }...
Ramin Bateni's user avatar
  • 16.6k
0 votes
0 answers
67 views

How to parallelly run several crawlers by Crawlee? I get some errors

I use Crawlee in my project. I want run 2 crawler parallelly by this way: await Promise.all([ crawler1.run(), crawler2.run(), ]); But I get this error: ENOENT: no such file or directory, open 'D:\...
Ramin Bateni's user avatar
  • 16.6k
0 votes
0 answers
27 views

Compiling error importing Crawlee in react

I'm trying to integrate the crawlee library (https://crawlee.dev) inside my react app for a social scrape project, but as soon as I import the PlaywrightCrawler module I get the following compile ...
samuele nencioni's user avatar
0 votes
1 answer
206 views

crawlee - How to add the same URL back to the requestQueue

How do i enqueue the same URL that i am currently handling the request for? I have this code and want to scrape the same URL again (possibly with a delay), i added enviroment variables that cached ...
Jaanis's user avatar
  • 39
2 votes
2 answers
408 views

How to reset crawlee URL cache?

I'm running a crawler that is called via an expressjs call. When I call the same route again, my crawler runs again but shows that all routes have already finished. I'm even removing the './storage' ...
Justin Young's user avatar
  • 2,393
0 votes
0 answers
105 views

How to access page.getby* functions inside crawlee

I'm using crawlee with PlaywrightCrawler. I'm getting a new url to crawl after clicking a few elements in the starting page. The way that i'm clicking those elements is using page.getByRole().click(), ...
matrs's user avatar
  • 59
0 votes
0 answers
791 views

Reclaiming failed request back to the list or queue. waiting for selector `ul` failed: timeout 30000ms exceeded Puppeteer Crawler

WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. waiting for selector .show-more-less-html__markup failed: timeout 30000ms exceeded {"id":"nWhJU8Vn62Xzbr3&...
Sachin Chillal's user avatar
0 votes
1 answer
116 views

timeoutSecs for RequestQueue ignoring user config?

I use RequestQueue like so: const requestQueue = await RequestQueue.open(); requestQueue.timeoutSecs = 60; await requestQueue.addRequest(...); but when running scraper I still see the ...
susdu's user avatar
  • 852
0 votes
1 answer
114 views

Number of dataset files incorrect

I am following this article. The number of files generated are sometimes < 3. I am expecting 000000001.json, 000000002.json and 000000003.json files to be generated in the storage/datasets/default ...
Rajesh's user avatar
  • 91
4 votes
1 answer
415 views

Extending Crawlee scraper requestHandler

I'm using [email protected], following the quick tutorial here to spin up a scraper. import { PlaywrightCrawler } from 'crawlee'; const crawler = new PlaywrightCrawler({ requestHandler: async ({ page,...
toanphan19's user avatar
0 votes
1 answer
253 views

Electron Project: Crawlee Puppeteer can't find module "puppeteer" in compiled version

I recently added crawlee 3.0.2 to a project that has worked fine for months using a standard puppeteer setup (both with and without puppeteer-extra/stealth). Everything runs fine when debugging and ...
drewness's user avatar
  • 109
1 vote
1 answer
1k views

Blocking specific resources (css, images, videos, etc) using crawlee and playwright

I'm using [email protected] (not released yet, from github), and I'm trying to block specific resources from loading with playwrightUtils.blockRequests (which isn't available in previous versions). When I ...
matrs's user avatar
  • 59