I'm trying to scrape some web pages in the TOR network, using Puppeteer and the tor package (apt install tor
).
Probably due to the nature of TOR connections sometimes I get a timeout.
In addition, I'm new to asynchronous programming in JavaScript.
Usually I have a try-catch-construct like these:
await Promise.all([
page.goto(url),
page.waitForNavigation({
waitUntil: 'domcontentloaded'
}),
]).catch((err) => { logMyErrors(err, true); });
or
let langMenu = await page.waitForXPath('//*[contains(@class, ".customer_name")]/ancestor::li').catch((err) => { logMyErrors(err, true); });
But I think often one or more retries would help to finally get the desired resource. Is there any best practice to implement retries?
let langMenu
example?