7

I'm trying to scrape some web pages in the TOR network, using Puppeteer and the tor package (apt install tor). Probably due to the nature of TOR connections sometimes I get a timeout. In addition, I'm new to asynchronous programming in JavaScript.

Usually I have a try-catch-construct like these:

await Promise.all([
  page.goto(url),
  page.waitForNavigation({
    waitUntil: 'domcontentloaded'
  }),
]).catch((err) => { logMyErrors(err, true); });

or

let langMenu = await page.waitForXPath('//*[contains(@class, ".customer_name")]/ancestor::li').catch((err) => { logMyErrors(err, true); });

But I think often one or more retries would help to finally get the desired resource. Is there any best practice to implement retries?

4
  • It depends on your requirements. Possibilites are to increase the timeout and/or set a maximum number of retries before canceling and then repeating the request while its timeouting (?). So there is no law and you have to decide based on your goal/system/resources etc. Jul 4, 2019 at 9:04
  • Yeah, but in case of multiple retries I'm not sure about how to implement the retries without producing a mess of nested code. Or is it just too simple? How would you do 3 retries of the above let langMenuexample?
    – R0byn
    Jul 4, 2019 at 11:01
  • I have added an answer, is it working? Jul 4, 2019 at 13:01
  • I tried to integrate it in my code but didn't manage to get it work. But this will rather be due to my still insufficient knowledge concerning asynchronous JavaScript programming. I was able to integrate Thomas Dondorf's more compact suggestion.
    – R0byn
    Jul 5, 2019 at 9:54

3 Answers 3

14

I would recommend this rather simple approach:

async function retry(promiseFactory, retryCount) {
  try {
    return await promiseFactory();
  } catch (error) {
    if (retryCount <= 0) {
      throw error;
    }
    return await retry(promiseFactory, retryCount - 1);
  }
}

This function calls the promiseFactory, and waits for the returned Promise to finish. In case an error happens the process is (recursively) repeated until retryCount reaches 0.

Code Sample

You can use the function like this:

await retry(
  () => page.waitForXPath('//*[contains(@class, ".customer_name")]/ancestor::li'),
  5 // retry this 5 times
);

You can also pass any other function returning a Promise like Promise.all:

await retry(
  () => Promise.all([
    page.goto(url),
    page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
  ]),
  1 // retry only once
);

Don't combine await and catch

Another advice: You should not combine await with .then or .catch as this will result in unexpected problems. Either use await and surround your code with a try..catch block or use .then and .catch. Otherwise your code might be waiting for the results of a catch function to finish, etc.

Instead, you use try..catch like this:

try {
  // ...
} catch (error) {
  logMyErrors(error);
}
0
1

Workflow

  • Define a retry function
  • run retry function if promise rejects
  • control retries with retryCount

Concise copy+paste solution:

const GotoWithRetry = async (page, url, retryCount = 2) => {
  if (retryCount < 0) {
    throw new Error(`Failed to navigate to ${url} after 3 retries.`);
  }
  await Promise.all([
    page.goto(url, {
      timeout: 120 * 1000,
      waitUntil: 'load',
    }),
    page.waitForResponse((response) => response.ok(), { timeout: 8000 }),
  ]).catch(() => {
    GotoWithRetry(page, url, retryCount - 1);
  });
};
0

Heres an example of retrying without pyramid of doom. Im no ES expert though and there may be some new async/await features which could make the code more cleaner but this works for now:

function retry(callback, retries) {
    let tries = 0;

    function tryRequest() {
        tries++;
        return callback().catch(e => {
            logMyErrors(e);

            if (tries < retries) {
                return tryRequest();
            }
        });
    }

    return tryRequest();
}

const logMyErrors = console.log;


retry(() => {
    console.log("retry");
    return new Promise((resolve, reject) => {
        //Emulate some rejections here
        if (Math.random() > 0.2) {
            throw new Error("Something went wrong."); 
        }

        resolve("Success");
    });
}, 10).then((result) => {}, (rejected) => {});

The function retry accepts a callback which must return a promise. callback is executed until either the Promise finishes (resolves, rejects) or the amount of retries is reached.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.