5

I want to use Puppeteer to respond to page updates. The page shows items and when I leave the page open new items can appear over time. E.g. every 10 seconds a new item is added.

I can use the following to wait for an item on the initial load of the page:

await page.waitFor(".item");
console.log("the initial items have been loaded")

How can I wait for / catch future items? I would like to achieve something like this (pseudo code):

await page.goto('http://mysite');
await page.waitFor(".item");
// check items (=these initial items)

// event when receiving new items:
// check item(s) (= the additional [or all] items)

2 Answers 2

6

You can use exposeFunction to expose a local function:

await page.exposeFunction('getItem', function(a) {
    console.log(a);
});

Then you can use page.evaluate to create an observer and listen to new nodes created inside a parent node.

This example scrapes (it's just an idea, not a final work) the python chat in Stack Overflow, and prints new items being created in that chat.

var baseurl =  'https://chat.stackoverflow.com/rooms/6/python';
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto(baseurl);

await page.exposeFunction('getItem', function(a) {
    console.log(a);
});

await page.evaluate(() => {
    var observer = new MutationObserver((mutations) => { 
        for(var mutation of mutations) {
            if(mutation.addedNodes.length) {
                getItem(mutation.addedNodes[0].innerText);
            }
        }
    });
    observer.observe(document.getElementById("chat"), { attributes: false, childList: true, subtree: true });
});
1
0

As an alternative to the excellent current answer which injects a MutationObserver using evaluate which forwards the data to an exposed Node function, Puppeteer offers a higher-level function called page.waitForFunction that blocks on an arbitrary predicate and uses either a MutationObserver or requestAnimationFrame under the hood to determine when to re-evaluate the predicate.

Calling page.waitForFunction in a loop might add overhead since each new call involves registering a fresh observer or RAF. You'd have to profile for your use case -- this isn't something I'd worry much about prematurely, though.

That said, the RAF option may provide tighter latency than MO for the cost of some extra CPU cycles to poll constantly.

Here's a minimal example on the following site that offers a periodically updating feed:

const wait = ms => new Promise(r => setTimeout(r, ms));
const r = (lo, hi) => ~~(Math.random() * (hi - lo) + lo);

const randomString = n =>
  [...Array(n)].map(() => String.fromCharCode(r(97, 123))).join("")
;

(async () => {
  for (let i = 0; i < 500; i++) {
    const el = document.createElement("div");
    document.body.appendChild(el);
    el.innerText = randomString(r(5, 15));
    await wait(r(1000, 5000));
  }
})();

const puppeteer = require("puppeteer");

const html = `
<html><body><div class="container"></div><script>
const wait = ms => new Promise(r => setTimeout(r, ms));
const r = (lo, hi) => ~~(Math.random() * (hi - lo) + lo);
const randomString = n =>
  [...Array(n)].map(() => String.fromCharCode(r(97, 123))).join("")
;
(async () => {
  for (;;) {
    const el = document.createElement("div");
    document.querySelector(".container").appendChild(el);
    el.innerText = randomString(r(5, 15));
    await wait(r(1000, 5000));
  }
})();
</script></body></html>
`;
let browser;
(async () => {
  browser = await puppeteer.launch({headless: false});
  const [page] = await browser.pages();
  await page.setContent(html);
  
  for (;;) {
    await page.waitForFunction((el, oldLength) =>
      el.children.length > oldLength,                           // predicate
      {polling: "mutation" /* or: "raf" */, timeout: 0},        // wFF options
      await page.$(".container"),                               // elem to watch
      await page.$eval(".container", el => el.children.length), // oldLength
    );
    const selMostRecent = ".container div:last-child";
    console.log(await page.$eval(selMostRecent, el => el.textContent));
  }
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

See also:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.