128 votes
Accepted

How to delete existing text from input using Puppeteer?

You can use page.evaluate to manipulate DOM as you see fit: await page.evaluate( () => document.getElementById("inputID").value = "") However sometimes just manipulating a ...
Vaviloff's user avatar
  • 16.6k
100 votes
Accepted

How can I set the browser window size when using `google-chrome --headless`?

I found it. Simply pass the --window-size command line argument to Google Chrome, for example --window-size=1920,1080. In a Protractor configuration this would look like this: capabilities: { ...
Leo Gallucci's user avatar
  • 16.6k
71 votes
Accepted

Headless automation with Nodejs Selenium Webdriver

Updated answer circa FEB-2018. Referencing the Selenium Webdriver NodeJS Examples (commit 5bf50c4) const chrome = require('selenium-webdriver/chrome'); const firefox = require('selenium-webdriver/...
Josh Peak's user avatar
  • 6,056
66 votes

Make Chrome Headless to Wait for Ajax Before Printing to PDF

chrome --headless http://localhost:8080/banana/key --run-all-compositor-stages-before-draw --print-to-pdf=C:\\tmp\\tmp.pdf --virtual-time-budget=10000 Try this, virtual-time-budget is the parameter ...
Izayoi Sakuya's user avatar
59 votes

Collect elements by class name and then click each one - Puppeteer

Iterating puppeteer async methods in for loop vs. Array.map()/Array.forEach() As all puppeteer methods are asynchronous it doesn't matter how we iterate over them. I've made a comparison and a rating ...
theDavidBarton's user avatar
53 votes
Accepted

Limit chrome headless CPU and memory usage

There had been a lot of discussion going around about the unpredictable CPU and Memory Consumption by Chrome Headless sessions. As per the discussion Building headless for minimum cpu+mem usage the ...
undetected Selenium's user avatar
46 votes
Accepted

Want to scrape table using Puppeteer. How can I get all rows, iterate through rows, and then get "td's" for each row?

One way to achieve this is to use evaluate that first gets an array of all the TDs then returns the textContent of each TD: const puppeteer = require('puppeteer'); const html = ` <html> <...
Rippo's user avatar
  • 22.3k
45 votes
Accepted

Puppeteer Button Press

I've figured it out finally. I found inside that same form an anchor element whose type was submit. I then clicked on it and the form was submitted. Here's the code I've used: const form = await ...
elena's user avatar
  • 3,950
40 votes

Want to scrape table using Puppeteer. How can I get all rows, iterate through rows, and then get "td's" for each row?

Two-Dimensional Array Method You can also scrape the innerText into a two-dimensional array representing your table. [ ['A1', 'B1', 'C1'], // Row 1 ['A2', 'B2', 'C2'], // Row 2 ['A3', 'B3', 'C3']...
Grant Miller's user avatar
  • 28.4k
40 votes
Accepted

Puppeteer: How to get the contents of each element of a nodelist?

page.$$(): You can use a combination of elementHandle.getProperty() and jsHandle.jsonValue() to obtain the innerText from an ElementHandle obtained with page.$$(): const tweets = await page.$$('....
Grant Miller's user avatar
  • 28.4k
40 votes
Accepted

Headless Chrome Node API and Puppeteer installation

Based on https://github.com/GoogleChrome/puppeteer You only have to run the following command in Ubuntu 18.04 npm i puppeteer Unfortunately this is not enough. You will require the following ...
CodeGuru's user avatar
  • 3,705
39 votes
Accepted

How to get all links from the DOM?

In your example code you're using page.$eval, not page.$$eval. Since the former uses document.querySelector instead of document.querySelectorAll, the behaviour you describe is the expected one. Also, ...
Miguel Calderón's user avatar
36 votes

Detect Chrome running in headless mode from JavaScript

The user agent string includes HeadlessChrome instead of Chrome. This is probably the signal that you're intended to look for, so you could use: /\bHeadlessChrome\//.test(navigator.userAgent) Other ...
Josh Lee's user avatar
  • 175k
33 votes

How to delete existing text from input using Puppeteer?

You can use the page.keyboard methods to change input values, or you can use page.evaluate(). Replace All Text: // Using page.keyboard: await page.focus('#example'); await page.keyboard.down('...
Grant Miller's user avatar
  • 28.4k
31 votes
Accepted

How can I dynamically inject functions to evaluate using Puppeteer?

You can add function to page context with addScriptTag: const browser = await puppeteer.launch(); const page = await browser.newPage(); function functionToInject (){ return 1+1; } function ...
Everettss's user avatar
  • 15.8k
28 votes

How to avoid being detected as bot on Puppeteer and Phantomjs?

If anyone need in future for the same problem. Using puppeteer-extra I have tested the code on a server. On 2nd run there is google Captcha. You can solve it your self and restart the bot or use a ...
Shimul D's user avatar
  • 443
27 votes

Collect elements by class name and then click each one - Puppeteer

Use page.evaluate to execute JS: const puppeteer = require('puppeteer'); puppeteer.launch().then(async browser => { const page = await browser.newPage(); await page.evaluate(() => { ...
chridam's user avatar
  • 102k
27 votes

Headless browser detection

There is a headless browser detection test which tests for the following: Does the User-Agent contain the string "HeadlessChrome"? Is navigator.webdriver set? Is window.chrome unset? Does ...
Thomas Dondorf's user avatar
25 votes

How to hide Chrome Driver in python?

Google announced in 4/2017 you can run in headless. https://developers.google.com/web/updates/2017/04/headless-chrome chrome_options = Options() # Chrome v75 and lower: # chrome_options.add_argument(&...
sealabr's user avatar
  • 1,611
25 votes

Why can't I access 'window' in an exposeFunction() function with Puppeteer?

evaluate the function You can pass the dynamic script using evaluate. (async function(){ var puppeteer = require('puppeteer'); const browser = await puppeteer.launch(); const page = await ...
Md. Abu Taher's user avatar
24 votes
Accepted

Difference of Headless browsers for automation

Browser A Browser is an application program that provides a way to look at and interact with all the information on the World Wide Web. Technically a Browser, alternatively referred as a Web Browser ...
undetected Selenium's user avatar
23 votes

How to avoid being detected as bot on Puppeteer and Phantomjs?

Things that can help in general : Headers should be similar to common browsers, including : User-Agent : use a recent one (see https://developers.whatismybrowser.com/useragents/explore/), or better, ...
Grubshka's user avatar
  • 573
23 votes
Accepted

Function similar to document.ready() in Puppeteer?

You can use page.waitForNavigation() as an alternative to jQuery's document.ready() function: await page.waitForNavigation({waitUntil: 'load'}); // consider navigation to be finished when ...
Grant Miller's user avatar
  • 28.4k
20 votes
Accepted

Detect Chrome running in headless mode from JavaScript

You can check navigator.webdriver property which is: The webdriver read-only property of the navigator interface indicates whether the user agent is controlled by automation. ... The ...
Justinas Jakavonis's user avatar
20 votes
Accepted

How can I pause and wait for user input with Puppeteer?

You can use page.waitForFunction() to wait for a function to return a truthy value before continuing. The following function will wait until the #username and #password fields contain a value before ...
Grant Miller's user avatar
  • 28.4k
20 votes
Accepted

How to point Puppeteer to local images/fonts?

The page URL is about:blank and Chrome does not allow to load local resources in non-local pages. So maybe something like this? 'use strict'; const { readFileSync } = require('fs'); const puppeteer ...
vsemozhebuty's user avatar
  • 13.4k
18 votes

How to hide Chrome Driver in python?

Try This! https://beomi.github.io/2017/09/28/HowToMakeWebCrawler-Headless-Chrome/ options = webdriver.ChromeOptions() options.add_argument('headless') options.add_argument('window-size=1920x1080') ...
Max Lee's user avatar
  • 241
17 votes

Collect elements by class name and then click each one - Puppeteer

To get all elements, you should use page.$$ method, which is the same as [...document.querySelectorAll] (spread inside an array) from reqular browser API. Then you could loop through it (map, for, ...
Rafael 'BSIDES' Pereira's user avatar
16 votes

What is a good headless browser to run with protractor?

If anyone reached here - the answers are outdated. Chromium (on next release) now supports headless mode. no need to work hard. You can read more here: https://developers.google.com/web/updates/...
guy mograbi's user avatar
16 votes
Accepted

How to hide Chrome Driver in python?

REF: how-could-i-start-a-selenium-browserlike-firefox-minimized You can move browser window over the monitor, like this: driver.set_window_position(-10000,0)
Beomi's user avatar
  • 1,717

Only top scored, non community-wiki answers of a minimum length are eligible