5

We tried for a few days to setup Proxy Authentication with selenium chromedriver in Python. We couldn't set ip up because Chrome throws a pop-up for authentication. Problem is that selenium can't switch to that window and so, can't type. The only solution that worked for us was using pyautogui which is a bad solution for us because we want to use the headless function.

Here are all the methods we tried:

driver.switch_to_window()
driver.switch_to_active_element()
driver.switch_to_alert()
ActionChains(driver).send_keys("test").perform()
driver.switch_to_alert()
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=http://user:[email protected]:31112')
driver.get("https://user:[email protected]")
proxy = {'address': '3.209.253.119:31112',
         'username': 'user',
         'password': 'pass'}


capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
                         'httpProxy': proxy['address'],
                         'ftpProxy': proxy['address'],
                         'sslProxy': proxy['address'],
                         'noProxy': '',
                         'class': "org.openqa.selenium.Proxy",
                         'autodetect': False}

capabilities['proxy']['socksUsername'] = proxy['username']
capabilities['proxy']['socksPassword'] = proxy['password']

driver = webdriver.Chrome(executable_path="chromedriver.exe", desired_capabilities=capabilities)
driver.get("http://google.com")

Any help would be really appreciated :)

4 Answers 4

2

If you need to use a proxy with no authentication (no username or password) with python and Selenium library with chromedriver you usually use the following code:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port)
driver = webdriver.Chrome(chrome_options=chrome_options)

It works fine unless proxy requires authentication. if the proxy requires you to log in with a username and password it will not work. In this case, you have to use more tricky solution that is explained below.

To set up proxy authentication we will generate a file and upload it to chromedriver dynamically using the following code below. This effectively create a chrome extension. This code configures selenium with chromedriver to use HTTP proxy that requires authentication with user/password pair.

import os
import zipfile

from selenium import webdriver


def create_chromedriver(PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS, USER_AGENT):
    manifest_json = """
    {
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
    }
    """

    background_js = """
    var config = {
        mode: "fixed_servers",
        rules: {
        singleProxy: {
            scheme: "http",
            host: "%s",
            port: parseInt(%s)
        },
        bypassList: ["localhost"]
        }
    };

    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

    function callbackFn(details) {
        return {
            authCredentials: {
            username: "%s",
            password: "%s"
            }
        };
    }

    chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ['blocking']
    );
    """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)


    def get_chromedriver(use_proxy=True, user_agent=USER_AGENT):
        path = os.path.dirname(os.path.abspath(__file__))
        chrome_options = webdriver.ChromeOptions()
        if use_proxy:
            pluginfile = 'proxy_auth_plugin.zip'
            with zipfile.ZipFile(pluginfile, 'w') as zp:
                zp.writestr("manifest.json", manifest_json)
                zp.writestr("background.js", background_js)
            chrome_options.add_extension(pluginfile)
        if user_agent:
            chrome_options.add_argument('--user-agent=%s' % USER_AGENT)
        driver = webdriver.Chrome(
            os.path.join(path, 'chromedriver'),
            chrome_options=chrome_options)
        return driver

    driver = get_chromedriver(use_proxy=True)
    # driver.get('https://www.google.com/search?q=my+ip+address')
    driver.get('https://httpbin.org/ip')


user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'
create_chromedriver('192.168.3.2', 8080, 'user', 'pass', user_agent)

Function get_chromedriver returns configured selenium web driver that you can use in your application. This code is tested and works just fine.

To implement this into your own code just call the function create_chromedriver with args of host, port, user, pass and user-agent. If you do not want to use proxy or user-agent just edit get_chromedriver accordingly so the args are false.

3
  • Yes, we came across this solution become it seems like this is the only one that works. Unfortunately it's still not a perfect solution since you can't use extensions in headless mode. Nov 16, 2020 at 0:38
  • @DavidKoskasa any solution to this? Jun 28, 2022 at 21:32
  • 1
    Also, the above solution currently uses Manifest V2 extension syntax, which will stop working in January 2023 stackoverflow.com/questions/55582136/… Mar 14 at 8:20
1

@luke-hamilton You can configure an authenticated proxy via an extension like this and run headlessly if you include this option

chrome_options.add_argument("--headless=chrome")
0
0

Using Selenium-Profiles, this is allready implemented into the driver.

Example script:

from selenium_profiles.webdriver import Chrome
from selenium_profiles.profiles import profiles

profile = profiles.Windows() # or .Android()
profile["proxy"] = {
  "proxy":"http://user1:pass1@example_host.com:41149"
  }

driver = Chrome(profile=profile)

driver.quit()  # Execute on the End!

Note: Setting the proxy dynamically is supported as well

Disclaimer: I am the author of Selenium-Profiles

0

For proxy with auth in a headless environment, you can use SeleniumBase:

pip install seleniumbase, then run this with python (after setting variables):

import time
from seleniumbase import Driver

driver = Driver(headless=True, proxy="USER:PASS@IP:PORT")
driver.get("https://whatismyip.com")
time.sleep(6)
driver.quit()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.