On the old boto
library is was simple enough to use the proxy
, proxy_port
, proxy_user
and proxy_pass
parameters when you open a connection. However, I could not find any equivalent way of programmatically define the proxy parameters on boto3. :(
As of at least version 1.5.79, botocore accepts a proxies
argument in the botocore config.
e.g.
import boto3
from botocore.config import Config
boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))
boto3 resource https://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.resource
botocore config https://botocore.readthedocs.io/en/stable/reference/config.html#botocore.config.Config
-
1I'm using boto3==1.4.6, botocore==1.6.6, but this does not seem to be working for me. Could you please provide a full example loading a file into a bucket, or something similar? – albarji Aug 24 '17 at 10:26
-
Since AWS uses HTTPS for all endpoints, try this configuration:
boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))
(note that it's https) – Brad Sep 28 '17 at 0:02 -
2I had the same issue with STS. I've put that config into client and it worked, In case someone else need:
conn = boto3.client('sts', config=Config(proxies={'http': 'myproxy', 'https': 'myproxy'}))
– PedroMVM Aug 13 '19 at 12:02
If you user proxy server does not have a password try the following:
import os
os.environ["HTTP_PROXY"] = "http://proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://proxy.com:port"
if you user proxy server has a password try the following:
import os
os.environ["HTTP_PROXY"] = "http://user:password@proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://user:password@proxy.com:port"
-
Although ensure you correctly include quotations around your envirion values. This answer has them missing at the beginning of the proxy – mightymephisto Apr 7 '16 at 10:15
-
This works, but changing an environment variable is troublesome. If along your program you need to perform an http request to other server, such request will get routed through the s3 proxy server, which is not what you want. You can kind of solve this by restoring both env variables to their original values after you are finished querying s3, but I wish there was a better solution. – albarji Aug 24 '17 at 10:13
Apart from altering the environment variable, I'll present what I found in the code.
Since boto3 uses botocore, I had a look through the source code:
From this link, we end up at:
def _get_proxies(self, url):
# We could also support getting proxies from a config file,
# but for now proxy support is taken from the environment.
return get_environ_proxies(url)
...which is called by proxies = self._get_proxies(final_endpoint_url)
in the EndpointCreator
class.
Long story short, if you're using python2 it will use the getproxies
method from urllib2 and if you're using python3, it will use urllib3.
get_environ_proxies
is expecting a dict containing {'http:' 'url'}
(and I'm guessing https
too).
You could always patch
the code, but that is poor practice.
This is one of the rare occasions when I would recommend monkey-patching, at least until the Boto developers allow connection-specific proxy settings:
import botocore.endpoint
def _get_proxies(self, url):
return {'http': 'http://someproxy:1234/', 'https': 'https://someproxy:1234/'}
botocore.endpoint.EndpointCreator._get_proxies = _get_proxies
import boto3
import os ;os.environ["HTTP_PROXY"] = "http://yourproxy.com:1234"; os.environ["HTTPS_PROXY"] = "https://yourproxy1.com:1234"
– Mani Nov 3 '15 at 11:31HTTPS_PROXY
would not work, onlyHTTP_PROXY
. – asieira Nov 3 '15 at 12:22