44

What I want to achieve

To scrape an website using AWS Lambda and save the data on S3.

The issues I'm having

When I execute Lambda, the following error message appears.

{ "errorMessage": "Unable to import module 'lambda_function': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/opt/python/urllib3/util/ssl_.py)", "errorType": "Runtime.ImportModuleError", "requestId": "fb66bea9-cbad-4bd3-bd4d-6125454e21be", "stackTrace": [] }

Code

The minimum Lambda code is as follows.

import requests
import boto3 

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    upload_res = s3.put_object(Bucket='horserace-dx', Key='/raw/a.html', Body='testtext')
    
    return event

An layer was added to the Lambda. Files were save in python folder using the commands below , frozen in a zip file, then uploaded to AWS Lambda as a layer.

!pip install requests -t ./python --no-user
!pip install pandas -t ./python --no-user
!pip install beautifulsoup4 -t ./python --no-user
  • The bucket horserace-dx exists
  • The folder raw exists
  • The role of the Lambda is properly set. It can read from and write to S3
  • The runtime of the Lambda is Python 3.9. The python version of the local computer is 3.9.13.

What I did so far

I google "cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'" and found some suggestions. I made the layer with the following code and tried again in vain.

!pip install requests -t ./python --no-user
!pip install pandas -t ./python --no-user
!pip install beautifulsoup4 -t ./python --no-user
!pip install urllib3==1.26.15 -t ./python --no-user

So what should I do to achieve what I want to achieve? Any suggestions would be greatly appreciated.

4
  • Is there a reason you are installing that specific version of urllib?
    – Paolo
    Jun 6 at 12:14
  • @Paolo on this page (qiita.com/SatoshiGachiFujimoto/items/437b0ccaba817903fb72) there was a suggestion that the same error was solved using that version. I know the author was using docker while I'm not, but I tried that anyway.
    – dixhom
    Jun 6 at 12:28
  • Try urllib3<2
    – Paolo
    Jun 6 at 13:36
  • 2
    @Paolo I got the same error
    – dixhom
    Jun 6 at 20:42

12 Answers 12

48

cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'

You are encountering this issue because you’re using botocore which does not support urllib3 2.0 yet.

Since you’re deploying to AWS Lambda, you’ll need to explicitly pin to urllib3<2 in your project to ensure urllib3 2.0 isn’t brought into your environment.
(Source)

urllib3<2

Follow this guide for how to deploy Python Lambda functions with .zip file archives.

If you cannot get it to work via the .zip file, consider deploying via a container image instead by following this guide.

1
  • We noticed this change between urllib3-2.0.3 version and urllib3-2.1.0
    – Matt Cliff
    2 days ago
26

In my case I just specified requests version (runtime python3.9) -

requests==2.28.2

and it worked.

3
  • 7
    2.29.0 also worked for me the full command is pip install requests==2.29.0
    – hispeed
    Jun 23 at 5:28
  • pip install requests==2.28.2 -t ./. will install it only in your current folder Oct 9 at 19:01
  • 2.31.0 didn't work. used 2.28.2 and it worked. using image with public.ecr.aws/lambda/python:3.9-x86_64. thanks for this comment @ashrafminhaj
    – mangusta
    Nov 15 at 21:50
5

I solved it by setting urllib3<2. There is a breaking change with urllib3 2.0.

For example, if you have a requirement.txt file, then add a line urllib3<2 to the file.

Refer to this for a better explanation: https://github.com/psf/requests/issues/6443#issuecomment-1535667256

4

The resolution that worked for me was to update the AWS Lambda runtime to 3.10 and if applicable any AWS Lambda layers to python 3.10 and then ensure you have packaged the latest release of requests. If you are using botocore or boto3 you will also need to ensure that you have included them in the code package or in an AWS Lambda Layer including versions boto3>=1.26.153 and botocore>=1.29.153.

1
  • 1
    That's actually a little misleading, using previous stable version of libraries did solve the issue while using python3.9 as runtime. Jun 18 at 7:41
3
  1. Execute the following commands.

    pip install requests==2.25.0 -t ./python --no-user pip install beautifulsoup4 -t ./python --no-user pip install pytz -t ./python --no-user

  2. On PyPI, download the following whl files from the pages of numpy and pandas

  • numpy-1.24.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  • pandas-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  1. Unzip the files and move the contents to the python folder.

  2. Zip the python folder and upload it to AWS Lambda Layer.

  3. Set the layer to the Lambda.

  4. Then the code runs without errors.

2

I solved this problem by using python 3.11 when setting up my lambda function in AWS, the lower versions tends to throw this error the more

2
  • 1
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Aug 11 at 3:20
  • 1
    upgrading lambda environment wasn't enough in my case--still had to rebuild package with correct requests version per @ashraf minhad
    – varontron
    Aug 16 at 10:51
1

You can solve this adding in your deployment.txt:

urllib3==1.16

1
  • 1
    This does not work, you will get error 'is_ipaddress' from 'urllib3.util.ssl_' (/var/task/urllib3/util/ssl_.py)", 1.26.16 works fine Jun 23 at 6:56
1

This is because openai urllib3 issue

Solution : Open any folder and type this command “pip install openai requests==2.29.0 urllib3==1.26.16 -t python” this will install pip with these specific versions of requests and urllib3

1
  • 1
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Sep 14 at 23:37
1

Had a similar issue, with this and other libraries. The root cause of the problem is library compatibility, so to solve it without specifying the library versions, we can use the lambda's runtime Docker image:

mkdir layer
cp requirements.txt layer/requirements.txt
docker run -ti -v $(pwd)/layer:/app -w /app --entrypoint /bin/bash public.ecr.aws/lambda/python:3.11 -c "pip3 install --target ./python -r requirements.txt"

We get at layer/python, all the libraries that we need for the layer. Then we just zip it and create the layer in AWS.

Here I'm using Python 3.11 for the lambda, but any other version should work

Note: on windows powershell, change $(pwd) to ${pwd}.

1

In my case,I encounter this error when using transformers library. I solved this problem by pip install urllib3<2

0

In the requirements.txt add the line urllib3<2 and build and deploy the code, then it will resolve the issue. Even if requirements.txt doesnt have that please add it, because there might be some installations for which urllib3 will be a prerequisite and that will be downloaded even if its not mentioned directly.

0

The solution I tried for the above error is creating virtual environment in python using

python -m venv <name_of_virtualenv>

and then installing boto3 in the virtual environment.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.