19

We're developing an AWS Lambda function for Alexa skill in Python and using pip to install the ask-sdk package to our dist/ directory:

pip install -t dist/ ask-sdk

The trouble is with the -t dist/ because pip wants to have all the dependencies there, even if they are installed system-wide.

Now, ask-sdk has a dependency on boto3 which pulls in a whole lot of other packages. However the AWS Lambda runtime environment provides boto3 and there is no need to package that and its dependencies with our code. I do have boto3 installed in the system and import boto3 works, so I thought pip should be happy, but because of -t dist/ it always installs it.

Can I somehow install just ask-sdk and its dependencies that don't exist in the system, e.g. ask-sdk-core, but not those that are already installed?

One way is to list all the modules and use pip --no-deps but that means a constantly keeping track of the dependencies manually, we would like to avoid that.

Somehow I would like to tell pip: if the package is already installed, even if not in -t dist/ don't put a copy in dist/.

Is that possible?

Thanks!

2
  • how you deploy lambda function? did you check serverless? Sep 1, 2018 at 8:04
  • @ManojJadhav - We use AWS SAM, but deployment is not a problem. We have to package it first, while excluding boto3 and its dependencies.
    – MLu
    Sep 1, 2018 at 9:01

3 Answers 3

12

Although you can't tell pip to "install all dependencies except those required by boto3", you can generate the needed requirements.txt by computing the difference between boto3 and ask-sdk from pip freeze output (tested with Python 3.6.6):

# get boto3 requirements
pip install boto3 -t py_lib.boto3
PYTHONPATH=py_lib.boto3 pip freeze > requirements-boto3.txt

# get ask-sdk requirements
pip install ask-sdk -t py_lib.ask-sdk
PYTHONPATH=py_lib.ask-sdk pip freeze > requirements-ask-sdk.txt

# compute their difference
grep -v -x -f requirements-boto3.txt requirements-ask-sdk.txt > requirements-final.txt

# patch to add one missing dep from boto3
# aws don't have this for some reason
grep urllib3 requirements-boto3.txt >> requirements-final.txt

The requirements-final.txt contains the following:

ask-sdk==1.5.0
ask-sdk-core==1.5.0
ask-sdk-dynamodb-persistence-adapter==1.5.0
ask-sdk-model==1.6.2
ask-sdk-runtime==1.5.0
certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1

To install the final set of dependencies to a folder:

pip install --no-deps -r requirements-final.txt -t py_lib

By skipping the boto3 dependencies, you can save about 45M of data from your python dependencies. The ask-sdk dependencies are only about 7.5M (2.1M compressed), allow you to use the build-in lambda code editor if you need to.

1
  • For those finding this answer years later, urllib3 is now a standard part of AWS Python lambdas. You can check if your python verison has it and what other libraries are already pre installed here: insidelambda.com
    – lynkfox
    Jan 5, 2022 at 20:17
4

You can try the option

  --no-dependencies

To ignore all dependencies.

To exclude specific, you can put it in requirements file and pass it:

pip install --no-deps -r requirements.txt
1
  • 1
    As I mention in my question above regarding --no-deps - we would need to keep track of the dependencies (other than boto3, e.g. ask-sdk-core) manually which we would like to avoid.
    – MLu
    Sep 1, 2018 at 8:59
2

This will work

$ pip install -t dist --no-deps ask-sdk

After the above command I checked out the dist directory content with tree and it had installed only ask-sdk without its dependencies

dist/
├── ask_sdk
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── __version__.py
│   ├── __version__.pyc
│   ├── standard.py
│   └── standard.pyc
└── ask_sdk-0.1.3.dist-info
    ├── INSTALLER
    ├── METADATA
    ├── RECORD
    ├── WHEEL
    └── top_level.txt
1
  • 1
    As I mention in my question above regarding --no-deps - we would need to keep track of the dependencies (other than boto3, e.g. ask-sdk-core) manually which we would like to avoid.
    – MLu
    Sep 1, 2018 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.