23

I am building a python 3.6 AWS Lambda deploy package and was facing an issue with SQLite.

In my code I am using nltk which has a import sqlite3 in one of the files.

Steps taken till now:

  1. Deployment package has only python modules that I am using in the root. I get the error: Unable to import module 'my_program': No module named '_sqlite3'

  2. Added the _sqlite3.so from /home/my_username/anaconda2/envs/py3k/lib/python3.6/lib-dynload/_sqlite3.so into package root. Then my error changed to:

    Unable to import module 'my_program': dynamic module does not define module export function (PyInit__sqlite3)

  3. Added the SQLite precompiled binaries from sqlite.org to the root of my package but I still get the error as point #2.

My setup: Ubuntu 16.04, python3 virtual env

AWS lambda env: python3

How can I fix this problem?

6
  • 2
    It works with python 2.7 in the AWS Lambda environment. Could be that the python3 environment does not have the binaries to support sqlite3. But my question still stands. How can I package binaries into the aws lambda to leverage them in my python package. May 19, 2017 at 17:52
  • I even tried to copy the libsqlite3.so into the lib directory of the zip but that didn't help. Did anyone tackle this?
    – Stereo
    Jul 11, 2017 at 21:30
  • 1
    you are out of luck till AWS adds it to there AMI. People have proposed some nice solutions below Jul 11, 2017 at 22:32
  • thanks @Darthsidious not sure why it wouldn't be possible to add the library explicitly to the zip though. I guess I am still learning the limitations.
    – Stereo
    Jul 11, 2017 at 22:33
  • 1
    I have pretty much tried every location in the zip for the sqlite.so Jul 11, 2017 at 22:50

8 Answers 8

38

Depending on what you're doing with NLTK, I may have found a solution.

The base nltk module imports a lot of dependencies, many of which are not used by substantial portions of its feature set. In my use case, I'm only using the nltk.sent_tokenize, which carries no functional dependency on sqlite3 even though sqlite3 gets imported as a dependency.

I was able to get my code working on AWS Lambda by changing

import nltk

to

import imp
import sys
sys.modules["sqlite"] = imp.new_module("sqlite")
sys.modules["sqlite3.dbapi2"] = imp.new_module("sqlite.dbapi2")
import nltk

This dynamically creates empty modules for sqlite and sqlite.dbapi2. When nltk.corpus.reader.panlex_lite tries to import sqlite, it will get our empty module instead of the standard library version. That means the import will succeed, but it also means that when nltk tries to use the sqlite module it will fail.

If you're using any functionality that actually depends on sqlite, I'm afraid I can't help. But if you're trying to use other nltk functionality and just need to get around the lack of sqlite, this technique might work.

1
  • this works but after that I was asked to install python -m textblob.download_corpora. where should I keep this file?
    – Jun
    Sep 8, 2018 at 0:47
9

This is a bit of a hack, but I've gotten this working by dropping the _sqlite3.so file from Python 3.6 on CentOS 7 directly into the root of the project being deployed with Zappa to AWS. This should mean that if you can include _sqlite3.so directly into the root of your ZIP, it should work, so it can be imported by this line in cpython:

https://github.com/python/cpython/blob/3.6/Lib/sqlite3/dbapi2.py#L27

Not pretty, but it works. You can find a copy of _sqlite.so here:

https://github.com/Miserlou/lambda-packages/files/1425358/_sqlite3.so.zip

Good luck!

4
  • 1
    absolutely works.. thanks for the pre built ".so" file!
    – human
    Aug 19, 2018 at 13:35
  • 1
    Perfect. Thank you!
    – mmla
    Feb 16, 2019 at 20:36
  • I have a different situation, when I put the _sqlite3.so file in the folder, the error sqlite3 module is required for the SqliteCache is gone, but another error comes out:[Errno 30] Read-only file system: '/home/sbx_user1051' Any suggestion?
    – Yang L
    Apr 25, 2019 at 16:11
  • @YangL That looks to me like whatever is calling the Cache is configured to try to write to /home/sbx_user1051, which likely wouldn't exist within the context of AWS Lambda.
    – FlipperPA
    Apr 26, 2019 at 18:03
6

This isn't a solution, but I have an explanation why.

Python 3 has support for sqlite in the standard library (stable to the point of pip knowing and not allowing installation of pysqlite). However, this library requires the sqlite developer tools (C libs) to be on the machine at runtime. Amazon's linux AMI does not have these installed by default, which is what AWS Lambda runs on (naked ami instances). I'm not sure if this means that sqlite support isn't installed or just won't work until the libraries are added, though, because I tested things in the wrong order.

Python 2 does not support sqlite in the standard library, you have to use a third party lib like pysqlite to get that support. This means that the binaries can be built more easily without depending on the machine state or path variables.

My suggestion, which you've already done I see, is to just run that function in python 2.7 if you can (and make your unit testing just that much harder :/).

Because of the limitations (it being something baked into python's base libs in 3) it is more difficult to create a lambda-friendly deployment package. The only thing I can suggest is to either petition AWS to add that support to lambda or (if you can get away without actually using the sqlite pieces in nltk) copying anaconda by putting blank libraries that have the proper methods and attributes but don't actually do anything.

If you're curious about the latter, check out any of the fake/_sqlite3 files in an anaconda install. The idea is only to avoid import errors.

2

As apathyman describes, there isn't a direct solution to this until Amazon bundle the C libraries required for sqlite3 into the AMI's used to run Python on lambda.

One workaround though, is using a pure Python implementation of SQLite, such as PyDbLite. This side-steps the problem, as a library like this doesn't require any particular C libraries to be installed, just Python.

Unfortunately, this doesn't help you if you are using a library which in turn uses the sqlite3 module.

1
  • 1
    PyDbLite isn't an implementation of sqlite Jun 14, 2018 at 17:07
1

My solution may or may not apply to you (as it depends on Python 3.5), but hopefully it may shed some light for similar issue.

sqlite3 comes with standard library, but is not built with the python3.6 that AWS use, with the reason explained by apathyman and other answers.

The quick hack is to include the share object .so into your lambda package:

find ~ -name _sqlite3.so

In my case:

/home/user/anaconda3/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/_sqlite3.so

However, that is not totally sufficient. You will get:

ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

Because the _sqlite3.so is built with python3.5, it also requires python3.5 share object. You will also need that in your package deployment:

find ~ -name libpython3.5m.so*

In my case:

/home/user/anaconda3/pkgs/python-3.5.2-0/lib/libpython3.5m.so.1.0

This solution is likely not work if you are using _sqlite3.so that is built with python3.6, because the libpython3.6 built by AWS will likely not support this. However, this is just my educational guess. If anyone has successfully done, please let me know.

1

TL;DR

Although not a very good approach but it works just fine. Use pickle module to dump whatever functionality you want in a separate file using you Local System. Export that file to AWS Project Directory, load functionality from file and use it.

In Long

So, here's what I tried, I had this same problem when I was trying to import stopwords from nltk.corpus but AWS doesn't let me import it, even installing it wasn't seems to be possible for me on Amazon AMI because of the same error _sqlite3 module not found. So to do that, what I tried was, using my local system I generated a file and dump stopwords into it. Here's the code

# Only for Local System

from nltk.corpus import stopwords
import pickle

stop = list(stopwords.words('English'))
with open('stopwords.pkl', 'wb') as f:
    pickle.dump(stop, f)

Now, I exported this file to AWS Directory using winscp and then used that functionality by loading it from the file on my project.

import pickle

# loading trained model
with open('stopwords.pkl', 'rb') as f:
    stop = pickle.load(f)

and it works just fine

0

From AusIV's answer, This version works for me in AWS Lambda and NLTK, I created a dummysqllite file to mock the required references.

spec = importlib.util.spec_from_file_location("_sqlite3","/dummysqllite.py")
sys.modules["_sqlite3"] = importlib.util.module_from_spec(spec)
sys.modules["sqlite3"] = importlib.util.module_from_spec(spec)
sys.modules["sqlite3.dbapi2"] = importlib.util.module_from_spec(spec)
0

You need the sqlite3.so file (as others have pointed out), but the most robust way to get it is to pull from the (semi-official?) AWS Lambda docker images available in lambci/lambda. For example, for Python 3.7, here's an easy way to do this:

First, let's grab the sqlite3.so (library file) from the docker image:

mkdir lib
docker run -v $PWD:$PWD lambci/lambda:build-python3.7 bash -c "cp sqlite3.cpython*.so $PWD/lib/"

Next, we'll make a zipped executable with our requirements and code:

pip install -t output requirements.txt
pip install . -t output
zip -r output.zip output

Finally, we add the library file to our image:

cd lib && zip -r ../output.zip sqlite3.cpython*.so

If you want to use AWS SAM build/packaging, instead copy it into the top-level of the lambda environment package (i.e., next to your other python files).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.