5

Problem

In a serverless project, I want to import XGBoost into a lambda written on Python. But when I'm trying to invoke the lambda, I see this error on CloudWatch:

[ERROR] XGBoostError: XGBoost Library (libxgboost.so) could not be loaded.
Likely causes:
* OpenMP runtime is not installed (vcomp140.dll or libgomp-1.dll for Windows, libgomp.so for UNIX-like OSes)
* You are running 32-bit Python on a 64-bit OS
Error message(s): ['libgomp.so.1: cannot open shared object file: No such file or directory']

What I tried?

  1. I'm running sls deploy on macOS so I've added dockerizePip: true to my serverless.yml

  2. I tried to fix missing dependency by using custom Dockerfile:

    FROM lambci/lambda:build-python3.6
    
    RUN apt-get update && apt-get install libaio1
    

    I also must specify library path in dockerExtraFiles but I have no idea where libgomp.so should be located on Linux. So, I stuck on this point.

My code

serverless.yml:

app: improve
org: kvadrug
service: testservice

provider:
  name: aws
  runtime: python3.8
  versionFunctions: false
  stage: dev
  region: us-west-2
  timeout: 30

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerFile: Dockerfile
    zip: true
    dockerizePip: true

functions:
  hello:
    handler: hello.hello
    events:
      - http:
          path: hello
          method: post
          private: true

package.json:

{
    "name": "testservice",
    "version": "1.0.0",
    "description": "Test service",
    "dependencies": {},
    "devDependencies": {
        "serverless-python-requirements": "^5.1.0"
    }
}

requirements.txt:

xgboost==1.0.2
2
  • did you try to install libgomp1? packages.debian.org/buster/libgomp1
    – jkr
    Commented May 10, 2020 at 20:03
  • @jakub, no. But I have installed libomp. Anyway, I'm not sure if it's related, because XGBoost works locally on my mac. The problem is around AWS lambda.
    – kelin
    Commented May 10, 2020 at 20:09

3 Answers 3

5
+50

Working configuration - Python 3.7

Tested on Mojave 10.14.6 (18G4032) and Docker v2.1.0.2 (37877).

Steps:

sls requirements clean
rm -Rf ~/Library/Caches/serverless-python-requirements/
sls deploy

requirements.txt:

xgboost==1.0.2

serverless.yml:

service: xgboost
provider:
  name: aws
  timeout:60
  runtime: python3.7

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    zip: true
    dockerizePip: non-linux
    dockerExtraFiles:
      - /usr/lib64/libgomp.so.1

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: post
          private: true

handler.py :

    try:
        import unzip_requirements
    except ImportError:
        pass

    import sys
    import glob
    import os

    def hello(event, context):
        pkgdir = '/tmp/sls-py-req/'

        print("-------- Sys Path --------")
        for p in sys.path:
            print(p)

        if os.path.exists(pkgdir):
            print("-------- Pkg Dir ----------")
            os.chdir(pkgdir)
            for file in glob.glob("*"):
                print(file)

        print("-------- Var Task ----------")
        os.chdir("/var/task")
        for file in glob.glob("*"):
            print(file)

        try:
            import xgboost as xgb
            print(xgb)
        except Exception as ex:
            template = "An exception of type {0} occurred. Arguments:\n{1!r}"
            message = template.format(type(ex).__name__, ex.args)
            print(message)

        return True

Python 3.8

For some reason, Lambda running Python 3.8 ignores the entry for '/tmp/sls-py-req' on 'sys.path'. Therefore you need to manually add the library file 'libgomp.so.1' to the root of your application.

To give it a try, update 'runtime: python3.7' to 'runtime: python3.8' on serverless.yml and follow these steps:

sls requirements clean
rm -Rf ~/Library/Caches/serverless-python-requirements/
sls package
cp .serverless/requirements/libgomp.so.1 ./
sls deploy

FYI - How to find the libraries location within the Lambda docker image

Run:

docker run --rm -ti --entrypoint /bin/sh -u 0 lambci/lambda\:build-python3.8

Then:

find / -name libgomp.so.1
1
  • Thanks! The Python3.8 solution worked. The only problem is that I wasn't able to get libgomp.so.1 until I deleted old resources folder manually. Also, the information about finding library is very useful.
    – kelin
    Commented May 16, 2020 at 17:09
2

On Linux, the libraries you need are installed into

/usr/lib/x86_64-linux-gnu/libgomp.so.1
/usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0

by

apt-get install libgomp1

On newer Ubuntu and CentOS, libgomp1 is no longer installed by default (I have just checked - I have it on xenial, don't have it on a fresh bionic).

This is on a AWS machine, unfortunately not very recent (I have no other readily available):

lrwxrwxrwx 1 root root     16 Oct  4  2019 /usr/lib/x86_64-linux-gnu/libgomp.so.1 -> libgomp.so.1.0.0
-rw-r--r-- 1 root root 138448 Oct  4  2019 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
3
  • I'm on macOS. And how do you think, were the library is installed on Amazon Linux 2 (kernel - 4.14.165-102.205.amzn2.x86_64)?
    – kelin
    Commented May 14, 2020 at 8:26
  • I have checked on a AWS machine I had accessible. From what I see of AWS Linux 2, it should be in the same location.
    – LSerni
    Commented May 14, 2020 at 10:26
  • I tried to specify this path in dockerExtraFiles: and include: but I'm getting the same error. Also, I noticed that the library wasn't added to the serverless package.
    – kelin
    Commented May 14, 2020 at 17:04
0

For some reason, these solutions did not work for me (Python 3.6, xgboost==1.4, Ubuntu 18.04, aarch64).

However, I was able to get things working by pointing to the libgomp library packaged with the xgboost Python package.

For example, export LD_PRELOAD=/usr/local/lib/python3.6/dist-packages/xgboost/lib/../../xgboost.libs/libgomp-d22c30c5.so.1.0.0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.