6

I am working on a tensorflow notebook in a Docker container. I manage additional dependencies via pipenv which I install during the build phase of the container. Strangely enough some packages, which I explicitly install, "certifi" for example, do not seem to show up in the python environment.

I already tried to install it with pip in a RUN command, switched Python versions of the Pipfile and added pretty much everything to the PATH and PYTHONPATH env variables. Strangely enough, if I log in to the container and do it by hand it works like a charm.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-06bc95e4104d> in <module>
      6 # data wrangling
      7 import json
----> 8 import spacy
      9 from langdetect import detect
     10 import psycopg2

~usr/local/lib/python3.5/dist-packages/spacy/__init__.py in <module>
     10 from thinc.neural.util import prefer_gpu, require_gpu
     11 
---> 12 from .cli.info import info as cli_info
     13 from .glossary import explain
     14 from .about import __version__

~usr/local/lib/python3.5/dist-packages/spacy/cli/__init__.py in <module>
----> 1 from .download import download  # noqa: F401
      2 from .info import info  # noqa: F401
      3 from .link import link  # noqa: F401
      4 from .package import package  # noqa: F401
      5 from .profile import profile  # noqa: F401

~usr/local/lib/python3.5/dist-packages/spacy/cli/download.py in <module>
      3 
      4 import plac
----> 5 import requests
      6 import os
      7 import subprocess

~usr/local/lib/python3.5/dist-packages/requests/__init__.py in <module>
    110 from .__version__ import __copyright__, __cake__
    111 
--> 112 from . import utils
    113 from . import packages
    114 from .models import Request, Response, PreparedRequest

~usr/local/lib/python3.5/dist-packages/requests/utils.py in <module>
     22 
     23 from .__version__ import __version__
---> 24 from . import certs
     25 # to_native_string is unused here, but imported here for backwards compatibility
     26 from ._internal_utils import to_native_string

~usr/local/lib/python3.5/dist-packages/requests/certs.py in <module>
     13 packaged CA bundle.
     14 """
---> 15 from certifi import where
     16 
     17 if __name__ == '__main__':

ImportError: No module named 'certifi'


Did somebody face a similar problem already?

The minimal setup for reproduction would be the following:

docker-compose.yaml:

version: '3.1'
services:

  notebook:
    container_name: Notebook
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "5430:8888"
    user: ${CURRENT_UID}

Dockerfile:

FROM tensorflow/tensorflow:latest-py3-jupyter

RUN pip install --user pipenv
ENV PATH="/root/.local/bin:/.local/bin:${PATH}"
ENV PYTHONPATH="/usr/lib/python35.zip:/usr/lib/python3.5:/usr/lib/python3.5/plat-x86_64-linux-gnu:/usr/lib/python3.5/lib-dynload:/.local/lib/python3.5/site-packages:/usr/local/lib/python3.5/dist-packages:/usr/lib/python3/dist-packages:${PYTHONPATH}"

RUN apt-get update
RUN apt-get install -y libsm6 libxext6 libxrender-dev libglib2.0-0 postgresql libpq-dev

# We copy just the requirements.txt first to leverage Docker cache
COPY ./Pipfile ./Pipfile
COPY ./Pipfile.lock ./Pipfile.lock

RUN pipenv install --deploy --system --ignore-pipfile
RUN python -m spacy download xx

RUN pip install --user certifi

Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
numpy = "*"
scikit-learn = {extras = ["alldeps"]}
spacy = "*"
langdetect = "*"
ipykernel = "*"
bokeh = "*"
ipywidgets = "*"
plotly = "*"
lapjv = "*"
tensorflow = "*"
"psycopg2" = "*"
gensim = "*"
cmake = "*"
certifi = "*"

[dev-packages]

[requires]
python_version = "3.5"

Locking the Pipfile with pipenv lock, building and bringing the container up with docker-compose up --build and logging into the container reveals that the certifi package is not installed. Help would be greatly appreciated!

5
  • 1
    I think you'd be more likely to get an answer if you can create a Minimal, Reproducible Example
    – Mark A
    Commented Jun 13, 2019 at 16:34
  • Good point, I added a minimal reproducible example.
    – multiplex
    Commented Jun 13, 2019 at 16:47
  • Why worry about the Pipfile if --ignore-pipfile is set in pipenv?
    – C.Nivs
    Commented Jun 13, 2019 at 16:52
  • Because I don't want to post the whole Pipfile.lock file. It is huge and would probably be downvoted for length. The pipenv lock command should generate the same one afaik. But good point, I could actually remove the COPY line from the Dockerfile.
    – multiplex
    Commented Jun 13, 2019 at 16:54
  • Apparently pipenv still wants the Pipfile regardless of the flag. But well, this doesn't change much.
    – multiplex
    Commented Jun 13, 2019 at 17:14

2 Answers 2

0

From your dockerfile, it looks like certifi is being installed after you run spacy which from the stack trace seems to rely on certifi

RUN python -m spacy download xx

RUN pip install --user certifi

switch the order of those lines to install it before running the script which relies on it.

Also, make sure that everything is configured to use python3, or explicitly use python3 and pip3

3
  • I tried that, but that doesn't seem to be the solution. Actually I only added the RUN pip .. command because it was not working in the first place. The Pipfile has certifi as an explicite dependency as well.
    – multiplex
    Commented Jun 13, 2019 at 17:20
  • Perhaps spacy requires a specific version of certifi. If certifi is a dependency of spacy, then pip would install that version as it installs spacy but when it gets to installing certifi, it would install most recent and thus not be compatible with spacy. I'm unfamiliar with these specific packages so just an idea
    – ekmcd
    Commented Jun 13, 2019 at 18:19
  • I also only added the explicit dependency because of the problem - these are artifacts of my debugging process if you want :) I also thought of such a problem, but normally pipenv would resolve exactly such problems afaik.
    – multiplex
    Commented Jun 14, 2019 at 13:37
0

Reviving the dead question: Apparently there was a version conflict that pip was not able to resolve. I switched to using Poetry as a module management solution and circumvented the issua that way.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.