11

I am writing a project that relies on tensorflow, but that can be provided by either of two pip packages: tensorflow or tensorflow-gpu. My project works fine with either, but I don't want people running it on a machine without gpu support to have to install the extra overhead, but I still want people running on machines with gpu support to be able to leverage that. Is there a way to mark in my requirements.txt file that I require either tensorflow or tensorflow-gpu but not both?

EDIT:

In this specific case I should note that from the programmer's point of view, both tensorflow and tensorflow-gpu are identical, as they both provide a module tensorflow which has the same functions/classes/methods etc., and only differ in that tensorflow-gpu benefits from GPU acceleration. The problem that I am having is that if I put tensorflow in requirements.txt then in order to run with GPU acceleration, users would have to do pip install -r requirements.txt && pip uninstall tensorflow && pip install tensorflow-gpu which is not ideal, and if I instead put tensorflow-gpu in requirements.txt, then it will require a bunch of unnecessary system libraries (CUDNN etc) and wont work out-of-the-box for non-gpu users.

EDIT AGAIN

As a work-around, I've decided to provide two different requirement files, requirements.txt and requirements-gpu.txt, both of which include a shared -r .requirements-core.txt and add their respective version of tensorflow. That way people who want GPU support can pip install -r requirements-gpu.txt but the standard pip install -r requirements.txt will still work out-of-the box for everyone.

1
  • 1
    No way. pip is rather a simplistic package manager and requirements.txt is just a straightforward list of dependencies. Conditions in requirements.txt can be set for Python version or architecture but that's all. No alterations at all.
    – phd
    Jun 5, 2018 at 20:18

2 Answers 2

6

You cannot condition download packages with requirements.txt, but you might do one of the following solutions:

1 - Install both packages tensorflow and tensorflow-gpu as dependencies and do a try / except to choose which package will actually be used, like:

tensorflow = null

try:
  tensorflow = __import__("tensorflow-gpu")
  tensorflow.operation_that_requires_gpu()
except:
  tensorflow = __import__("tensorflow")
enter code here

2 - On your project you ask the client to pass the dependency direct to you:

def my_function_that_uses_tensorflow(tensorflow):
  # do stuff

from my_module import my_function_that_uses_tensorflow
import tensorflow # or tensorflow = __import__("tensorflow-gpu")
my_function_that_uses_tensorflow(tensorflow)

3 - If tensorflow-gpu and tensorflow both install their package with the same tensorflow name on your site-packages, then my suggestion is do a try /except as I said on option number 1, but don't include tensorflow-gpu or tensorflow as a dependency of your package (treat it as a "peer dependency" that the code using your package should include as their dependency in order to use it):

try:
  import tensorflow
except:
  raise ImportError('You need to include tensorflow or tensorflow-gpu as a dependency in order to use this package')
2
  • 1
    In this specific case, these steps are not useful because both tensorflow and tensorflow-gpu act the same from the programmer's perspective (i.e. they both provide a module tensorflow which has identical functions/classes/methods/etc). This is useful for the general case, however.
    – phsyron
    Jun 6, 2018 at 20:24
  • 1
    @phsyron My second suggestion than would work, ask for the user to include the correct tensorflow lib as a dependency (like a peer dependency of your lib) and pass it as an argument to you Or just try to import tensorflow with the try / except on your lib and say the user have to include the correct tensorflow as his dependency if it is missing Jun 6, 2018 at 22:34
0

There have been a handful of proposals for such functionality, but this currently does not exist within pip environment variables:

... currently PEP 508 does not provide environment markers for GPU/CUDA availability, which leads to problems for projects that want to provide distributions for environments with and without GPU support.

As far as I can tell, there's been multiple suggestions to bring this issue to distutils-sig, but no one has actually done it.

Relevant issues:

There is however a 3rd party library for TensorFlow specifically, but it has its own set of caveats:

There is now a third-party project which attempts to amend this for tensorflow (https://github.com/akatrevorjay/tensorflow-auto-detect) but this approach is somewhat fragile (depends on version numbers being in sync), doesn't directly scale to all similar projects, and would require maintainers for a given project to maintain three separate projects, instead of just one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.