441

I have a requirements.txt file that I'm using with Travis-CI. It seems silly to duplicate the requirements in both requirements.txt and setup.py, so I was hoping to pass a file handle to the install_requires kwarg in setuptools.setup.

Is this possible? If so, how should I go about doing it?

Here is my requirements.txt file:

guessit>=0.5.2
tvdb_api>=1.8.2
hachoir-metadata>=1.3.3
hachoir-core>=1.3.3
hachoir-parser>=1.3.4
4
  • 9
    install_requires is used to declare dependencies on packages that are required for the package to work and are used by developer of the package, while requirements.txt is used to automate installing of environments, which allows installing extra software and do the version pinning and are used by sysadmins deploying the package. Their role and target audience differ significantly, so trying to combine them like OP wishes is a genuine design mistake imho.
    – Zart
    Jul 2, 2016 at 16:40
  • 9
    My 2 cents. Do not use requirements.txt in your setup.py. The purposes are different, ared caremad.io/2013/07/setup-vs-requirement Sep 6, 2016 at 20:27
  • 8
    I see lots of complicated answers. What's wrong with plain old [line.strip() for line in open("requirements.txt").readlines()]? Aug 27, 2019 at 12:12
  • 1
    It is not recommended to do this. But if really needed then it is straightforward: setuptools itself already has everything necessary pkg_resources.parse_requirements()
    – sinoroc
    Jan 29, 2020 at 16:29

20 Answers 20

381

On the face of it, it does seem that requirements.txt and setup.py are silly duplicates, but it's important to understand that while the form is similar, the intended function is very different.

The goal of a package author, when specifying dependencies, is to say "wherever you install this package, these are the other packages you need, in order for this package to work."

In contrast, the deployment author (which may be the same person at a different time) has a different job, in that they say "here's the list of packages that we've gathered together and tested and that I now need to install".

The package author writes for a wide variety of scenarios, because they're putting their work out there to be used in ways they may not know about, and have no way of knowing what packages will be installed alongside their package. In order to be a good neighbor and avoid dependency version conflicts with other packages, they need to specify as wide a range of dependency versions as can possibly work. This is what install_requires in setup.py does.

The deployment author writes for a very different, very specific goal: a single instance of an installed application or service, installed on a particular computer. In order to precisely control a deployment, and be sure that the right packages are tested and deployed, the deployment author must specify the exact version and source-location of every package to be installed, including dependencies and dependency's dependencies. With this spec, a deployment can be repeatably applied to several machines, or tested on a test machine, and the deployment author can be confident that the same packages are deployed every time. This is what a requirements.txt does.

So you can see that, while they both look like a big list of packages and versions, these two things have very different jobs. And it's definitely easy to mix this up and get it wrong! But the right way to think about this is that requirements.txt is an "answer" to the "question" posed by the requirements in all the various setup.py package files. Rather than write it by hand, it's often generated by telling pip to look at all the setup.py files in a set of desired packages, find a set of packages that it thinks fits all the requirements, and then, after they're installed, "freeze" that list of packages into a text file (this is where the pip freeze name comes from).

So the takeaway:

  • setup.py should declare the loosest possible dependency versions that are still workable. Its job is to say what a particular package can work with.
  • requirements.txt is a deployment manifest that defines an entire installation job, and shouldn't be thought of as tied to any one package. Its job is to declare an exhaustive list of all the necessary packages to make a deployment work.
  • Because these two things have such different content and reasons for existing, it's not feasible to simply copy one into the other.

References:

17
  • 12
    It's still not clear to me why a developer would keep a version-controlled requirements.txt along with the source of the package that contains the concrete/frozen requirements for installation or test. Surely setup.py can be used for this purpose within the project itself? I can only imagine using such a file for tools used to support managing the project (e.g. refactoring, making releases etc.). Aug 24, 2016 at 14:00
  • 2
    @samBrightman I agree entirely, I don't think library packages or application packages should commit their requirements.txt file to the repository with the code. I think that should be an artifact generated during the build testing, and then used to document a build manifest and ultimately generate a deployment artifact. Aug 28, 2016 at 1:19
  • 6
    So you are saying requirements.txt is more documentation for the state of the world that produced a given build, even though it is not usually used in the build process itself? That makes sense. However, it looks like several systems rely on duplication: Travis installs some default (old) packages in your virtualenv and says to use requirements.txt. If I ask how to ensure dependencies are at latest using setup.py, people insist that I should use requirements.txt. Sep 20, 2016 at 6:03
  • 1
    Yea, I mean, I'm not an authority on python packaging, and I don't even think the opinion I've given above is commonly shared. The entire subject of python packaging is a confusing mess of "you do your thing, I'll do mine", so it's less that there's an official right way to do it that everyone needs to learn, and more that a lot of different people are trying to scrape together a sane model out of what's out there, and developing different opinions in the process. Sep 21, 2016 at 17:03
  • 2
    The best advice you can get out of any of this is to find a model that works for you, document it well, and make sure everyone you work with understands it. Think through why you're doing each bit and whether it really makes sense for your use case. And try to stay as well-read as you can about the current state of building, packaging, and publishing in Python, just in case things get better. But don't hold your breath. Sep 21, 2016 at 17:07
340

You can flip it around and list the dependencies in setup.py and have a single character — a dot . — in requirements.txt instead.


Alternatively, even if not advised, it is still possible to parse the requirements.txt file (if it doesn't refer any external requirements by URL) with the following hack (tested with pip 9.0.1):

install_reqs = parse_requirements('requirements.txt', session='hack')

This doesn't filter environment markers though.


In old versions of pip, more specifically older than 6.0, there is a public API that can be used to achieve this. A requirement file can contain comments (#) and can include some other files (--requirement or -r). Thus, if you really want to parse a requirements.txt you can use the pip parser:

from pip.req import parse_requirements

# parse_requirements() returns generator of pip.req.InstallRequirement objects
install_reqs = parse_requirements(<requirements_path>)

# reqs is a list of requirement
# e.g. ['django==1.5.1', 'mezzanine==1.4.6']
reqs = [str(ir.req) for ir in install_reqs]

setup(
    ...
    install_requires=reqs
)
15
  • 36
    What if the user does not have pip installed? Ka-boom? Jul 28, 2013 at 22:06
  • 150
    @GringoSuave If the user does not have pip installed, he needs to install it first.
    – guettli
    Sep 24, 2013 at 11:12
  • 8
    You also need to supply the urls in your requirements file, in case there are any -e or -f ("editable" git repo) lines pointing to non-pypi packages. Use this: setup(..., dependency_links=[str(req_line.url) for req_line in parse_requirements(<requirements_path>)], ...)
    – hobs
    Oct 8, 2013 at 22:39
  • 98
    You really don't want to do this. Speaking as a pip maintainer pip does not support being called as an API like this at all. In fact pip 1.6 (next version at this time) moves this function. Mar 26, 2014 at 0:59
  • 43
    This should no longer be the accepted answer, if it ever should have. It's blatantly broken. Even when it worked, it's blatantly unnecessary. Since pip defaults to parsing dependencies from setup.py in the absence of requirements.txt, the simple answer astutely noted by Tobu below is to list all dependencies in setup.py and remove requirements.txt. For applications requiring both, simply reduce the dependency list in requirements.txt to merely the . character. Done. Jun 30, 2016 at 5:40
149

It can't take a file handle. The install_requires argument can only be a string or a list of strings.

You can, of course, read your file in the setup script and pass it as a list of strings to install_requires.

import os
from setuptools import setup

with open('requirements.txt') as f:
    required = f.read().splitlines()

setup(...
install_requires=required,
...)
12
  • 8
    Although useful this changes specification of requirements from being declarative to imperative. This makes it impossible for some tools to find out what your requirements are. For instance, PyCharm offers automatic installation of all requirements specified in install_requires. However, it does not work if you don't use declarative syntax. Apr 28, 2013 at 10:36
  • 80
    @PiotrDobrogost Perhaps the PyCharm developer should fix their program then. setup.py is a program that should be run, not a data file that should be parsed. That doesn't make this answer any worse. Apr 28, 2013 at 18:26
  • 5
    I'm just pointing out possible problems; this answer is perfectly fine. It's not only PyCharm which has problem with information being "hidden" behind code. This is universal problem and thus there's general move towards declarative specification of metadata in Python packaging. Apr 28, 2013 at 18:50
  • 43
    Works fine as long as you put include requirements.txt into your MANIFEST.in or you won't be able to install your library from a source distribution.
    – Pankrat
    Jun 25, 2013 at 14:29
  • 7
    I know this is an old question, but you can at least nowadays configure PyCharm to parse a requirements file at Preferences->Tools->Python integrated tools->Package requirements file
    – lekksi
    Dec 28, 2014 at 18:56
70

Requirements files use an expanded pip format, which is only useful if you need to complement your setup.py with stronger constraints, for example specifying the exact urls some of the dependencies must come from, or the output of pip freeze to freeze the entire package set to known-working versions. If you don't need the extra constraints, use only a setup.py. If you feel like you really need to ship a requirements.txt anyway, you can make it a single line:

.

It will be valid and refer exactly to the contents of the setup.py that is in the same directory.

5
  • 13
    But in this case it would also try to install my app too. What if I don't need it and only want install_requires installed? Jul 24, 2016 at 20:33
  • 3
    To elaborate on what @ffeast is asking, if requirements exist only in setup.py, is there a way to install the requirements (equivalent of pip install -r requirements.txt ) without installing the package itself?
    – haridsv
    Nov 13, 2017 at 9:34
  • 1
    @ffeast @haridsv -e . should be enough. Check this page: caremad.io/posts/2013/07/setup-vs-requirement
    – dexhunter
    Jan 5, 2018 at 12:11
  • 6
    @DexD.Hunter it still tries to install the app itself. This is not what we want Apr 25, 2019 at 16:07
  • Do you give specific version or a version range to each dependency in setup.py? If a range is given, do you auto-test each possible dependency combination?
    – John
    Nov 25, 2021 at 5:54
40

While not an exact answer to the question, I recommend Donald Stufft's blog post at https://caremad.io/2013/07/setup-vs-requirement/ for a good take on this problem. I've been using it to great success.

In short, requirements.txt is not a setup.py alternative, but a deployment complement. Keep an appropriate abstraction of package dependencies in setup.py. Set requirements.txt or more of 'em to fetch specific versions of package dependencies for development, testing, or production.

E.g. with packages included in the repo under deps/:

# fetch specific dependencies
--no-index
--find-links deps/

# install package
# NOTE: -e . for editable mode
.

pip executes package's setup.py and installs the specific versions of dependencies declared in install_requires. There's no duplicity and the purpose of both artifacts is preserved.

4
  • 8
    This doesn't work when you want to provide a package for others to install via pip install my-package. If dependencies for my-package are not listed in my-package/setup.py, they are not installed by pip install my-package. I've been unable to determine how to provide a package for others that includes dependencies without explicitly stating them in setup.py. Would love to know if someone has figured out how to keep it DRY while allowing others to install my-package + dependencies without downloading the requirements file and manually calling pip install -r my-package/requirements.txt. Nov 16, 2014 at 16:01
  • 2
    @Malina The package here is perfectly installable without requirements.txt. That's the whole point. Updated the question to make things more clear. Also updated obsolete blog post link. Mar 6, 2015 at 8:15
  • so when running setup.py it will call requirements.txt for specific versions of the files listed in stup.py?
    – dtracers
    May 13, 2016 at 15:58
  • It's the other way around @dtracers. requirements.txt points to the package it self, where the setup.py's dependecies could be picked up. So when installing using requirements, it works and when installing through pip, it works too - in both cases using setup.py's dependecies, but also allowing to install more things when using requirements.txt
    – smido
    Mar 19, 2018 at 10:20
29

First, I believe parsing requirements.txt to fill the list of dependencies in package metadata is not a good idea. The requirements.txt file and the list of "install dependencies" are two different concepts, they are not interchangeable. It should be the other way around, the list of dependencies in package metadata should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. For example with a tool such as pip-compile. See the notes at the bottom of this answer.

But everyone has different needs, that lead to different workflows. So with that said... There are 3 possibilities to handle this, depending on where you want your project's package metadata to be written: pyproject.toml, setup.cfg, or setup.py.


Words of caution!

If you insist on having the list of dependencies in package metadata be read from a requirements.txt file then make sure that this requirements.txt file is included in the "source distribution" (sdist) otherwise installation will fail, for obvious reasons.

These techniques will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.


pyproject.toml

[project]
# ...
dynamic = ["dependencies"]

[tool.setuptools.dynamic]
# ...
dependencies = requirements.txt

setup.cfg

Since setuptools version 62.6 it is possible to write something like this in setup.cfg:

[options]
install_requires = file: requirements.txt

setup.py

It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.

It could more or less look like this:

#!/usr/bin/env python

import pathlib

import pkg_resources
import setuptools

with pathlib.Path('requirements.txt').open() as requirements_txt:
    install_requires = [
        str(requirement)
        for requirement
        in pkg_resources.parse_requirements(requirements_txt)
    ]

setuptools.setup(
    install_requires=install_requires,
)

Notes:

6
  • 1
    In case you weren't aware, the reason why many (myself included) have been using pip's parsing and not pkg_resources's since before 2015 are bugs such as github.com/pypa/setuptools/issues/470 . This exact one is fixed nowadays, but I'm still a bit scared to use it, since both implementations appear to be developed separately.
    – trevorj
    Feb 12, 2020 at 19:51
  • @trevorj Thanks for pointing this, I didn't know. Fact is nowadays it works and getting pip involved seems like a ridiculous idea to me (particularly in this fashion). Have a look at the other answers, most seem like slight variations of the same ill-advised idea, without barely any warning notice. And newcomers might just follow this trend. Hopefully initiatives such as PEP517 and PEP518 will steer the community away from this madness.
    – sinoroc
    Feb 13, 2020 at 6:49
  • @sinoroc Thanks! Your answer should have the most ups as it's the cleanest. Ideally the user would load requirements.in instead of requirements.txt here. (requirements.in used by pip-tools, may be the precise install_requirements 'truth' we are looking for) May 13, 2021 at 0:43
  • 1
    parse_requirements will fail with a syntax error if the requirements.txt file contains options such as -r.
    – mhucka
    Jul 7, 2022 at 18:31
  • 1
    @mhucka Yes, it is known shortcoming, I thought I had mentioned it, but apparently I had only mentioned it in my other (linked) answer. So now I added it and some other updates as well.
    – sinoroc
    Jul 7, 2022 at 20:20
24

Using parse_requirements is problematic because the pip API isn't publicly documented and supported. In pip 1.6, that function is actually moving, so existing uses of it are likely to break.

A more reliable way to eliminate duplication between setup.py and requirements.txt is to specific your dependencies in setup.py and then put -e . into your requirements.txt file. Some information from one of the pip developers about why that's a better way to go is available here: https://caremad.io/blog/setup-vs-requirement/

1
19

Most of the other answers above don't work with the current version of pip's API. Here is the correct* way to do it with the current version of pip (6.0.8 at the time of writing, also worked in 7.1.2. You can check your version with pip -V).

from pip.req import parse_requirements
from pip.download import PipSession

install_reqs = parse_requirements(<requirements_path>, session=PipSession())

reqs = [str(ir.req) for ir in install_reqs]

setup(
    ...
    install_requires=reqs
    ....
)

* Correct, in that it is the way to use parse_requirements with the current pip. It still probably isn't the best way to do it, since, as posters above said, pip doesn't really maintain an API.

13

Install the current package in Travis. This avoids the use of a requirements.txt file. For example:

language: python
python:
  - "2.7"
  - "2.6"
install:
  - pip install -q -e .
script:
  - python runtests.py
1
  • 2
    This is by far the best combination of "correct" and "practical". I'd add that if after the tests pass you can get Travis to generate a requirements.txt with pip freeze and export that file somewhere as an artifact (like S3 or something), then you'd have a great way to repeatably install exactly what you tested. Nov 13, 2015 at 4:23
7

from pip.req import parse_requirements did not work for me and I think it's for the blank lines in my requirements.txt, but this function does work

def parse_requirements(requirements):
    with open(requirements) as f:
        return [l.strip('\n') for l in f if l.strip('\n') and not l.startswith('#')]

reqs = parse_requirements(<requirements_path>)

setup(
    ...
    install_requires=reqs,
    ...
)
6

The following interface became deprecated in pip 10:

from pip.req import parse_requirements
from pip.download import PipSession

So I switched it just to simple text parsing:

with open('requirements.txt', 'r') as f:
    install_reqs = [
        s for s in [
            line.split('#', 1)[0].strip(' \t\n') for line in f
        ] if s != ''
    ]
0
3

BEWARE OF parse_requirements BEHAVIOUR!

Please note that pip.req.parse_requirements will change underscores to dashes. This was enraging me for a few days before I discovered it. Example demonstrating:

from pip.req import parse_requirements  # tested with v.1.4.1

reqs = '''
example_with_underscores
example-with-dashes
'''

with open('requirements.txt', 'w') as f:
    f.write(reqs)

req_deps = parse_requirements('requirements.txt')
result = [str(ir.req) for ir in req_deps if ir.req is not None]
print result

produces

['example-with-underscores', 'example-with-dashes']
2
  • 1
    Use unsafe_name to get the underscores version: [ir.req.unsafe_name for ir in req_deps if ir.req is not None]
    – alanjds
    Oct 5, 2015 at 14:59
  • 5
    As pointed out elsewhere, PIP is an application, not a library. It has no publicly agreed-upon API, and importing it into your code is not a supported use case. It's not surprising that it has unexpected behavior; its internal functions were never intended to be used this way. Nov 13, 2015 at 3:31
3

If you don't want to force your users to install pip, you can emulate its behavior with this:

import sys

from os import path as p

try:
    from setuptools import setup, find_packages
except ImportError:
    from distutils.core import setup, find_packages


def read(filename, parent=None):
    parent = (parent or __file__)

    try:
        with open(p.join(p.dirname(parent), filename)) as f:
            return f.read()
    except IOError:
        return ''


def parse_requirements(filename, parent=None):
    parent = (parent or __file__)
    filepath = p.join(p.dirname(parent), filename)
    content = read(filename, parent)

    for line_number, line in enumerate(content.splitlines(), 1):
        candidate = line.strip()

        if candidate.startswith('-r'):
            for item in parse_requirements(candidate[2:].strip(), filepath):
                yield item
        else:
            yield candidate

setup(
...
    install_requires=list(parse_requirements('requirements.txt'))
)
1
  • Note that distutils has been deprecated and is scheduled for removal in Python 3.12.
    – Asclepius
    Nov 23, 2022 at 1:51
1

I created a reusable function for this. It actually parses an entire directory of requirements files and sets them to extras_require.

Latest always available here: https://gist.github.com/akatrevorjay/293c26fefa24a7b812f5

import glob
import itertools
import os

# This is getting ridiculous
try:
    from pip._internal.req import parse_requirements
    from pip._internal.network.session import PipSession
except ImportError:
    try:
        from pip._internal.req import parse_requirements
        from pip._internal.download import PipSession
    except ImportError:
        from pip.req import parse_requirements
        from pip.download import PipSession


def setup_requirements(
        patterns=[
            'requirements.txt', 'requirements/*.txt', 'requirements/*.pip'
        ],
        combine=True):
    """
    Parse a glob of requirements and return a dictionary of setup() options.
    Create a dictionary that holds your options to setup() and update it using this.
    Pass that as kwargs into setup(), viola

    Any files that are not a standard option name (ie install, tests, setup) are added to extras_require with their
    basename minus ext. An extra key is added to extras_require: 'all', that contains all distinct reqs combined.

    Keep in mind all literally contains `all` packages in your extras.
    This means if you have conflicting packages across your extras, then you're going to have a bad time.
    (don't use all in these cases.)

    If you're running this for a Docker build, set `combine=True`.
    This will set `install_requires` to all distinct reqs combined.

    Example:

    >>> import setuptools
    >>> _conf = dict(
    ...     name='mainline',
    ...     version='0.0.1',
    ...     description='Mainline',
    ...     author='Trevor Joynson <[email protected],io>',
    ...     url='https://trevor.joynson.io',
    ...     namespace_packages=['mainline'],
    ...     packages=setuptools.find_packages(),
    ...     zip_safe=False,
    ...     include_package_data=True,
    ... )
    >>> _conf.update(setup_requirements())
    >>> # setuptools.setup(**_conf)

    :param str pattern: Glob pattern to find requirements files
    :param bool combine: Set True to set install_requires to extras_require['all']
    :return dict: Dictionary of parsed setup() options
    """
    session = PipSession()

    # Handle setuptools insanity
    key_map = {
        'requirements': 'install_requires',
        'install': 'install_requires',
        'tests': 'tests_require',
        'setup': 'setup_requires',
    }
    ret = {v: set() for v in key_map.values()}
    extras = ret['extras_require'] = {}
    all_reqs = set()

    files = [glob.glob(pat) for pat in patterns]
    files = itertools.chain(*files)

    for full_fn in files:
        # Parse
        reqs = {
            str(r.req)
            for r in parse_requirements(full_fn, session=session)
            # Must match env marker, eg:
            #   yarl ; python_version >= '3.0'
            if r.match_markers()
        }
        all_reqs.update(reqs)

        # Add in the right section
        fn = os.path.basename(full_fn)
        barefn, _ = os.path.splitext(fn)
        key = key_map.get(barefn)

        if key:
            ret[key].update(reqs)
            extras[key] = reqs

        extras[barefn] = reqs

    if 'all' not in extras:
        extras['all'] = list(all_reqs)

    if combine:
        extras['install'] = ret['install_requires']
        ret['install_requires'] = list(all_reqs)

    def _listify(dikt):
        ret = {}

        for k, v in dikt.items():
            if isinstance(v, set):
                v = list(v)
            elif isinstance(v, dict):
                v = _listify(v)
            ret[k] = v

        return ret

    ret = _listify(ret)

    return ret


__all__ = ['setup_requirements']

if __name__ == '__main__':
    reqs = setup_requirements()
    print(reqs)
2
  • very nice! even handles recursive requirements with latest pip :)
    – amohr
    Mar 2, 2018 at 0:15
  • @amohr Thanks! I recently updated it for an even later pip, I'm unsure why they they're acting the way they are, by moving things to pip._internal.. If you don't provide a usable external API, then you shouldn't break all those that are using all that you provide.
    – trevorj
    Aug 2, 2018 at 0:58
0

Another possible solution...

def gather_requirements(top_path=None):
    """Captures requirements from repo.

    Expected file format is: requirements[-_]<optional-extras>.txt

    For example:

        pip install -e .[foo]

    Would require:

        requirements-foo.txt

        or

        requirements_foo.txt

    """
    from pip.download import PipSession
    from pip.req import parse_requirements
    import re

    session = PipSession()
    top_path = top_path or os.path.realpath(os.getcwd())
    extras = {}
    for filepath in tree(top_path):
        filename = os.path.basename(filepath)
        basename, ext = os.path.splitext(filename)
        if ext == '.txt' and basename.startswith('requirements'):
            if filename == 'requirements.txt':
                extra_name = 'requirements'
            else:
                _, extra_name = re.split(r'[-_]', basename, 1)
            if extra_name:
                reqs = [str(ir.req) for ir in parse_requirements(filepath, session=session)]
                extras.setdefault(extra_name, []).extend(reqs)
    all_reqs = set()
    for key, values in extras.items():
        all_reqs.update(values)
    extras['all'] = list(all_reqs)
    return extras

and then to use...

reqs = gather_requirements()
install_reqs = reqs.pop('requirements', [])
test_reqs = reqs.pop('test', [])
...
setup(
    ...
    'install_requires': install_reqs,
    'test_requires': test_reqs,
    'extras_require': reqs,
    ...
)
2
  • where does tree come from?
    – roschach
    Dec 21, 2018 at 11:15
  • @FrancescoBoi if you forgive me a little for not presenting a fully working solution... tree is really just a scan of the local file system (very similar to a "tree" command in linux). Also, my solution above may not work entirely at this point because pip is constantly being updated and I used pip internals. Jun 11, 2020 at 15:25
0

A workaround that works for me

WARNING: Only use when you are not using setup.cfg already !!! This will overwrite setup.cfg

  1. Keep requirements.txt and requirements-dev.txt at same level as setup.py

  2. Remove these lines from setup.py

    install_requires=...
    extras_require=...
    
  3. Add this to setup.py to create setup.cfg and load requirements from files - requirements.txt and requirements-dev.txt

    # Read the dependencies from requirements.txt
    try:
        with open("requirements.txt", "r", encoding="utf8") as f:
            requirements = f.read().splitlines()
        requirements = [
            f"\t{req.strip()}\n" for req in requirements if req.strip()[0] != "#"
        ]
    except FileNotFoundError:
        requirements = []
    finally:
        if requirements:
            requirements = ["[options]\n", "install_requires =\n"] + requirements
            with open("setup.cfg", "w", encoding="utf8") as cfg:
                cfg.writelines(requirements)
    
    # Read the extra dependencies from requirements_extra.txt
    try:
        with open("requirements-dev.txt", "r", encoding="utf8") as f:
            dev_requirements = f.read().splitlines()
        dev_requirements = [
            f"\t{req.strip()}\n" for req in dev_requirements if req.strip()[0] != "#"
        ]
    except FileNotFoundError:
        dev_requirements = []
    finally:
        if dev_requirements:
            dev_requirements = [
                "\n[options.extras_require]\n",
                "dev =\n",
            ] + dev_requirements
            with open("setup.cfg", "a", encoding="utf8") as cfg:
                cfg.writelines(dev_requirements)
    

setup.cfg should look like this after you run python setup.py sdist

[options]
install_requires =
   package_1
   ...

[options.extras_require]
dev =
    package_2
    ...
-1

Cross posting my answer from this SO question for another simple, pip version proof solution.

try:  # for pip >= 10
    from pip._internal.req import parse_requirements
    from pip._internal.download import PipSession
except ImportError:  # for pip <= 9.0.3
    from pip.req import parse_requirements
    from pip.download import PipSession

requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())

if __name__ == '__main__':
    setup(
        ...
        install_requires=[str(requirement.req) for requirement in requirements],
        ...
    )

Then just throw in all your requirements under requirements.txt under project root directory.

-2

Yet another parse_requirements hack that also parses environment markers into extras_require:

from collections import defaultdict
from pip.req import parse_requirements

requirements = []
extras = defaultdict(list)
for r in parse_requirements('requirements.txt', session='hack'):
    if r.markers:
        extras[':' + str(r.markers)].append(str(r.req))
    else:
        requirements.append(str(r.req))

setup(
    ...,
    install_requires=requirements,
    extras_require=extras
)

It should support both sdist and binary dists.

As stated by others, parse_requirements has several shortcomings, so this is not what you should do on public projects, but it may suffice for internal/personal projects.

1
  • pip 20.1 changed their API and markers are not available via parse_requirements() anymore, so this now fails. Jun 12, 2020 at 14:20
-2

I did this:

import re

def requirements(filename):
    with open(filename) as f:
        ll = f.read().splitlines()
    d = {}
    for l in ll:
        k, v = re.split(r'==|>=', l)
        d[k] = v
    return d

def packageInfo():
    try:
        from pip._internal.operations import freeze
    except ImportError:
        from pip.operations import freeze

    d = {}
    for kv in freeze.freeze():
        k, v = re.split(r'==|>=', kv)
        d[k] = v
    return d

req = getpackver('requirements.txt')
pkginfo = packageInfo()

for k, v in req.items():
    print(f'{k:<16}: {v:<6} -> {pkginfo[k]}')
-4

Here is a complete hack (tested with pip 9.0.1) based on Romain's answer that parses requirements.txt and filters it according to current environment markers:

from pip.req import parse_requirements

requirements = []
for r in parse_requirements('requirements.txt', session='hack'):
    # check markers, such as
    #
    #     rope_py3k    ; python_version >= '3.0'
    #
    if r.match_markers():
        requirements.append(str(r.req))

print(requirements)
3
  • 1
    This is only partially true. If you call r.match_markers() you're actually evaluating the markers, which is correct thing to do for a sdist. However, if you're building a binary dist (e.g. wheel), the package would only list those libraries that matched your build-time environment. Dec 13, 2016 at 9:34
  • @TuukkaMustonen, so where to find this wheel environment (if it is the thing person tries to do) to evaluate markers against it? Dec 15, 2016 at 17:42
  • See stackoverflow.com/a/41172125/165629 that should also support bdist_wheel. It doesn't evaluate markers, it just adds them to extras_require. Dec 15, 2016 at 19:43

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.