1174

Sometimes I download the Python source code from GitHub and don't know how to install all the dependencies. If there isn't any requirements.txt file I have to create it by hand.

Given the Python source code directory, is it possible to create requirements.txt automatically from the import section?

4
  • 85
    You can do it by running pip freeze > requirements.txt will save all your python libraries with current version into requirements.txt file
    – Shaikhul
    Jul 28, 2015 at 18:36
  • 28
    @Shaikhul but that doesn't help in the case where you don't have the dependencies installed because you've only just downloaded the package from GitHub...
    – jonrsharpe
    Jul 28, 2015 at 18:58
  • 38
    Please note: a) pip freeze will dump current versions of all the installed modules on that system irrespective of there usage in the current project. b) pip will only list modules that have been installed via pip
    – akskap
    Jan 17, 2017 at 3:10
  • If you needed to generate requirements.txt for a lambda layer like I did, you can use pip freeze with the --path argument as suggested in this post: stackoverflow.com/questions/31210165/… Sep 8 at 13:35

26 Answers 26

1343

You can use the following code to generate a requirements.txt file:

pip install pipreqs
pipreqs /path/to/project

The benefits of using pipreqs from its GitHub.

Why not pip freeze?

  • pip freeze only saves the packages that are installed with pip install in your environment.
  • pip freeze saves all packages in the environment including those that you don't use in your current project (if you don't have virtualenv).
  • and sometimes you just need to create requirements.txt for a new project without installing modules.
22
  • 1
    @IgorBarinov thats a Warning and won't create a problem to run pipreqs, I installed it on python 2.6 and got same warning but it does create a requirements.txt file, thanks @DJanssens
    – Shaikhul
    Jul 28, 2015 at 19:29
  • 3
    @Shaikhul if you got these warnings you will not have a full file, you will just get a part of requirements. You can check it on github.com/npow/MemNN/tree/hyperopt repo. I have only 'hyperopt == 0.0.2 numpy == 1.9.2 scikit_learn == 0.16.1 theano == 0.7.0' Jul 28, 2015 at 19:59
  • 26
    I used pipreqs and needed to specify the used charset in my scanned projectfolder - otherwise I received an error generated by pipreqs: pipreqs --encoding utf-8
    – squeezer44
    Oct 9, 2020 at 22:51
  • 3
    Note: This will not include installed apps in your settings that are not explicitly imported in your views but nonetheless being used in the background.
    – Tamdim
    Mar 19, 2021 at 18:22
  • 7
    Important notice: this variant can be bad. Use virtualenv and pip3 freeze > requirements.txt. Let me explain: pipreqs only saves packages which are imported in project, not their dependencies. Sometimes package maintainer write requirement rule like this: "Jinja2>=2.10" (this is from setup.py of Flask 1.02 package). So the incompatible major version can be installed (in my example i got Jinja 3.1.0) and breaks everything.
    – khokm
    Apr 11, 2022 at 20:43
1208

Use Pipenv or other tools is recommended for improving your development flow.

pip3 freeze > requirements.txt  # Python3
pip freeze > requirements.txt  # Python2

If you do not use a virtual environment, pigar will be a good choice for you.

8
  • 11
    For anyone erroring, I recommend trying pip3 freeze > requirements.txt before asking a question. Aug 18, 2017 at 15:39
  • 9
    I execute pip3 freeze > requirements.txt and I'm surprised to not find some packages I am actually using such as argparse and configparser. Is it because they are part of Python core? I believe a better answer is to use pipreqs below because it will only list the requirements your project is using. Nov 3, 2017 at 17:15
  • 94
    @damnever Ok thank you! pip3 freeze gives me the list of all the Python packages installed in my environment whereas pipreqs gives me all the ones actually used in my source code which is what I was looking for. Beside that, nothing's wrong with it. Nov 4, 2017 at 4:22
  • 2
    pipenv has serious issues, see this discussion: news.ycombinator.com/item?id=18612590 Jan 4, 2019 at 14:53
  • 9
    Piping into requirements.txt can be quite a bad idea.
    – pfabri
    Mar 21, 2019 at 19:42
128

For python3: (I have both python 2 and 3 on my machine, where python2 is the default)

# install
pip3 install pipreqs

# Run in current directory
python3 -m  pipreqs.pipreqs .

python2:

pip install pipreqs
python -m  pipreqs.pipreqs .

To check your python version:

python --version
3
  • 4
    Worked for me with a slight modifcation: pip install pipreqs and then python -m pipreqs.pipreqs .
    – Tirbo06
    Sep 17, 2021 at 11:29
  • I needed python -m pipreqs.pipreqs --encoding=utf8 --force because of some UTF8 characters (in my code, I guess?), as seen here May 15 at 17:10
  • github.com/bndr/pipreqs appears to be dead: "build no longer available" badge, "Looking for maintainers to move this project forward." Version installed from pip crashed on a small toy project. Jul 28 at 7:27
54

Kinda mind-blowing how this simple task is so complicated in Python. Here is what I think is the best way to do it automatically.

You need two tools:

1.pipreqs

pip3 install pipreqs

pipreqs will go through your project and only install the packages that your project use. Instead of all the packages in your python environment as pip freeze would do.

But there's a problem with this approach. It does not install the sub-packages.

For example, your project uses pandas==1.3.2. pandas itself uses numpy==1.21.2 among other packages. But pipreqs itself does not write the sub-packages (i.e. numpy) in requirments.txt

This is where you need to combine pipreqs with the second tool.

  1. pip-tools

pip3 install pip-tools

pip-tools will take the packages in requirements.in and generate the requirements.txt with all the sub-packages. For example, if you have pandas==1.3.2 in requirements.in, pip-tools would generate

numpy==1.21.2 # via pandas in requirements.txt.

But you need to manually add the package in requirements.in. Which is prone to mistake and you might forget to do this once in a while.

This is where you can use the first tool.

But both the tools write to requirements.txt. So how do you fix it?

Use the --savepath for pipreqs to write in requirements.in instead of the default requirements.txt.

To do it in one command; just do

pipreqs --savepath=requirements.in && pip-compile

There you go. Now you don't need to worry about manually maintaining the packages and you're requirements.txt will have all the sub-packages so that your build is deterministic.

TL;DR

  1. pip3 install pipreqs
  2. pip3 install pip-tools

Use the following to build a deterministic requirements.txt

pipreqs --savepath=requirements.in && pip-compile

5
  • 6
    It should be pip install pip-tools instead of pip install pip-tool Feb 21, 2022 at 11:20
  • When I ran pipreqs --savepath=requirements.in && pip-compile, pipreqs created a requirements.in file in addition to requirements.txt. Can you elaborate in your post or a comment why requirements.in is generated? Dec 20, 2022 at 2:44
  • 1
    @SergeiWallace i don't know if this is still useful to you, but it created requriements.in because pipreqs --savepath=requirements.in && pip-compile has --savepath=requirements.in with a in extension not txt Jan 9 at 19:32
  • @SergeiWallace The requirements.in file is only used as an intermediate file which pip-compile then uses to generate the final requirements.txt.
    – Cory Gross
    May 23 at 14:09
  • github.com/bndr/pipreqs appears to be dead: "build no longer available" badge, "Looking for maintainers to move this project forward." Version installed from pip crashed on a small toy project. Jul 28 at 7:28
54

In my case, I use Anaconda, so running the following command from a Conda terminal inside my environment solved it, and created this requirements.txt file for me automatically:

conda list -e > requirements.txt

This was taken from this Github link pratos/condaenv.txt.

If an error have been seen, and you are using Anaconda, try to use the .yml option:

conda env export > <environment-name>.yml

For another person to use the environment or if you are creating a new environment on another machine:

conda env create -f <environment-name>.yml

.yml option been found here

3
  • 1
    if you're not using a virtual environment, this will create an overly long requirements.txt file
    – SBFRF
    Jul 14, 2019 at 14:28
  • 3
    Does this actually work? Whenever I have tried this there are dependencies and syntax particulars that pip install requirements.txt rejects.
    – conner.xyz
    Jul 15, 2019 at 20:33
  • I found some errors using the .txt, so just added the other option that worked with me too..."Using the .yml" mentioned above. Mar 3, 2020 at 23:50
41

I blindly followed the accepted answer of using pip3 freeze > requirements.txt

It generated a huge file that listed all the dependencies of the entire solution, which is not what I wanted.

So you need to figure out what sort of requirements.txt you are trying to generate.

If you need a requirements.txt file that has ALL the dependencies, then use the pip3

pip3 freeze > requirements.txt

However, if you want to generate a minimal requirements.txt that only lists the dependencies you need, then use the pipreqs package. Especially helpful if you have numerous requirements.txt files in per component level in the project and not a single file on the solution wide level.

pip install pipreqs
pipreqs [path to folder]
e.g. pipreqs .
     pipreqs . --force --ignore=tests (Overwrites exisiting requirements.txt, ignores the tests directory)
3
  • If you don't to include subdirectories, you can add the --ignore <dirs> flag to your command
    – Francis
    Jun 17, 2021 at 1:55
  • If you require Juptyer notebook support, you can see jltz2's answer below stackoverflow.com/a/68544207/1622880
    – Francis
    Oct 24, 2021 at 23:00
  • github.com/bndr/pipreqs appears to be dead: "build no longer available" badge, "Looking for maintainers to move this project forward." Version installed from pip crashed on a small toy project. Jul 28 at 7:28
27

As most of the answers using pipreqs didn't work for me. Here, is my answer.

To generate the requirements.txt file:

pip install pipreqs

python -m  pipreqs.pipreqs --encoding utf-8  /path/to/project

I prefer using pipreqs more than pip freeze, as pip freeze saves all packages in the environment including those that you don't use in your current project. However, pipreqs only save the ones you are using in your project.

To install the requirements use:

pip3 install -r requirements.txt
3
  • I tried this but it missed some packages. I think this only grabs from the import statements, so it can miss some things you need, and will cause errors on deployment.
    – McGrew
    Aug 26, 2022 at 19:20
  • 1
    Yes, as I have mentioned, use it if you only want those packages, not all the packages in the environment. Aug 27, 2022 at 1:19
  • github.com/bndr/pipreqs appears to be dead: "build no longer available" badge, "Looking for maintainers to move this project forward." Version installed from pip crashed on a small toy project. Jul 28 at 7:29
20

Firstly, your project file must be a py file which is direct python file. If your file is in ipynb format, you can convert it to py type by using the line of code below:

jupyter nbconvert --to=python

Then, you need to install pipreqs library from cmd (terminal for mac).

pip install pipreqs

Now we can create txt file by using the code below. If you are in the same path with your file, you can just write ./ . Otherwise you need to give path of your file.

pipreqs ./

or

pipreqs /home/project/location

That will create a requirements.txt file for your project.

2
  • I get No template sub-directory with name 'python' found in the following paths
    – koppor
    Jun 8, 2021 at 15:31
  • You can try uninstalling and installing nbconvert using: pip uninstall nbconvert pip install nbconvert @koppor
    – berkayln
    Jun 9, 2021 at 7:03
11

Make sure to run pip3 for python3.7.

pip3 freeze >> yourfile.txt

Before executing the above command make sure you have created a virtual environment.

python3:

pip3 install virtualenv
python3 -m venv <myenvname> 

python2:

pip install virtualenv
virtualenv <myenvname>

After that put your source code in the directory. If you run the python file now, probably it won't launch if you are using non-native modules. You can install those modules by running pip3 install <module> or pip install <module>.

This will not affect you entire module list except the environment you are in.

Now you can execute the command at the top and now you have a requirements file which contains only the modules you installed in the virtual environment. Now you can run the command at the top.

I advise everyone to use environments as it makes things easier when it comes to stuff like this.

11

Create file requirement.txt

For the Python 3 version, the command is:

pip3 freeze > requirements.txt

For the Python 2 version, the command is:

pip freeze > requirements.txt

Install requirements.txt

For the Python 3 version, the command is:

pip3 install -r requirements.txt

For the Python 2 version, the command is:

pip install -r requirements.txt
9

Simple Pythonic Way

To get a list of all the REQUIREMENTS in a standard requirements.txt file, you can use the following command.

pip freeze > requirements.txt

Now, this should automatically create a standard requirements file with all of the packages installed alongside their corresponding versions.

Pretty Print on Terminal

If you just want to get a pretty print on the terminal you can use the following approach.

pip list

This lists all of the installed packages, in a pretty print format.

Custom Dependency

If you have a project folder like say, a Github Repo, and you want to get a custom requirements.txt for project You can use the following Package. https://pypi.org/project/pipreqs/ pipreqs

Usage

$ pipreqs /home/project/location
Successfully saved requirements file in /home/project/location/requirements.txt

Contents of requirements.txt

wheel==0.23.0
Yarg==0.1.9
docopt==0.6.2
8

If you have installed many dependencies in your system and you need requirements.txt for a specific project, you can install first pipreqs:

$ pip install pipreqs

and execute the below command under the project folder.

$ pipreqs

This command will generate requirements.txt file for the particular project.

1
  • github.com/bndr/pipreqs appears to be dead: "build no longer available" badge, "Looking for maintainers to move this project forward." Version installed from pip crashed on a small toy project. Jul 28 at 7:29
8

Automatic requirements.txt updating approach

While developing a Python application with requirements.txt we have several choices:

  1. Generate requirements.txt after development, when we want to deploy it. It is performed by pip freeze > requirements.txt or pipreqs for less messy result.
  2. Add every module to file requirements.txt manually after each install.
  3. Install manager that will handle requirements.txt updates for us.

There are many answers for the first option, the second option is self-explanatory, so I would like to describe the third approach. There is a library called to-requirements.txt. To install it, type this:

pip install to-requirements.txt  # Pip install to requirements.txt

If you read the whole command at once, you would see what it does. After installing you should set it up. Run:

requirements-txt setup

It overrides the pip scripts so that each pip install or pip uninstall updates the requirements.txt file of your project automatically with required versions of packages. The overriding is made safely, so that after uninstalling this package the pip will behave ordinary.

And you could customize the way it works. For example, disable it globally and activate it only for the required directories, activate it only for git repositories, or allow / disallow to create requirements.txt file if it does not exist.

References:

  1. Documentation
  2. GitHub
  3. PyPI
7

If Facing the same issue as mine i.e. not on the virtual environment and wants requirements.txt for a specific project or from the selected folder(includes children) and pipreqs is not supporting.

You can use :

import os
import sys
from fuzzywuzzy import fuzz
import subprocess

path = "C:/Users/Username/Desktop/DjangoProjects/restAPItest"


files = os.listdir(path)
pyfiles = []
for root, dirs, files in os.walk(path):
      for file in files:
        if file.endswith('.py'):
              pyfiles.append(os.path.join(root, file))

stopWords = ['from', 'import',',','.']

importables = []

for file in pyfiles:
    with open(file) as f:
        content = f.readlines()

        for line in content:
            if "import" in line:
                for sw in stopWords:
                    line = ' '.join(line.split(sw))

                importables.append(line.strip().split(' ')[0])

importables = set(importables)

subprocess.call(f"pip freeze > {path}/requirements.txt", shell=True)

with open(path+'/requirements.txt') as req:
    modules = req.readlines()
    modules = {m.split('=')[0].lower() : m for m in modules}


notList = [''.join(i.split('_')) for i in sys.builtin_module_names]+['os']

new_requirements = []
for req_module in importables:
    try :
        new_requirements.append(modules[req_module])

    except KeyError:
        for k,v in modules.items():
            if len(req_module)>1 and req_module not in notList:
                if fuzz.partial_ratio(req_module,k) > 90:
                    new_requirements.append(modules[k])

new_requirements = [i for i in set(new_requirements)]

new_requirements

with open(path+'/requirements.txt','w') as req:
    req.write(''.join(new_requirements))

P.S: It may have a few additional libraries as it checks on fuzzylogic.

7

You can just do it with the following commands. It will create file requirement.txt and add relevant modules automatically.

For Unix:

pip3 freeze > requirements.txt

For Windows:

pip freeze > requirements.txt
1
  • I think the difference between using pip3 and pip is coming from the Python Version difference. Not necessarily from the operating system that is used, CMIIW. For Python3 : pip3 freeze > requirements.txt For Python2: pip freeze > requirements.txt Jan 6 at 7:19
6

best way for Python 3 is:

pip3 freeze > requirements.txt

it worked for me...

1
6

@Francis has it right - https://stackoverflow.com/a/65728461/1021819

But just to add:

With additional support for Jupyter notebooks - i.e. .ipynb files - you can now use https://pypi.org/project/pipreqsnb (same syntax as pipreqs):

pip install pipreqsnb
pipreqsnb .

[I am not an author]

1
  • 1
    This works great, but you need to use the command as pipreqsnb ., not pipreqs .
    – c0d3rman
    Feb 2 at 19:44
5

If you want to list only packages used inside a virtualenv use:

pip freeze -l > requirements.txt 
0
4

Not a complete solution, but may help to compile a shortlist on Linux.

grep --include='*.py' -rhPo '^\s*(from|import)\s+\w+' . | sed -r 's/\s*(import|from)\s+//' | sort -u > requirements.txt
2

Or if you are using something like virtualenv, you can just run this command to generate a requirements.txt file:

./.venv/bin/pip freeze > requirements.txt
0
1

I created this bash command.

for l in $(pip freeze); do p=$(echo "$l" | cut -d'=' -f1); f=$(find . -type f -exec grep "$p" {} \; | grep 'import'); [[ ! -z "$f" ]] && echo "$l" ; done;
0
1

Using pip freeze > requirements.txt is a bad way to create the requirements file! It can serve as a temporary solution for your problem, but when managing requirements for a Python project it is best to do it manually.

A simple search for "import" or "from x import" will give you the list of all dependencies that need to be installed (nothing extra).

The problem with pip freeze it that it simply dumps all installed packages with strict versions, every dependency has its own dependencies and they are included in the dump. For example, you have lib==1.0 installed, that needs sub-lib==0.5, if you use pip freeze you'll get both, but later when you wish to update the version of lib to 2.0, most likely you'll get conflicts since lib v2.0 now uses sub-lib v1.0, not the v0.5 that you require... This gets complex fast for multiple dependencies.

We got into those problems in a couple of projects, and since then I created an automated script to clean pip freeze's dumps. It is safe (comments unneeded dependencies) and works great.

0

To help solve this problem, always run requirements.txt on only local packages. By local packages I mean packages that are only in your project folder. To do this do: Pip freeze —local > requirements.txt

Not pip freeze > requirements.txt. Note that it’s double underscore before local.

However installing pipreqs helps too. Pip install pipreqs.

The perfect solution though is to have a pipfile. The pipfile updates on its own whenever you install a new local package. It also has a pipfile.lock similar to package.json in JavaScript. To do this always install your packages with pipenv not pip. So we do pipenv

2
  • I have a project which only needs robotframework and rpaframework. I have tried both pipreqs -> it gives me just robotframework then pip freeze > requirements.txt gives many libraries which not in used in the project. how do I use these tool/command to output the requiremnt correctly? it will helps in setting up the new environment. Jun 28 at 10:03
  • What do you mean by "run requirements.txt"? Create? Use? Change? Something else? Overwrite? Please respond by editing (changing) your answer, not here in comments (but *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** without *** *** *** *** *** *** *** *** *** *** *** *** *** *** "Edit:", "Update:", or similar - the answer should appear as if it was written today). Sep 2 at 17:22
0

Pipenv users can generate the requirement.txt file from the project's Pipfile with:

pipenv lock --requirements
1
  • Error: No such option: --requirements Did you mean --quiet?
    – melbic
    May 9 at 16:04
0

For a different use case from the original question, where you do have the environment but have not pinned your dependencies, I found pipreqs to be inconsistent as it would sometimes duplicate dependencies with different versions like two entries of pillow with version 9.0.0 and 9.5.0 as well as fail to catch some needed dependencies.

I realized pipreqs is overkill for me personally.
I saw several people mention that pip freeze comes with caveat of storing ALL of your environments packages.
However, there is a simple solution that works well for me:

Sort out unpinned top level requirements into dev and core files like so:

requirements
├── core.txt
└── dev.txt

Since you probably only want your core requirements for production,

pip freeze | grep -F -f requirements/core.txt > requirements.txt

And let's say you wanted to omit extra like trailing +cpu at the end of the requirement names (which I needed to do for a Hugging Face deployment), you could chain a piped filter beforehand.

pip freeze | grep -F -f requirements/core.txt | sed 's/+cpu//g' > requirements.txt

I find this much simpler to maintain.

-1

Use:

pipreqs --savepath=requirements.in && pip-compile

It won't be working if you are on Windows PowerShell.

Try this;

( pipreqs --savepath=requirements.in ) -and (pip-compile)

Not the answer you're looking for? Browse other questions tagged or ask your own question.