268

I am installing packages from requirements.txt

pip install -r requirements.txt

The requirements.txt file reads:

Pillow
lxml
cssselect
jieba
beautifulsoup
nltk

lxml is the only package failing to install and this leads to everything failing (expected results as pointed out by larsks in the comments). However, after lxml fails pip still runs through and downloads the rest of the packages.

From what I understand the pip install -r requirements.txt command will fail if any of the packages listed in the requirements.txt fail to install.

Is there any argument I can pass when running pip install -r requirements.txt to tell it to install what it can and skip the packages that it cannot, or to exit as soon as it sees something fail?

5
  • 5
    Remove lxml from your requirements.txt
    – RickyA
    Mar 7, 2014 at 12:49
  • 3
    Thanks, that would work in this case, but in general, is there any way around this? Or is it normal just to run the command, see that it fails and then prune the package list?
    – e h
    Mar 7, 2014 at 12:52
  • 8
    If a package is listed in requirements.txt it is presumably required, so it makes sense that pip would fail if the package can't be installed. If the code runs anyway without that package then it was hardly a requirement. Randomly pruning failed packages from requirements.txt seems like it's just going to cause problems with missing dependencies.
    – larsks
    Mar 7, 2014 at 13:03
  • 2
    @larsks it can be very common to have multiple requirements.txt files for a single development environment or package. For example, there may be one full of optional tools that can be used to enhance the unit/integration test environment or performance profiling, but which are not strictly required. You still want to version control a single source file expressing these packages and any pinned versions, and rely on the same pip install -r <some file> workflow to standardize creation of the appropriate environment.
    – ely
    Jun 28, 2019 at 15:08
  • It's very surprising to me that 1) pip continues installing packages once it encounters a failure (download, build or otherwise) and 2) there's no inbuilt way to opt out of this behavior. Nov 7, 2020 at 10:29

12 Answers 12

437

Running each line with pip install may be a workaround.

cat requirements.txt | xargs -n 1 pip install

Note: -a parameter is not available under MacOS, so old cat is more portable.

10
  • 27
    for mac: cat requirements.txt | xargs -n 1 pip install Sep 4, 2016 at 1:32
  • 7
    I had to do: cat requirements.txt | cut -f1 -d"#" | sed '/^\s*$/d' | xargs -n 1 pip install to remove anything in the comments and get rid of empty lines.
    – Narek
    Apr 17, 2018 at 20:47
  • 1
    At least with GNU xargs, there's -a flag option, which allows xargs read arguments from file, so this can be done as xargs -n 1 -a requirements.txt pip install. Prevents UUOC and excessive plumbing Sep 3, 2018 at 22:28
  • 39
    For windows :) FOR /F %k in (requirements.txt) DO pip install %k
    – wcyn
    Aug 25, 2019 at 8:22
  • Even without the -a flag, consider < requirements.txt xargs -n 1 pip install, or if you don't like the redirect at the front, xargs -n 1 pip install < requirements.txt. :-)
    – torek
    Jul 6, 2020 at 17:21
33

This solution handles empty lines, whitespace lines, # comment lines, whitespace-then-# comment lines in your requirements.txt.

cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 pip install

Hat tip to this answer for the sed magic.

3
  • 1
    Works nicely. I used pip freeze instead of cat requirements.txt.
    – Vishal
    May 14, 2019 at 15:57
  • This doesn't handle comments on the same line as a requirement is declared. For that you need to add -e 's/\s*#.*$//' to the sed command. Apr 6, 2021 at 2:15
  • I got the following: 'cat' is not recognized as an internal or external command, operable program or batch file.
    – Laurent T
    Feb 14, 2022 at 4:32
17

For windows users, you can use this:

FOR /F %k in (requirements.txt) DO ( if NOT # == %k ( pip install %k ) )

Logic: for every dependency in file(requirements.txt), install them and ignore those start with "#".

12

For Windows:

pip version >=18

import sys
from pip._internal import main as pip_main

def install(package):
    pip_main(['install', package])

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)

pip version <18

import sys
import pip

def install(package):
    pip.main(['install', package])

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)
1
  • 'main' is not a part of pip. Error.
    – Pranzell
    Jul 29, 2019 at 14:29
5

Building on the answer by @MZD, here's a solution to filter out all text starting with a comment sign #

cat requirements.txt | grep -Eo '(^[^#]+)' | xargs -n 1 pip install
0
4

The xargs solution works but can have portability issues (BSD/GNU) and/or be cumbersome if you have comments or blank lines in your requirements file.

As for the usecase where such a behavior would be required, I use for instance two separate requirement files, one which is only listing core dependencies that need to be always installed and another file with non-core dependencies that are in 90% of the cases not needed for most usecases. That would be an equivalent of the Recommends section of a debian package.

I use the following shell script (requires sed) to install optional dependencies:

#!/bin/sh

while read dependency; do
    dependency_stripped="$(echo "${dependency}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
    # Skip comments
    if [[ $dependency_stripped == \#* ]]; then
        continue
    # Skip blank lines
    elif [ -z "$dependency_stripped" ]; then
        continue
    else
        if pip install "$dependency_stripped"; then
            echo "$dependency_stripped is installed"
        else
            echo "Could not install $dependency_stripped, skipping"
        fi
    fi
done < recommends.txt
4

One line PowerShell:

Get-Content .\requirements.txt | ForEach-Object {pip install $_}

If you need to ignore certain lines then:

Get-Content .\requirements.txt | ForEach-Object {if (!$_.startswith("#")){pip install $_}}

OR

Get-Content .\requirements.txt | ForEach-Object {if ($_ -notmatch "#"){pip install $_}}

3

For Windows using PowerShell:

foreach($line in Get-Content requirements.txt) {
    if(!($line.StartsWith('#'))){
        pip install $line
    }
}
2

Another option is to use pip install --dry-run to get a list of packages that you need to install and then keep trying it and remove the ones that don't work.

1
  • FYI only available in pip 22.2+
    – Geordie
    Jun 8 at 20:38
1

A very general solution

The following code installs all requirements for:

  • multiple requirement files (requirements1.txt, requirements2.txt)
  • ignores lines with comments #
  • skips packages, which are not instalable
  • runs pip install each line (not each word as in some other answers)
$ (cat requirements1.txt; echo ""; cat requirements2.txt) | grep "^[^#]" | xargs -L 1 pip install
0

Thanks, Etienne Prothon for windows cases.

But, after upgrading to pip 18, pip package don't expose main to public. So you may need to change code like this.

 # This code install line by line a list of pip package 
 import sys
 from pip._internal import main as pip_main

 def install(package):
    pip_main(['install', package])

 if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        for line in f:
            install(line)
-2

For Windows:

import os
from pip.__main__ import _main as main

error_log = open('error_log.txt', 'w')

def install(package):
    try:
        main(['install'] + [str(package)])
    except Exception as e:
        error_log.write(str(e))

if __name__ == '__main__':
    f = open('requirements1.txt', 'r')
    for line in f:
        install(line)
    f.close()
    error_log.close()
  1. Create a local directory, and put your requirements.txt file in it.
  2. Copy the code above and save it as a python file in the same directory. Remember to use .py extension, for instance, install_packages.py
  3. Run this file using a cmd: python install_packages.py
  4. All the packages mentioned will be installed in one go without stopping at all. :)

You can add other parameters in install function. Like: main(['install'] + [str(package)] + ['--update'])

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.