267

Background

I was about to try Python package downloaded from GitHub, and realized that it did not have a setup.py, so I could not install it with

pip install -e <folder>

Instead, the package had a pyproject.toml file which seems to have very similar entries as the setup.py usually has.

What I found

Googling lead me into PEP-518 and it gives some critique to setup.py in Rationale section. However, it does not clearly tell that usage of setup.py should be avoided, or that pyproject.toml would as such completely replace setup.py.

Questions

Is the pyproject.toml something that is used to replace setup.py? Or should a package come with both, a pyproject.toml and a setup.py?
How would one install a project with pyproject.toml in an editable state?

2
  • 4
    See PEP-518.
    – Klaus D.
    Jul 19, 2020 at 18:02
  • 12
    Thanks, @KlausD, that was in the top Google results but the PEP-518 did not take stance on should developers avoid using setup.py, and how to install the packages in editable state, if setup.py is not used, etc.
    – np8
    Jul 19, 2020 at 18:31

5 Answers 5

118

Yes, pyproject.toml is the specified file format of PEP 518 which contains the build system requirements of Python projects.

This solves the build-tool dependency chicken and egg problem, i.e. pip can read pyproject.toml and what version of setuptools or wheel one may need.

If you need a setup.py for an editable install, you could use a shim in setup.py:

#!/usr/bin/env python

import setuptools

if __name__ == "__main__":
    setuptools.setup()
5
  • 35
    Thanks for the shim! So it seems that the pyproject.toml is almost a replacement for setup.py and developers are expected to use both so that the project can be installed in editable state? Weirdly, I have been using setup.py for few years in various small projects and never had a need to remove a "chicken and egg problem".
    – np8
    Jul 19, 2020 at 18:38
  • This is also useful if you aren't use setuptools, for example flit or poetry
    – pce
    Jul 19, 2020 at 19:56
  • 2
    I'm confused... what does pip have to do with building? Isn't pip just for installing dependencies?
    – Shannon
    Jul 2, 2021 at 2:15
  • 2
    @Shannon To build a Project you may need to install at least some abstract dependencies to execute the build system. pip can act as a installer backend, while pip wheel target-dir pip is acting as a build frontend and with pip install lxml==2.4.0 as a integration fronted, see PEP517
    – pce
    Jul 4, 2021 at 4:56
  • 2
    A common chicken and egg problem is building an extension with pybind11. setup.py needs to add pybind11.get_include() to the compiler flags, but can't do that unless pybind11 is installed. And if this is specified as a dependency in setup.py ...
    – ChrisD
    Jul 20, 2021 at 21:14
99

pyproject.toml is the new unified Python project settings file that replaces setup.py. Editable installs still need a setup.py: import setuptools; setuptools.setup()

To use pyproject.toml, run python -m pip install .

Then, if the project is using poetry instead of pip, you can install dependencies (into %USERPROFILE%\AppData\Local\pypoetry\Cache\virtualenvs) like this:

poetry install

And then run dependencies like pytest:

poetry run pytest tests/

And pre-commit (uses .pre-commit-config.yaml):

poetry run pre-commit install
poetry run pre-commit run --all-files
9
  • 3
    In what sense is setup.py non-editable?
    – Dave
    May 23, 2021 at 15:57
  • 22
    @Dave An editable install is an installation where the installed files are symlinks pointing back to the source directory. This is particularly useful for developers, so they don't need to reinstall the package every time they change a line of code.
    – gerrit
    Jun 25, 2021 at 7:14
  • what if I need to run arch -arm64 pip3 install psycopg2 --no-binary :all: AND update pyproject.toml?
    – Ren
    Apr 10, 2022 at 18:44
  • @Ren stackoverflow.com/a/27043037/819417 with poetry update instead of pip install Apr 15, 2022 at 4:06
  • @CeesTimmerman yeah but that doesn't add the arch -arm64 command nor the --no-binary :all: options.
    – Ren
    Apr 16, 2022 at 5:10
80

What is it for?

Currently there are multiple packaging tools being popular in Python community and while setuptools still seems to be prevalent it's not a de facto standard anymore. This situation creates a number of hassles for both end users and developers:

  1. For setuptools-based packages installation from source / build of a distribution can fail if one doesn't have setuptools installed;
  2. pip doesn't support the installation of packages based on other packaging tools from source, so these tools had to generate a setup.py file to produce a compatible package. To build a distribution package one has to install the packaging tool first and then use tool-specific commands;
  3. If package author decides to change the packaging tool, workflows must be changed as well to use different tool-specific commands.

pyproject.toml is a new configuration file introduced by PEP 517 and PEP 518 to solve these problems:

... think of the (rough) steps required to produce a built artifact for a project:

  1. The source checkout of the project.
  2. Installation of the build system.
  3. Execute the build system.

This PEP [518] covers step #2. PEP 517 covers step #3 ...

Any tool can also extend this file with its own section (table) to accept tool-specific options, but it's up to them and not required.

PEP 621 suggests using pyproject.toml to specify package core metadata in static, tool-agnostic way. Which backends currently support this is shown in the following table:

Does it replace setup.py?

For setuptools-based packages pyproject.toml is not strictly meant to replace setup.py, but rather to ensure its correct execution if it's still needed. For other packaging tools – yes, it is:

Where the build-backend key exists, this takes precedence and the source tree follows the format and conventions of the specified backend (as such no setup.py is needed unless the backend requires it). Projects may still wish to include a setup.py for compatibility with tools that do not use this spec.

How to install a package in editable mode?

Originally "editable install" was a setuptools-specific feature and as such it was not supported by PEP 517. Later on PEP 660 extended this concept to packages using pyproject.toml.

There are two possible conditions for installing a package in editable mode using pip:

  • Modern:
    Both the frontend (pip) and a backend must support PEP 660.
    pip supports it since version 21.3;
  • Legacy:
    Packaging tool must provide a setup.py file which supports the develop command.
    Since version 21.1 pip can also install packages using only setup.cfg file in editable mode.

The following table describes the support of editable installs by various backends:

12

Answering this part only, as the rest has nicely been explained by others:

How would one install a project with pyproject.toml in an editable state?

Solution

Since the release of poetry-core v1.0.8 in Feb 2022 you can do this:

a) you need this entry in your pyproject.toml:

[build-system]
requires = ["poetry-core>=1.0.8"]
build-backend = "poetry.core.masonry.api"

b) run:

pip install -e .

Sources

3
  • 4
    Update 2022: PEP 660 is implemented in pip (from 21.3.0) and some backends, e.g. flit. However no progress in setuptools.
    – kap
    Jan 8, 2022 at 13:50
  • Note that the poetry install command already makes an editable install by default, which is also explained in the issue you've linked.
    – EvgenKo423
    May 8, 2022 at 13:32
  • 1
    The PR in poetry seems to be merged already. Does this means that editable versions with poetry do not require this workaround anymore? Feb 19 at 11:28
2

pyproject.toml can declare the files in your python package and all the metadata for it that will show in PyPi.

A tool like flit can process the pyproject.toml file into a package that can be uploaded to PyPi or installed with pip.

Other tools use pyproject.toml for other purposes. For example, pytest stores information about where to find and how to run tests, and instructions to pytest about modifying pythonpath (sys.path) before running the tests. Many IDEs can use this to help developers conveniently run tests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.