There is nothing incorrect about the answers added so far, but they will only take you so far.
You may find yourself in the same situation as I did recently if your project begins to scale up.
What is the limitation of modules?
Consider a project structure consisting of a "main script" and some library code which is organized into 1 or more modules.
What is the limitation of this? What happens if you want to have more than one binary?
Asside: Python does not have binaries as such. But you can have multiple scripts each with their own entry point. (A def main()
function analagous to int main()
in C, and similar main
functions in other languages.) You may at some point find yourself wanting to write a project which consists of 1 or more libraries which are shared by multiple Python "binaries" (scripts). This is exactly analagous to the kind of project structure you might write in other languages including C, C++, Rust, Java, C#, and many others.
Back to the point: Using only the modules approach described above you are limited, and will be forced down a route of putting all library code in the same directory as a collection of things which are meant to be used like executables.
This is obviously not great.
A possible solution using Virtual Environments, a build backend like setuptools, and a pyproject.toml
file:
In summary, do the following things:
- create a Virtual Environment with
python3 -m venv .venv
- activate it (enable the venv)
source .venv/bin/activate
Asside: I like to create a symlink to .venv/bin/activate
so I can simply run . activate
from the project root to activate my venvs.
- install any libraries you need with
pip3 install ...
- create a folder for your "binaries" called
bin
, put your "executable" Python scripts here
- create a folder called
src
for your library code
- for each library module create a directory under
src
- add an
__init__.py
file to each library module
- add your module code for each library module
- create
pyproject.toml
in the project root (see below)
- "build" your libraries, and "install" them
pip3 install -e .
from the project root
What does all of this do?
What you are effectively doing here is moving module code out of one directory, where the python interpreter can find it, into another where it cannot. That doesn't seem to make a lot of sense, except to say the project directory structure is now improved. You should have some minimal "executable" python scripts in a bin
directory and everything else logically organized under a src
directory.
In order for the Python interpreter to find it, what you are doing here is "installing" your library code into the active Virtual Environment using pip3
. In order to install it, you first need to build it with setuptools
. When you build it and install it you use the -e
flag. This installs it using symlinks instead of the usual way, which means if you modify your code, you can use the modifications immediatly without having to run the install step again.
Hopefully that all makes sense, I only recently learned about this, so this is my best attempt to provide a straightforward explanation.
Example toml
file
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "my_project_name"
version = "1.0.0"
description = "My Project Description"
[tools.setuptools.packages.find]
where = ["src"]