20

Consider the following Python project skeleton:

proj/
├── foo
│   └── __init__.py
├── README.md
└── scripts
    └── run.py

In this case foo holds the main project files, for example

# foo/__init__.py
class Foo():
    def run(self):
        print('Running...')

And scripts holds auxiliary scripts that need to import files from foo, which are then invoked via:

[~/proj]$ python scripts/run.py

There are two ways of importing Foo which both fail:

  1. If a relative import is attempted from ..foo import Foo then the error is ValueError: attempted relative import beyond top-level package
  2. If an absolute import is attempted from foo import Foo then the error is ModuleNotFoundError: No module named 'foo'

My current workaround is to append the running path to sys.path:

import sys
sys.path.append('.')

from foo import Foo
Foo().run()

But this feels like a hack, and has to be added to every new script in scripts/.

Is there a better way to structure scripts in such projects?

1
  • 1
    Is there a reason you do not put a __init__.py into proj, or scripts diretly into foo since they are all auxillary functions anyway?
    – Andreas
    Sep 4, 2020 at 11:50

7 Answers 7

17
+100

There's two ways you could resolve this.

(1) Turn your project into an installable package

Add a proj/setup.py file with the following contents:

import setuptools

setuptools.setup(
    name="my-project",
    version="1.0.0",
    author="You",
    author_email="[email protected]",
    description="This is my project",
    packages=["foo"],
)

create a virtualenv:

python3 -m venv virtualenv  # this creates a directory "virtualenv" in your project
source ./virtualenv/bin/activate  # this switches you into the new environment
python setup.py develop  # this places your "foo" package in the environment

inside the virtualenv, foo behaves as an installed package and is importable via import foo.

So you can use absolute imports in your scripts.

To make them run from anywhere, without needing to activate the virtualenv, you can then specify the path as a shebang.

In scripts/run.py (the first line is important):

#!/path/to/proj/virtualenv/bin/python

import foo

print(foo.callfunc())

(2) Make the scripts part of the foo package

Instead of a separate subdirectory scripts, make a subpackage. In proj/foo/commands/run.py:

from .. import callfunc()

def main():
    print(callfunc())

if __name__ == "__main__":
    main()

Then execute the script from the top-level proj/ directory with:

python -m foo.commands.run

If you combine this with (1) and install your package, you can then run python -m foo.commands.run from anywhere.

1
  • 1
    While this solution does make the module available–in order for this to be callable as a typical script–one needs to add the script to setup.py using one of two argument options, as detailed in my answer.
    – pygeek
    Sep 10, 2020 at 4:21
8

Best practice? Put a single entry-point in the root

I know this might sound absurd, if you have lots of scripts you want to be able to execute... But it's actually the cleanest option and it's the one that is most often used in big Python projects like magage.py in Django, for example. It also doesn't need to be a huge undertaking. Even more importantly, it is always more secure to have a single entry point than several smaller ones.

proj/
├── run.py
├── foo
│   └── __init__.py
├── README.md
└── scripts
    └── my_script.py

When run.py lives in the root directory, it can be very lightweight... Basically just a wrapper to call the function you need from my_scripts.py. It just ties everything together so now all of your imports just work.

Just keep in mind that your entrypoint is your root. The parent of a root doesn't exist. So put your entrypoint in the root, and then import packages relative to the root, aka import foo from scripts.

But how do I call multiple scripts!?

If you need to be able to call multiple scripts, this is a good argument for... Well... arguments! Keep run.py as your single entrypoint/command, and leverage subcommands to pass functionality to the script you care about.

Reinventing the wheel?

Generally, frameworks have already done the architecture for you to add your own subcommands, such as Django and, for a smaller footprint, Flask.

You can easily wrap up a small project without that help, though, as I've illustrated.

Security

No one ever wishes their code was less refactorable after a few years of working with it. No one ever wishes their codebase has less security. As we drive to more secure systems in general, it would make sense to create some gatekeeper script that determines what is and isn’t a safe operation and by whom. Moving the code to an LDAP based system, and need to lock things down by group? No problem. You can either change the single file or add LDAP security in your codebase, even creating your own internal API.

With distributed scripts, security options are much less flexible and much harder to maintain, and a single vulnerability could leave you wide open to exploit.

Bonus advantage You're adding abstraction to your script base. If you ever want to change the structure of your codebase (maybe you want scripts to have subfolders with more organization), you/your users don't need to do any refactoring for any dependencies, or change paths to longer, more verbose names. Your package is self-contained, and the only thing a user will ever need to touch is your proj/run.py entry-point.

And, obviously, you don't need to play with Python paths as much!

1
  • That's really a nice idea. I have multiple checkouts of different versions of my project to run various tests in parallel. Solutions like pip setup -e . works poorly in case because work only with a single version of a package.
    – reddot
    Aug 16, 2022 at 7:41
6

Solution

There are multiple ways to achieve this. Both require creating a python package by adding a setup.py (building on @matejcik's answer).

Option 1 (recommended): entry_point + console_scripts register a function in your project as the entry point to script execution (ie: proj:foo:cli:run).

Option 2: scripts: Use this keyword argument in the setup() method to reference the path to your script (ie: `bin/script.py).

Note

I recommend using a CLI library/framework like Click so that your codebase is only concerned with maintaining application specific business logic rather than CLI robust framework feature logic. Also, click recommends using entry_point + console_scripts method of script integration due to cross-platform compatibility.

Setup Tools - Automatic script creation: https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation

Setup Tools - keyword arguments: https://setuptools.readthedocs.io/en/latest/setuptools.html#new-and-changed-setup-keywords

Click GitHub: https://github.com/pallets/click/

Click Setuptools integration: https://click.palletsprojects.com/en/master/setuptools/

2

You need to add __init__.py files to scripts and to proj folders for those to be considered Python packages and for you to be able to import from those.

One way this is also commonly done, is to place your foo and scripts folders into a proj/src folder, which then has a __init__.py file, and thus is a Python package.

2
  • 1
    That doens't really answer my question, adding an __init__.py to the top-level dir is bad taste, nor does it actually resolve the error. The class in the init file is just for simplicity in this example.
    – Yuval Adam
    Sep 1, 2019 at 8:54
  • @YuvalAdam Yeah the comment about the class in the __init__.py was uwarranted, it' not terrible to have it there. But I stand by the fact that adding an __init__.py to the top-level directory is the way to go for sure.
    – ruohola
    Sep 9, 2020 at 19:17
1

If you like simplicity, and there are no additional restrictions on what you asked, add one __init__.py to the scripts folder, and to any other sibling folders, making them packages, then always use the absolute import form, as you said you do not want proj as a parent package of those and so there is no __init__.py there, and then call your scripts (instead) from inside the proj folder with:

python -m scripts.run

or whatever name you give to other scripts other than run.py

This is similar to option 2 of @matejcik answer, but even simpler.

1

another solution is you add a.pth file on your Python directory

enter image description here

and write the content of the following,

# your.pth 

#↓ input the directory of proj
C:\...\proj  

done

# scripts.py
from foo import Foo
Foo().run()

It will work well.

.. note:: If your IDE is PyCharm, then you can use the Source roots to help you too.

0

Python looks for packages/modules in the directories listed in sys.path. There are several ways of ensuring that your directories of interest, in this case proj, is one of those directories:

  1. Move your scripts to the proj directory. Python adds the directory containing the input script to sys.path.
  2. Put the directory proj into the contents of the PYTHONPATH environment variable.
  3. Make the module part of an installable package and install it, either in a virtual environment or not.
  4. At run time, dynamically add the directory proj to sys.path.

Option 1 is the most logical and requires no source changes. If you are afraid that might break something, you can perhaps make scripts a symbolic link pointing back to proj?

If you are unwilling to do that, then ...

You may consider it a hack, but I would recommend that you do modify your scripts to update sys.path at runtime. But instead append an absolute path so that the scripts can be executed regardless of what the current directory is. In your case, directory proj is the parent directory of directory scripts, where the scripts reside, so:

import sys
import os.path

parent_directory = os.path.split(os.path.dirname(__file__))[0]
if parent_directory not in sys.path:
    #sys.path.insert(0, parent_directory) # the first entry is directory of the running script, so maybe insert after that at index 1
    sys.append(parent_directory)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.