56

Background:

I have a directory structure like so:

Package/
    setup.py
    src/
        __init__.py
        __main__.py 
        code.py

I want to be able to run the code in a lot of different ways.

  1. pip install Package and then python and then from Package import *

  2. python -m Package which should do the thing in __main__.py

  3. python __main__.py which should also do the thing in __main__.py but this time, we assume you've downloaded source rather than pip installing.

Now I've gotten the first two to work, but with a messy setup:

setup.py:

setup(
    name='Package',
    packages=['Package'],
    package_dir={'Package': 'src'},
    ...
    entry_points={ 'console_scripts': ['Package = src.__main__:main' ] }

__init__.py:

from Package.code import .......

__main__.py:

from . import .......

What would make more sense to me would be in both cases to write

from code import ........

but that gives me import errors.

Question:

Is the way I have it really the only way?

And most importantly, how do I support the third use case? Right now, python __main__.py throws

File "__main__.py", line 10, in <module>
    from . import code
ImportError: cannot import name 'class defined in code.py'

Notes:

I have read

6
  • from code import is implicit relative import which is not recommended (and doesn't work in Python 3). Either use absolute import or explicit relative import.
    – phd
    Jul 7, 2017 at 18:58
  • @phd could you elaborate a little bit as to what each of those are? Jul 7, 2017 at 19:04
  • 1
    Absolute import means to use full path: from Package.somemodule.submodule import even inside Package or somemodule. Relative import is to import from neighbours without using full path. from code import is implicit rel. import because it looks like absolute but really is relative. Relative import was the default mode in Python 2 but its implicit mode is now forbidden in Python 3 (default is now abs. import). Explicit relative import is from .code import — see the dot in .code? It means "import from the current module".
    – phd
    Jul 7, 2017 at 19:10
  • Got it, so now we should always do from Package.somemodule.submodule import ? Jul 7, 2017 at 19:13
  • This is what PEP8 recommends (though officialy PEP8 is only for stdlib).
    – phd
    Jul 7, 2017 at 19:20

3 Answers 3

47
+50

You have almost everything you need (even a bit more)! I'd go with the following setup:

code.py:

foo = 1

__init__.py:

from .code import foo

Doing a relative import here because __init__.py will be used when importing the whole package. Note that we explicitly mark the import as relative by using the .-syntax because this is required for Python 3 (and in Python 2 if you did from __future__ import absolute_import).

__main__.py:

from Package import foo

print('foo = ', foo)

This is the package's main script and so we use an absolute import statement. By doing so we assume that the package has been installed (or at least has been put on the path); and that is the way packages should be dealt with! You might think that this conflicts with your third use case but actually there is no reason not to pip install when dealing with a package. And it really isn't a big deal (especially when using a virtualenv)!

If your concern is to tinker with the source files and readily observe the changes by running the __main__.py file then you can simply install the package using the -e ("editable") switch: pip install -e . (assuming you are in directory Package). With your current directory structure, however, this won't work because the -e switch will place an egg-link to the directory containing the setup.py file; this directory does not contain a package named Package but src instead (I have a question about that).

Instead, if you follow the convention to name the root directory of a package's source after the package itself (that is Package for your example) then installing with -e is not a problem: Python does find the required package Package in the corresponding directory:

$ tree Package/
Package/
├── setup.py
└── Package   <-- Renamed "src" to "Package" because that's the package's name.
    ├── code.py
    ├── __init__.py
    └── __main__.py

This also lets you omit the extra definition of package_dir={'Package': 'src'} in setup.py.

A note about setup.py: For the three use cases which you've specified there is no need to define an entry point. That is you can skip the line entry_points={ 'console_scripts': ['Package = src.__main__:main' ] }. By shipping a __main__.py module python -m Package will readily execute the code in this module. You can also add an extra if-clause:

def main():
    print('foo = ', foo)

if __name__ == '__main__':
    main()

The entry point on the other hand lets you directly execute the code in __main__.main from the CLI; that is running $ Package will execute the corresponding code.

Recap

The bottom line is that I'd always use pip install when dealing with packages. And why not, especially if you've already created a setup.py file? If changes to the package are to be applied "in real-time" then you can install with the -e switch (this might require a renaming of the src folder, see above). So your third use case would read as "Download the source and pip install (-e) Package (within a virtualenv); then you can run python __main__.py".


Edit

Run __main__.py without pip install

If you don't want to install the package via pip but still be able to run the __main__.py script, I'd still go with the above setup. Then we need to make sure that the from Package import ... statement(s) are still succeeding and this can be achieved by extending the import path (note that the this requires the src directory to be renamed to the package's name!).

Modify PYTHONPATH

For Linux bash you can set the Pythonpath as follows:

export PYTHONPATH=$PYTHONPATH:/path/to/Package

Or if you're in the same directory as __main__.py:

export PYTHONPATH=$PYTHONPATH:`cd ..; pwd`

Of course there are different ways for different operating systems.

Extend the path in __main__.py

You (or rather your colleague) could add the following lines to the top of the script (before the from Package import ... statements):

import sys
sys.path.append('/path/to/Package')

Extend the path in sitecustomize.py

You can place a module named sitecustomize.py in the lib/python3.5/site-packages/ directory of your Python installation which contains the following lines:

import sys
sys.path.append('/path/to/Package')

Create a separate, top-level main.py script

So you'd have the following layout:

$ tree Package/
Package/
├── main.py   <-- Add this file.
├── setup.py
└── src
    ├── code.py
    ├── __init__.py
    └── __main__.py

where main.py contains

import src.__main__

Now __main__.py is treated as a part of the src package and the relative import will work. Instead of running python src/__main__.py you would run python main.py now.

7
  • I'm accepting this answer because it's the most complete and thorough. Thanks for your answer a_guest! I'm only bummed there's no easy way to clone the repo (instead of pip installing) and running main.py directly -- one of my colleagues was hoping to use the code that way... I might put a try-except block at the top of the file for importing, but that feels icky... Jul 19, 2017 at 16:52
  • @AlexLenail Now I understand a bit better what you need the third use case for. I made an edit to my answer which contains a few suggestions on how to run __main__.py without having done pip install beforehand. They are based on extending the import path with the directory in which the source folder is located (they require the src directory to be renamed to the package's name). No changes to the actual code are required though. On the other hand I'd expect the repo to contain the setup.py file and you can pip install from that file. So it really isn't that difficult.
    – a_guest
    Jul 19, 2017 at 22:02
  • 1
    Nice answer. Note that you need not and should not modify the Python path to run the Package/__main__.py file from the file system. Simply type python Package/ instead of python Package/__main__.py and this will work (the python command accepts directories).
    – Géry Ogam
    Nov 21, 2019 at 17:14
  • 1
    Also about Python path manipulations, note that you can also create a path configuration file in one of the four sys.prefix, sys.exec_prefix, sys.prefix/lib/site-packages, sys.exec_prefix/lib/site-packages directories of your Python environment. It is a file with a .pth extension whose contents are additional paths (one per line) appended to sys.path by the site module which is automatically imported at python startup unless the -S option is used.
    – Géry Ogam
    Nov 21, 2019 at 17:29
  • 1
    @Konrad There are different approaches, either including the tests with the actual package (inside src) or, more commonly, having a separate test directory (next to src). In any case I would use only absolute imports inside tests as it makes it clear what is being tested and independent of the test's location. There exist many opinions on how to structure the test directory, but loosely it should follow the package's structure. You can check out popular projects such as requests or attrs to explore how they did it.
    – a_guest
    Mar 17, 2022 at 8:52
3

from code import ......... fails because there is no Python package installed on your system named code. There is a Python module on your system named code, but in your import statement you don't specify the package that your code module can be found in.

The purpose of the __init__.py file that you have in src/ tells Python that the src/ directory should be treated a Python package, with its contents as the modules within the package. Since code.py is located in src/ along with your __init__.py file, your code module is located in your src package.

Now that you know which package your code module can be found in, you can import stuff from it with:

from src.code import .........

Also, as a side note: The __init__.py does its job just by being present in your src/ directory, so it doesn't even need to contain any code. For that reason it's generally a good idea to leave the __init__.py file blank.

2
  • Hi @MatTheWhale! Thanks for answering here. Unfortunately, I'm not sure you've answered my question about __main__.py, specifically, how to support the third use case, which probably requires something to replace from . import ..... Also, leaving __init__.py blank for this package doesn't work when other folks install the package. Jul 10, 2017 at 14:11
  • @MatTheWhale I cannot follow your argumentation; code is located in the directory src but the package's name is Package, so I don't see how from src.code import ... would help. Also "the purpose" of an __init__.py is much different in Python 2 than in Python 3, and I don't agree that "it's generally a good idea to leave the __init__.py file blank" - there are many cases where __init__.py is actively used; just imagine making contents available at the package level.
    – a_guest
    Jul 19, 2017 at 15:28
1

I often use this setup because it works better with python setup.py develop

Package_root/
    setup.py
    src/
        Package/
            __init__.py
            __main__.py 
            code.py

It's probably not (yet) the detailed answer you're expecting but I think it's worth trying for the three use cases.

setup( ...
    package_dir = {'': 'src'},
    entry_points = {'console_scripts': ['Package = Package.__main__:main'],},
    packages = find_packages(exclude=["Package.egg_info",]),
...)
2
  • 1
    what does package_dir = {'': 'src'} do?
    – ikamen
    Apr 19, 2019 at 12:34
  • @ikamen I think I found this syntax in this doc page Apr 21, 2019 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.