ANOTHER SUGGESTION
I explored this question and various others on SO and elsewhere... all the stuff about adding (or removing) an empty __init__.py
in and/or conftest.py in various parts of the project directory structure, all the stuff about PYTHONPATH, etc., etc.: NONE of these solutions worked for me, in what is actually a very simple situation, and which shouldn't be causing any grief.
I regard this as a flaw in pytest's current setup. In fact I got a message recently from someone on SO who clearly knew his stuff. He said that pytest is not designed to work with (as per Java/Groovy/Gradle) separate "src" and "test" directory structures, and that test files should be mingled in amongst the application directories and files. This perhaps goes some way to providing an explanation ... however, tests, particularly integration/functional tests, don't always necessarily correspond neatly to particular directories, and I think pytest should give users more choice in this regard.
Structure of my project:
project_root
src
core
__init__.py
__main__.py
my_other_file.py
tests
basic_tests
test_first_tests.py
The import problem posed: very simply, __main__.py
has a line import my_other_file
. This (not surprisingly) works OK when I just run the app, i.e. run python src/core
from the root directory.
But when I run pytest
with a test which imports __main__.py
I get
ModuleNotFoundError: No module named 'my_other_file'
on the line in __main__.py
that tries to import "my_other_file". Note that the problem here, in my case, is that, in the course of a pytest test, one application file fails to find another application file in the same package.
USING PYTHONPATH
After a lot of experimentation, and putting an __init__.py
file and a confest.py file in just about every directory I could find (I think the crucial files were __init__.py
added to "tests" and "basic_tests", see above directory structure), and then setting PYTHONPATH to as follows
PYTHONPATH=D:\My Documents\software projects\EclipseWorkspace\my_project\src\core
... I found it worked. Imports in the testing files had to be tweaked a bit, the general pattern being from core import app, project
, but the test files were able to "see" core
, and crucially there was no need to mess around with the import
commands in the app files.
HOWEVER... for some reason the tests now run much slower using this method! Compared to my solution below, where the core
package can be seen to be loaded just once, my suspicion is that the PYTHONPATH solution probably results in vast amounts of code being reloaded again and again. I can't yet confirm this, but I can't see any other explanation.
THE ALTERNATIVE
My alternative fairly simple solution is this:
1 - in __init__.py
in that application package (i.e. directory "core"), put the following two lines:
import pathlib, sys
sys.path.append(str(pathlib.Path(__file__).parent))
NB normally there isn't anything in an __init__.py
of course. It turns out, as I confirmed by experimentation, that pytest usually (see update below) executes __init__.py
in this situation, after pytest has done whatever it has done to mess up the sys.path
entries.
2 - UPDATE 2022-01:
The original solution I had found involved putting a conftest.py file in the application directory(ies) - without which things didn't work. This is obviously undesirable. I find that another solution is to put this code in your conftest.py file in your root directory:
def pytest_configure(config):
import src.core # NB this causes `src/core/__init__.py` to run
# set up any "aliases" (optional...)
import sys
sys.modules['core'] = sys.modules['src.core']
... indeed, from my experiments, the effect of putting conftest.py in the application directory seems to be that pytest then runs __init__.py
in that directory. This appears to imply that the module is being imported...
(previous suggestion:)
yes, you also HAVE to include an empty "conftest.py" file in the directory "core". Hopefully this should be the only conftest.py you'll need: I experimented with all this, and putting one in the root directory was not necessary (nor did it solve the problem without the suggested code in __init__.py
).
3 - finally, in my test function, before calling core.__main__
in my example, I have to import the file I know is about to be imported:
import core.my_other_file
import core.__main__
If you do this in the first test in your file, you will find that sys.modules is set up for all other tests in that file. Better yet, put import core.my_other_file
at the very start of the file, before the first test. Unfortunately, from core import *
does not seem to work.
Later: this method has some idiosyncracies and limitations. For example, although the -k
switch works OK to filter in/out tests or entire files, if you do something like pytest tests/tests_concerning_module_x
, it appears that core.__init__.py
does NOT get run... so the files in the core module are once again mutually unimportable during testing. Other limitations will probably come to light...
As I say, I regard this as a flaw in pytest's setup. I have absolutely no idea what pytest does to establish a common-sense setup for sys.path
, but it's obviously getting it wrong. There should be no need to rely on PYTHONPATH, or whatever, and if indeed this is the "official" solution, the documentation on this subject is sorely lacking.
NB this suggestion of mine has a problem: by adding to sys.path
every time pytest runs __init__.py
in a module, it means that this new path thereafter becomes permanent in sys.path
, both during testing and, more worryingly, during runs of the application itself, if there is anything which actually calls __init__.py
. (Incidentally, just going python src/core
(as in my example) does NOT cause this to happen. But other things might.)
To cater for this I have a clunky but effective solution:
import pathlib, sys, traceback
frame_list = traceback.extract_stack()
if len(frame_list) > 2:
path_parts = pathlib.Path(frame_list[2].filename).parts
sys_platform = sys.platform
if sys_platform.startswith('win'):
if len(path_parts) > 2 and path_parts[-3:-1] == ('Scripts', 'pytest.exe'):
sys.testing_context = True
sys.path.append(str(pathlib.Path(__file__).parent))
elif sys_platform.startswith('lin'):
if len(path_parts) > 2 and path_parts[-3:-1] == ('_pytest', 'config'):
sys.testing_context = True
sys.path.append(str(pathlib.Path(__file__).parent))
This is based on my examination of the stacktrace when pytest runs something, in a W10 context and Linux Mint 20 context. It means that in an application run there will be no messing with sys.path
.
Of course, this may break with future versions of pytest. My version is 6.2.5.