How can I load a Python module given its full path? Note that the file can be anywhere in the filesystem, as it is a configuration option.
For Python 3.5+ use:
import importlib.util
spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
foo.MyClass()
For Python 3.3 and 3.4 use:
from importlib.machinery import SourceFileLoader
foo = SourceFileLoader("module.name", "/path/to/file.py").load_module()
foo.MyClass()
(Although this has been deprecated in Python 3.4.)
For Python 2 use:
import imp
foo = imp.load_source('module.name', '/path/to/file.py')
foo.MyClass()
There are equivalent convenience functions for compiled Python files and DLLs.
See also http://bugs.python.org/issue21436.
-
57If I knew the namespace - 'module.name' - I would already use
__import__. – Sridhar Ratnakumar Aug 10 '09 at 21:54 -
64@SridharRatnakumar the value of the first argument of
imp.load_sourceonly sets the.__name__of the returned module. it doesn't effect loading. – Dan D. Dec 14 '11 at 4:51 -
18@DanD. — the first argument of
imp.load_source()determines the key of the new entry created in thesys.modulesdictionary, so the first argument does indeed affect loading. – Brandon Rhodes Apr 21 '13 at 16:32 -
10The
impmodule is deprecated since version 3.4: Theimppackage is pending deprecation in favor ofimportlib. – Chiel ten Brinke Dec 8 '13 at 11:20 -
11@AXO and more to the point one wonders why something as simple and basic as this has to be so complicated. It isn't in many many other languages. – rocky May 22 '16 at 17:04
The advantage of adding a path to sys.path (over using imp) is that it simplifies things when importing more than one module from a single package. For example:
import sys
# the mock-0.3.1 dir contains testcase.py, testutils.py & mock.py
sys.path.append('/foo/bar/mock-0.3.1')
from testcase import TestCase
from testutils import RunTests
from mock import Mock, sentinel, patch
-
14How do we use
sys.path.appendto point to a single python file instead of a directory? – Phani Jan 13 '14 at 17:46 -
29:-) Perhaps your question would be better suited as a StackOverflow question, not a comment on an answer. – Daryl Spitzer Mar 6 '15 at 0:12
-
3The python path can contain zip archives, "eggs" (a complex kind of zip archives), etc. Modules can be imported out of them. So the path elements are indeed containers of files, but they are not necessarily directories. – alexis Apr 30 '15 at 21:21
-
14Beware of the fact that Python caches import statements. In the rare case that you have two different folders sharing a single class name (classX), the approach of adding a path to sys.path, importing classX, removing the path and repeating for the reamaining paths won't work. Python will always load the class from the first path from its cache. In my case I aimed at creating a plugin system where all plugins implement a specific classX. I ended up using SourceFileLoader, note that its deprecation is controversial. – ComFreek Jul 3 '15 at 17:18
-
3Note this approach allows the imported module to import other modules from the same dir, which modules often do, while the accepted answer's approach does not (at least on 3.7).
importlib.import_module(mod_name)can be used instead of the explicit import here if the module name isn't known at runtime I would add asys.path.pop()in the end, though, assuming the imported code doesn't try to import more modules as it is used. – Eli_B May 6 '19 at 21:45
If your top-level module is not a file but is packaged as a directory with __init__.py, then the accepted solution almost works, but not quite. In Python 3.5+ the following code is needed (note the added line that begins with 'sys.modules'):
MODULE_PATH = "/path/to/your/module/__init__.py"
MODULE_NAME = "mymodule"
import importlib
import sys
spec = importlib.util.spec_from_file_location(MODULE_NAME, MODULE_PATH)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
Without this line, when exec_module is executed, it tries to bind relative imports in your top level __init__.py to the top level module name -- in this case "mymodule". But "mymodule" isn't loaded yet so you'll get the error "SystemError: Parent module 'mymodule' not loaded, cannot perform relative import". So you need to bind the name before you load it. The reason for this is the fundamental invariant of the relative import system: "The invariant holding is that if you have sys.modules['spam'] and sys.modules['spam.foo'] (as you would after the above import), the latter must appear as the foo attribute of the former" as discussed here.
-
1Thanks a lot! This method enables relative imports between submodules. Great! – tebanep Oct 10 '19 at 14:09
-
This answer matches the documentation here: docs.python.org/3/library/…. – Tim Ludwinski Dec 9 '19 at 16:08
-
1
-
@Gulzar, it is whatever name you'd like to give your module, such that you can later do: "from mymodule import myclass" – Idodo Feb 16 at 22:44
-
So...
/path/to/your/module/is actually/path/to/your/PACKAGE/? and bymymoduleyou meanmyfile.py? – Gulzar Feb 17 at 7:03
To import your module, you need to add its directory to the environment variable, either temporarily or permanently.
Temporarily
import sys
sys.path.append("/path/to/my/modules/")
import my_module
Permanently
Adding the following line to your .bashrc file (in linux) and excecute source ~/.bashrc in the terminal:
export PYTHONPATH="${PYTHONPATH}:/path/to/my/modules/"
Credit/Source: saarrrr, another stackexchange question
-
5This "temp" solution is a great answer if you want to prod a project around in a jupyter notebook elsewhere. – fordy Nov 16 '18 at 17:20
-
-
@ShaiAlon You are adding paths, so no danger other than when you transfer codes from one computer to another, paths might get messed up. So, for package development, I only import local packages. Also, package names should be unique. If you are worried, use the temporary solution. – Miladiouss Nov 13 '19 at 0:11
-
It sounds like you don't want to specifically import the configuration file (which has a whole lot of side effects and additional complications involved), you just want to run it, and be able to access the resulting namespace. The standard library provides an API specifically for that in the form of runpy.run_path:
from runpy import run_path
settings = run_path("/path/to/file.py")
That interface is available in Python 2.7 and Python 3.2+
-
I like this method but when I get the result of run_path its a dictionary which I cannot seem to access? – Stephen Ellwood Sep 11 '18 at 9:00
-
What do you mean by "cannot access"? You can't import from it (that's why this is only a good option when import-style access isn't actually required), but the contents should be available via the regular dict API (
result[name],result.get('name', default_value), etc) – ncoghlan Sep 13 '18 at 3:16 -
This answer is way underrated. It's very short and simple! Even better, if you need a proper module namespace, you can do something like
from runpy import run_path; from argparse import Namespace; mod = Namespace(**run_path('path/to/file.py'))– RuRo Mar 18 at 21:26 -
1@Maggyero The command line never goes through
runpy.run_path, but if a given path is a directory or zipfile, then it ends up delegating torunpy.run_modulefor the__main__execution. The duplicated logic for "Is it a script, directory, or zipfile?" isn't complicated enough to be worth delegating to Python code. – ncoghlan Sep 22 at 7:39 -
1Also by looking at the implementation of the C function
pymain_run_module, it seems that CPython delegates to the Python functionrunpy._run_module_as_maininstead ofrunpy.run_module—though if I understood correctly the only difference is that the first function executes the code in the built-in__main__environment (cf. here) while the second function executes it in a new environment? – Maggyero Sep 22 at 9:33
You can also do something like this and add the directory that the configuration file is sitting in to the Python load path, and then just do a normal import, assuming you know the name of the file in advance, in this case "config".
Messy, but it works.
configfile = '~/config.py'
import os
import sys
sys.path.append(os.path.dirname(os.path.expanduser(configfile)))
import config
-
-
I tried: config_file = 'setup-for-chats', setup_file = get_setup_file(config_file + ".py"), sys.path.append(os.path.dirname(os.path.expanduser(setup_file))), import config_file >> "ImportError: No module named config_file" – Shai Alon Nov 10 '19 at 16:17
-
-
36
Do you mean load or import?
You can manipulate the sys.path list specify the path to your module, then import your module. For example, given a module at:
/foo/bar.py
You could do:
import sys
sys.path[0:0] = ['/foo'] # puts the /foo directory at the start of your path
import bar
I have come up with a slightly modified version of @SebastianRittau's wonderful answer (for Python > 3.4 I think), which will allow you to load a file with any extension as a module using spec_from_loader instead of spec_from_file_location:
from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader
spec = spec_from_loader("module.name", SourceFileLoader("module.name", "/path/to/file.py"))
mod = module_from_spec(spec)
spec.loader.exec_module(mod)
The advantage of encoding the path in an explicit SourceFileLoader is that the machinery will not try to figure out the type of the file from the extension. This means that you can load something like a .txt file using this method, but you could not do it with spec_from_file_location without specifying the loader because .txt is not in importlib.machinery.SOURCE_SUFFIXES.
def import_file(full_path_to_module):
try:
import os
module_dir, module_file = os.path.split(full_path_to_module)
module_name, module_ext = os.path.splitext(module_file)
save_cwd = os.getcwd()
os.chdir(module_dir)
module_obj = __import__(module_name)
module_obj.__file__ = full_path_to_module
globals()[module_name] = module_obj
os.chdir(save_cwd)
except:
raise ImportError
import_file('/home/somebody/somemodule.py')
-
39Why write 14 lines of buggy code when this is already addressed by the standard library? You haven't done error checking on format or content of full_path_to_module or the os.whatever operations; and using a catch-all
except:clause is rarely a good idea. – Chris Johnson Jun 7 '13 at 19:17 -
You should use more "try-finally"s in here. E.g.
save_cwd = os.getcwd()try: …finally: os.chdir(save_cwd)– kay Sep 21 '14 at 1:33 -
11@ChrisJohnson
this is already addressed by the standard libraryyeah, but python has nasty habit of not being backward-compatible... as the checked answer says there're 2 different ways before and after 3.3. In that case I'd rather like to write my own universal function than check version on the fly. And yes, maybe this code isn't too well error-protected, but it shows an idea (which is os.chdir(), I haven't though about it), basing on which I can write a better code. Hence +1. – Sushi271 May 15 '15 at 10:27 -
Here is some code that works in all Python versions, from 2.7-3.5 and probably even others.
config_file = "/tmp/config.py"
with open(config_file) as f:
code = compile(f.read(), config_file, 'exec')
exec(code, globals(), locals())
I tested it. It may be ugly but so far is the only one that works in all versions.
-
1This answer worked for me where
load_sourcedid not because it imports the script and provides the script access to the modules and globals at the time of importing. – Klik Nov 22 '17 at 19:13
I believe you can use imp.find_module() and imp.load_module() to load the specified module. You'll need to split the module name off of the path, i.e. if you wanted to load /home/mypath/mymodule.py you'd need to do:
imp.find_module('mymodule', '/home/mypath/')
...but that should get the job done.
If we have scripts in the same project but in different directory means, we can solve this problem by the following method.
In this situation utils.py is in src/main/util/
import sys
sys.path.append('./')
import src.main.util.utils
#or
from src.main.util.utils import json_converter # json_converter is example method
-
1
You can use the pkgutil module (specifically the walk_packages method) to get a list of the packages in the current directory. From there it's trivial to use the importlib machinery to import the modules you want:
import pkgutil
import importlib
packages = pkgutil.walk_packages(path='.')
for importer, name, is_package in packages:
mod = importlib.import_module(name)
# do whatever you want with module now, it's been imported!
Create python module test.py
import sys
sys.path.append("<project-path>/lib/")
from tes1 import Client1
from tes2 import Client2
import tes3
Create python module test_check.py
from test import Client1
from test import Client2
from test import test3
We can import the imported module from module.
This area of Python 3.4 seems to be extremely tortuous to understand! However with a bit of hacking using the code from Chris Calloway as a start I managed to get something working. Here's the basic function.
def import_module_from_file(full_path_to_module):
"""
Import a module given the full path/filename of the .py file
Python 3.4
"""
module = None
try:
# Get module name and path from full path
module_dir, module_file = os.path.split(full_path_to_module)
module_name, module_ext = os.path.splitext(module_file)
# Get module "spec" from filename
spec = importlib.util.spec_from_file_location(module_name,full_path_to_module)
module = spec.loader.load_module()
except Exception as ec:
# Simple error printing
# Insert "sophisticated" stuff here
print(ec)
finally:
return module
This appears to use non-deprecated modules from Python 3.4. I don't pretend to understand why, but it seems to work from within a program. I found Chris' solution worked on the command line but not from inside a program.
I'm not saying that it is better, but for the sake of completeness, I wanted to suggest the exec function, available in both python 2 and 3.
exec allows you to execute arbitrary code in either the global scope, or in an internal scope, provided as a dictionary.
For example, if you have a module stored in "/path/to/module" with the function foo(), you could run it by doing the following:
module = dict()
with open("/path/to/module") as f:
exec(f.read(), module)
module['foo']()
This makes it a bit more explicit that you're loading code dynamically, and grants you some additional power, such as the ability to provide custom builtins.
And if having access through attributes, instead of keys is important to you, you can design a custom dict class for the globals, that provides such access, e.g.:
class MyModuleClass(dict):
def __getattr__(self, name):
return self.__getitem__(name)
To import a module from a given filename, you can temporarily extend the path, and restore the system path in the finally block reference:
filename = "directory/module.py"
directory, module_name = os.path.split(filename)
module_name = os.path.splitext(module_name)[0]
path = list(sys.path)
sys.path.insert(0, directory)
try:
module = __import__(module_name)
finally:
sys.path[:] = path # restore
This should work
path = os.path.join('./path/to/folder/with/py/files', '*.py')
for infile in glob.glob(path):
basename = os.path.basename(infile)
basename_without_extension = basename[:-3]
# http://docs.python.org/library/imp.html?highlight=imp#module-imp
imp.load_source(basename_without_extension, infile)
-
4A more general way to cut the extension out is:
name, ext = os.path.splitext(os.path.basename(infile)). Your method works because the previous restriction to .py extension. Also, you should probably import the module to some variable/dictionary entry. – ReneSac Dec 6 '12 at 13:16
There's a package that's dedicated to this specifically:
from thesmuggler import smuggle
# À la `import weapons`
weapons = smuggle('weapons.py')
# À la `from contraband import drugs, alcohol`
drugs, alcohol = smuggle('drugs', 'alcohol', source='contraband.py')
# À la `from contraband import drugs as dope, alcohol as booze`
dope, booze = smuggle('drugs', 'alcohol', source='contraband.py')
It's tested across Python versions (Jython and PyPy too), but it might be overkill depending on the size of your project.
I made a package that uses imp for you. I call it import_file and this is how it's used:
>>>from import_file import import_file
>>>mylib = import_file('c:\\mylib.py')
>>>another = import_file('relative_subdir/another.py')
You can get it at:
http://pypi.python.org/pypi/import_file
or at
-
1
-
I've spent all day troubleshooting an import bug in a pyinstaller generated exe. In the end this is the only thing that worked for me. Thank you so much for making this! – frakman1 Nov 29 '18 at 22:12
Import package modules at runtime (Python recipe)
http://code.activestate.com/recipes/223972/
###################
## #
## classloader.py #
## #
###################
import sys, types
def _get_mod(modulePath):
try:
aMod = sys.modules[modulePath]
if not isinstance(aMod, types.ModuleType):
raise KeyError
except KeyError:
# The last [''] is very important!
aMod = __import__(modulePath, globals(), locals(), [''])
sys.modules[modulePath] = aMod
return aMod
def _get_func(fullFuncName):
"""Retrieve a function object from a full dotted-package name."""
# Parse out the path, module, and function
lastDot = fullFuncName.rfind(u".")
funcName = fullFuncName[lastDot + 1:]
modPath = fullFuncName[:lastDot]
aMod = _get_mod(modPath)
aFunc = getattr(aMod, funcName)
# Assert that the function is a *callable* attribute.
assert callable(aFunc), u"%s is not callable." % fullFuncName
# Return a reference to the function itself,
# not the results of the function.
return aFunc
def _get_class(fullClassName, parentClass=None):
"""Load a module and retrieve a class (NOT an instance).
If the parentClass is supplied, className must be of parentClass
or a subclass of parentClass (or None is returned).
"""
aClass = _get_func(fullClassName)
# Assert that the class is a subclass of parentClass.
if parentClass is not None:
if not issubclass(aClass, parentClass):
raise TypeError(u"%s is not a subclass of %s" %
(fullClassName, parentClass))
# Return a reference to the class itself, not an instantiated object.
return aClass
######################
## Usage ##
######################
class StorageManager: pass
class StorageManagerMySQL(StorageManager): pass
def storage_object(aFullClassName, allOptions={}):
aStoreClass = _get_class(aFullClassName, StorageManager)
return aStoreClass(allOptions)
In Linux, adding a symbolic link in the directory your python script is located works.
ie:
ln -s /absolute/path/to/module/module.py /absolute/path/to/script/module.py
python will create /absolute/path/to/script/module.pyc and will update it if you change the contents of /absolute/path/to/module/module.py
then include the following in mypythonscript.py
from module import *
-
1This is the hack I used, and it has caused me some problems. One of the more painful ones was that IDEA has an issue where it doesn't pickup altered code from within the link, but yet attempts to save what it thinks is there. A race condition where the last to save is what sticks... I lost a decent amount of work because of this. – Gripp Jun 16 '15 at 23:23
-
@Gripp not sure if I am understanding your issue, but I frequently (almost exclusively) edit my scripts on a remote server from my desktop via SFTP with a client like CyberDuck, and in that case as well it is a bad idea to try and edit the symlinked file, instead its much safer to edit the original file. You can catch some of these issues by using
gitand checking yourgit statusto verify that your changes to the script are actually making it back to the source document and not getting lost in the ether. – user5359531 Aug 1 '17 at 19:39
Adding this to the list of answers as I couldn't find anything that worked. This will allow imports of compiled (pyd) python modules in 3.4:
import sys
import importlib.machinery
def load_module(name, filename):
# If the Loader finds the module name in this list it will use
# module_name.__file__ instead so we need to delete it here
if name in sys.modules:
del sys.modules[name]
loader = importlib.machinery.ExtensionFileLoader(name, filename)
module = loader.load_module()
locals()[name] = module
globals()[name] = module
load_module('something', r'C:\Path\To\something.pyd')
something.do_something()
quite simple way: suppose you want import file with relative path ../../MyLibs/pyfunc.py
libPath = '../../MyLibs'
import sys
if not libPath in sys.path: sys.path.append(libPath)
import pyfunc as pf
But if you make it without a guard you can finally get a very long path
A simple solution using importlib instead of the imp package (tested for Python 2.7, although it should work for Python 3 too):
import importlib
dirname, basename = os.path.split(pyfilepath) # pyfilepath: '/my/path/mymodule.py'
sys.path.append(dirname) # only directories should be added to PYTHONPATH
module_name = os.path.splitext(basename)[0] # '/my/path/mymodule.py' --> 'mymodule'
module = importlib.import_module(module_name) # name space of defined module (otherwise we would literally look for "module_name")
Now you can directly use the namespace of the imported module, like this:
a = module.myvar
b = module.myfunc(a)
The advantage of this solution is that we don't even need to know the actual name of the module we would like to import, in order to use it in our code. This is useful, e.g. in case the path of the module is a configurable argument.
-
This way you are modifying the
sys.path, which does not fit every use case. – bgusach Jul 19 '18 at 14:26 -
@bgusach This may be true, but it is also desirable in some cases (adding a path to sys.path simplifies things when importing more than one module from a single package). At any rate, if this not desirable, one can immediately afterwards do
sys.path.pop()– Ataxias Jul 20 '18 at 16:38
I have wrote my own global and portable import function, based on importlib module, for:
- Be able to import both module as a submodule and to import content of a module to a parent module (or into a globals if has no parent module).
- Be able to import modules with a period characters in a file name.
- Be able to import modules with any extension.
- Be able to use a standalone name for a submodule instead of a file name without extension which is by default.
- Be able to define the import order based on previously imported module instead of dependent on
sys.pathor on a what ever search path storage.
The examples directory structure:
<root>
|
+- test.py
|
+- testlib.py
|
+- /std1
| |
| +- testlib.std1.py
|
+- /std2
| |
| +- testlib.std2.py
|
+- /std3
|
+- testlib.std3.py
Inclusion dependency and order:
test.py
-> testlib.py
-> testlib.std1.py
-> testlib.std2.py
-> testlib.std3.py
Implementation:
Latest changes store: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/python/tacklelib/tacklelib.py
test.py:
import os, sys, inspect, copy
SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("test::SOURCE_FILE: ", SOURCE_FILE)
# portable import to the global space
sys.path.append(TACKLELIB_ROOT) # TACKLELIB_ROOT - path to the library directory
import tacklelib as tkl
tkl.tkl_init(tkl)
# cleanup
del tkl # must be instead of `tkl = None`, otherwise the variable would be still persist
sys.path.pop()
tkl_import_module(SOURCE_DIR, 'testlib.py')
print(globals().keys())
testlib.base_test()
testlib.testlib_std1.std1_test()
testlib.testlib_std1.testlib_std2.std2_test()
#testlib.testlib.std3.std3_test() # does not reachable directly ...
getattr(globals()['testlib'], 'testlib.std3').std3_test() # ... but reachable through the `globals` + `getattr`
tkl_import_module(SOURCE_DIR, 'testlib.py', '.')
print(globals().keys())
base_test()
testlib_std1.std1_test()
testlib_std1.testlib_std2.std2_test()
#testlib.std3.std3_test() # does not reachable directly ...
globals()['testlib.std3'].std3_test() # ... but reachable through the `globals` + `getattr`
testlib.py:
# optional for 3.4.x and higher
#import os, inspect
#
#SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
#SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("1 testlib::SOURCE_FILE: ", SOURCE_FILE)
tkl_import_module(SOURCE_DIR + '/std1', 'testlib.std1.py', 'testlib_std1')
# SOURCE_DIR is restored here
print("2 testlib::SOURCE_FILE: ", SOURCE_FILE)
tkl_import_module(SOURCE_DIR + '/std3', 'testlib.std3.py')
print("3 testlib::SOURCE_FILE: ", SOURCE_FILE)
def base_test():
print('base_test')
testlib.std1.py:
# optional for 3.4.x and higher
#import os, inspect
#
#SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
#SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("testlib.std1::SOURCE_FILE: ", SOURCE_FILE)
tkl_import_module(SOURCE_DIR + '/../std2', 'testlib.std2.py', 'testlib_std2')
def std1_test():
print('std1_test')
testlib.std2.py:
# optional for 3.4.x and higher
#import os, inspect
#
#SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
#SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("testlib.std2::SOURCE_FILE: ", SOURCE_FILE)
def std2_test():
print('std2_test')
testlib.std3.py:
# optional for 3.4.x and higher
#import os, inspect
#
#SOURCE_FILE = os.path.abspath(inspect.getsourcefile(lambda:0)).replace('\\','/')
#SOURCE_DIR = os.path.dirname(SOURCE_FILE)
print("testlib.std3::SOURCE_FILE: ", SOURCE_FILE)
def std3_test():
print('std3_test')
Output (3.7.4):
test::SOURCE_FILE: <root>/test01/test.py
import : <root>/test01/testlib.py as testlib -> []
1 testlib::SOURCE_FILE: <root>/test01/testlib.py
import : <root>/test01/std1/testlib.std1.py as testlib_std1 -> ['testlib']
import : <root>/test01/std1/../std2/testlib.std2.py as testlib_std2 -> ['testlib', 'testlib_std1']
testlib.std2::SOURCE_FILE: <root>/test01/std1/../std2/testlib.std2.py
2 testlib::SOURCE_FILE: <root>/test01/testlib.py
import : <root>/test01/std3/testlib.std3.py as testlib.std3 -> ['testlib']
testlib.std3::SOURCE_FILE: <root>/test01/std3/testlib.std3.py
3 testlib::SOURCE_FILE: <root>/test01/testlib.py
dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'os', 'sys', 'inspect', 'copy', 'SOURCE_FILE', 'SOURCE_DIR', 'TackleGlobalImportModuleState', 'tkl_membercopy', 'tkl_merge_module', 'tkl_get_parent_imported_module_state', 'tkl_declare_global', 'tkl_import_module', 'TackleSourceModuleState', 'tkl_source_module', 'TackleLocalImportModuleState', 'testlib'])
base_test
std1_test
std2_test
std3_test
import : <root>/test01/testlib.py as . -> []
1 testlib::SOURCE_FILE: <root>/test01/testlib.py
import : <root>/test01/std1/testlib.std1.py as testlib_std1 -> ['testlib']
import : <root>/test01/std1/../std2/testlib.std2.py as testlib_std2 -> ['testlib', 'testlib_std1']
testlib.std2::SOURCE_FILE: <root>/test01/std1/../std2/testlib.std2.py
2 testlib::SOURCE_FILE: <root>/test01/testlib.py
import : <root>/test01/std3/testlib.std3.py as testlib.std3 -> ['testlib']
testlib.std3::SOURCE_FILE: <root>/test01/std3/testlib.std3.py
3 testlib::SOURCE_FILE: <root>/test01/testlib.py
dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__annotations__', '__builtins__', '__file__', '__cached__', 'os', 'sys', 'inspect', 'copy', 'SOURCE_FILE', 'SOURCE_DIR', 'TackleGlobalImportModuleState', 'tkl_membercopy', 'tkl_merge_module', 'tkl_get_parent_imported_module_state', 'tkl_declare_global', 'tkl_import_module', 'TackleSourceModuleState', 'tkl_source_module', 'TackleLocalImportModuleState', 'testlib', 'testlib_std1', 'testlib.std3', 'base_test'])
base_test
std1_test
std2_test
std3_test
Tested in Python 3.7.4, 3.2.5, 2.7.16
Pros:
- Can import both module as a submodule and can import content of a module to a parent module (or into a globals if has no parent module).
- Can import modules with periods in a file name.
- Can import any extension module from any extension module.
- Can use a standalone name for a submodule instead of a file name without extension which is by default (for example,
testlib.std.pyastestlib,testlib.blabla.pyastestlib_blablaand so on). - Does not depend on a
sys.pathor on a what ever search path storage. - Does not require to save/restore global variables like
SOURCE_FILEandSOURCE_DIRbetween calls totkl_import_module. - [for
3.4.xand higher] Can mix the module namespaces in nestedtkl_import_modulecalls (ex:named->local->namedorlocal->named->localand so on). - [for
3.4.xand higher] Can auto export global variables/functions/classes from where being declared to all children modules imported through thetkl_import_module(through thetkl_declare_globalfunction).
Cons:
- [for
3.3.xand lower] Require to declaretkl_import_modulein all modules which calls totkl_import_module(code duplication)
Update 1,2 (for 3.4.x and higher only):
In Python 3.4 and higher you can bypass the requirement to declare tkl_import_module in each module by declare tkl_import_module in a top level module and the function would inject itself to all children modules in a single call (it's a kind of self deploy import).
Update 3:
Added function tkl_source_module as analog to bash source with support execution guard upon import (implemented through the module merge instead of import).
Update 4:
Added function tkl_declare_global to auto export a module global variable to all children modules where a module global variable is not visible because is not a part of a child module.
Update 5:
All functions has moved into the tacklelib library, see the link above.
This answer is a supplement to Sebastian Rittau's answer responding to the comment: "but what if you don't have the module name?" This is a quick and dirty way of getting the likely python module name given a filename -- it just goes up the tree until it finds a directory without an __init__.py file and then turns it back into a filename. For Python 3.4+ (uses pathlib), which makes sense since Py2 people can use "imp" or other ways of doing relative imports:
import pathlib
def likely_python_module(filename):
'''
Given a filename or Path, return the "likely" python module name. That is, iterate
the parent directories until it doesn't contain an __init__.py file.
:rtype: str
'''
p = pathlib.Path(filename).resolve()
paths = []
if p.name != '__init__.py':
paths.append(p.stem)
while True:
p = p.parent
if not p:
break
if not p.is_dir():
break
inits = [f for f in p.iterdir() if f.name == '__init__.py']
if not inits:
break
paths.append(p.stem)
return '.'.join(reversed(paths))
There are certainly possibilities for improvement, and the optional __init__.py files might necessitate other changes, but if you have __init__.py in general, this does the trick.
For those like me that just want to import a simple pure python script from somewhere on their machine, this works nicely:
import os
from shutil import copy2
copy2("path/to/module.py", os.getcwd() + "\\module.py")
import module
The reason shutil's copy2 method is nice is because of this. It can copy a directory and can copy permissions and meta data.
This is my 2 utility functions using only pathlib. It infers the module name from the path By default, it recursively loads all python files from folders and replaces init.py by the parent folder name. But you can also give a Path and/or a glob to select some specific files.
from pathlib import Path
from importlib.util import spec_from_file_location, module_from_spec
from typing import Optional
def get_module_from_path(path: Path, relative_to: Optional[Path] = None):
if not relative_to:
relative_to = Path.cwd()
abs_path = path.absolute()
relative_path = abs_path.relative_to(relative_to.absolute())
if relative_path.name == "__init__.py":
relative_path = relative_path.parent
module_name = ".".join(relative_path.with_suffix("").parts)
mod = module_from_spec(spec_from_file_location(module_name, path))
return mod
def get_modules_from_folder(folder: Optional[Path] = None, glob_str: str = "*/**/*.py"):
if not folder:
folder = Path(".")
mod_list = []
for file_path in sorted(folder.glob(glob_str)):
mod_list.append(get_module_from_path(file_path))
return mod_list
import,virtualenv,pip,setuptoolswhatnot should all be thrown out and replaced with working code. I just tried to grokvirtualenvor was itpipenvand had to work thru the equivalent of a Jumbo Jet manual. How that contrivance is paraded as The Solution to dealing with deps totally escapes me. – John Frazer Jun 26 at 20:59