31

I have a trusted remote server that stores many custom Python modules. I can fetch them via HTTP (e.g. using urllib2.urlopen) as text/plain, but I cannot save the fetched module code to the local hard disk. How can I import the code as a fully operable Python module, including its global variables and imports?
I suppose I have to use some combination of exec and imp module's functions, but I've been unable to make it work yet.

2
  • are you downloading them over ssl or is every intermediate router trusted as well? :P Sep 26, 2010 at 19:58
  • 1
    Actually, yes - both the module storage and the front-end server are parts of a single system (and are even located in the same server room). For now, suppose there are no security implications: even if I download the code over SSL, the question of how to interpret it still stands.
    – dpq
    Sep 26, 2010 at 20:04

4 Answers 4

48

It looks like this should do the trick: importing a dynamically generated module

>>> import imp
>>> foo = imp.new_module("foo")
>>> foo_code = """
... class Foo:
...     pass
... """
>>> exec foo_code in foo.__dict__
>>> foo.Foo.__module__
'foo'
>>>

Also, as suggested in the ActiveState article, you might want to add your new module to sys.modules:

>>> import sys
>>> sys.modules["foo"] = foo
>>> from foo import Foo
<class 'Foo' …>
>>>
3
  • Is there a trick so that I could allow the user to "from foo import Foo"? In this example, I get "ImportError: No module named foo".
    – D. A.
    Mar 4, 2013 at 15:29
  • After sys.modules["foo"] = foo, the sys module breaks and equals None. Is that supposed to happen or can you avoid that? See my question on that here.
    – fredrik
    May 28, 2016 at 14:18
  • 3
    @DavidWolever It seems that imp.new_module is deprecated since Python 3.4 (ref). Use importlib.util.module_from_spec instead.
    – nekketsuuu
    Jan 24, 2018 at 15:47
5

Here's something I bookmarked a while back that covers something similar:

It's a bit beyond what you want, but the basic idea is there.

1

I recently encountered trying to do this while trying to write unit tests for source code examples I put into a project's readme (I wanted to avoid just linking to small files or duplicating the text in a way that could get out of sync).

I came up with the following

import sys
import types
from importlib import import_module


def compile_and_install_module(module_name: str, source_code: str) -> types.ModuleType:
    """Compile source code and install it as a module.

    End result is that `import <module_name>` and `from <module_name> import ...` should work.
    """
    module = types.ModuleType(module_name, "Module created from source code")

    # Execute source in context of empty/fake module
    exec(source_code, module.__dict__)

    # Insert fake module into sys.modules. It's now a real module
    sys.modules[module_name] = module

    # Imports should work now
    return import_module(module_name)

And a quick example of how you can use it

$ cat hello.py 
def foo():
    print("Hello world")


bar = 42

$ python
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from compile import compile_and_install_module
>>> compile_and_install_module("hello", open("hello.py").read())
<module 'hello'>
>>> import hello
>>> hello.foo()
Hello world
>>> from hello import bar
>>> bar
42

You can remove the return value and import_lib import if you

1
  • The types.ModuleType seems to be the new way of creating modules dynamically, since imp is deprecated now. +1 to you.
    – winwin
    May 2 at 10:26
0

Python3 version
(attempted to edit other answer but the edit que is full)

import imp

my_dynamic_module = imp.new_module("my_dynamic_module")
exec("""
class Foo:
    pass
""", my_dynamic_module.__dict__)

Foo = my_dynamic_module.Foo
foo_object = Foo()

# register it on sys
import sys
sys.modules[my_dynamic_module.__name__] = my_dynamic_module

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.