572

Currently I am working on a python project that contains sub modules and uses numpy/scipy. Ipython is used as interactive console. Unfortunately I am not very happy with workflow that I am using right now, I would appreciate some advice.

In IPython, the framework is loaded by a simple import command. However, it is often necessary to change code in one of the submodules of the framework. At this point a model is already loaded and I use IPython to interact with it.

Now, the framework contains many modules that depend on each other, i.e. when the framework is initially loaded the main module is importing and configuring the submodules. The changes to the code are only executed if the module is reloaded using reload(main_mod.sub_mod). This is cumbersome as I need to reload all changed modules individually using the full path. It would be very convenient if reload(main_module) would also reload all sub modules, but without reloading numpy/scipy..

2
  • Would you care to elaborate more on However, it is often necessary to change code in one of the submodules of the framework. So why it's necessary to change code? Thanks
    – eat
    Commented Mar 19, 2011 at 19:13
  • 33
    @eat: The framework is continually being developed, so there are constant changes to the code base.
    – Alain
    Commented Mar 19, 2011 at 19:26

14 Answers 14

885

IPython comes with some automatic reloading magic:

%load_ext autoreload
%autoreload 2

It will reload all changed modules every time before executing a new line. The way this works is slightly different than dreload. Some caveats apply, type %autoreload? to see what can go wrong.


If you want to always enable this settings, modify your IPython configuration file ~/.ipython/profile_default/ipython_config.py[1] and appending:

c.InteractiveShellApp.extensions = ['autoreload']     
c.InteractiveShellApp.exec_lines = ['%autoreload 2']

Credit to @Kos via a comment below.

[1] If you don't have the file ~/.ipython/profile_default/ipython_config.py, you need to call ipython profile create first. Or the file may be located at $IPYTHONDIR.

17
  • 72
    I have c.InteractiveShellApp.extensions = ['autoreload'] and c.InteractiveShellApp.exec_lines = ['%autoreload 2'] in my ~/.ipython/profile_default/ipython_config.py.
    – Kos
    Commented Jan 10, 2013 at 12:24
  • 4
    That might be a performance hit, though, so use with caution.
    – asmeurer
    Commented Jan 21, 2013 at 7:10
  • 3
    The reload is only performed when you hit Enter in Ipython shell, and not usually noticeable.
    – pv.
    Commented Feb 4, 2013 at 23:32
  • 4
    This works pretty fine when debugging packages, so what is the purpose of dreload, it seems dreload is too invasive and prints error when some packages like matplotlib are loaded.
    – dashesy
    Commented Jun 2, 2013 at 15:28
  • 4
    If you use @Kos method, make sure c is defined: c = get_config()
    – Tickon
    Commented Apr 1, 2015 at 12:19
124

Module named importlib allow to access to import internals. Especially, it provide function importlib.reload():

import importlib
importlib.reload(my_module)

In contrary of %autoreload, importlib.reload() also reset global variables set in module. In most cases, it is what you want.

importlib is only available since Python 3.1. For older version, you have to use module imp.

I suggest to read documentation of importlib.reload() to get the list of all caveats of this function (recursive reload, cases where definitions of old objects remain, etc...).

4
  • 21
    Over the past two years I've googled this question multiple times, and each time your answer is the perfect one I'm looking for. Thank you.
    – A Kareem
    Commented Nov 3, 2020 at 7:16
  • 1
    can only reload modules though.
    – bonobo
    Commented Sep 7, 2022 at 11:20
  • This works provided that if you want to reload a submodule named mymodule.data_class, then you have to write the code- importlib.reload(mymodule.data_class).
    – Prem
    Commented Jun 28, 2023 at 6:42
  • 1
    If you use "from ... " statement: "If a module imports objects from another module using from … import …, calling reload() for the other module does not redefine the objects imported from it — one way around this is to re-execute the from statement, another is to use import and qualified names (module.name) instead."
    – syviad
    Commented Jan 22 at 16:37
43

In IPython 0.12 (and possibly earlier), you can use this:

%load_ext autoreload
%autoreload 2

This is essentially the same as the answer by pv., except that the extension has been renamed and is now loaded using %load_ext.

1
  • 1
    For me in Jupyter lab this work and it is short enough to put into all notebooks who need it. Commented Sep 10, 2021 at 17:05
26

For some reason, neither %autoreload, nor dreload seem to work for the situation when you import code from one notebook to another. Only plain Python reload works:

reload(module)

Based on [1].

2
21

IPython offers dreload() to recursively reload all submodules. Personally, I prefer to use the %run() magic command (though it does not perform a deep reload, as pointed out by John Salvatier in the comments).

2
  • 13
    I think that (unfortunately) %run script.py only reloads the script you're calling, not the packages it imports. If you're trying to debug a package you're building, this can be a pain. Commented Mar 20, 2011 at 21:29
  • 5
    NB. dreload has been replaced in recent IPython (e.g. IPython 6.0) by deepreload. Commented Apr 18, 2018 at 3:05
7

http://shawnleezx.github.io/blog/2015/08/03/some-notes-on-ipython-startup-script/

To avoid typing those magic function again and again, they could be put in the ipython startup script(Name it with .py suffix under .ipython/profile_default/startup. All python scripts under that folder will be loaded according to lexical order), which looks like the following:

from IPython import get_ipython
ipython = get_ipython()

ipython.magic("pylab")
ipython.magic("load_ext autoreload")
ipython.magic("autoreload 2")
1
  • This also seems to work if you're running your script with e.g. %run script.pyfrom the IPython REPL Commented Oct 3, 2019 at 8:42
6

How about this:

import inspect

# needs to be primed with an empty set for loaded
def recursively_reload_all_submodules(module, loaded=None):
    for name in dir(module):
        member = getattr(module, name)
        if inspect.ismodule(member) and member not in loaded:
            recursively_reload_all_submodules(member, loaded)
    loaded.add(module)
    reload(module)

import mymodule
recursively_reload_all_submodules(mymodule, set())

This should effectively reload the entire tree of modules and submodules you give it. You can also put this function in your .ipythonrc (I think) so it is loaded every time you start the interpreter.

5
  • That looks good, however, it might be that it is not covering modules or members of modules that are imported using from ... import ... or import ... as. At least that's giving me often some trouble when working interactively on the terminal. I've moved on to use stored macros in IPython that do the necessary imports and setups to start working in a predefined state.
    – Bernhard
    Commented Mar 19, 2011 at 22:39
  • It actually does cover from ... import ... and import ... as as long as the thing you imported is a module. The only thing that it doesn't cover is modules in a package that weren't loaded from it's __init__.py file. For packages, you can probably check if the module's __path__ attribute is a directory. If it is, traverse it and recursively import all the modules you could find. I didn't write this part because the author has not asked for a solution for packages.
    – Y.H Wong
    Commented Mar 19, 2011 at 23:23
  • Indeed this looks good. I thought about this possibility, at the same I would expect that there would be some built in functionality, i.e. based on this. However, it was not clear to me how to use this. After some digging that should have occurred before I posted the original question, I found this extension.
    – Alain
    Commented Mar 20, 2011 at 0:27
  • Also you can use pkgutil to get all submodules in a package, even if the package does not import the submodules into the top module. stackoverflow.com/a/1707786/1243926 Commented Jan 2, 2015 at 18:45
  • you also have to do:for module in sys.modules:
    – aoeu256
    Commented Sep 12, 2019 at 17:13
4

My standard practice for reloading is to combine both methods following first opening of IPython:

from IPython.lib.deepreload import reload
%load_ext autoreload
%autoreload 2

Loading modules before doing this will cause them not to be reloaded, even with a manual reload(module_name). I still, very rarely, get inexplicable problems with class methods not reloading that I've not yet looked into.

3

Another option:

$ cat << EOF > ~/.ipython/profile_default/startup/50-autoreload.ipy
%load_ext autoreload
%autoreload 2
EOF

Verified on ipython and ipython3 v5.1.0 on Ubuntu 14.04.

3

Prior to your module imports include these lines, where the first tests whether or not the autoreload extension has already been loaded:

if 'autoreload' not in get_ipython().extension_manager.loaded:
    %load_ext autoreload
%autoreload 2

import sys
    .
    .
    .
2

I hate to add yet another answer to a long thread, but I found a solution that enables recursive reloading of submodules on %run() that others might find useful (I have anyway)

del the submodule you wish to reload on run from sys.modules in iPython:

In[1]: from sys import modules
In[2]: del modules["mymodule.mysubmodule"] # tab completion can be used like mymodule.<tab>!

Now your script will recursively reload this submodule:

In[3]: %run myscript.py
1

Note that the above mentioned autoreload only works in IntelliJ if you manually save the changed file (e.g. using ctrl+s or cmd+s). It doesn't seem to work with auto-saving.

1
  • 1
    I confirm this is the case also for PyCharm.
    – tsando
    Commented Feb 12, 2020 at 17:23
0

On Jupyter Notebooks on Anaconda, doing this:

%load_ext autoreload
%autoreload 2

produced the message:

The autoreload extension is already loaded. To reload it, use: %reload_ext autoreload

It looks like it's preferable to do:

%reload_ext autoreload
%autoreload 2

Version information:

The version of the notebook server is 5.0.0 and is running on: Python 3.6.2 |Anaconda, Inc.| (default, Sep 20 2017, 13:35:58) [MSC v.1900 32 bit (Intel)]

0

Any subobjects will not be reloaded by this, I believe you have to use IPython's deepreload for that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.