176

I am trying to implement multiprocessing in my code, and so, I thought that I would start my learning with some examples. I used the first example found in this documentation.

from multiprocessing import Pool
def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

When I run the above code I get an AttributeError: can't get attribute 'f' on <module '__main__' (built-in)>. I do not know why I am getting this error. I am also using Python 3.5 if that helps.

10
  • works perfectly for me (also on python 3.5). Commented Dec 29, 2016 at 19:02
  • 8
    I should add that I am also using Jupyter Notebook and anaconda as my interpreter. But anaconda is using python 3.5.
    – PiccolMan
    Commented Dec 29, 2016 at 19:11
  • 11
    I get the error on Windows 10 with Anaconda using Python 3.6.
    – devinbost
    Commented Nov 20, 2018 at 20:34
  • 9
    I get the error on a regular python shell windows 10 Python 3.6
    – rockikz
    Commented Jan 22, 2019 at 21:29
  • 1
    I got this error using spyder, but the error goes away when running the script from the command line. Commented Aug 6, 2020 at 19:43

5 Answers 5

166

This problem seems to be a design feature of multiprocessing.Pool. See https://bugs.python.org/issue25053. For some reason Pool does not always work with objects not defined in an imported module. So you have to write your function into a different file and import the module.

File: defs.py

def f(x):
    return x*x

File: run.py

from multiprocessing import Pool
import defs

 if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(defs.f, [1, 2, 3]))

If you use print or a different built-in function, the example should work. If this is not a bug (according to the link), the given example is chosen badly.

10
  • 2
    When I run the scripts, I got: AttributeError: exit. Turned out the problem was with the "with" statement, which requires an object with "_ _ enter " and " exit __" method. So I had to change it to: p = Pool(5) and it worked. Thank you very much! Commented Nov 4, 2017 at 11:16
  • 47
    You have to define the f() function before you create the instance of Pool, otherwise the workers cannot see your function. However, as per my understanding, you do NOT forcefully have to use imports. Commented Aug 19, 2019 at 1:03
  • 1
    Very funny, when running code under pycharm it gives me this error, but not in vscode or jupyter. So
    – Jay
    Commented Sep 26, 2019 at 13:45
  • 31
    For me, just defining the function f() above Pool creation (or import) did not solve the issue (Win 10, Python 3.6.8)
    – FlorianH
    Commented Mar 26, 2020 at 9:49
  • 3
    Me neither - Pool() still does not work with these suggestions. ThreadPool() works as intended, but it is a totally different function. Commented Apr 10, 2020 at 19:56
147

The multiprocessing module has a major limitation when it comes to IPython use:

Functionality within this package requires that the __main__ module be importable by the children. [...] This means that some examples, such as the multiprocessing.pool.Pool examples will not work in the interactive interpreter. [from the documentation]

Fortunately, there is a fork of the multiprocessing module called multiprocess which uses dill instead of pickle to serialization and overcomes this issue conveniently.

Just install multiprocess and replace multiprocessing with multiprocess in your imports:

import multiprocess as mp

def f(x):
    return x*x

with mp.Pool(5) as pool:
    print(pool.map(f, [1, 2, 3, 4, 5]))

Of course, externalizing the code as suggested in this answer works as well, but I find it very inconvenient: That is not why (and how) I use IPython environments.

<tl;dr> multiprocessing does not work in IPython environments right away, use its fork multiprocess instead.

6
  • 2
    the multiprocess fork does not seem to have the same modules - it cant find Process, freeze_support and Queue for example? Commented May 8, 2021 at 20:16
  • This one works with Jupyter notebook / IPython env, better than earlier one, but both works!
    – q43dom
    Commented Sep 2, 2021 at 15:14
  • 13
    Thanks, it's absolutely bonkers that simple multiprocessing doesn't work in 2021.
    – thc
    Commented Nov 20, 2021 at 1:47
  • 2
    This works perfectly in IPython (Python 3.9, macOS 11)! Commented Feb 14, 2022 at 12:51
  • 1
    The "AttributeError: Can't get attribute 'run_model' on <module 'main' (built-in)>" is still occurring in the relatively new MacOS Ventura 13.0+ (I am using 13.3) with PyTorch 2.0.0 when trying to access the torch.backends.mps.is_available() and associated MPS ways to utilise things like the AMD Radeon Pro 5500M GPU. The suggested solution seems to work here, too.
    – pudepied
    Commented Apr 10, 2023 at 15:06
13

This answer is for those who get this error on Windows 10 in 2021.

I've researched this error a bit since I got it myself. I get this error when running any examples from the official Python 3 documentation on multiprocessing.

Test environment:

  • x86 Windows 10.0.19043.1165 + Python 3.9.2 - there is an error
  • x86 Windows 10.0.19043.1165 + Python 3.9.6 - there is an error
  • x86 Windows 10.0.19043.1110 + Python 3.9.6 - there is an error
  • ARM Windows 10.0.21354.1 + Python 3.9.6 - no error (version from DEV branch)
  • ARM macOS 11.5.2 + Python 3.9.6 - no errors

I have no way to test this situation in other conditions. But my guess is that the problem is on Windows as there is no such bug in the developer version "10.0.21354.1", but this ARM version probably has x86 emulation.

Also note that there was no such bug at the time Python 3.9.2 was released (February). Since all this time I was working on the same computer, I was surprised by the situation when the previously working code stopped working, and only the version for Windows changed.

I was unable to find a bug request with a similar situation in the Python bug tracker (I probably did a poor search). And the message marked "Correct answer" refers to a different situation. The problem is easy to reproduce, you can try to follow any example from the multiprocessing documentation on a freshly installed Windows 10 + Python 3.

Later, I will have the opportunity to check out Python 3.10 and the latest version of Windows 10. I am also interested in this situation in the context of Windows 11.

If you have information about this error (link to the bug tracker or something similar), be sure to share it.

At the moment I switched to Linux to continue working.

3
  • 1
    I can reproduce the issue on Windows 1902 x64. No issues with WSL. p = Process(target = f('bob')) doesn't "work" since f('bob') is computed in current process and instead Process's target gets the f's return value i.e. None. If you change f to def f(name): return name then it will throw error since string name is not callable.
    – tejasvi
    Commented Aug 26, 2021 at 8:03
  • Thanks a lot for the clarification. Edited the answer, removed all the code - it still doesn't help) Commented Oct 14, 2021 at 22:09
  • Very interesting indeed. Wİn11 with py 3.12.3. I had to move mp handlers into my local library to make it work... Commented Apr 23 at 14:29
6

Why not use joblib? Your code is equivalent to:

# pip install joblib

from joblib import Parallel, delayed


def f(x):
    return x*x

res = Parallel(
    n_jobs=5
)(
    delayed(f)(x) for x in [1, 2, 3]
)
5
  • 2
    This has nothing to do with the question. The OP probably wants to use multiprocessing since it's a native solution.
    – Neoares
    Commented Aug 16, 2022 at 8:53
  • 1
    works like a charm in IPython! Commented Sep 15, 2022 at 9:53
  • 2
    @Neoares Why stick to the native solution when there's a better workable choice?
    – Wenmin Wu
    Commented Dec 16, 2022 at 3:25
  • @Wenmin Wu Is joblib a fully functional replacement of torch.multiprocessing e.g. support handaling multiple GPU's and DataSampler the same way?
    – MosQuan
    Commented Mar 5, 2023 at 23:26
  • Thanks, very useful, I didn't know about joblib and had a usecase where multiprocessing was difficult to use. joblib did the job.
    – ingo-m
    Commented Jun 19, 2023 at 15:36
-2

If you're using Jupyter notebook (like the OP), then defining the function in a separate cell and executing that cell first fixes the problem. The accepted answer works too, but it's more work. Defining the function before, i.e. above the pool, isn't adequate. It has to be in a completely different notebook cell which is executed first.

1
  • 29
    This does not work (anymore), unfortunately. Commented Dec 4, 2020 at 6:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.