409

I have not seen clear examples with use-cases for Pool.apply, Pool.apply_async and Pool.map. I am mainly using Pool.map; what are the advantages of others?

3 Answers 3

563

Back in the old days of Python, to call a function with arbitrary arguments, you would use apply:

apply(f,args,kwargs)

apply still exists in Python2.7 though not in Python3, and is generally not used anymore. Nowadays,

f(*args,**kwargs)

is preferred. The multiprocessing.Pool modules tries to provide a similar interface.

Pool.apply is like Python apply, except that the function call is performed in a separate process. Pool.apply blocks until the function is completed.

Pool.apply_async is also like Python's built-in apply, except that the call returns immediately instead of waiting for the result. An AsyncResult object is returned. You call its get() method to retrieve the result of the function call. The get() method blocks until the function is completed. Thus, pool.apply(func, args, kwargs) is equivalent to pool.apply_async(func, args, kwargs).get().

In contrast to Pool.apply, the Pool.apply_async method also has a callback which, if supplied, is called when the function is complete. This can be used instead of calling get().

For example:

import multiprocessing as mp
import time

def foo_pool(x):
    time.sleep(2)
    return x*x

result_list = []
def log_result(result):
    # This is called whenever foo_pool(i) returns a result.
    # result_list is modified only by the main process, not the pool workers.
    result_list.append(result)

def apply_async_with_callback():
    pool = mp.Pool()
    for i in range(10):
        pool.apply_async(foo_pool, args = (i, ), callback = log_result)
    pool.close()
    pool.join()
    print(result_list)

if __name__ == '__main__':
    apply_async_with_callback()

may yield a result such as

[1, 0, 4, 9, 25, 16, 49, 36, 81, 64]

Notice, unlike pool.map, the order of the results may not correspond to the order in which the pool.apply_async calls were made.


So, if you need to run a function in a separate process, but want the current process to block until that function returns, use Pool.apply. Like Pool.apply, Pool.map blocks until the complete result is returned.

If you want the Pool of worker processes to perform many function calls asynchronously, use Pool.apply_async. The order of the results is not guaranteed to be the same as the order of the calls to Pool.apply_async.

Notice also that you could call a number of different functions with Pool.apply_async (not all calls need to use the same function).

In contrast, Pool.map applies the same function to many arguments. However, unlike Pool.apply_async, the results are returned in an order corresponding to the order of the arguments.

15
  • 12
    Should there be if __name__=="__main__" before apply_async_with_callback() on Windows?
    – jfs
    Commented Dec 16, 2011 at 12:38
  • 47
    Look inside multiprocessing/pool.py and you will see that Pool.map(func,iterable) is equivalent to Pool.map_async(func,iterable).get(). So the relationship between Pool.map and Pool.map_async is similar to that of Pool.apply and Pool.apply_async. The async commands return immediately, while the non-async commands block. The async commands also have a callback.
    – unutbu
    Commented Dec 17, 2011 at 11:38
  • 9
    Deciding between using Pool.map and Pool.apply is similar to deciding when to use map or apply in Python. You just use the tool that fits the job. Deciding between using the async and non-async version depends on if you want the call to block the current process and/or if you want to use the callback.
    – unutbu
    Commented Dec 17, 2011 at 11:39
  • 6
    @falsePockets: Yes. Each call to apply_async returns an ApplyResult object. Calling that ApplyResult's get method will return the associated function's return value (or raise mp.TimeoutError if the call times-out.) So if you put the ApplyResults in an ordered list, then calling their get methods will return the results in the same order. You could just use pool.map in this situation however.
    – unutbu
    Commented May 22, 2017 at 10:11
  • 4
    @galactica: Each time the worker function ends successfully (without raising an exception), the callback function is called in the main process. The worker functions put return values in a queue, and the pool._result_handler thread in the main process handles the returned values one at a time, passing the returned value to the callback function. So you are guaranteed that the callback function will be called once for each returned value and there is no concurrency problem here because the callback is being called sequentially by a single thread in the main process.
    – unutbu
    Commented Jul 29, 2019 at 22:46
189

Here is an overview in a table format in order to show the differences between Pool.apply, Pool.apply_async, Pool.map and Pool.map_async. When choosing one, you have to take multi-args, concurrency, blocking, and ordering into account:

                  | Multi-args   Concurrence    Blocking     Ordered-results
---------------------------------------------------------------------
Pool.map          | no           yes            yes          yes
Pool.map_async    | no           yes            no           yes
Pool.apply        | yes          no             yes          no
Pool.apply_async  | yes          yes            no           no
Pool.starmap      | yes          yes            yes          yes
Pool.starmap_async| yes          yes            no           no

Notes:

  • Pool.imap and Pool.imap_async – lazier version of map and map_async.

  • Pool.starmap method, very much similar to map method besides it acceptance of multiple arguments.

  • Async methods submit all the processes at once and retrieve the results once they are finished. Use get method to obtain the results.

  • Pool.map(or Pool.apply)methods are very much similar to Python built-in map(or apply). They block the main process until all the processes complete and return the result.

Examples:

map

Is called for a list of jobs in one time

results = pool.map(func, [1, 2, 3])

apply

Can only be called for one job

for x, y in [[1, 1], [2, 2]]:
    results.append(pool.apply(func, (x, y)))

def collect_result(result):
    results.append(result)

map_async

Is called for a list of jobs in one time

pool.map_async(func, jobs, callback=collect_result)

apply_async

Can only be called for one job and executes a job in the background in parallel

for x, y in [[1, 1], [2, 2]]:
    pool.apply_async(worker, (x, y), callback=collect_result)

starmap

Is a variant of pool.map which support multiple arguments

pool.starmap(func, [(1, 1), (2, 1), (3, 1)])

starmap_async

A combination of starmap() and map_async() that iterates over iterable of iterables and calls func with the iterables unpacked. Returns a result object.

pool.starmap_async(calculate_worker, [(1, 1), (2, 1), (3, 1)], callback=collect_result)

Reference:

Find complete documentation here: https://docs.python.org/3/library/multiprocessing.html

6
  • 3
    Pool.starmap() is blocking Commented Mar 20, 2020 at 0:43
  • I like this answer, +1 Commented Apr 13, 2021 at 20:50
  • 5
    if apply does not have concurrence, then what is its point? use?
    – Ritwik
    Commented May 2, 2021 at 19:37
  • 3
    A table/picture is worth a thousand words, I always come back to this answer again and again to pick the function I want.
    – lqi
    Commented May 2, 2022 at 18:52
  • This is all very nice in theory, but in practice you often have to very carefully tune the size of work delegated to each parallel worker in order to get any speedup in the face of all of the overheads involved in using the pool. In particular you have to be using shared memory rather than passing the arguments and the return values directly, to avoid the serialization time of them between your main process and the workers when they are large. Otherwise it's no more than a way of warming yourself up through hardware heat dissipation on a cold day.
    – matanox
    Commented Jan 31 at 5:49
90

Regarding apply vs map:

pool.apply(f, args): f is only executed in ONE of the workers of the pool. So ONE of the processes in the pool will run f(args).

pool.map(f, iterable): This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. So you take advantage of all the processes in the pool.

6
  • 7
    what if the iterable is a generator Commented Jun 21, 2017 at 19:36
  • Hmm... Good question. To be honest I haven't ever used pools with generators, but this thread might be helpful: stackoverflow.com/questions/5318936/…
    – kakhkAtion
    Commented Jun 21, 2017 at 20:10
  • @kakhkAtion Regarding apply, if only one of the workers execute the function, what do the rest of the workers do? Do I have to call apply multiple times to have the rest of the workers perform a task?
    – Moondra
    Commented Jul 27, 2017 at 17:33
  • 4
    True. Also take a look at pool.apply_async if you want to lunch workers asynchronously. "pool_apply blocks until the result is ready, so apply_async() is better suited for performing work in parallel"
    – kakhkAtion
    Commented Jul 27, 2017 at 18:50
  • 1
    What happens when I have 4 processes but have called apply_async() 8 times? Will it automatically handle it with a queue? Commented Dec 23, 2019 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.