Many of the answers suggest how to choose only 1 option, but why not be able to use all 3? In this answer I explain how you can use asyncio
to manage combining all 3 forms of concurrency instead as well as easily swap between them later if need be.
The short answer
Many developers that are first-timers to concurrency in Python will end up using processing.Process
and threading.Thread
. However, these are the low-level APIs which have been merged together by the high-level API provided by the concurrent.futures
module. Furthermore, spawning processes and threads has overhead, such as requiring more memory, a problem which plagued one of the examples I showed below. To an extent, concurrent.futures
manages this for you so that you cannot as easily do something like spawn a thousand processes and crash your computer by only spawning a few processes and then just re-using those processes each time one finishes.
These high-level APIs are provided through concurrent.futures.Executor
, which are then implemented by concurrent.futures.ProcessPoolExecutor
and concurrent.futures.ThreadPoolExecutor
. In most cases, you should use these over the multiprocessing.Process
and threading.Thread
, because it's easier to change from one to the other in the future when you use concurrent.futures
and you don't have to learn the detailed differences of each.
Since these share a unified interfaces, you'll also find that code using multiprocessing
or threading
will often use concurrent.futures
. asyncio
is no exception to this, and provides a way to use it via the following code:
import asyncio
from concurrent.futures import Executor
from functools import partial
from typing import Any, Callable, Optional, TypeVar
T = TypeVar("T")
async def run_in_executor(
executor: Optional[Executor],
func: Callable[..., T],
/,
*args: Any,
**kwargs: Any,
) -> T:
"""
Run `func(*args, **kwargs)` asynchronously, using an executor.
If the executor is None, use the default ThreadPoolExecutor.
"""
return await asyncio.get_running_loop().run_in_executor(
executor,
partial(func, *args, **kwargs),
)
# Example usage for running `print` in a thread.
async def main():
await run_in_executor(None, print, "O" * 100_000)
asyncio.run(main())
In fact it turns out that using threading
with asyncio
was so common that in Python 3.9 they added asyncio.to_thread(func, *args, **kwargs)
to shorten it for the default ThreadPoolExecutor
.
The long answer
Are there any disadvantages to this approach?
Yes. With asyncio
, the biggest disadvantage is that asynchronous functions aren't the same as synchronous functions. This can trip up new users of asyncio
a lot and cause a lot of rework to be done if you didn't start programming with asyncio
in mind from the beginning.
Another disadvantage is that users of your code will also become forced to use asyncio
. All of this necessary rework will often leave first-time asyncio
users with a really sour taste in their mouth.
Are there any non-performance advantages to this?
Yes. Similar to how using concurrent.futures
is advantageous over threading.Thread
and multiprocessing.Process
for its unified interface, this approach can be considered a further abstraction from an Executor
to an asynchronous function. You can start off using asyncio
, and if later you find a part of it you need threading
or multiprocessing
, you can use asyncio.to_thread
or run_in_executor
. Likewise, you may later discover that an asynchronous version of what you're trying to run with threading already exists, so you can easily step back from using threading
and switch to asyncio
instead.
Are there any performance advantages to this?
Yes... and no. Ultimately it depends on the task. In some cases, it may not help (though it likely does not hurt), while in other cases it may help a lot. The rest of this answer provides some explanations as to why using asyncio
to run an Executor
may be advantageous.
- Combining multiple executors and other asynchronous code
asyncio
essentially provides significantly more control over concurrency at the cost of you need to take control of the concurrency more. If you want to simultaneously run some code using a ThreadPoolExecutor
along side some other code using a ProcessPoolExecutor
, it is not so easy managing this using synchronous code, but it is very easy with asyncio
.
import asyncio
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
async def with_processing():
with ProcessPoolExecutor() as executor:
tasks = [...]
for task in asyncio.as_completed(tasks):
result = await task
...
async def with_threading():
with ThreadPoolExecutor() as executor:
tasks = [...]
for task in asyncio.as_completed(tasks):
result = await task
...
async def main():
await asyncio.gather(with_processing(), with_threading())
asyncio.run(main())
How does this work? Essentially asyncio
asks the executors to run their functions. Then, while an executor is running, asyncio
will go run other code. For example, the ProcessPoolExecutor
starts a bunch of processes, and then while waiting for those processes to finish, the ThreadPoolExecutor
starts a bunch of threads. asyncio
will then check in on these executors and collect their results when they are done. Furthermore, if you have other code using asyncio
, you can run them while waiting for the processes and threads to finish.
- Narrowing in on what sections of code needs executors
It is not common that you will have many executors in your code, but what is a common problem that I have seen when people use threads/processes is that they will shove the entirety of their code into a thread/process, expecting it to work. For example, I once saw the following code (approximately):
from concurrent.futures import ThreadPoolExecutor
import requests
def get_data(url):
return requests.get(url).json()["data"]
urls = [...]
with ThreadPoolExecutor() as executor:
for data in executor.map(get_data, urls):
print(data)
The funny thing about this piece of code is that it was slower with concurrency than without. Why? Because the resulting json
was large, and having many threads consume a huge amount of memory was disastrous. Luckily the solution was simple:
from concurrent.futures import ThreadPoolExecutor
import requests
urls = [...]
with ThreadPoolExecutor() as executor:
for response in executor.map(requests.get, urls):
print(response.json()["data"])
Now only one json
is unloaded into memory at a time, and everything is fine.
The lesson here?
You shouldn't try to just slap all of your code into threads/processes, you should instead focus in on what part of the code actually needs concurrency.
But what if get_data
was not a function as simple as this case? What if we had to apply the executor somewhere deep in the middle of the function? This is where asyncio
comes in:
import asyncio
import requests
async def get_data(url):
# A lot of code.
...
# The specific part that needs threading.
response = await asyncio.to_thread(requests.get, url, some_other_params)
# A lot of code.
...
return data
urls = [...]
async def main():
tasks = [get_data(url) for url in urls]
for task in asyncio.as_completed(tasks):
data = await task
print(data)
asyncio.run(main())
Attempting the same with concurrent.futures
is by no means pretty. You could use things such as callbacks, queues, etc., but it would be significantly harder to manage than basic asyncio
code.