176

I would like to use asyncio to get webpage html.

I run the following code in jupyter notebook:

import aiofiles
import aiohttp
from aiohttp import ClientSession

async def get_info(url, session):
    resp = await session.request(method="GET", url=url)
    resp.raise_for_status()
    html = await resp.text(encoding='GB18030')
    with open('test_asyncio.html', 'w', encoding='utf-8-sig') as f:
        f.write(html)
    return html
    
async def main(urls):
    async with ClientSession() as session:
        tasks = [get_info(url, session) for url in urls]
        return await asyncio.gather(*tasks)

if __name__ == "__main__":
    url = ['http://huanyuntianxiazh.fang.com/house/1010123799/housedetail.htm', 'http://zhaoshangyonghefu010.fang.com/house/1010126863/housedetail.htm']
    result = asyncio.run(main(url))

However, it returns RuntimeError: asyncio.run() cannot be called from a running event loop

What is the problem?

How to solve it?

8 Answers 8

251

The asyncio.run() documentation says:

This function cannot be called when another asyncio event loop is running in the same thread.

In your case, jupyter (IPython ≥ 7.0) is already running an event loop:

You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races.

Therefore you don't need to start the event loop yourself and can instead call await main(url) directly, even if your code lies outside any asynchronous function.

Modern Jupyter lab/notebook

Use the following for newer versions of Jupyter (IPython ≥ 7.0):

async def main():
    print(1)
    
await main()

Python or older IPython

If you are using Python ≥ 3.7 or IPython < 7.0, use the following:

import asyncio

async def main():
    print(1)
    
asyncio.run(main())

That's also that form you should use if you are running this in a python REPL or in an independent script (a bot, a web scrapper, etc.).

If you are using an older version of python (< 3.7), the API to run asynchronous code was a bit less elegant:

import asyncio

async def main():
    print(1)
    
loop = asyncio.get_event_loop()
loop.run_until_complete(hello_world())

Using await in your code

In your case, you can call await main(url) as follows:

url = ['url1', 'url2']
result = await main(url)

for text in result:
    pass # text contains your html (text) response

This change to recent versions of IPython makes notebook code simpler and more intuitive for beginers.

Further notices

Few remarks that might help you in different use cases.

Jupyter vs. IPython caution

There is a slight difference on how Jupyter uses the loop compared to IPython.

[...] IPykernel having a persistent asyncio loop running, while Terminal IPython starts and stops a loop for each code block.

This can lead to unexpected issues.

Google Colab

In the past, Google colab required you to do more complex loop manipulations like presented in some other answers here. Now plain await main() should just work like in IPython ≥ 7.0 (tested on Colab version 2023/08/18).

Python REPL

You can also run the python REPL using the asyncio concurrent context. As explained in asyncio's documentation:

$ python -m asyncio
asyncio REPL ...
Use "await" directly instead of "asyncio.run()".
>>> import asyncio
>>> await asyncio.sleep(10, result='hello')
'hello'

The asyncio REPL should be available for python ≥ 3.8.1.

When does asyncio.run matters and why?

Older versions of IPython were running in a synchronous context, which is why calling asyncio.run was mandatory.

The asyncio.run function allows to run asynchronous code from a synchronous context by doing the following:

  • starts an event loop,
  • runs the async function passed as argument in this (new) event loop,
  • stops the event loop once the function returned

In more technical terms (notice how the function is called a coroutine):

This function runs the passed coroutine, taking care of managing the asyncio event loop, finalizing asynchronous generators, and closing the threadpool.

What happen when using await in synchronous context?

If you happen to use await in a synchronous context you would get the one of the following errors:

  • SyntaxError: 'await' outside function
  • SyntaxError: 'await' outside async function

In that case that means you need to use asyncio.run(main()) instead of await main().

11
  • Thank you, cglacet. However, there is a warning : c:\program files\python37\lib\site-packages\ipykernel_launcher.py:29: RuntimeWarning: coroutine 'main' was never awaited
    – Chan
    Commented Mar 29, 2019 at 2:32
  • 1
    That's probably because you called main(url) instead of await main(url).
    – cglacet
    Commented Mar 29, 2019 at 2:33
  • 3
    Is there a way to have a code snippet that works both inside and outside of Jupyter? Commented Apr 2, 2020 at 10:58
  • I don't know if it's possible, but I'm not sure you would really want that anyway. Maybe ask this as a separate question and see if people have ideas on whether it's possible or desirable.
    – cglacet
    Commented Apr 11, 2020 at 13:06
  • 7
    Interestingly, when I run the above in jupyter, I get: SyntaxError: 'await' outside function
    – Luca
    Commented Jul 27, 2020 at 22:55
74

To add to cglacet's answer - if one wants to detect whether a loop is running and adjust automatically (ie run main() on the existing loop, otherwise asyncio.run()), here is a snippet that may prove useful:

# async def main():
#     ...

try:
    loop = asyncio.get_running_loop()
except RuntimeError:  # 'RuntimeError: There is no current event loop...'
    loop = None

if loop and loop.is_running():
    print('Async event loop already running. Adding coroutine to the event loop.')
    tsk = loop.create_task(main())
    # ^-- https://docs.python.org/3/library/asyncio-task.html#task-object
    # Optionally, a callback function can be executed when the coroutine completes
    tsk.add_done_callback(
        lambda t: print(f'Task done with result={t.result()}  << return val of main()'))
else:
    print('Starting new event loop')
    result = asyncio.run(main())
6
  • 1
    Nice, indeed adding a callback here is probably the only way of getting the result back. That would probably be interesting to have this code in a function so you would only have to call something like run(main(*args, **kwargs)). The function could also make sure you get your output back, not sure that's really simple tho (because of the callback).
    – cglacet
    Commented Apr 29, 2020 at 22:35
  • This is an intriguing solution. How would you get t.result() in a variable instead of printing it out?
    – alec_djinn
    Commented Aug 10, 2022 at 12:13
  • 1
    @alec_djinn no immediate easy answer (at least not from within the sync code), since introducing a blocking method in the current thread ie. in sync code (such as Queue.get(), with a Queue.put(t.result()) in callback) would also block the existing async loop from running (in current thread), thus no result would arrive -> deadlock. If you're in a Jupyter NB you could still use a messaging system and manually check for the result (in non-blocking manner). Note using time.sleep() in sync code (current thread) doesn't help since it would also pause the existing async loop.
    – Jean Monet
    Commented Aug 10, 2022 at 13:08
  • 1
    @JeanMonet this is awesome! Deserve more upvotes.
    – tyson.wu
    Commented Jul 14, 2023 at 7:59
  • This doesn't block execution after tsk.add_done_callback(...) and that's the problem if later code depends on result. Commented Oct 18, 2023 at 17:48
51

Just use this:

https://github.com/erdewit/nest_asyncio

import nest_asyncio
nest_asyncio.apply()
4
  • 8
    The docs for nest_asyncio mention a issue report with asyncio where its explicitly stated that this is no intended behaviour for asyncio. So I would consider nest_asyncio a big fat hack which I wouldn't trust with in my codebase to not break everything with a new Python version. Commented Oct 14, 2020 at 11:29
  • Another issue is that nest_asyncio requires Python 3.5 or above, which is not helpful to me as I am stuck on Python 2.7.
    – mknote
    Commented Nov 17, 2020 at 21:23
  • 2
    this allowed me to solve the issue for the imported code (where you can't change it yourself)
    – Alex S.
    Commented Nov 30, 2020 at 17:16
  • 1
    This worked so well fixing a imported asyncio issue, thanks alot
    – Mika C.
    Commented May 9, 2022 at 19:15
31

Combining the methods from Pankaj Sharma and Jean Monet, I wrote the following snippet that acts as asyncio.run (with slightly different syntax), but also works within a Jupyter notebook.

import threading

class RunThread(threading.Thread):
    def __init__(self, func, args, kwargs):
        self.func = func
        self.args = args
        self.kwargs = kwargs
        self.result = None
        super().__init__()

    def run(self):
        self.result = asyncio.run(self.func(*self.args, **self.kwargs))

def run_async(func, *args, **kwargs):
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = None
    if loop and loop.is_running():
        thread = RunThread(func, args, kwargs)
        thread.start()
        thread.join()
        return thread.result
    else:
        return asyncio.run(func(*args, **kwargs))

Usage:

async def test(name):
    await asyncio.sleep(5)
    return f"hello {name}"

run_async(test, "user")  # blocks for 5 seconds and returns "hello user"
1
  • 1
    thank you for this clever solution. Do you have any suggestion for "asyncio attached to a different loop" error? Inside child loops i need to use 2 coroutines from main loop.
    – Hakan
    Commented Aug 11, 2021 at 6:35
10

To add to existing answers, a somewhat shorter version:

  • Without external libraries
  • Allows running inside or outside of a Jupyter Notebook
  • Allows fetching the return value
try:
    asyncio.get_running_loop() # Triggers RuntimeError if no running event loop
    # Create a separate thread so we can block before returning
    with ThreadPoolExecutor(1) as pool:
        result = pool.submit(lambda: asyncio.run(myfunc())).result()
except RuntimeError:
    result = asyncio.run(myfunc())

Note: if myfunc's internals throw a RuntimeError and was run in the ThreadPoolExecutor, myfunc will be re-run in the except clause. So if your myfunc can throw a RuntimeError, consider checking for the RuntimeError's cause within the except clause

2
  • 1
    Attention: calling result() on the Future returned after submitting the lambda to the pool may raise RuntimeError on its own, depending on what is inside myfunc. This is then caught and causes running myfunc() again in the default event loop.
    – lnstadrum
    Commented Apr 9, 2023 at 15:01
  • To save some a google search from concurrent.futures import ThreadPoolExecutor. Although it's worth looking at the documentation as well, docs.python.org/3/library/… Commented Apr 18 at 18:39
6

I found the unsync package useful for writing code that behaves the same way in a Python script and the Jupyter REPL.

import asyncio
from unsync import unsync


@unsync
async def demo_async_fn():
    await asyncio.sleep(0.1)
    return "done!"

print(demo_async_fn().result())
0
4

Slight simplification of the solution by Mark:

import threading

class RunThread(threading.Thread):
    def __init__(self, coro):
        self.coro = coro
        self.result = None
        super().__init__()

    def run(self):
        self.result = asyncio.run(self.coro)


def run_async(coro):
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = None
    if loop and loop.is_running():
        thread = RunThread(coro)
        thread.start()
        thread.join()
        return thread.result
    else:
        return asyncio.run(coro)

Use run_async() like async.run(), i.e., run_async(test("user")).

1

As cglacet mentioned that documentation says

This function cannot be called when another asyncio event loop is running in the same thread.

You can use another thread i.e -

class ResolveThread(threading.Thread):
            def __init__(self,result1,fun,url):
                self.result1= result1
                self.fun = fun
                self.url = url
                threading.Thread.__init__(self)
            def run(self):
                result1[0] = asyncio.run(self.fun(self.url))


result1 = [None]
sp = ResolveThread(result1)
sp.start()
sp.join() # connect main thread
result = result1[0]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.