18

I have been trying all kinds of things to be able to use an asyncio loop inside another asyncio loop. Most of the time my test just end in errors, such as:

RuntimeError: This event loop is already running

My example code below is just the base test I started with, so you can see the basics of what I am trying to do. I tried so many things after this test, it was just too confusing, so I figured I should keep it simple when asking for help. If anyone can point me in the right direction, that would be great. Thank you for your time!

import asyncio

async def fetch(data):
    message = 'Hey {}!'.format(data)
    other_data = ['image_a.com', 'image_b.com', 'image_c.com']
    images = sub_run(other_data)
    return {'message' : message, 'images' : images}

async def bound(sem, data):
    async with sem:
        r = await fetch(data)
        return r

async def build(dataset):
    tasks = []
    sem = asyncio.Semaphore(400)

    for data in dataset:
        task = asyncio.ensure_future(bound(sem, data))
        tasks.append(task)

    r = await asyncio.gather(*tasks)
    return r

def run(dataset):
    loop = asyncio.get_event_loop()
    future = asyncio.ensure_future(build(dataset))
    responses = loop.run_until_complete(future)
    loop.close()
    return responses

async def sub_fetch(data):
    image = 'https://{}'.format(data)
    return image

async def sub_bound(sem, data):
    async with sem:
        r = await sub_fetch(data)
        return r

async def sub_build(dataset):
    tasks = []
    sem = asyncio.Semaphore(400)

    for data in dataset:
        task = asyncio.ensure_future(sub_bound(sem, data))
        tasks.append(task)

    r = await asyncio.gather(*tasks)
    return r

def sub_run(dataset):
    loop = asyncio.get_event_loop()
    future = asyncio.ensure_future(sub_build(dataset))
    responses = loop.run_until_complete(future)
    loop.close()
    return responses

if __name__ == '__main__':
    dataset = ['Joe', 'Bob', 'Zoe', 'Howard']
    responses = run(dataset)
    print (responses)

1 Answer 1

17

Running loop.run_until_compete inside a running event loop would block the outer loop, thus defeating the purpose of using asyncio. Because of that, asyncio event loops aren't recursive, and one shouldn't need to run them recursively. (See this issue for details.) Instead of creating an inner event loop, await a task on the existing one.

In your case, remove sub_run and simply replace its usage:

images = sub_run(other_data)

with:

images = await sub_build(other_data)

And it will work just fine, running the sub-coroutines and not continuing with the outer coroutine until the inner one is complete, as you likely intended from the sync code.

11
  • 2
    That is awesome! I get it now! I can just use the same loop and create more task for that loop. You explained it very clearly, thank you very much. Commented Feb 19, 2018 at 16:52
  • Don't we need to change sub_run() from sync to async?
    – Emerson Xu
    Commented Dec 18, 2019 at 22:14
  • @EmersonXu The idea is to completely get rid of sub_run. Commented Dec 19, 2019 at 7:39
  • @user4815162342 yes, it's very clear to remove sub_run() in this case. But say if we have many sync function calls inside run(), do we need to convert all sync functions to async?
    – Emerson Xu
    Commented Dec 20, 2019 at 10:43
  • 1
    @EmersonXu Either convert them, or use run_in_executor to execute legacy blocking code. Commented Dec 20, 2019 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.