1

Imagine that I have a main program which starts many async activities which all wait on queues to do jobs, and then on ctrl-C properly closes them all down: it might look something like this:

async def run_act1_forever():
  # this is the async queue loop
  while True:
    job = await inputQueue1.get()
    # do something with this incoming job

def run_activity_1(loop):
  # run the async queue loop as a task
  coro = loop.create_task(run_act1_forever())
  return coro

def mainprogram():
  loop = asyncio.get_event_loop()
  act1 = run_activity_1(loop)
  # also start act2, act3, etc here

  try:
    loop.run_forever()
  except KeyboardInterrupt: 
    pass
  finally:
    act1.cancel()
    # also act2.cancel(), act3.cancel(), etc
    loop.close()

This all works fine. However, starting up activity 1 is actually more complex than this; it happens in three parts. Part 1 is to wait on the queue until a particular job comes in, one time; part 2 is a synchronous part which has to run in a thread with run_in_executor, one time, and then part 3 is the endless waiting on the queue for jobs as above. How do I structure this? My initial thought was:

async def run_act1_forever():
  # this is the async queue loop
  while True:
    job = await inputQueue1.get()
    # do something with this incoming job

async def run_act1_step1():
  while True:
    job = await inputQueue1.get()
    # good, we have handled that first task; we're done
    break

def run_act1_step2():
  # note: this is sync, not async, so it's in a thread
  # do whatever, here, and then exit when done
  time.sleep(5)

def run_activity_1(loop):
  # run step 1 as a task
  step1 = loop.create_task(run_act1_step1())
  # ERROR! See below

  # now run the sync step 2 in a thread
  self.loop.run_in_executor(None, run_act1_step2())

  # finally, run the async queue loop as a task
  coro = loop.create_task(run_act1_forever())
  return coro

def mainprogram():
  loop = asyncio.get_event_loop()
  act1 = run_activity_1(loop)
  # also start act2, act3, etc here

  try:
    loop.run_forever()
  except KeyboardInterrupt: 
    pass
  finally:
    act1.cancel()
    # also act2.cancel(), act3.cancel(), etc
    loop.close()

but this does not work, because at the point where we say "ERROR!", we need to await the step1 task and we never do. We can't await it, because run_activity_1 is not an async function. So... what should I do here?

I thought about getting the Future back from calling run_act1_step1() and then using future.add_done_callback to handle running steps 2 and 3. However, if I do that, then run_activity_1() can't return the future generated by run_act1_forever(), which means that mainprogram() can't cancel that run_act1_forever() task.

I thought of generating an "empty" Future in run_activity_1() and returning that, and then making that empty Future "chain" to the Future returned by run_act1_forever(). But Python asyncio doesn't support chaining Futures.

3 Answers 3

2

You say that things are difficult because run_activity_1 is not an async function, but don't really detail why it can't be async.

async def run_activity_1(loop):
    await run_act1_step1()
    await loop.run_in_executor(None, run_act1_step2)
    await run_act1_forever()

The returned coroutine won't be the same as the one returned by run_act1_forever(), but cancellation should propagate if you've got as far as executing that step.

With this change, run_activity_1 is no longer returning a task, so the invocation inside mainprogram would need to change to:

act1 = loop.create_task(run_activity_1(loop))
2
  • mainprogram() isn't async (and can't easily be, for boring reasons). So when mainprogram() calls run_activity_1(), it can't await the result (and should not do so anyway because it needs to run a bunch of activities).
    – sil
    Dec 30, 2021 at 15:59
  • 1
    You will still need to do a loop.create_task call at some level. With this change, you'd probably need to change mainprogram to run loop.create_task(run_activity_1(loop)). Dec 31, 2021 at 1:18
1

I think you were on the right track when you said, "I thought about getting the Future back from calling run_act1_step1() and then using future.add_done_callback to handle running steps 2 and 3." That's the logical way to structure this application. You have to manage the various returned objects correctly, but a small class solves this problem.

Here is a program similar to your second code snippet. It runs (tested with Python3.10) and handles Ctrl-C gracefully.

Python3.10 issues a deprecation warning when the function asyncio.get_event_loop() is called without a running loop, so I avoided doing that.

Activities.run() creates task1, then attaches a done_callback that starts task2 and the rest of the activities. The Activities object keeps track of task1 and task2 so they can be cancelled. The main program keeps a reference to Activities, and calls cancel_gracefully() to do the right thing, depending on how far the script progressed through the sequence of start-up activities.

Some care needs to be taken to catch the CancelledExceptions; otherwise stuff gets printed on the console when the program terminates.

The important difference between this program and your second code snippet is that this program immediately stores task1 and task2 in variables so they can be accessed later. Therefore they can be cancelled any time after their creation. The done_callback trick is used to launch all the steps in the proper order.

#! python3.10

import asyncio
import time

async def run_act1_forever():
    # this is the async queue loop
    while True:
        await asyncio.sleep(1.0)
        # job = await inputQueue1.get()
        # do something with this incoming job
        print("Act1 forever")

async def run_act1_step1():
    while True:
        await asyncio.sleep(1.0)
        # job = await inputQueue1.get()
        # good, we have handled that first task; we're done
        break
    print("act1 step1 finished")

def run_act1_step2():
    # note: this is sync, not async, so it's in a thread
    # do whatever, here, and then exit when done
    time.sleep(5)
    print("Step2 finished")

class Activities:
    def __init__(self, loop):
        self.loop = loop
        self.task1: asyncio.Task = None
        self.task2: asyncio.Task = None
        
    def run(self):
        # run step 1 as a task
        self.task1 = self.loop.create_task(run_act1_step1())
        self.task1.add_done_callback(self.run2)
        # also start act2, act3, etc here
        
    def run2(self, fut):
        try:
            if fut.exception() is not None:  # do nothing if task1 failed
                return
        except asyncio.CancelledError:  # or if it was cancelled
            return
        # now run the sync step 2 in a thread
        self.loop.run_in_executor(None, run_act1_step2)
        
        # finally, run the async queue loop as a task
        self.task2 = self.loop.create_task(run_act1_forever())
        
    async def cancel_gracefully(self):
        if self.task2 is not None:
            # in this case, task1 has already finished without error
            self.task2.cancel()
            try:
                await self.task2
            except asyncio.CancelledError:
                pass
        elif self.task1 is not None:
            self.task1.cancel()
            try:
                await self.task1
            except asyncio.CancelledError:
                pass
        # also act2.cancel(), act3.cancel(), etc

def mainprogram():
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    acts = Activities(loop)
    loop.call_soon(acts.run)

    try:
        loop.run_forever()
    except KeyboardInterrupt: 
        pass
    loop.run_until_complete(acts.cancel_gracefully())
        
if __name__ == "__main__":
    mainprogram()
1

You can do this with a combination of threading events and asyncio events. You'll need two events, one to signal the first item has arrived. The thread will wait on this event, so it needs to be a threading Event. You'll also need one to signal the thread is finished. Your run_act1_forever coroutine will await this, so it will need to be an asyncio Event. You can then return the task for run_act1_forever normally and cancel it as you need.

Note that when setting the asyncio event from the separate thread you'll need to use loop.call_soon_threadsafe as asyncio Events are not thread safe.

import asyncio
import time
import threading
import functools
from asyncio import Queue, AbstractEventLoop

async def run_act1_forever(inputQueue1: Queue,
                           thread_done_event: asyncio.Event):
  await thread_done_event.wait()
  print('running forever')
  while True:
    job = await inputQueue1.get()

async def run_act1_step1(inputQueue1: Queue,
                         first_item_event: threading.Event):
  print('Waiting for queue item')
  job = await inputQueue1.get()
  print('Setting event')
  first_item_event.set()


def run_act1_step2(loop: AbstractEventLoop,
                   first_item_event: threading.Event,
                   thread_done_event: asyncio.Event):
  print('Waiting for event...')
  first_item_event.wait()
  print('Got event, processing...')
  time.sleep(5)
  loop.call_soon_threadsafe(thread_done_event.set)


def run_activity_1(loop):
  inputQueue1 = asyncio.Queue(loop=loop)

  first_item_event = threading.Event()
  thread_done_event = asyncio.Event(loop=loop)
  loop.create_task(run_act1_step1(inputQueue1, first_item_event))

  inputQueue1.put_nowait('First item to test the code')

  loop.run_in_executor(None, functools.partial(run_act1_step2,
                                               loop,
                                               first_item_event,
                                               thread_done_event))
  return loop.create_task(run_act1_forever(inputQueue1, thread_done_event))

def mainprogram():
  loop = asyncio.new_event_loop()
  act1 = run_activity_1(loop)
  # also start act2, act3, etc here

  try:
    loop.run_forever()
  except KeyboardInterrupt:
    pass
  finally:
    act1.cancel()

    # also act2.cancel(), act3.cancel(), etc
    loop.close()

mainprogram()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.