6

I'm writing a command to randomly create 5M orders in a database.

def constrained_sum_sample(
    number_of_integers: int, total: Optional[int] = 5000000
) -> int:
    """Return a randomly chosen list of n positive integers summing to total.

    Args:
        number_of_integers (int): The number of integers;
        total (Optional[int]): The total sum. Defaults to 5000000.

    Yields:
        (int): The integers whose the sum is equals to total.
    """

    dividers = sorted(sample(range(1, total), number_of_integers - 1))
    for i, j in zip(dividers + [total], [0] + dividers):
        yield i - j


def create_orders():
    customers = Customer.objects.all()
    number_of_customers = Customer.objects.count()
    for customer, number_of_orders in zip(
        customers,
        constrained_sum_sample(number_of_integers=number_of_customers),
    ):
        for _ in range(number_of_orders):
            create_order(customer=customer)

number_of_customers will be at least greater than 1k and the create_order function does at least 5 db operations (one to create the order, one to randomly get the order's store, one to create the order item (and this can go up to 30, also randomly), one to get the item's product (or higher but equals to the item) and one to create the sales note.

As you may suspect this take a LONG time to complete. I've tried, unsuccessfully, to perform these operations asynchronously. All of my attempts (dozen at least; most of them using sync_to_async) have raised the following error:

SynchronousOnlyOperation you cannot call this from an async context - use a thread or sync_to_async

Before I continue to break my head, I ask: is it possible to achieve what I desire? If so, how should I proceed?

Thank you very much!

10
  • Could you have a number of async methods that are run concurrently, but within them you have the await sync_to_async(write_to_db)(**props_as_primitives)?
    – schillingt
    Jan 7, 2021 at 16:31
  • Sorry, but I didn't understand your question. My problem is quite simple actually: I want to to make the first loop of the create_orders function to be async. Jan 8, 2021 at 0:55
  • First of all, you are creating 5M rows, definitely, it will take time (also depends on high scalable your Database is). Secondly, calling async db connection may not help you, since they must take place in "sequential" order (simple ex: You must create customer instance before creating the order).
    – JPG
    Jan 10, 2021 at 2:35
  • 1
    I would rather try to clean my code before trying to port to an async solution.
    – JPG
    Jan 10, 2021 at 2:36
  • @JPG as you can see by the code the customer are already created. Also, the orders dont need do be in "sequential" order. Finally, simple saying "clean your code" helps literally zero. Jan 10, 2021 at 13:04

3 Answers 3

5

Not yet supported but in development.

Django 3.1 has officially asynchronous support for views and middleware however if you try to call ORM within async function you will get SynchronousOnlyOperation.

if you need to call DB from async function they have provided helpers utils like: async_to_sync and sync_to_async to change between threaded or coroutine mode as follows:

from asgiref.sync import sync_to_async

results = await sync_to_async(Blog.objects.get, thread_sensitive=True)(pk=123)

If you need to queue call to DB, we used to use tasks queues like celery or rabbitMQ.

  • By the way if you really know what you are doing you can call it but on your responsibility just turn off the Async safety but watch out for data lost and integrity errors
#settings.py
DJANGO_ALLOW_ASYNC_UNSAFE=True

The reason this is needed in Django is that many libraries, specifically database adapters, require that they are accessed in the same thread that they were created in. Also a lot of existing Django code assumes it all runs in the same thread, e.g. middleware adding things to a request for later use in views.

More fun news in the release notes: https://docs.djangoproject.com/en/3.1/topics/async/

1

It's possible to achieve what you desire, however you need a different perspective to solve this problem.

Try using asynchronous workers, and a simple one would be rq workers or celery.

Use one of these libraries to process async long-running tasks defined in django in different threads or processes.

-2

you can use bulk_create() to create large number of objects , this will speed up the process , additionally put the bulk_create() under a separate thread.

1
  • Yeah bulk_create are going to help but not solve the problem. How can I put it in a separate thread? Jan 10, 2021 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.