0

I have a Task that generates a PDF file for an order (it takes about 10 seconds to create one PDF):

public async Task GeneratePDF(Guid Id) {
   var order = await 
      _context
      .Orders
      .Include(order => order.Customer)
      ... //a lot more Include and ThenInclude statements
      .FirstOrDefaultAsync(order ==> order.Id == Id);
   var document = ...  //PDF generated here, takes about 10 seconds
   order.PDF = document ;
   _context.SaveChangesAsync();
}

I tried the following:

public async Task GenerateAllPDFs() {
   var orderIds = await _context.Orders.Select(order=> order.Id).ToListAsync();
   foreach (var id in orderIds)
   {
      _ = GeneratePDF(id).ContinueWith(t => Console.WriteLine(t.Exception), TaskContinuationOptions.OnlyOnFaulted);
   }
}

this gives me the error:

System.ObjectDisposedException: Cannot access a disposed object. A common cause of this error is disposing a context that was resolved from dependency injection and then later trying to use the same context instance elsewhere in your application. This may occur if you are calling Dispose() on the context, or wrapping the context in a using statement. If you are using dependency injection, you should let the dependency injection container take care of disposing context instances.

If I change the task as follows...

public async Task GenerateAllPDFs() {
   var orderIds = await _context.Orders.Select(order=> order.Id).ToListAsync();
   foreach (var id in orderIds)
   {
      _ = await GeneratePDF(id);
   }
}

...it runs the task for each order in series, taking ages to complete (I have a few thousands orders, taking about 10 seconds per order)...

How can I run this task in parallel for all orders in the context, so that the time it takes to complete is much less than sequential processing?

7
  • it looks for me as if GeneratePDF does too much. The part with the writing the stuff back to the server should not be in there. You need to separate it. 1) get all orders, 2) then do a Parallel.Foreach to generate all documents in parallel. and assign each document to the proper order and in the end 4) do a single ` _context.SaveChangesAsync();` to make a bulk update on the data on the server
    – Mong Zhu
    Oct 30, 2019 at 17:35
  • this line: var orderIds = await _context.Orders.Select(order=> order.Id).ToListAsync(); is actually superfluous. If you are querying the server, then you could already get all orders, since you need them anyway: var allOrders = await _context.Orders.Select(order=> order).ToListAsync();
    – Mong Zhu
    Oct 30, 2019 at 17:38
  • 1
    @MongZhu that makes sense. let me try to refactor it.
    – intrixius
    Oct 30, 2019 at 17:41
  • @MongZhu Although your comment makes sense, it does not change the execution time.
    – intrixius
    Oct 30, 2019 at 17:46
  • did you measure the execution time of each step? which one takes the longest? did you change your code like I suggested in my answer? could you post it?
    – Mong Zhu
    Oct 30, 2019 at 17:48

4 Answers 4

2

You can map your order IDs to tasks and await them all like:

public async Task GeneratePDF(Order order) {
   var document = ...  //PDF generated here, takes about 10 seconds
   order.PDF = document ;
}

public async Task GenerateAllPDFs() {
   var orderIds = await _context.Orders.ToListAsync();
   var tasks = orderIds.Select((order) => GeneratePDF(order).ContinueWith(t => Console.WriteLine(t.Exception), TaskContinuationOptions.OnlyOnFaulted));
   await Task.WhenAll(tasks);
   await _context.SaveChangesAsync();
}
15
  • this method works only partially, I get this error intermittently: System.InvalidOperationException: A second operation started on this context before a previous operation completed. This is usually caused by different threads using the same instance of DbContext, however instance members are not guaranteed to be thread safe. This could also be caused by a nested query being evaluated on the client, if this is the case rewrite the query avoiding nested invocations.
    – intrixius
    Oct 30, 2019 at 18:08
  • 1
    i have the feeling that this line: var order = await _context.Orders.FirstOrDefaultAsync(order ==> order.Id == Id); is actually causing the exception
    – Mong Zhu
    Oct 30, 2019 at 18:12
  • 1
    Good one, big warning message about this on EF Cores docs.. :) learn.microsoft.com/en-us/ef/core/querying/async
    – Grabofus
    Oct 30, 2019 at 18:14
  • 1
    @MongZhu Updated, though it's getting really similar to your answer.. :) If the generation of the PDF is async or need the generation to be reusable, you can use this answer, otherwise you're better off sticking with Parallel.ForEach.
    – Grabofus
    Oct 30, 2019 at 18:20
  • 1
    @intrixius the generation of the document itself is still independent of all your db access code. Simply move this part into an extra method. and you can reuse it however you like
    – Mong Zhu
    Oct 30, 2019 at 18:41
1

Here is my suggestion from the comment as an answer. I would split it in 3 parts:

1) get all orders,

2) then do a Parallel.Foreach to generate all documents in parallel. and assign each document to the proper order and in the end

3) do a single _context.SaveChangesAsync(); to make a bulk update on the data on the server

public async Task GenerateAllPDFs()
{
    var allOrders = await _context.Orders.ToListAsync();
    System.Threading.Tasks.Parallel.ForEach(allOrders, order => 
    {
        var document = ...  //PDF generated here, takes about 10 seconds
        order.PDF = document ;
    });
    await _context.SaveChangesAsync();
}
4
  • 1
    What's this part for? .Select(order => order)
    – Grabofus
    Oct 30, 2019 at 17:58
  • @Grabofus you are right, I did not think that it access already the Orders. Thank you
    – Mong Zhu
    Oct 30, 2019 at 18:04
  • 1
    Also (I know it's just copy paste of the original), but you should await the SaveChangesAsync() :)
    – Grabofus
    Oct 30, 2019 at 18:05
  • @Grabofus thanx again :D
    – Mong Zhu
    Oct 30, 2019 at 18:06
0

You need to implement parallel programing.

https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/task-based-asynchronous-programming

public class Example
{
   public static void Main()
   {
      Task[] taskArray = new Task[10];
      for (int i = 0; i < taskArray.Length; i++) {
         taskArray[i] = Task.Factory.StartNew( (Object obj ) => {
                                                  CustomData data = obj as CustomData;
                                                  if (data == null) 
                                                     return;

                                                  data.ThreadNum = Thread.CurrentThread.ManagedThreadId;
                                               },
                                               new CustomData() {Name = i, CreationTime = DateTime.Now.Ticks} );
      }
      Task.WaitAll(taskArray);     
      foreach (var task in taskArray) {
         var data = task.AsyncState as CustomData;
         if (data != null)
            Console.WriteLine("Task #{0} created at {1}, ran on thread #{2}.",
                              data.Name, data.CreationTime, data.ThreadNum);
      }                     
   }
}
1
  • that seems interesting, but those MS docs are not very clear on how to implement it in (my) real life scenarios
    – intrixius
    Oct 30, 2019 at 17:48
0

I think I will have to "duplicate" the GeneratePDF method to facilitate the batch processing by implementing the other answers , since I need this method also in non-batch mode...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.