1

We have are developing an application that will receive events from various systems via a message queue (Azure) but it is just possible that some events (messages) will not arrive in the order they were sent. These events will be received and processed by a central CQRS/ES based system but my worry is that if the events are placed in the event store in the wrong order we will get garbage out (for example "order create" after "add order item").

Are typical ES systems meant to resolve this issue or are we meant to ensure that such messages are put in the right order before being pushed into the event store? If you have links to articles that back up either view it would help.

Edit: I think my description is clearly far too vague so the responses, while helpful in understanding CQRS/ES, do not quite answer my problem so I'll add a little more detail and hopefully someone will recognise the problem.

Firstly the players.

  • the front end web site (not actually relevant to this problem) delivers orders to the management system.
  • our management system which takes orders from the web site and passes them to the warehouse and is hosted on site.
  • the warehouse which accepts orders, fulfils them if possible and notifies us when an order is fulfilled or cannot be partially or completely fulfilled.

Linking the warehouse to the management system is a fairly thin Azure cloud based coupling. Messages from the warehouse are sent to a WCF/Soap layer in the cloud, parsed, and sent over the messages bus. Message to the warehouse are sent over the message bus and then, again in the cloud, converted into Soap calls to a server at the warehouse.

The warehouse is very careful to ensure that messages it sends have identifiers that increment without a gap so we can know when a message is missed. However when we take those messages and forward them to the management system they are transported over the message bus and could, in theory, arrive in the wrong order.

Now given that we have a sequence number in the messages we could ensure the messages are put back in the right order before they are sent to the CQRS/ES system but my questions is, is that necessary, can the ES actually be used to reorder the events into the logical order they were intended?

2
  • Are the events generated by different sources? e.g. the OrderCreated event is generated by another source than the OrderItemAdded event and at different times? Commented Jan 5, 2016 at 11:30
  • 1
    For pre and post dated events, see Greg Youngs post on the matter goodenoughsoftware.net/2014/03/02/… Commented Jan 5, 2016 at 11:35

5 Answers 5

1

Each message that arrives in Service Bus is tagged with a SequenceNumber. The SequenceNumber is a monotonically increasing, gapless 64-bit integer sequence, scoped to the Queue (or Topic) that provides an absolute order criterion by arrival in the Queue. That order may different from the delivery order due to errors/aborts and exists so you can reconstitute order of arrival.

Two features in Service Bus specific to management of order inside a Queue are:

  1. Sessions. A sessionful queue puts locks on all messages with the same SessionId property, meaning that FIFO is guaranteed for that sequence, since no messages later in the sequence are delivered until the "current" message is either processed or abandoned.
  2. Deferral. The Defer method puts a message aside if the message cannot be processed at this time. The message can later be retrieved by its SequenceNumber, which pulls from the hidden deferral queue. If you need a place to keep track of which messages have been deferred for a session, you can put a data structure holding that information right into the message session, if you use a sessionful queue. You can then pick up that state again elsewhere on an accepted session if you, for instance, fail over processing onto a different machine.

These features have been built specifically for document workflows in Office 365 where order obviously matters quite a bit.

2
  • Clemens, thanks. I think you are suggesting we fix the ordering before using the messages and for what it's worth I agree. However others disagree and actually a point they make is that multiple senders exist in the cloud so the original order of the messages can be lost.
    – naskew
    Commented Jan 7, 2016 at 6:31
  • 1
    The "right" message order always depends on some sort of context. For some it's the absolute order of arrival in the queue. If you care about order related to an originator or subject, use sessions (sessionid identifies originator/subject). If you care about order related to an external context you manage (like a workflow), also with multiple senders, stash "early" messages back into the queue/sub with deferral and retrieve them when it is their turn, i.e. sort the message stream on the receive side. [I work on Service Bus, FWIW] Commented Jan 7, 2016 at 14:00
1

I would have commented on KarlM's answer but stackoverflow won't allow it, so here goes...

It sounds like you want the transport mechanism to provide transactional locking on your aggregate. To me this sounds inherently wrong.

It sounds as though the design being proposed is flawed. Having had this exact problem in the past, I would look at your constraints. Either you want to provide transactional guarantees to the website, or you want to provide them to the warehouse. You can't do both, one always wins.

To be fully distributed: If you want to provide them to the website, then the warehouse must ask if it can begin to fulfil the order. If you want to provide them to the warehouse, then the website must ask if it can cancel the order.

Hope that is useful.

2
  • Simon, that is certainly true and we have had conversations along these lines here. The likelihood is that we will have a system where the warehouse tells us it is fulfilling the order and the website will display orders as locked once fulfilment starts. However the possibility exists that editing commences before the lock and is submitted after, in this case the user will be informed that it was not possible to edit.
    – naskew
    Commented Jan 5, 2016 at 14:21
  • If you're able to implement a request/response pattern for the website call I think that's fine. The trade off is just that the client system has to wait for the response before confirming the action. Commented Jan 5, 2016 at 18:59
0

For events generated from a single command handler/aggregate in an "optimistic locking" scenario, I would assume you would include the aggregate version in the event, and thus those events are implicitly ordered.

Events from multiple aggregates should not care about order, because of the transactional guarantees of an aggregate.

Check out http://cqrs.nu/Faq/aggregates , http://cqrs.nu/Faq/command-handlers and related FAQs

For an intro to ES and optimistic locking, look at http://www.jayway.com/2013/03/08/aggregates-event-sourcing-distilled/

1
  • Thanks for your reply Karl, the events are being sent from the website that takes customer orders and the warehouse that fulfils those orders. There are some decisions being taken I'm not sure how to handle, for example a customer may modify an order up to the moment the warehouse starts to fulfil that order but those two events could in theory overlap. The problem is not with the warehouse or the website, it is with the transport (Azure Service Bus) which does not guarantee messages will arrive in the order they are sent.
    – naskew
    Commented Jan 5, 2016 at 8:36
0

You say: "These events will be received and processed by a central CQRS/ES based system but my worry is that if the events are placed in the event store in the wrong order we will get garbage out (for example "order create" after "add order item")." There seems to be a misunderstanding about what CQRS pattern with Event Sourcing is. Simply put Event Sourcing means that you change Aggregates (as per DDD terminology) via internally generated events, the Aggregate persistence is represented by events and the Aggregate can be restored by replaying events. This means that the scope is quite small, the Aggregate itself. Now, CQRS with Event Sourcing means that these events from the Aggregates are published and used to create Read projections, or other domain models that have different purposes. So I don't really get your question given the explanations above.

Related to Ordering:

  • there is already an answer mentioning optimistic locking, so events generated inside a single Aggregate must be ordered and optimistic locking is a solution
  • Read projections processing events in order. A solution I used in the past was to to publish events on RabbitMQ and process them with Storm. RabbitMQ has some guarantees about ordering and Storm has some processing affinity features. For Storm, (as far as I remember) allows you to specify that for a given ID (for example an Aggregate ID) the same handler would be used, hence the events are processed in the same order as received from RabbitMQ.
0

The article on MSDN https://msdn.microsoft.com/en-us/library/jj591559.aspx states "Stored events should be immutable and are always read in the order in which they were saved" under "Performance, Scalability, and consistency". This clearly means that appending events out of order is not tolerated. The same article also states multiple times that while events cannot be altered, corrective events can be made. This would imply again that events are processed in the order they are received to determine the current truth (state of of the aggregate). My conclusion is that we should fixed the messaging order problem before posting events to the event store.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.