4

I'm tasked with creating an event stream whereby I have an automated poller (set at 10minute intervals) to retrieve all entities that changed within the last 10minutes.

Now, the business logic dictates that we only create new update events when specific fields change in the entity. Since we don't have granularity with regards to what specific field changed (all we know is that something changed), I have to create some kind of differentiator method that performs the following:

  1. Obtains the previous entity state
  2. Compares this previous state with the newest state (i.e. a diff)
  3. Create an update event if at least one of the fields that the business logic has determined to be important
  4. If the update event has been created, replace the previous entity state with the latest one

Given this architectural problem, is there an known pattern or set of patterns, or guidelines in how to build out this type of system?

2 Answers 2

3
+50

A canonical solution to this problem inverses it: instead of detecting the changes after they are made, the system should produce the change events in the first place. Put otherwise, the API should not allow storing new entity versions; instead it should only allow applying the changes. From system design standpoint, working backwards (detecting changes after they are already made) creates a more complex system because it includes more steps.

Most common implementation of this approach is Event Sourcing, which also fits well with CQRS. The two are typically put in place together. Properly implementing Event Sourcing may be a difficult task and it certainly should not be used without a good reason (your use case certainly sounds like one though). Depending on where the changes are produced you may consider using frameworks that facilitate the change tracking (e.g. Redux in UI layer).


I understand that this modification maybe needs to be added to the system that is already designed around storing the entity versions. One possible solution in this case is to crate an isolation level between two worlds: a layer in the system that will transform entity versions to the stream of the changes. For example, keep the existing interface, but replace its implementation by the one based on events; create the events as early as possible the system, instead of mixing the two approaches.

Hope this makes sense.

3
  • Hi, thanks for the answer. Can you please expand further on the isolation layer solution? To provide further context, I'm trying to create events (based on business rules) when polling for created and updated entities from an external modular API which is my only interface into the relational database where the data resides. This itself is an off-the-shelf application, so I can't modify it, the implementation has to take place after I've consumed from the API. What options do I have to keep this simple?
    – ugotchi
    Nov 19, 2019 at 18:19
  • i.e. do I have any other options, apart from having to storing all the entities, in order to create events based on changes, in the case that I need to create an event based on specific changes in an entity (e.g. a status field change from "pending" to "accepted"), when the API only tells me that the entity overall has been updated.
    – ugotchi
    Nov 19, 2019 at 18:24
  • 1
    Oh, my answer concerns the application which stores the data itself. I don't think above easily applies if you work outside of it. A polling option has a big disadvantage - there exists no correct polling frequency: too often will create a lot of load on both apps, too infrequent will miss intermediate changes. If you host this application, maybe you can consider listening to the underlying DB commit log or intercept the changes elsewhere (e.g.: listening to HTTP requests), provided that your license for this application allows that. Disadvantage of this solution: it is very fragile.
    – Tim
    Nov 19, 2019 at 20:37
3

Take a look at the observer pattern and the iterator pattern, and also at the paradigm of reactive programming.

Note that these are rather general approaches addressing the handling of state changes (or events). The implementation depends on the programming language and the operational environment that you are using. You should really look for existing frameworks and libraries such as ReactiveX (cross-platform) or Spring WebFlux (Java).

Here is an interesting discussion about the relationship between the concepts.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.