1

I'm looking at donne martin's design for a web crawler. the crawler service processes a newly crawled url, and then:

  • Adds a job to the Reverse Index Service queue to generate a reverse index
  • Adds a job to the Document Service queue to generate a static title and snippet

what would happen if instead the crawler service would synchronously call these 2 services? I would still be able to horizontally scale all 3 services according to the load on each, right? what came to me as a possible reason is just more complex flow control if one of them fails. are there other more compelling reasons for these async jobs?

2 Answers 2

1
+50

what would happen if instead the crawler service would synchronously call these 2 services?

The first point — then the slowest service will become a bottleneck for the crawler. Synchronous call means that the crawler needs to wait for the request to be processed by the service. In case of queue, crawler will be working faster, processing new links and not waiting for other services. We could assume that the crawler could have its own internal queue tho.

The second point — durability. Maybe it's not that important if one link or several will be lost if any of the services will get down and wouldn't be able to process a request from the crawler. But queues can be durable, saving state on the disk, restoring its work at the point where it's been stopped. Could be very useful if all services will go down at the same time and many links will be lost.

what came to me as a possible reason is just more complex flow control if one of them fails

That approach isn't flexible. Normally you should be able to add as many new services as you want easily to scale workload, without any changes in code. So the “flow control” should not exist as code that needs modification each time you add or remove instances of a service. In real applications that can scale up and down, all such things are done automatically without redeploy of the application.

-1

There are likely more reasons behind this design choice, but one is almost certainly use of Microservices. It is a popular technique, so demonstrating command of it is a good idea for answering design questions and benefits of it are well described on Wikipedia:

  • Modularity: This makes the application easier to understand, develop, test, and become more resilient to architecture erosion.[6] This benefit is often argued in comparison to the complexity of monolithic architectures.[33]
  • Scalability: Since microservices are implemented and deployed independently of each other, i.e. they run within independent processes, they can be monitored and scaled independently.[34]
  • Integration of heterogeneous and legacy systems: microservices is considered as a viable mean for modernizing existing monolithic software application.[35][36] There are experience reports of several companies who have successfully replaced (parts of) their existing software by microservices, or are in the process of doing so.[37] The process for Software modernization of legacy applications is done using an incremental approach.[38]
  • Distributed development: it parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently.[39] It also allows the architecture of an individual service to emerge through continuous refactoring.[40] Microservice-based architectures facilitate continuous integration, continuous delivery and deployment.[41] [42]

All of those apply in this case. Indeed, well-defined API makes the modules separate, reusable, easy to understand. Most likely each of the 3 modules will have very different execution time and CPU/memory requirements, so scaling them separately makes a lot of sense. Some companies like Amazon mentioned on the page might go much further splitting those modules into microservices based on the team number, so this split into 3 services can very well be chosen based on the assumption of having 3 teams, rather than technical constraints.

The page also describes criticism of the technique.

2
  • thanks! but I think there's a mis-communication here: I wasn't asking why we shouldn't separate them into their own micro-service, I completely agree that it's a good idea, due to all the benefits you mentioned. I'm contemplating why should we call them asynchronously and not directly via an API. They would still be micro-services, right?
    – ihadanny
    Apr 10, 2020 at 12:38
  • Not exactly as you wouldn't be able to load-balance them separately which is exactly what I mentioned in the notes. If it is a direct synchronous call, performance would be bound by the slowest operation. Downvoter cares to explain or post a better answer?
    – isp-zax
    Apr 12, 2020 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.