3

we're planing to deploy a web-application with Amazon OpsWork and I just wanted to check with you, if our architecture might have any design flaws.

We've 4 components:

  1. A load balanacer (Amazon preferably)
  2. Express based on Node.js
  3. MongoDB
  4. ElasticSearch

Here's a communication diagram of our components:

components communication diagram

At the front is a load balancer which distributes http requests to multiple web servers.

The web server is stateless and therefore can be cloned each time the load requires it. All web server instances are equal. Session information is saved in the MongoDB.

In the "backend" we're planing to use the build-in cluster functionalities from MongoDB and ElasticSearch. Therefore each web server instance only connects to a single MongoDB and ElasticSearch master instance. MongoDB and ElasticSearch are then scaling accordingly. Furthermore the the ElasticSearch master speaks to the MongoDB master to retrieve data for building the index.

How we see it, the most challenging task to setup such a system, is to configure OpsWorks with the MongoDB and ElasticSearch cluster.

Many thanks in advance!

4
  • 2
    Is multi region deployment considered as well? Good to think about it as it might be much better to have replicas scaled over regions if your audience is all around the world. But then you will need to make sure that web workers will access nearest MongoDB replica instead of master that can be very far physically. Same with ElasticSearch. Profit from your architecture is high load performance, but not really nice latency from different regions.
    – moka
    Jun 28, 2013 at 10:18
  • Setup could be made in each region though, if the data allows it. Therefor you would loose performance over different regions!
    – Kersten
    Jun 28, 2013 at 10:21
  • At the moment our customers are from Germany and maybe Europe. So we do not consider multi region deployment right now.
    – Zebi
    Jun 28, 2013 at 10:45
  • If you are storing session data, I think store it in cache server might be an option (still can have fallback option to access it from persistent storage). Your design looks similar with what I've done and in production now: masonzhang.com/2013/07/…. I also have the same goal to scale web server easily, and every request is stateless. BTW: I also use node.js Aug 8, 2013 at 8:02

3 Answers 3

3

if our architecture might have any design flaws.

Well, keep in mind that we can't tell much from a generic diagram. But here are some notes:

1) MongoDB isn't as easy to scale as other databases such as DynamoDB, Riak or Cassandra. For example, if you ever exceed the capacity of a single master (no matter how many slaves you have, all writes go to the single master), you'll have to shard. But switching to sharding is very disruptive and very tedious to set up.

If you don't expect to exceed the write capacity of one node, then you'll be fine on MongoDB.

2) What will you do for async tasks such as sending emails, creating long reports, etc?

It's possible to do these things in the request loop, and that's probably a fine way to get started. But as you have more boxes, the chances of failure go up. When a box dies, all the async tasks go away and nobody will know what they were. You also can have problems where one box gets heavily loaded with async tasks (using too much CPU or memory), and the problem will get worse and worse as it gets more tasks and completes them more slowly.

Also, a front-end like ELB will have a 60-second limit, which can cause problems if some of your requests could take longer. (Spin them off into async jobs with polling or something.)

3) ELB doesn't support web sockets. Consider that if you think you might want websockets down the road.

1
  • 1) nearly 100% data in the application is generated by an internal process (it's a shipment tracking service) so we don't expect to exceed write capacity of one node any time soon. 2) Async tasks will be handled by a queuing based system independently of the web interface. We should not run into timeouts of the ELB. 3) This is a very good information. For now we cancelled web sockets because we ran into other problems with it. Thank you for your feedback!
    – Zebi
    Jul 1, 2013 at 9:14
2

There's no such thing as a master in elastic search. You have master copies of shards and replicas of shards but they are basically moved around through your cluster by elastic search. Nodes might be master for one shard and a replica for another. So, you could simply put a load balancer in front of it.

However, you can specialize nodes to be data nodes or routing nodes as explained here: http://www.elasticsearch.org/guide/reference/modules/node/

The routing nodes effectively become load balancers. You could have a few of those (redundancy) and distribute load between those. Alternatively, you could run a dedicated router node on each web server. Basically routing nodes are pretty light and you save a bit of bandwidth/latency since your web server talks to localhost and from there it is all elastic search internal cluster traffic.

1

I'd recommend to replace MongoDB with Amazon Dynamo DB (it has node.js SDK).

2
  • What are the reasons to switch to DynamoDB on amazon? We need mongo, too because we have to support local installtions without relying on cloud computing solutions too.
    – Zebi
    Jul 1, 2013 at 8:27
  • can you explain why you wanted to replace mongo ?
    – nur farazi
    Aug 15, 2015 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.