3

I'm designing a server daemon for a project that has to take a large number of simultaneous requests and process them asynchronously. I'm aware of the sheer scale of such a project, but I'm serious about it and am trying to make a clear design and plan before going further.

Here's a list of my goals:

  • Scalability - Must be able to parallelize the architecture onto multiple processors or even multiple servers.
  • Ability to cope with a huge number of parallel connections.
  • Must not cause blocking problems if a single request takes a long time to process.
  • Request to response turnaround time must be minimal.
  • Built around the .NET framework (will be writing this in C#)

My proposed architecture and flow is rather complicated, so here's a chart of my initial design:

Architecture Flow Chart

(and here it is on tinypic in case it resizes badly)

The idea is that requests come in via the network (though I've not decided if TCP or UDP would be best yet) and are passed immediately to a high-speed load balancer. The load balancer then selects a request queue (RQ) to place the request, using a weighted random number generator. The weights are derived from the size of each queue. The reason for using a weighted RNG, rather than just placing the requests into the least busy queue, is that it prevents an empty but blocked queue (due to a hung request) from locking up the whole server. If all RQs exceed a certain size, the load balancer drops the request and places a "server too busy" response into the output queue (OPQ) - this part isn't shown in the diagram.

Each queue corresponds to a thread whose affinity is set to one CPU core on the server. These threads are part of the parallel request processor, which consumes requests from each queue. The requests are categorized into one of three types:

  1. Immediate - Immediate requests are, as the name suggests, processed immediately.

  2. Deferrable - Deferrable requests are considered to be low priority. They are processed immediately during low load, or placed into the deferred request queue (DRQ) if load is high. The load balancer fetches these deferred requests from the DRQ, marks them as immediate, then places them back into appropriate RQs.

  3. Timed - Timed requests are placed into the timed request queue (TRQ) along with their target timestamp. These requests are often generated as a result of another request, rather than being explicitly sent in by a client. When the request timestamp is exceeded, the next available request processor thread consumes it and processes it.

When a request is processed, data may be fetched from a key/value pair cache in memory, a key/value pair cache or on disk, or from a dedicated SQL database server. The values cached will be BSON, and the index will be a string. I'm thinking of using Dictionary<T1,T2> to implement this in memory, and a btree (or similar) for the disk cache.

The response is created when processing is complete, and it is placed into the output queue (OPQ). A loop then consumes responses from the OPQ and transmits them back to the client over the network. If the OPQ reaches 80% of its maximum size, one quarter of the request processor threads are halted. If the OPQ reaches 90% of its maximum size, half of the request processor threads are halted. If the OPQ reaches its maximum size, all request processor threads are halted. This will be achieved with a semaphore, which should also prevent individual request processor threads from getting blocked and leaving stale requests.

What I'm looking for are suggestions on a few areas:

  • Are there any major flaws to this architecture that I missed?
  • Is there anything I should consider changing for performance reasons?
  • Would TCP or UDP be more appropriate for requests? It'd be very useful to have the "proof of delivery" that TCP offers, but the lightweight nature of UDP is appealing too.
  • Are there any special considerations I need to think about when dealing with 100k+ simultaneous connections on a Windows server? I know Linux's TCP stack deals well, but I'm not so sure with Windows.
  • Are there any other questions that I should be asking? Have I forgotten to consider anything?

I know this was a lot to read, and is probably quite a lot to ask too, so thank you for your time.

Updated version of the diagram here.

1
  • How did this project go/how is it going? Any blog posts on it? I'm very interested to hear what you have learnt along the way and what conclusions you came to.
    – Tyson
    Apr 25, 2013 at 3:32

3 Answers 3

2

Also you can consider following:

  • Failover. You could design an approach to persist requests whilst possible service crashes so all pending requests will be processed even after a service restart
  • Error Queue. (also known as Dead Letter Channel pattern)
  • Pipes and Filters. By providing such feature you would achieve a great level of flexibility and extensibility of the service
  • Request acknowledgment. In some predefined time interval a client which sent a request to the service waiting for the Ack message with a CorrelationId set to initial RequestId, in this way service can notify clients that specific request is received and placed in inbound queue, if a client does not receive Ack for just sent request - it can resend it or mark as failed.

PS: Also I would suggest great book "Enterprise Integration Patterns"

2
  • That's a good point on failover. Since each instance would run on a separate server I was going to load balance between them (doesn't matter which one a request is sent to) and just have it rebalance if a server goes down. However, I hadn't considered the idea of doing something with the pending requests. Perhaps I should keep a duplicate of them on disk in case the daemon crashes, but consider the requests lost if the whole machine goes down? Any better ideas?
    – Polynomial
    Nov 3, 2011 at 22:18
  • Just to clarify: The required response time would mean that if the actual server went down, the reboot time would be too long for the requests to still be relevant, so I'd have to throw away the on-disk queue anyway.
    – Polynomial
    Nov 3, 2011 at 22:28
1

I don't understand why you need multiple request queues. Seems to me you need only one request queue, with many processors all reading from it. It shouldn't be a problem with any queue system. Having just one queue decouples the input from the processors, allowing better scalability- fire up more processors when needed, nobody else needs to care about it.

As for TCP vs. UDP - what kind of performance are you looking for? And wouldn't it be better to use some existing communications infrastructure such as ZeroMQ to take care of these technicalities for you?

Itay.

6
  • The idea of having multiple request queues stems from an idea I had about specializing certain queues to prefer one subset of request types. That should allow me to tweak my code to make processing certain message types go a bit faster. On the TCP/UDP/something else side of things, I'm not really sure what I need. I'd like to be directly interfacing with the network protocol, but that means I'm limited to whatever .NET supports (which is pretty much just TCP and UDP).
    – Polynomial
    Nov 3, 2011 at 22:23
  • I wouldn't go there. If your queue supported priorities (some queue systems do, I don't recall if MSMQ does), you're home free. In general I think you're trying to reinvent a lot of stuff that has already been invented - and open sourced. I would really look into several existing queuing systems before I start implementing things over the network. Also take a look at WCF, although I'm not sure how performance-oriented iti s.
    – zmbq
    Nov 3, 2011 at 22:31
  • Perhaps I should keep the multiple queues, but use them simply to store different message priorities. That way I can just move my load balancer to the other side of the queues and simplify a lot of internal stuff in the process.
    – Polynomial
    Nov 3, 2011 at 22:36
  • Well, multiple queues - one per priority - is a very common way of implementing a few priorities on a queueing system that doesn't support priorities. And indeed it provides the decoupling you need for better scalability. Which queues are you using?
    – zmbq
    Nov 3, 2011 at 22:57
  • I'm using Queue<T> from the .NET framework, which doesn't support priorities. I'll use a weighted RNG to load balance the fetches, as I was planning to do before with the inserts. Update: I might end up using ConcurrentQueue<T> instead, if I need explicit thread safety.
    – Polynomial
    Nov 3, 2011 at 23:04
0

If you want this to scale very well, you'll need to make sure that all components are scalable - the processing elements, the input/output pieces, and the queues. If you're intending to do this on the Microsoft stack, I'd seriously recommend looking into Windows Azure, which offers most, if not all, of the key functionality you'll need. One thing you haven't mentioned - will there be a persistent storage layer (e.g. a database)? If so, be prepared to scale that too, or it will become your single point of failure.

3
  • The database was shown in the diagram and mentioned in my question. I also don't really want to use Azure, as I'd much rather have my application do the logic. The reason for this is that I want it to be installable on a range of different hosts (including customer hosts) and have them act as their own instance or as part of a shared instance.
    – Polynomial
    Nov 3, 2011 at 22:14
  • Sorry - the image is blocked from my current location, and I missed the DB reference in the post. As for "my application doing the logic", I don't see how using Azure's scalability features takes away your ability to tweak the logic. Applications installed on customer hosts can use the "common" instances, or use separate accounts to "privatize" their installation. Nov 3, 2011 at 22:23
  • I meant that I'd like certain customers to be able to run their own "farm" of my server without having to purchase or install Azure. I'd also like to stay away from costly software dependancies myself.
    – Polynomial
    Nov 3, 2011 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.