19

I'm writing a Java server which uses plain sockets to accept connections from clients. I'm using the fairly simple model where each connection has its own thread reading from it in blocking mode. Pseudo code:

handshake();

while(!closed) {
  length = readHeader(); // this usually blocks a few seconds
  readMessage(length);
}

cleanup();

(Threads are created from an Executors.newCachedThreadPool() so there shouldn't be any significant overhead in starting them)

I know this is a bit of a naive setup and it wouldn't really scale well to many connections if the threads were dedicated OS threads. However, I've heard that multiple threads in Java can share one hardware thread. Is that true?

Knowing that I'll be using the Hotspot VM on Linux, on a server with 8 cores and 12GB of RAM, do you think this setup will work well for thousands of connections? If not, what are the alternatives?

0

10 Answers 10

7

This will scale well for up to hundreds of connections, not to thousands. One issue is that a Java thread takes quite a bit of stack as well (e.g. 256K), and the OS will have problems scheduling all your threads.

Look at Java NIO or framworks that will help you get started doing complex stuff more easily (e.g. Apache Mina)

5
  • 1
    Each thread will take some stack, but if I transform it into a non-blocking model each connection will need more data on the heap, such as "in what phase of reading a message are we now", which is currently simply determined by the instruction pointer (correct word?) of a thread. The scheduling might be an issue though. Oct 5, 2010 at 20:08
  • 1
    @Bart: The extra space per connection is nowhere near as large as a stack; going to NIO will improve scalability. The cost is that most folk find it harder to grok what's going on (and Java doesn't have coroutines, which can be used to balance the concerns). Oct 5, 2010 at 21:29
  • As ivy said, the main consideration to scaling to thousands is really the stack they will end up consuming. On the other hand, I am not fully convinced that scheduling will be a major factor. It will be some factor, but perhaps not as great as one might perceive. If your dominant activities are I/O, then your threads will promptly yield the CPUs in most cases, and they will get paged in only if data is received. If you think about it, that picture is not too different even if you use NIO. It's just that more number of threads are involved in asking for roughly the same number of CPU cycles.
    – sjlee
    Oct 5, 2010 at 23:35
  • You can configure the stack smaller. Also, won't parts of the stack not in use be swapped out or never even allocated to physical pages? It seems like an idle thread will have a working set of just one memory page. Probably that is still a lot more than it needs to complete the request (a handful of pointers) but pretty manageable. Mar 24, 2012 at 12:54
  • You can configure the stack smaller (e.g. 4 times smaller) but this will only help a bit. Asynchronous frameworks are the key to solve this problem. I'm quite happy with using Netty in the Java world.
    – ivy
    Mar 9, 2013 at 16:31
5

It is possible this will scale to thousands of clients. But how many thousands is the next question.

A common alternative is to use Selectors and non-blocking I/O found in the java.nio package.

Eventually you get into the question of whether it's useful to set up your server in a clustered configuration, balancing the load over multiple physical machines.

3

To have a good perfomance when handling many sockets you usually use a select approach that is how Unix API handles single-threaded multi-socket applications that need many resources.

This can be done through the java.nio package that has a Selector class which basically is able to go through all the opened sockets and notify you when new data is available.

You register all the opened streams inside a single Selector and then you can handle all of them from just one thread.

You can get additional infos with a tutorial here

4
  • By accident I stumbled upon a SO question about the method InputStream.available(), which I'd forgotten about. Could you tell me the advantage of using a Selector as opposed to having a thread handle multiple connections, using available() to prevent blocking? Oct 5, 2010 at 20:38
  • Because you avoid to reimplement something that goes over the various Sockets programmatically since you will be using something specifically tailored for this purpose :) The main advantage would be saving time for debug and implementation..
    – Jack
    Oct 5, 2010 at 20:43
  • Hmm, but the implementation of that, at least in my case, would be extremely simple. My protocol is 1. read integer that specifies message length. 2. read that number of bytes in one go Oct 5, 2010 at 20:47
  • In that case it's a matter of preference. Of course avoiding to use nio when you don't know it neither you need it would be overkill but if you plan to have many sockets why don't give channels and asynchronous IO a chance? Mind that you will have to care about interrupting the thread to avoid having it going in an infinite loop when data isn't ready..
    – Jack
    Oct 5, 2010 at 20:51
3

The JVM for Linux is using one to one thread mapping. This means that every Java thread is mapped to one native OS thread.

So creating a thousand of threads or more is not a good idea because it will impact your performance (context switching, cache flushes / misses, synchronization latency etc). It also doesn't make any sense if you have less than a thousand of CPUs.

The only adequate solution for the serving many clients in parallel is to use asynchronous I/O. Please see this answer on Java NIO for details.

See also:

3
  • 1
    Thanks, this is a helpful answer, but I don't agree with "It also doesn't make any sense if you have less than a thousand of CPUs.". It makes sense for keeping the current state of the connection ("are we now reading a header or data", "how much of the message did we receive yet", etc.) on the stack rather than the heap, which is faster (right?) and makes it easier to program. Oct 5, 2010 at 20:11
  • It only makes you feel like it is easier to program (both approaches are relatively easy IMO). Using stack doesn't make it faster de facto, it all depends on what you are doing. In any case, trade-off having 1K threads will be greater than maintaining a list of sessions with pre-allocated state.
    – user405725
    Oct 5, 2010 at 20:18
  • @Bart: No, it won't be efficient, even tho you will be using stack. Async I/O is a lot more efficient than creating thousand of threads and checking anything using each thread`s stack.
    – user405725
    Jul 26, 2011 at 13:52
2

Try Netty.

the "one thread per request" model is the way most Java app servers are written. Your implementation can scale as well as they do.

1
  • 1
    Just because "most" people do something does that mean it is right or the best way. The only way to answer this for sure would be to write your code using both methods and test to see which one performs better in terms of CPU and memory usage. Due to differences between windows and linux you may need to consider testing in both environments to know for sure. That said I believe the one thread per request is not scalable.
    – Jay
    Jul 26, 2011 at 3:33
1

Threads aren't as expensive as they used to be, so an "ordinary" IO implementation can be ok to a point. However if you are looking at scaling to thousands or beyond it is probably worth investigating something more sophisticated.

The java.nio package solves this by providing socket multiplexing/non-blocking IO which allows you bind several connections to one Selector. However this solution is much harder to get right than the simple blocking approach because of the multithreading and non-blocking aspect.

If you wish to pursue something beyond the simple IO then I would suggest looking at one of the good quality network abstraction libraries out there. From personal experience I can recommend Netty which does most of the fiddly NIO handling for you. It does however have a bit of a learning curve but once you get used to the event based approach it is very powerful.

1

If you have any interest in leveraging deployment and management of an existing container, you might look at making a new protocol handler inside of Tomcat. See this answer to a related question.

UPDATE: This post from Matthew Schmidt claims the NIO-based connector (written by Filip Hanik) in Tomcat 6 achieved 16,000 concurrent connections.

If you want to write your own connector, take a look at MINA to help with NIO abstractions. MINA also has management features which may eliminate need for another container (should you be concerned about deployment of many units and their operation, etc.)

1
  • Interesting, but my application is not currently using a container. I'm also interesting in working with NIO myself (if it turns out to be necessary) for education purposes. Oct 5, 2010 at 20:41
1

I'd suggest it depends more on exactly what else the server is doing when it processes the messages. If it's relatively lightweight then your machines spec should EASILY cope with merely handling the connections of thousands of such processes. Tens of thousands is another question perhaps, but you only need two machines on the same network to actually empirically test it and get a definite answer.

0

I think a better approach is to not handle threads yourself. Create a pool (ThreadExecutor or some other stuff) and simple dispatch work to your pool.

Of course, I think asynchronous I/O will make it better and faster, but will help you with socket and networking problems. Only. When your threads block because of I/O, the JVM will put it to sleep and change for another thread until the blocking I/O return. But this will block only the thread. Your processor will continue to run and start to proccess other thread. So, minus the time to create a thread, the way you use I/O not affect not much your model. If you do not create threads (using a pool) your problem is solved.

2
  • I do use an Executor, but since each thread basically does while(!closed) read(); that doesn't actually decrease the number of threads, it just diminishes the overhead of creating them. Oct 5, 2010 at 20:16
  • Its exactly my point. When you call read() your thread will block until it has something to do. Doesnt matter how many threads you have, but how many RUNNABLE threads you have, and the blocked threads dont count. They will not compete for processor time. So, in the end you will have only the threads that have things to do competing by processor time. Oct 6, 2010 at 13:02
0

Why roll your own? You could use a servlet container with servlets, a message queue, or ZeroMQ.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.