0

I have asked this question on serverfault, but I am actually looking for answers from MongoDB configuration perspective. I tried to compare the performance of a sharded database server against a sharded and replicated database server.

The sharded configuration consists of 8 shards each running on three different machines thereby constituting a total of 24 shards. All 8 of these shards run in the same partition on each machine.

The sharded and replicated version is 8 shards again just like plain sharding, and all 8 mongods run on the same partition in each machine. But apart from this, each of these three machine now run additional 16 threads on another partition which serve as the secondary for the 8 mongods running on other machines. This is the way I prepared a sharded and replicated configuration with data chunks having replication factor of 3.

Important point to note is that once the data has been loaded, it is not modified. So after primary and secondaries have synchronized then it doesn't matter which one i read from.

To run the queries, I use an entirely different machine (let's call it config) which runs mongos and this machine's only purpose is to receive queries and run them on the cluster.

Contrary to my expectations, plain sharding of 8 threads on each machine (total = 3 * 8 = 24) is performing better for queries than the sharded + replicated configuration.

I have a script written to perform the query. So in order to time the scripts, I use time ./testScript and see the result. I tried changing the reading preference for replicated cluster by logging to mongo of config and run db.getMongo().setReadPref('secondary') and then exit the shell and run the queries like time ./testScript.

The questions are:

  1. Where am i going wrong in the replication? Why is it slower than its plain sharding version?
  2. Does the db.getMongo().ReadPref('secondary') persist when i leave the shell and try to perform the query?

All the four machines are running Linux and i have already increased the ulimit -n to 2048 from initial value of 1024 to allow more connections. The collections are properly distributed and all the mongods have equal number of chunks. Goes without saying that indices in both configurations are the same.

Hardware Specifications - Architecture : AMD64 - RAM: 64 GB - No. of cores : 32

5
  • Please clarify: how can 8 shards be running on three different machines each as opposed to replicated shards with 3 data bearing nodes each? Aug 19, 2014 at 18:30
  • 8 shards per machine. So total 24 shards. Aug 19, 2014 at 19:37
  • What is the sense behind that? One mongod instance is capable of eating up 64GB and the complete IO bandwidth, both mass storage (data center grade SSDs) and dual bonded network interfaces? And more. Aug 19, 2014 at 22:18
  • The entire database <20GB and the queries' (which use aggregation) result would never ever reach 1GB. So it makes sense to let multiple mongods run on one machine. Doesn't it? Aug 19, 2014 at 22:26
  • No since you would divide the resources at your disposal, reduced by the overhead each mongod and the IO scheduling imposes. See my answer for details. Aug 19, 2014 at 22:47

1 Answer 1

1

Doing replications on the same machine, probably even on the same physical disk leads to the situation that the same data is written multiple times on the same machine. Disk latency and IO scheduling issues can arise here. It is only natural that writing data to a disk 3 times is slower than doing it once. It may well be that a new insert has to wait because the last insert still is written on the same disk or server (which eats up CPU cycles) by the slave mongod.It does not make sense at all to do sth like this. Setting up replica sets is meant to enhance availability of the data in case a complete server fails. For everything else, there is RAID.

Plus, there is another factor why standalone shards are faster with your current setup: In a sharded collection, each document is only written to one specific shard. So this is why the standalone setup is faster.

One note about your setup: running multiple mongods on a single machine does not make sense at all. I have seen single mongod processes eating up 64GB of RAM and the whole IO bandwidth for both dual bonded network interfaces and enterprise grade SSDs. Those mongod processes do have a considerable overhead, too. So in general, there should only one data bearing mongod process per host.

What you should ask yourself is what you want to achieve: high parallelism or high availability. With a three node setup, sharding doesn't really make sense. I'd set up a replica set on these nodes with a RF of 3, using single mongod processes. This way, you get the maximum performance out of the nodes and you have high availability, the possibility to do rolling maintenance and such. In case you need to scale out, you need to add another three servers in order to maintain the RF of 3 and achieve parallelism (plus the config servers, of course).

Edit As for your second question: No, write concern is either set on connection level (the current connection, that is) or on query level. And the current connection is what is returned by db.getMongo().

8
  • Writing happens only and only once during data loading. The queries aren't writing anything. They are only reading as I have mentioned in the question. Also the replication is not on the same machine. The replication of the data of one machine is being done on the other two. So the 8 primary mongods on one host are secondaries on the other, entirely different, machines. I saw the memory usage of the processors on the servers while running the queries and none of the cores are beyond 60%. Aug 19, 2014 at 23:04
  • Thanks for the second question's answer though. But can you please link the documentation to it? Aug 19, 2014 at 23:08
  • That wasn't clear. "But apart from this, each of these three machine now run additional 16 threads on another partition which serve as the secondary". (I assume you are talking of mongod instances, as each instance can have many more threads tun 16). I stand to my point. Put a single mongod replset member on each of the machines, and you will be faster. The scheduling between the different processes still needs some time. Also, to nail this problem down, please provide the test code, how many times you have run it and the standard deviance. Aug 19, 2014 at 23:13
  • The default read preference is "primary" and it has to be changed explicitly, so this behavior is implied. Aug 19, 2014 at 23:21
  • To explain my comment, let's say I have machines A, B and C. All of them run 8 mongods each. Now a particular mongod on A would have 1 mongod running on B and 1 on C thus constituting 3 members of one replica set. This happens with all 8 mongods on A. Further, the 8 mongods of B and C also do the same thing. Aug 19, 2014 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.