6

I have a MongoDB sharded setup with 3 shards: shard0000, shard0001 and shard0002. The machine that runs shard0002 is down now, which causes all my queries to fail. I'd like to temporarily remove shard0002 from my setup and keep working with the first two shards. That should be doable assuming I only use unsharded collections that reside in the first two shards, right?

What I tried first is: db.runCommand({removeshard: 'IP:PORT'}) which obviously doesn't help, because it just puts the shard in draining mode, which will never end (since it's down). Then I tried connected to my config server and did db.shards.remove({_id: 'shard0002'}) on the config DB then restart mongos so it reloads the config. Now whenever I try to do anything I get "can't find shard for: shard0002".

Is there any way to just let Mongo know that I don't care about that shard for now, then re-enable it later when it becomes available.

1
  • You say all queries fail? Does this include queries that can work with just a single shard that is still alive?
    – Thilo
    Oct 5, 2010 at 6:26

3 Answers 3

12

I had a different problem, and I manually removed the shard with:

use config
db.shards.remove({"_id":"shard0002"});
1
  • Does not work (in version 4.2). If the client tries to access any chunk from removed shard, then you still get an error. Nov 21, 2019 at 13:36
2

Manually modify the the shard entry in the config db, then removeshard

2
  • 4
    UNHELPFUL. Need detail or link to how to do this. What collection? where? how to modify it? ???? Aug 19, 2014 at 14:31
  • 1
    @KevinJ.Rice, in the config db, there is a collection called shards. You will want to remove the shard entry from there manually.
    – brismuth
    Sep 9, 2014 at 15:33
1

I tried several options to do this in version 4.2.

At the end I ended to these commands to be executed on Config Server:

use config

db.databases.updateMany( {primary: "shard0002"}, {$set: {primary: "shard0000"} })
db.shards.deleteOne({_id : "shard0002" })
db.chunks.updateMany( {shard : "shard0002"}, {$set: {shard: "shard0000"} })

while ( db.chunks.updateMany( {"history.shard" : "shard0002"}, 
      {$set: {"history.$.shard": "shard0000"} }).modifiedCount > 0 ) { print("Updated") }

It works to a certain extent, i.e. CRUD operations are working. However, when you run getShardDistribution() then you get an error Collection 'db.collection' is not sharded.

Finally I see only one reliable & secure solution:

  • Shut down all mongod and mongos in your sharded cluster
  • Start available shards as standalone service (see Perform Maintenance on Replica Set Members)
  • Take a backup from available shards with mongodump.
  • Drop data folders from all hosts.
  • Build your application newly from scratch. Startup all mongod and mongos
  • Load data into new cluster with mongorestore

Perhaps for large cluster you have to shuffle around a bit like this:

  • Deploy Config servers and mongos server, with one empty shard
    1. Start one old shard as standalone
    2. Take backup from this old Shard
    3. Tear down this old shard
    4. build a fresh empty new shard
    5. add new shard to your new cluster
    6. restore data into the new cluster
    7. backup can be dropped and shard can be reused in new cluster
  • Repeat above for each shard you have in your cluster (the broken shard might be skipped most likely)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.