0

It's taking a long time to save 500+ Facebook friends in MongoDB and I think I'm doing it so wrong. I'll paste how I'm doing the insertion:

models.js:

Friends = new Meteor.Collection('friends');
Friend = {
set : function(owner, friend) {
    var user_id = get_user_by_uid(friend['uid']);
    return Friends.update({uid: friend['uid'], owner: owner}, {$set:{
        name : friend['name'],
        pic_square : 'https://graph.facebook.com/'+friend['uid']+'/picture?width=150&height=150',
        pic_cover : friend['pic_cover'],
        uid : friend['uid'],
        likes_count : friend['likes_count'],
        friend_count : friend['friend_count'],
        wall_count : friend['wall_count'],
        age : get_age(friend['birthday_date']),
        mutual_friend_count : friend['mutual_friend_count'],
        owner : owner,
        user_id : user_id ? user_id['_id'] : undefined
    }}, {upsert: true});
}
}

server.js:

// First get facebook list of friends   
friends = friends['data']['data'];

_.each(friends, function(friend){
    Friend.set(user_id, friend);
});

The loads go high with 2+ users and it takes ages to insert on the database. What should I change here ?

2
  • Why are you using Friends.update instead of Friends.insert?
    – Rahul
    Jan 22, 2013 at 3:06
  • Try inserting the data and see if the speed improves. Also note that upsert isn't supported in minimongo yet (the client-side implementation of mongodb in Meteor)
    – Rahul
    Jan 22, 2013 at 3:19

1 Answer 1

4

The performance is bad for two reasons I think.

First, you are experiencing minimongo performance, not mongodb performance, on the client. minimongo can't index, so upsert is expensive—it is O(n^2) expensive on database size. Simply add a if (Meteor.isSimulation) return; statement into your model just before the database updating.

Take a look at some sample code to see how to organize your code a bit, because Friend.set(user_id, friend) should be occurring in a method call, conventionally, defined in your model.js. Then, it should escape if it is detected as a client simulating the call, as opposed to the server executing it.

Second, you are using uid and owner like a key without making them a key. On your server startup code, add Friends._ensureIndex({uid:1, owner:1}).

If none of this works, then your queries to Facebook might be rate-limited in some way.

Check out https://stackoverflow.com/a/8805436/1757994 for some discussion about the error message you'd receive if you are being rate limited.

They almost certainly do not want you copying the graph the way you are doing. You might want to consider not copying the graph at all and only getting the data on a use-basis, because it very rapidly becomes stale anyway.

4
  • Friend.set is occuring inside a method call. I'll try the other points. BRB Jan 22, 2013 at 19:08
  • Did: set : function(owner, friend) {var user_id = get_user_by_uid(friend['uid']); if (Meteor.isSimulation) return; [...] and put the _ensureIndex in Meteor.startup of server. Is it required to erase the collection/db or something ? Jan 22, 2013 at 19:14
  • No, I don't think you have to erase anything. If you don't see a performance improvement, and you are indeed querying Facebook for each line, then you are being rate limited. See: stackoverflow.com/a/8805436/1757994 Jan 23, 2013 at 17:14
  • I don't see any huge improvements. I'll need to change my algorithm and don't do upserts. Thanks for the help. Jan 24, 2013 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.