2

I have the following code:

client = MongoClient()
data_base = client.hkpr_restore
agents_collection = data_base.agents
agent_ids = agents_collection.find({},{"_id":1})

This gives me a result of:

{u'_id': ObjectId('553020a8bf2e4e7a438b46d9')}
{u'_id': ObjectId('553020a8bf2e4e7a438b46da')}
{u'_id': ObjectId('553020a8bf2e4e7a438b46db')}

How do I just get at the ObjectId's so I can then use each ID to search another collection?

1
  • I'm not getting you. how exactly you were output looking for. Commented Apr 21, 2015 at 11:43

5 Answers 5

6

Use distinct

In [27]: agent_ids = agents_collection.distinct('_id')

In [28]: agent_ids
Out[28]: 
[ObjectId('553662940acf450bef638e6d'),
 ObjectId('553662940acf450bef638e6e'),
 ObjectId('553662940acf450bef638e6f')]

In [29]: agent_id2 = [str(id) for id in agents_collection.distinct('_id')]

In [30]: agent_id2
Out[30]: 
['553662940acf450bef638e6d',
 '553662940acf450bef638e6e',
 '553662940acf450bef638e6f']
1
  • 1
    No good, pymongo.errors.OperationFailure: distinct too big, 16mb cap
    – radtek
    Commented Sep 23, 2020 at 12:01
2

I solved the problem by following this answer. Adding hint to the find syntax then simply iterate through the cursor returned.

db.c.find({},{_id:1}).hint(_id:1);

I am guessing without the hint the cursor would get the whole documentation back when iterated, causing the iteration to be extremely slow. With hint, the cursor would only return ObjectId back and the iteration would finish very quickly.

The background is I am working on an ETL job that require sync one mongo collection to another while modify the data by some criteria. The total number of Object id is around 100000000.

I tried using distinct but got the following error:

Error in : distinct too big, 16mb cap

I tried using aggregation and did $group as answered from other similar question. Only to hit some memory consumption error.

2
  • 1
    You may want to rearrange this answer. At first glance, it looks like you are asking another question.
    – Tom Zych
    Commented Sep 2, 2019 at 8:48
  • @TomZych Thanks for the advice. I have rearranged the answer. Commented Sep 2, 2019 at 9:55
1

Try creating a list comprehension with just the _ids as follows:

>>> client = MongoClient()
>>> data_base = client.hkpr_restore
>>> agents_collection = data_base.agents
>>> result = agents_collection.find({},{"_id":1})
>>> agent_ids = [x["_id"] for x in result]
>>> 
>>> print agent_ids
[ ObjectId('553020a8bf2e4e7a438b46d9'),  ObjectId('553020a8bf2e4e7a438b46da'),  ObjectId('553020a8bf2e4e7a438b46db')]
>>>
0
0

I would like to add something which is more general than querying for all _id.

import bson
[...]
results = agents_collection.find({}})
objects = [v for result in results for k,v in result.items()
          if isinstance(v,bson.objectid.ObjectId)]

Context: saving objects in gridfs creates ObjectIds, to retrieve all of them for further querying, this function helped me out.

0

Although I wasn't searching for the _id, I was extracting another field. I found this method was fast (assuming you have an index on the field):

list_of_strings = {x.get("MY_FIELD") for x in db.col.find({},{"_id": 0, "MY_FIELD": 1}).hint("MY_FIELDIdx")}

Where MY_FIELDIdx is the name of the index for the field I'm trying to extract.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.