0

complete MongoDB/database noob here so any tip would be appreciated. I scraped data using scrapy straight into a locally-hosted MongoDB server. I would like to compare the "price" data from one collection to the "price7" data in the other collection. The names field are the same across collections. What would be the best way of doing this?

Sloppy screenshot of the data here: enter image description here

1
  • MongoDB is for documents and isn't meant for queries across collections. Why not store them all in one collection?
    – taco
    Commented Apr 10, 2015 at 0:22

1 Answer 1

3

Unfortunately you can't compare directly between 2 collections in mongo without peppering in some fancy javascript.

Here's an example of how you could accomplish that, https://stackoverflow.com/a/9240952/4760274

Since you're already using scrapy, and seemingly not comfortable with crazy mongodb internals, its easy enough to whip up a python script to do the evaluation

import pymongo
conn = pymongo.Connection('localhost', 27017)
db = conn['databasename']

for item in db.collection1.find():
    _id = item['_id']
    item2 = db.collection2.find({'_id':_id})
    print "{}: {}, {}: {}, diff: {}, a>b?:{}".format(
        item['name'], item['price'], item1['name'], 
        item1['price'], item['price'] - item1['price'],
        item['price'] > item1['price'])

finally, you can modify your scrapy modules to just insert both into the same collection, tweak field names to identify distinct values from different sources and just allow mongo to coalesce it, in a single collection you can do a simpler query to compare prices

db.unified_collection.find({$where: "this.price1 > this.price2"})  

(This doesn't allow you however to get the difference in a single query like a SQL query could)

edit: port must be int :)

update: It'll also be wise to note the comparison above^ is assuming you're setting ID and not using mongo's generated _id (which it appears you may be using), those are randomly generated so there's no relation between 2 identical entities. In order to properly match them with either approach mentioned above (script, or having separate crawlers use the same data model), you'll need something to qualify uniqueness on in order to do any sane comparison between the 2 sources.
From the image of your data, it looks like the safest bet is on the "name" field, but if there's even a slight amount of variance you're going to get insufficient results. Whether iterating through 2 collections and comparing or coalescing you'll need some rule to clean and compare to get a good match (regex, soundex, other string manipulation tricks), If done in the crawler/model side you'd need to make your unified collection unique on a field, and a hash of your cleaned names would make a good candidate value (as to keep the original values in tact).

Another option is SQL, useful for analytic tests you look to be doing, but again you face the problem of how to relate (better, how to manipulate to relate), and also the holdups of schema changes/migrations (and lack of ability to store extra misc. data where available).

1
  • Fantastic! I really appreciate your help! Going to give this a shot now.
    – 5Y3WS
    Commented Apr 10, 2015 at 0:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.