1

We are working for a client to redesign an existing system which basicaly deals with a lot of files.

The files(more than 5 million) are currently stored on the servers filesystem.The client wants the new system to store the file in S3. The files also have metadata associated(name,authors name,price ,description etc.).

The search functionality is also to be redesigned.The following are the basic requirements

  1. Full text search should be available on file descriptions.
  2. Filtering should be possible on other attributes of files.

Also , based on the file description, the system should also be able to give recommendation for similar files.

I do not have experience with creating such solution before,so asking for help and suggestion. I was thinking on the lines of following solutions:

  1. Store the file meta data in MongoDB ,and use the search functionality (http://www.mongodb.org/display/DOCS/Full+Text+Search+in+Mongo)
  2. Use Amazon DynamoDB.It provides api to scan/query the dataset.
  3. Use Lucene/Solr(I havent worked with these yet,I still need to look deeper)

There was this project that I found,that is very similar to what I require http://www.thriftdb.com - On the home page it says its a datastore with search builtin.

Please let me know if this question should be a community wiki.

Thanks in advance.

3 Answers 3

6

You're in luck, announced today:

http://aws.amazon.com/about-aws/whats-new/2012/04/11/aws-announces-cloudsearch/

1

About searching files and filtering by attributes, the best would be Sphinx Search Engine which is used in filestube (google was using it also years ago). I dont know if it will work on amazon servers.

2
  • Rafał Łużyński do you know if Sphinx works with nosql databases
    – gt5050
    Mar 24, 2012 at 11:59
  • sphinxsearch.com/about as you can see, they have support for nosql in xml. Non-SQL storage indexing. Data can also be streamed to batch indexer in a simple XML format called XMLpipe, or inserted directly into an incremental RT index Apr 7, 2012 at 9:39
0

Amazon has a custom AMI for Lucene/Solr and we have been happily using it in our projects. Lucene has a powerful indexing capability and executes at exceptional speeds. I would strongly recommend using Apache Lucene/Solr for all your search needs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.