I'm researching strategies for storing what could potentially be a large amount of data and I want to find the best storage technologies. I'm open to any suggestions MySQL, NoSQL, flat files, anything.
What I'm doing:
- I will distribute all 4-byte integer numbers amongst a set of users.
- Numbers will be 0 through 4,294,967,295, unsigned 4-byte integer range.
- Users can have many thousands of these numbers.
- Expecting thousands of users but small possibility of millions.
- Numbers will be distributed over a period of time, not all at once
- Using AWS, a few servers and EBS volumes
My main concern is storage space. I need to do this on the cheap and large volumes cost a bit of money on AWS.
I've done a little research on representing the number allocation mathematically but I found too many issues with that.
Requirements
- Somewhat efficient retrieval for analytics and realtime data display. Doesn't have to be lightning fast but reasonable.
- I'll need to do lookups in both directions user id -> number and number -> user id
- As little storage space as possible.
- Reasonable (less than 8 GB) memory usage.
- Must be accurate, lost numbers and mis-assigned numbers are not an option.
Here's what I've found so far:
The lower bound for storing all 4-byte numbers associated with 2-byte user IDs is ((4+2) * 2^32 / 1024 / 1024 / 1024) = 24 GB.
Cassandra is a key/value pair database. Based on this http://www.datastax.com/docs/0.8/cluster_architecture/cluster_planning I calculate that if I used every 4-byte number as keys and 2-byte user IDs as values the I'd need approx 260GB of storage (without replication).
Redis is in-memory. I think this would eliminate it as a possibility since that much memory would be crazy expensive on AWS.
I'm currently looking for similar info on MySQL and Mongo.
Here's my question. Is there any references I could use for determining the best solution or are there alternate solutions to this that I'm not thinking of.
Thanks all.
UPDATE - I added an additional requirement, I need to lookup numbers based on user id and user id based on number. Also, Redis is in-memory so a direct implementation would make Redis more expensive than a disk based solution.
:-)