0

We have an application deployed across AWS with using EC2, EBS services. The infrastructure dropped by layers (independent instances):

  • application (with load balancer)

  • database (master-slave standard schema)

  • media server (streaming)

  • background processing (redis, delayed_job)

Application and Database instance use number of EBS block storage devices (root, data), which help us to attach/detach them and do EBS snapshots to S3. It's pretty default way how AWS works.

But EBS should be located in a specific zone and can be attached to one instance only in the same time.

Media server is one of bottlenecks, so we'd like to scale them with master/slave schema. So for the media server storage we'd like to try distributed file systems can be attached to multiple servers. What do you advice?

0

2 Answers 2

1
+50

If you're not Facebook or Amazon, then you have no real reason to use something as elaborate as Hadoop or Cassandra. When you reach that level of growth, you'll be able to afford engineers who can choose/design the perfect solution to your problems.

In the meantime, I would strongly recommend GlusterFS for distributed storage. It's extremely easy to install, configure and get up and running. Also, if you're currently streaming files from local storage, you'll appreciate that GlusterFS also acts as local storage while remaining accessible by multiple servers. In other words, no changes to your application are required.

I can't tell you the exact configuration options for your specific application, but there are many available such as distributed, replicated, striped data. You can also play with cache settings to avoid hitting disks on every request, etc.

One thing to note, since GlusterFS is a layer above the other storage layers (particularly with Amazon), you might not get impressive disk performance. Actually it might be much worst than what you have now, for the sake of scalability... basically you could be better-off designing your application to serve streaming media from a CDN who already has the correct infrastructure for your type of application. It's something to think about.

0

HBase/Hadoop

Cassandra

MogileFS

Good same question (if I understand correctly):
Lustre, Gluster or MogileFS?? for video storage, encoding and streaming

There are many distributed file systems, just find the one you need.
The above are just part which I personally know (haven't tested them).

6
  • Worth mentioning is that Hadoop and Cassandra are/were used by Facebook, Amazon and other big ones so..
    – Poni
    Sep 9, 2011 at 11:26
  • And there's also "Project Voldemort" used by LinkedIn
    – Poni
    Sep 9, 2011 at 11:29
  • voldermort/cassandra/hbase are databases, which is not what I seek. Do you have an experience with using DFS across cloud infrastructure? I can choose any DFS I found and integrate with our system. Again, It would be great to hear about work solution, best practices.
    – Anatoly
    Sep 9, 2011 at 11:48
  • another way for us is Dedicated Servers with common practices
    – Anatoly
    Sep 9, 2011 at 11:51
  • You say "for the media server storage we'd like to try distributed file systems can be attached to multiple servers"...... the above are indeed databases, in other words are distributed file systems as you request. They are not as MySQL though. Not sure then what you're looking for.
    – Poni
    Sep 9, 2011 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.