5

The Cloudera documentation says that Hadoop does not support on disk encryption. Would it be possible to use hardware encrypted hard drives with Hadoop?

4 Answers 4

7

eCryptfs can be used to do per-file encryption on each individual Hadoop node. It's rather tedious to setup, but it certainly can be done.

Gazzang offers a turnkey commercial solution built on top of eCryptfs to secure "big data" through encryption, and partners with several of the Hadoop and NoSQL vendors.

Gazzang's cloud-based Encryption Platform for Big Data helps organizations transparently encrypt data stored in the cloud or on premises, using advanced key management and process-based access control lists, and helping meet security and compliance requirements.

Full disclosure: I am one of authors and current maintainers of eCryptfs. I am also Gazzang's Chief Architect and a lead developer.

5

If you have mounted a file system on the drive then Hadoop can use the drive. HDFS stores its data in the normal OS file system. Hadoop will not know whether the drive is encrypted or not and it will not care.

3

Hadoop doesn't directly support encryption, though a compression codec can be used used for encryption/decryption. Here are more details about encryption and HDFS.

Regarding h/w based encryption, I think Hadoop should be able to work on it. As Spike mentioned, HDFS is like any other Java application and stores it's data in the normal OS file systems. FYI, MapR uses Direct I/O for better HDFS performance.

0

See also Intel's Rhino. Not open source yet...

https://github.com/intel-hadoop/project-rhino/ https://hadoop.intel.com/pdfs/IntelEncryptionforHadoopSolutionBrief.pdf

1
  • I talked to Intel at StrataConf CA this year and I'm pretty sure they aren't planning to open source it (they are selling it.) Dec 24, 2013 at 17:03

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.