8

I have a file that contains java serialized objects like "Vector". I have stored this file over Hadoop Distributed File System(HDFS). Now I intend to read this file (using method readObject) in one of the map task. I suppose

FileInputStream in = new FileInputStream("hdfs/path/to/file");

wont' work as the file is stored over HDFS. So I thought of using org.apache.hadoop.fs.FileSystem class. But Unfortunately it does not have any method that returns FileInputStream. All it has is a method that returns FSDataInputStream but I want a inputstream that can read serialized java objects like vector from a file rather than just primitive data types that FSDataInputStream would do.

Please help!

2 Answers 2

7

FileInputStream doesn't give you facitily to read serialized objects directly. You need to wrap it into ObjectInputStream. You can do the same with FSDataInputStream, just wrap it into ObjectInputStream and then you can read your objects from it.

In other words, if you have fileSystem of type org.apache.hadoop.fs.FileSystem, just use:

ObjectInputStream in = new ObjectInputStream(fileSystem.open(path));
3
  • 2
    Using CDH5 this results an error. fcont=hfs.FileContext.getFileContext(); f=fcont.open(hfs.Path('/tmp/test.txt')); ois = java.io.ObjectInputStream(f); java.io.StreamCorruptedException: java.io.StreamCorruptedException: invalid stream header: 6C6F6700
    – Andor
    Apr 16, 2014 at 10:06
  • 1
    @Andor: I've been getting the same exception and have been trying to figure out a solution since a couple of days, without any luck. Were you able to find out what was the problem?
    – Aditya
    Apr 20, 2014 at 22:58
  • My Java guru colleagues said that's not possible, so I have looked for an other way, which was the NFS gateway of HDFS. Though I'm convinced, that if the interface is appropriate, this should work also.
    – Andor
    Apr 23, 2014 at 8:03
-1

You need to convert the FSDataInputStream like this (scala code)

val hadoopConf = new org.apache.hadoop.conf.Configuration()
val hdfs = org.apache.hadoop.fs.FileSystem.get(new     java.net.URI("hdfs://nameserv"), hadoopConf)

val in = hdfs.open(new org.apache.hadoop.fs.Path("hdfs://nameserv/somepath/myfile")).asInstanceOf[java.io.InputStream]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.