Questions tagged [sequencefile]

A SequenceFile is a Hadoop binary file containing key/value pairs.

sequencefile
Filter by
Sorted by
Tagged with
2 votes
1 answer
563 views

Is it possible to check if a file on HDFS is a SequenceFile without (mis)using exceptions?

I want to read a specific SequenceFile from HDFS from a client application. I can do this by using the SequenceFile.Reader and it works fine. But is it also possible to check whether a file is a ...
rabejens's user avatar
  • 7,852
0 votes
1 answer
757 views

Fail to write SequenceFile with Pig

I want to store some Pig variable to Hadoop SequenceFile, so as to run external MapReduce jobs. Assume that my data has the (chararray, int) schema: (hello,1) (test,2) (example,3) I wrote this ...
Aslan986's user avatar
  • 10.1k
2 votes
1 answer
356 views

I'm in trouble in K-Means using Mapreduce (modified)

I think my code is not wrong but, it doesn't work correctly. This is K-means clustering using mapreduce. (https://github.com/30stm/K-Means-using-mapreduce/tree/master) Make a dataset using ...
Sungmin Alan Kang's user avatar
0 votes
1 answer
357 views

how to change the hadoop sequence file value to jackson parser?

I have one problem which I really don't know what to do. I have a Hadoop sequence file containing the links of the webpage. Each entry of the Hadoop sequence file, the key will be the URL of one ...
user2970089's user avatar
1 vote
2 answers
2k views

appending to existing sequence file

In my use case, I need a find a way to append key/value pairs to the existing sequence file. How to do it? Any clue would be greatly helpful. I am using hadoop 2x. Also, I came across the below ...
user3721344's user avatar
1 vote
0 answers
90 views

SequenceFiles and Hadoop streaming

I have a use case wherein I use hadoop streaming to run an executable as map process. In the input side, I have large number of sequence files. Each seq file has, says 8 keys and corresponding values ...
user3721344's user avatar
2 votes
4 answers
9k views

How to copy the output of -text HDFS command into another file?

Is there any way we can copy text content of hdfs file into another file system using HDFS command: hadoop fs -text /user/dir1/abc.txt Can I print the output of -text into another file by using -...
dewet's user avatar
  • 31
1 vote
0 answers
62 views

Convert a space separated file (eachy row = vector) to SequenceFile

I created the large text file (4 GB) as followings. 0 1 2 3 2 1 3 6 2 0 6 4 3 0 6 3 0 0 1 6 7 3 9 4 Each row describes a vector, and each column denotes each element of the vector. Each element is ...
Benben's user avatar
  • 1,375
0 votes
1 answer
328 views

Increase number of splits for SequenceFileInputFormat

I'm using SequenceFileInputFormat as my input for my map, where the key is text and the value is text. There are 106 files each of them is between 500 MB to 750 MB. I saw my logs and it says there ...
user3690321's user avatar
0 votes
1 answer
400 views

classcastException when sorting sequenceFile in hadoop?

I am following Hadoop-The definitive guide 3rd edition by Tom White. I have successfully written a sequenceFile into HDFS. I followed the example the author gave in book. but when I try to run the ...
eagertoLearn's user avatar
  • 9,972
2 votes
0 answers
2k views

Reading SequenceFile written by Spark

I have bunch of sequence files that I want to read using Scalding and I am having some troubles. This is my code: class ReadSequenceFileApp(args:Args) extends ConfiguredJob(args) { SequenceFile(...
Rob Schneider's user avatar
3 votes
2 answers
2k views

NegativeArraySizeException when creating a SequenceFile with large (>1GB) BytesWritable value size

I have tried different ways to create a large Hadoop SequenceFile with simply one short(<100bytes) key but one large (>1GB) value (BytesWriteable). The following sample works for out-of-box: ...
user815613's user avatar
1 vote
1 answer
1k views

Avro file type for images?

I try to...figure that case in Hadoop. What is best file format Avro or SequenceFile, in case storing images in HDFS and process them after, with Python? SequenceFile are key-value oriented, so I ...
Cugar's user avatar
  • 79
0 votes
0 answers
80 views

SeqFilesFromDirectory() error on amazon EMR

I am trying to run a simple program on Amazon EMR which converts text files in a directory into sequence files. The program runs fine on my local machine but gives me following error on Amazon EMR. ...
user3376898's user avatar
10 votes
2 answers
13k views

Write and read raw byte arrays in Spark - using Sequence File SequenceFile

How do you write RDD[Array[Byte]] to a file using Apache Spark and read it back again?
samthebest's user avatar
  • 31.2k
0 votes
1 answer
347 views

Read sequencefile in Hadoop 2.0

I am trying to read a sequencefile in hadoop 2.0 but I am unable to achieve it. I am using the below code which works perfectly fine in hadoop 1.0. Please let me know if I am missing something wrt 2.0 ...
Arjun's user avatar
  • 31
0 votes
2 answers
1k views

Exporting sequence file to Oracle by Sqoop

I have been trying to find some documentations about how we can export sequence file to Oracle by using Sqoop. Is that possible? Currently I have my files(in HDFS) in text based format and I am using ...
dreamer's user avatar
  • 1,051
2 votes
1 answer
3k views

How to extract data from Hadoop sequence file?

Hadoop sequence file is really strange. I pack the images into sequence file and can't recovery image. I do some simple test. And I found the size of bytes even not same before and after use sequence ...
hakunami's user avatar
  • 2,401
0 votes
2 answers
402 views

Hive SequenceFile with Java Class; just pass to toString()

I've got a Hadoop SequenceFile where the key is IntWritable and the value is some arbitrary Java class implementing Writable, and with an interesting toString() method. I would love to make a two ...
Joseph Victor's user avatar
0 votes
1 answer
433 views

Error when getting original images from Hadoop sequenceFile

I first pack all my images into Hadoop sequenceFile: FSDataInputStream in = null; in = fs.open(new Path(uri)); //uri is the image location in HDFS byte buffer[] = new byte[in.available()]; in....
hakunami's user avatar
  • 2,401
1 vote
0 answers
437 views

Is there a javascript library for Hadoop sequencefile

I try to build nodejs server which collect binary data from user and storing it to Hadoop sequencefile. As a good tutorial, there's approach using the Hadoop executable. My question: Is there ...
emesday's user avatar
  • 6,158
7 votes
6 answers
30k views

hadoop mapreduce: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

I am trying to write a snappy block compressed sequence file from a map-reduce job. I am using hadoop 2.0.0-cdh4.5.0, and snappy-java 1.0.4.1 Here is my code: package jinvestor.jhouse.mr; import ...
msknapp's user avatar
  • 1,645
0 votes
2 answers
1k views

Mahout : Cannot convert into sequence file

I'm trying to convert some text files into mahout sequence files. So I do mahout seqdirectory -i inputFolder -o outputFolder But I always get this exception java.lang.Exception: java.lang....
IrishDog's user avatar
  • 470
0 votes
2 answers
755 views

If I store all my images in SequenceFile, how I can I design mapper to process a selection of them?

I do have lots of image files and need to store them in HDFS, in order to avoid the Small Files Problem, I am planning to store my image files using Sequence Files. My problem is that I need to ...
zaz's user avatar
  • 25
0 votes
0 answers
616 views

Convert Text File to Sequence File

I am a newbie to Hadoop and Mahout. I wanted to know how to convert a simple text file containing a set of vectors to sequence file. I have tried the MR framework and changed outputFormat to ...
Jayant's user avatar
  • 346
1 vote
1 answer
1k views

How to Convert SequenceFile in Hadoop to Image file? Following code returns error (bImageFromConvert is NULL)

import java.awt.image.BufferedImage; import java.io.ByteArrayInputStream; import java.io.File; import javax.imageio.ImageIO; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs....
Gargi's user avatar
  • 113
2 votes
1 answer
2k views

writing/reading key/value pairs in sequence file format in Hadoop.

I have a mapreduce program whose output is all in text files right now. A sample of the program is below. What I do not understand how to do is output the key/value pairs from the reducer in sequence ...
user2654569's user avatar
1 vote
1 answer
205 views

hadoop sequence file collection

How can a reducer (with a Text key and an Iterable MapWritable value) output all its Map's to a sequence file so as to preserve grouping on its key? For example, suppose the mappers send records to ...
user2250400's user avatar
0 votes
0 answers
79 views

Store Complex data with SequenceFile Hadoop

my question is how to generate a sequenceFile from text to output some format like this: <string, string>,<double, double> the left side is the key, and the right side is the value.
Yan's user avatar
  • 153
2 votes
2 answers
1k views

Is there a simple way to migrate from SequenceFiles to Avro?

I'm currently using hadoop mapreduce jobs with SequenceFiles of writables. The same Writable type are used for serialization also in the non-hadoop related parts of the system. This method is hard to ...
Ophir Yoktan's user avatar
  • 8,309
6 votes
2 answers
1k views

Handling Writables fully qualified name changes in Hadoop SequenceFile

I have a bunch of Hadoop SequenceFiles that have been written with some Writable subclass I wrote. Let's call it FishWritable. This Writable worked out well for a while, until I decided there was ...
Alex A.'s user avatar
  • 2,696
7 votes
1 answer
1k views

Extend SequenceFileInputFormat to include file name+offset

I would like to be able to create a custom InputFormat that reads sequence files, but additionally exposes the file path and offset within that file where the record is located. To take a step back, ...
Joe K's user avatar
  • 18.3k
0 votes
1 answer
187 views

Sequencefiles which map a single key to multiple values

I am trying to do some preprocessing on data that will be fed to LucidWorks Big Data for indexing. LWBD accepts SolrXML in the form of Sequencefile files. I want to create a Pig script which will take ...
johnothegrouch's user avatar
1 vote
1 answer
2k views

Sequence File of Objects into Hive

We started with a bunch of data stored in NetCDF files. From there, some Java code was written to create sequence files from the NetCDF files. We don't know much about the original intentions of the ...
Jason H Mindle's user avatar
0 votes
0 answers
826 views

Cannot write the output of the reducer to a sequence file

I have a Map function and a Reduce function outputting kep-value pairs of class Text and IntWritable.. This is just the gist of the Map part in the Main function : TableMapReduceUtil....
Pavan's user avatar
  • 668
1 vote
1 answer
1k views

Lease mismatch LeaseExpiredException

I have seen some posts on this topic, but I could not figure the fix to my problem. I am using Hadoop version Hadoop 2.0.0-cdh4.2.0, and java version "1.7.0_09-icedtea". I am running a program that ...
H W's user avatar
  • 121
0 votes
1 answer
2k views

sequence files created by HBASE export utility aren't readable

I tried the HBase export tool to transfer a table to HDFS. I tried to hadoop dfs -text the file to see a collection of contents. However, I got a fatal error: java.lang.RuntimeException: java.io....
user2325080's user avatar
1 vote
1 answer
431 views

Is Hadoop SequenceFile binary safe?

I read the SequenceFile.java in hadoop-1.0.4 source codes. And I find the sync(long) method which is used to find a "sync marker" (a 16 bytes MD5 when generated at file creation time) in SequenceFile ...
Shawn H's user avatar
  • 103
1 vote
1 answer
2k views

HDFS sequence file performance tuning

I'm trying to use Hadoop to process a lot of small files which are stored in sequence file. My program is highly IO bound so I want to make sure that IO throughput is high enough. I wrote a MR ...
avhacker's user avatar
  • 677
0 votes
1 answer
2k views

how to limit size of Hadoop Sequence file?

I am writing Hadoop seq file using txt as input. I know how to write Sequence file from text file. But i want to limit the output sequence file to some specific size say, 256MB. Is there any inbuilt ...
Pradeep Bhadani's user avatar
0 votes
2 answers
5k views

use SequenceFile class write file

I use the follow codes to write some data into a SequenceFile Format file . when the program run a while , I interrupte the program via red button on eclipse console . however , when I check the data ...
user2193944's user avatar
0 votes
1 answer
1k views

Write binary data from HDFS files to SequenceFile

I a lot of files in HDFS and want to copy them into sequence files by MR job. The key type of the seq file is TEXT (I use SHA1), and the value type is BytesWritable(the file content). I find some ...
avhacker's user avatar
  • 677
0 votes
1 answer
1k views

Read Hadoop SequenceFile: weird hex number stream

I am trying to convert a piece of Hadoop SequenceFile into plain text with the following code: Configuration config = new Configuration(); Path path = new Path( inputPath ); SequenceFile....
Yuhao's user avatar
  • 1,590
3 votes
1 answer
3k views

How does Mapper class identify the SequenceFile as inputfile in hadoop?

In my one MapReduce task, I override the BytesWritable as KeyBytesWritable, and override the ByteWritable as ValueBytesWritable. Then I output the result using SequenceFileOutputFormat. My question ...
JoJo's user avatar
  • 1,407
1 vote
4 answers
4k views

Hadoop HDFS: Read sequence files that are being written

I am using Hadoop 1.0.3. I write logs to an Hadoop sequence file into HDFS, I call syncFS() after each bunch of logs but I never close the file (except when I am performing daily rolling). What I ...
Guillaume Perrot's user avatar
0 votes
1 answer
64 views

Is there an efficient way to find the sequence file for a given key?

My map-reduce job writes the final results to sequence files. Is there any efficient way to find the sequence file for a given key?
flyingfromchina's user avatar
1 vote
1 answer
889 views

Hadoop HDFS and Sequence File

I have one more very basic concept level question coming to my mind, what are the differences and relationship between HDFS and various file format -- sequence file (map file based on it), HAR file? ...
Lin Ma's user avatar
  • 9,989
6 votes
1 answer
6k views

Reading Hadoop SequenceFiles with Hive

I have some mapred data from the Common Crawl that I have stored in a SequenceFile format. I have tried repeatedly to use this data "as is" with Hive so I can query and sample it at various stages. ...
codingmonk's user avatar
1 vote
1 answer
4k views

Mahout: converting one large text file to SequenceFile format

I have done a lot of searching on the web for this, but I've found nothing, even though I feel like it has to be somewhat common. I have used Mahout's seqdirectory command to convert a folder ...
Nick's user avatar
  • 13
1 vote
1 answer
2k views

Hadoop SequenceFile size

I'm creating a HashMap of key value pairs of a Hadoop Vector that is stored inside a SequenceFile. For efficiency purposes I want to know how long the Vector of key value pairs is so that I can ...
Calavoow's user avatar
  • 492