7

How can I decompress and view few lines of a compressed file in hdfs. The below command displays the last few lines of the compressed data

hadoop fs -tail /myfolder/part-r-00024.gz

Is there a way I can use the -text command and pipe the output to tail command? I tried this but this doesn't work.

hadoop fs -text /myfolder/part-r-00024.gz > hadoop fs -tail /myfolder/

4 Answers 4

19

The following will show you the specified number of lines without decompressing the whole file:

hadoop fs -cat /hdfs_location/part-00000.gz | zcat | head -n 20

The following will page the file, also without first decompressing the whole of it:

hadoop fs -cat /hdfs_location/part-00000.gz | zmore
4

Try the following, should work as long as your file isn't too big (since the whole thing will be decompressed):

hadoop fs -text /myfolder/part-r-00024.gz | tail
3
  • I've tried that.That created a file called tail on my local machine which was about 9 GB in size. My server is a Windows 2008 server using hadoop MapR client not a linux server.
    – nobody
    Commented Aug 12, 2015 at 15:31
  • Then you'll have to run a hadoop job which decompresses the data to HDFS, and then tail the decompressed file, I think.
    – mattinbits
    Commented Aug 12, 2015 at 15:38
  • Ok,I ended up writing a pig script to view the content. Thanks
    – nobody
    Commented Aug 12, 2015 at 15:57
1

I ended up writing a pig script.

A = LOAD '/myfolder/part-r-00024.gz' USING PigStorage('\t');
B = LIMIT A 10;
DUMP B;
0

Use gunzip to view the compressed file contents:

 hdfs dfs -cat /path/filename.gz | gunzip

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.