17

In Hadoop fs how to lookup the block size for a particular file?

I was primarily interested in a command line, something like:

hadoop fs ... hdfs://fs1.data/...

But it looks like that does not exist. Is there a Java solution?

2
  • Do you mean the particular file size? Dec 7, 2011 at 6:27
  • 1
    Nope, block size. The default is HDFS block size is 64MB and it can be changed. Dec 7, 2011 at 6:49

5 Answers 5

46

The fsck commands in the other answers list the blocks and allow you to see the number of blocks. However, to see the actual block size in bytes with no extra cruft do:

hadoop fs -stat %o /filename

Default block size is:

hdfs getconf -confKey dfs.blocksize

Details about units

The units for the block size are not documented in the hadoop fs -stat command, however, looking at the source line and the docs for the method it calls we can see it uses bytes and cannot report block sizes over about 9 exabytes.

The units for the hdfs getconf command may not be bytes. It returns whatever string is being used for dfs.blocksize in the configuration file. (This is seen in the source for the final function and its indirect caller)

1
  • Can you provide the unit of the output by the provided command.
    – awadhesh14
    Apr 16, 2019 at 14:29
14

Seems hadoop fs doesn't have options to do this.

But hadoop fsck could.

You can try this

$HADOOP_HOME/bin/hadoop fsck /path/to/file -files -blocks
1

I think it should be doable with:

hadoop fsck /filename -blocks

but I get Connection refused

1
  • I have done test, yes, it works. You mean connection refused or permission refused? Dec 7, 2011 at 7:06
1

Try to code below

path=hdfs://a/b/c

size=`hdfs dfs -count ${path} | awk '{print $3}'`
echo $size
0

For displaying the actual block size of the existing file within HDFS I used:

[pety@master1 ~]$ hdfs dfs -stat %o /tmp/testfile_64
67108864

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.