1

I have few questions regarding Hadoop architecture

  1. In Mapreduce can we dynamically modify the block size and no of mappers,if so how do we do?

  2. Hows does the block gets created in HDFS. For example the hadoop framework is installed on say redhat linux machine. The default block size of linux filesystem is 4k. Is the HDFS block a logical wrapper on the 4k blocks or how does a block gets created. also is it parallel or sequential? because for example a file has only 32 MB since the block size is 64 MB. Is the remaining 32 Mb reusable?

  3. I want to see the location(data node) of all the blocks of particular file I just copied to the HDFS. Is there any command to do that from a single location?

  4. If I move the video file to HDFS, how does the block allocation happen for this video file

2
  • 1
    Too many questions in one question :) Mar 3, 2016 at 4:57
  • Any hint on any of the above questions will be very useful for me :)
    – Karthi
    Mar 3, 2016 at 4:58

1 Answer 1

1
  1. In Mapreduce can we dynamically modify the block size and no of mappers?

I assume that you are looking for HDFS file system.

HDFS is distributed storage system and Mapreduce is distributed processing framework.

HDFS block size can be changed with hdfs-site.xml

Have a look at documentation page for various HDFS configurations.

dfs.blocksize

134217728 ( default value)

The default block size for new files, in bytes. You can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.), Or provide complete size in bytes (such as 134217728 for 128 MB).

Relate SE question:

How to set data block size in Hadoop ? Is it advantage to change it?

  1. Hows does the block gets created in HDFS. For example the hadoop framework is installed on say redhat linux machine. The default block size of linux filesystem is 4k. Is the HDFS block a logical wrapper on the 4k blocks or how does a block gets created. also is it parallel or sequential? because for example a file has only 32 MB since the block size is 64 MB. Is the remaining 32 Mb reusable?

Remaining 32 MB is re-usable.

Have a look at this SE question for HDFS block write operation :

Hadoop file write

  1. I want to see the location(data node) of all the blocks of particular file I just copied to the HDFS. Is there any command to do that from a single location?
hadoop fsck /path/to/file -files -blocks

Related SE question:

Viewing the number of blocks for a file in hadoop

  1. If I move the video file to HDFS, how does the block allocation happen for this video file?

Number of blocks = File size in MB / DFS block size in MB

Once number of blocks have been identified, those blocks will be written as explained in Hadoop file write question.

Few more good questions:

Hadoop chunk size vs split vs block size

How hadoop decides how many nodes will do map and reduce tasks

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.