20

I am going through hadoop definitive guide, where it clearly explains about input splits. It goes like

Input splits doesn’t contain actual data, rather it has the storage locations to data on HDFS

and

Usually,Size of Input split is same as block size

1) let’s say a 64MB block is on node A and replicated among 2 other nodes(B,C), and the input split size for the map-reduce program is 64MB, will this split just have location for node A? Or will it have locations for all the three nodes A,b,C?

2) Since data is local to all the three nodes how the framework decides(picks) a maptask to run on a particular node?

3) How is it handled if the Input Split size is greater or lesser than block size?

7 Answers 7

31
  • The answer by @user1668782 is a great explanation for the question and I'll try to give a graphical depiction of it.

  • Assume we have a file of 400MB with consists of 4 records(e.g : csv file of 400MB and it has 4 rows, 100MB each)

enter image description here

  • If the HDFS Block Size is configured as 128MB, then the 4 records will not be distributed among the blocks evenly. It will look like this.

enter image description here

  • Block 1 contains the entire first record and a 28MB chunk of the second record.
  • If a mapper is to be run on Block 1, the mapper cannot process since it won't have the entire second record.
  • This is the exact problem that input splits solve. Input splits respects logical record boundaries.

  • Lets Assume the input split size is 200MB

enter image description here

  • Therefore the input split 1 should have both the record 1 and record 2. And input split 2 will not start with the record 2 since record 2 has been assigned to input split 1. Input split 2 will start with record 3.

  • This is why an input split is only a logical chunk of data. It points to start and end locations with in blocks.

Hope this helps.

2
  • 11
    why is the input split 2 and 3 is of size 100 MB and the first one is 200 MB? Jan 20, 2017 at 20:02
  • 1
    can we assigned different inputsplit to each block? or it is just the logical representation
    – bajran
    Feb 23, 2018 at 3:41
20

Block is the physical representation of data. Split is the logical representation of data present in Block.

Block and split size can be changed in properties.

Map reads data from Block through splits i.e. split act as a broker between Block and Mapper.

Consider two blocks:

Block 1

aa bb cc dd ee ff gg hh ii jj

Block 2

ww ee yy uu oo ii oo pp kk ll nn

Now map reads block 1 till aa to JJ and doesn't know how to read block 2 i.e. block doesn't know how to process different block of information. Here comes a Split it will form a Logical grouping of Block 1 and Block 2 as single Block, then it forms offset(key) and line (value) using inputformat and record reader and send map to process further processing.

If your resource is limited and you want to limit the number of maps you can increase the split size. For example: If we have 640 MB of 10 blocks i.e. each block of 64 MB and resource is limited then you can mention Split size as 128 MB then then logical grouping of 128 MB is formed and only 5 maps will be executed with a size of 128 MB.

If we specify split size is false then whole file will form one input split and processed by one map which it takes more time to process when file is big.

2
  • 2
    If the block1 is on machine1 and block2 is on machine2. Lets say map is running on machine 1, If the split size is double the block size. Does the map function on machine1 get the block2 from machine2 to process? Apr 10, 2015 at 10:06
  • 1
    Can input split be less than block size ? May 22, 2017 at 12:17
4

Input splits are a logical division of your records whereas HDFS blocks are a physical division of the input data. It’s extremely efficient when they’re the same, but in practice it’s never perfectly aligned. Records may cross block boundaries. Hadoop guarantees the processing of all records . A machine processing a particular split may fetch a fragment of a record from a block other than its “main” block and which may reside remotely. The communication cost for fetching a record fragment is inconsequential because it happens relatively rarely.

0

To 1) and 2): i'm not 100% sure, but if the task cannot complete - for whatever reason, including if something is wrong with the input split - then it is terminated and another one started in it's place: so each maptask gets exactly one split with file info (you can quickly tell if this is the case by debugging against a local cluster to see what information is held in the input split object: I seem to recall it's just the one location).

to 3): if the file format is splittable, then Hadoop will attempt to cut the file down to "inputSplit" size chunks; if not, then it's one task per file, regardless of the file size. If you change the value of minimum-input-split, then you can prevent there being too many mapper tasks that are spawned if each of your input files are divided into the block size, but you can only combine inputs if you do some magic with the combiner class (I think that's what it's called).

2
  • I think the concept of "node proximity" answers questions 1 and 2.
    – rohith
    Jul 18, 2013 at 16:56
  • input split is logical, it doesn't actually contain file data. it has references to locations(nodes) where blocks are stored.
    – rohith
    Jul 18, 2013 at 16:59
0

Hadoop framework strength is its data locality.So whenever a client request for the hdfs data, framework always checks for the locality else it looks for little I/O utilization.

0

Input splits are logical data units that fed to each mapper. Data is split across valid records. Input splits contain addresses of blocks and byte offsets.

Let's say, you have a text file that spanned across 4 blocks.

File:

a b c d
e f g h
i j k l
m n o p

Blocks:

block1: a b c d e
block2: f g h i j
block3: k l m n o
block4: p

Splits:

Split1: a b c d e f h
Split2: i j k l m n o p

Observe the splits are inline with boundaries (records) from file. Now, each split is fed to a mapper.

If Input split size is less than the block size, you will end up with using more no.of mappers vice versa.

Hope that helps.

0

HDFS block size is an exact number but Input split size is based on our data logic which may be a little different with the configured number

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.