5

Do hadoop datanodes register themselves with the namenode by calling the namenode, or does the namenode have a list of datanodes and it reaches out to them.

I want to understand to better troubleshoot a problem with a new namenode I brought up (after a namenode failure) where it doesn't see any of the datanodes (but has the fsimage correct).

3
  • Data nodes heartbeat in to the name node. The name node does not reach out to data nodes. But that doesn't seem to be the question you really want to ask. ;)
    – Ilion
    May 21, 2013 at 9:12
  • No, that is the question, I wanted to keep it simple, it helps me know the architecture, what to expect so I can figure out why they're not heartbeating to the new namenode. Knowing the architecture is step 1. You should post that as an answer. May 21, 2013 at 9:25
  • The new namenode - is this on the same address / port as the old name node? May 21, 2013 at 10:21

3 Answers 3

7

Data nodes heartbeat in to the name node. The name node does not reach out to data nodes.

Even when retrieving data, the name node does not reach out to the data nodes. The name node will inform the client where the data is and the client will retrieve it from the data nodes. (To clarify, during an MR workflow the Job Tracker finds from the name node where the data is and assigns task trackers appropriately.)

3

Each datanode keeps the namenode details in hdfs.conf file. And namenode keep names of all data nodes in slaves file. I think you should update your slaves files in namenode and master file in datanodes.

3

I suppose you have a working cluster (with fs.default.name in core-site.xml properly configured in datanodes) before hard shutting down the namenode.

When I shut down my namenode with kill -9 pid, my datanodes start to show in log:

INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 0 time(s).

INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 1 time(s).

...

INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 9 time(s).

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: java.net.ConnectException: Call to namenodehost/192.168.0.35:8020 failed on connection exception: java.net.ConnectException: Connection refused at ...

INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 0 time(s).

INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 1 time(s).

...

repeatedly until I load again my namenode. At that moment, datanodes' logs shows:

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished generating blocks being written report for 1 volumes in 0 seconds

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous block report scan

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished asynchronous block report scan in 10ms

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reconciled asynchronous block scan with filesystem. 0 blocks concurrently deleted during scan, 0 blocks concurrently added during scan, 4 ongoing creations ignored

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reconciled asynchronous block report against current state in 0 ms

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 411 blocks took 0 msec to generate and 68 msecs for RPC and NN processing

Each datanode reconnects to the namenode and everything works ok.

Does this helps?

3
  • 1
    The last line forced me to think what you mean. Thanks :) Jul 30, 2014 at 11:13
  • Can datanodes start before namenode? or there is always a startup dependency between namenode and datanodes? Jul 9, 2018 at 3:05
  • @user1870400 You got me there, I don't know the answer with 100% confidence :P Usually namenode is started prior datanodes, and if you start HDFS service in a Cloudera CDH it always start the namenode first. Jul 10, 2018 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.