I suppose you have a working cluster (with fs.default.name
in core-site.xml
properly configured in datanodes) before hard shutting down the namenode.
When I shut down my namenode with kill -9 pid
, my datanodes start to show in log:
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 0 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 1 time(s).
...
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 9 time(s).
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: java.net.ConnectException: Call to namenodehost/192.168.0.35:8020 failed on connection exception: java.net.ConnectException: Connection refused at ...
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 0 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenodehost/192.168.0.35:8020. Already tried 1 time(s).
...
repeatedly until I load again my namenode. At that moment, datanodes' logs shows:
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished generating blocks being written report for 1 volumes in 0 seconds
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous block report scan
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished asynchronous block report scan in 10ms
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reconciled asynchronous block scan with filesystem. 0 blocks concurrently deleted during scan, 0 blocks concurrently added during scan, 4 ongoing creations ignored
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reconciled asynchronous block report against current state in 0 ms
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 411 blocks took 0 msec to generate and 68 msecs for RPC and NN processing
Each datanode reconnects to the namenode and everything works ok.
Does this helps?