27

I have constructed a single-node Hadoop environment on CentOS using the Cloudera CDH repository. When I want to copy a local file to HDFS, I used the command:

sudo -u hdfs hadoop fs -put /root/MyHadoop/file1.txt /

But,the result depressed me:

put: '/root/MyHadoop/file1.txt': No such file or directory

I'm sure this file does exist.

Please help me,Thanks!

4 Answers 4

27

As user hdfs, do you have access rights to /root/ (in your local hdd)?. Usually you don't. You must copy file1.txt to a place where local hdfs user has read rights before trying to copy it to HDFS.

Try:

cp /root/MyHadoop/file1.txt /tmp
chown hdfs:hdfs /tmp/file1.txt
# older versions of Hadoop
sudo -u hdfs hadoop fs -put /tmp/file1.txt /
# newer versions of Hadoop
sudo -u hdfs hdfs dfs -put /tmp/file1.txt /

--- edit:

Take a look at the cleaner roman-nikitchenko's answer bellow.

3
  • Thanks,but I tried to move this file to another place,type the follow command: {sudo -u hdfs hadoop fs -put /home/skfeng/Downloads/hadoop-book-master/test-data/ch1/file1.txt /} it still doesn't work and has the same results: {put: `/home/skfeng/Downloads/hadoop-book-master/test-data/ch1/file1.txt': No such file or directory }
    – skfeng
    Aug 28, 2013 at 10:47
  • 1
    And does user hdfs have access rights to the home of the user skfeng? Aug 28, 2013 at 10:55
  • Yes! I understand!User hdfs have no access to home of other users,I make a new directory for hdfs,and it works!Thank you very much!:)
    – skfeng
    Aug 28, 2013 at 12:06
18

I had the same situation and here is my solution:

 HADOOP_USER_NAME=hdfs hdfs fs -put /root/MyHadoop/file1.txt /

Advantages:

  1. You don't need sudo.
  2. You don't need actually appropriate local user 'hdfs' at all.
  3. You don't need to copy anything or change permissions because of previous points.
2
  • 2
    I like this solution, but I always forget what is the environment variable name :P +1 Oct 2, 2014 at 19:07
  • 1
    Thanks, i have to use DFS instead of FS, like this : HADOOP_USER_NAME=PUT_YOUR_USER_HERE hdfs dfs -put /source/xxx/yyy /destination/zzz
    – mourad m
    Apr 30, 2020 at 9:12
9

try to create a dir in the HDFS by usig: $ hadoop fs -mkdir your_dir and then put it into it $ hadoop fs -put /root/MyHadoop/file1.txt your_dir

0

Here is a command for writing df directly to hdfs file system in python script:

df.write.save('path', format='parquet', mode='append')

mode can be append | overwrite

If you want to put in in hdfs using shell use this command: hdfs dfs -put /local_file_path_location /hadoop_file_path_location

You can then check on localhost:50070 UI for verification


Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.