Hadoop filesystem

Hadoop Filesystem 101

• Useful command on HDFS file operation.

  
      #List directory contents in hdfs
      hdfs dfs -ls hdfs:/user/shoh
      #Display the disk space used by files
      hdfs dfs -du -h hdfs:/user/shoh
      #Copy data to HDFS
      hdfs dfs -mkdir hdfs:/user/shoh/nanoaod
      hdfs dfs -ls hdfs:/user/shoh/
      hdfs dfs -copyFromLocal ~/nanoaods.root hdfs:/user/shoh/nanoaod
      hdfs dfs -ls -R hdfs:/user/shoh/nanoaod
      #Copy the file back to local filesystem
      hdfs dfs -copyToLocal hdfs:/user/shoh/nanoaod/nanoaods.root ~
      #Removing files
      hdfs dfs -rm hdfs:/user/shoh/nanoaod/nanoaods.root
      

  More infomation about hdfs can be found here.

  Trobleshooting

• Safe mode in Hadoop can be turn off by running.

  
      hdfs dfsadmin -safemode leave
      

  msg: Safe mode is OFF

• </ul>