1 / 11

An introduction to the Apache Hadoop command set

An introduction to Apache Apache Hadoop command set.What commands are available and what do they do ? A brief introduction to each command without indepth detail.

semtechs
Download Presentation

An introduction to the Apache Hadoop command set

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Apache Command Set • What types ? • What are they ? • What do they do ? • Environment • Configuration www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  2. Hadoop commands – What types ? • User commands • Administration commands • Generic options for all commands • Configuration options • Environment • Variables i.e. HADOOP_PREFIX • Aliases i.e. hls = hadoop fs -ls www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  3. Hadoop commands – What are they ? User Commands • archive – save files to a har archive • distcp – copy files or directories recursively • fs – file system commands • cat – copies file to stdout • chgrp – change group associated with file • chmod – change file permissions • chown – change file ownership • CopyFromLocal – copy from local file reference • CopyToLocal – copy to local file reference www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  4. Hadoop commands – What are they ? User Commands • fs – file system commands • count – count of dir / files/ bytes • cp – copy files • du – size of files and directories • dus – display file lengths • expunge – empty trash • get – copy files to local file system • getmerge – get but merge files • ls – file listing • lsr recursive ls www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  5. Hadoop commands – What are they ? User Commands • fs – file system commands • mkdir – make directory • moveFromLocal – put with delete of origin • mv – move from source to destination • put – copy between file systems • rm – remove a file • rmr – recursive delete • setrep – change file replication factor • stat – returns file stat information www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  6. Hadoop commands – What are they ? User Commands • fs – file system commands • tail – display end of file • test – check file existence / type • text – output file as text • touchz – create zero length file • fsck – HDFS file system check • fetchdt – get delegation token from name node • jar – run jar file • Job – manage mapreduce jobs www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  7. Hadoop commands – What are they ? User Commands • pipes – run a pipe job • queue – interact and view job queue • version – get Hadoop version • CLASSNAME – run class named CLASSNAME • classpath – print the class path www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  8. Hadoop commands – What are they ? Administration Commands • balancer – run cluster balancing • daemonlog – get/set daemon log level • datanode – run hdfs data node • dfsadmin – run dfsadmin client • mradmin – run map reduce admin client • jobtracker – run mr jobtracker node • namenode – runs the name node • secondarynamenode – run secondary name node • tasktracker – run task tracker node www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  9. Hadoop Environment See the .bashrc for environment set up ##export HADOOP_HOME=/usr/local/hadoop ## deprecated export HADOOP_PREFIX=/usr/local/hadoop export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386 unalias hfs &> /dev/null alias hfs="hadoop fs" unalias hls &> /dev/null ; alias hls="hfs -ls" unalias hup1 &> /dev/null ; alias hup1="cd $HADOOP_PREFIX/bin ; ./start-dfs.sh" unalias hup2 &> /dev/null ; alias hup2="cd $HADOOP_PREFIX/bin ; ./start-mapred.sh" unalias hdwn1 &> /dev/null ; alias hdwn1="cd $HADOOP_PREFIX/bin ; ./stop-mapred.sh" unalias hdwn2 &> /dev/null ; alias hdwn2="cd $HADOOP_PREFIX/bin ; ./stop-dfs.sh" # if using LZO compression then add entry here for viewing # LZO compressed files ##PATH=$PATH:$HADOOP_HOME/bin ## deprecated PATH=$PATH:$HADOOP_PREFIX/bin PATH=$PATH:$JAVA_HOME/bin export PATH www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  10. Hadoop Configuration • Configuration files under $HADOOP_PREFIX/conf • Initial set up in • core-site.xml • hdfs-site.xml • mapred-site.xml • Example from core-site.xml <property> <name>hadoop.tmp.dir</name> <value>/app/hadoop/tmp</value> <description>A base for other temporary directories.</description> </property> www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  11. Contact Us • Feel free to contact us at • www.semtech-solutions.co.nz • info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems

More Related