Introduction to the Linux Command Line for High-Performance Computing

1. Introduction to theLinux Command Line for High-Performance Computing Dr. Charles J Antonelli LSAIT ARSOctober, 2014

2. Roadmap • Linux overview • The command shell • Navigating the filesystem • Basic commands & wildcarding • Shell redirection & pipelining • Editing text files • Permissions • Processes cja 2014

3. Poll • How many here to learn the command line • For HPC, specifically Flux, work? • For other computing cluster work? • For desktops/other Linux/Unix/Mac OS X? • Other? cja 2014

4. Course Text William E Shotts, Jr.,“The Linux Command Line: A Complete Introduction,”No Starch Press, January 2012.Download Creative Commons Licensed version athttp://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf. cja 2014

5. Linux Overview cja 2014

6. A compute node 48 GB RAM 12 Intel cores Local disk Network cja 2014

7. cja 2014 http://linuxforums.org.uk/index.php?action=media;sa=item;in=262

8. The command shell cja 2014

9. The command line • A basic way of interacting with a Linux system • Execute commands • Create files and directories • Edit file content • Access the web • Copy files to and from other hosts • Run HPC jobs • … do things you can’t do from the conventional point-and-click Graphical User Interface (GUI) cja 2014

10. Why command line? • Linux was designed for the command line • You can create new Linux commands using the command line, without programming • Many systems provide only the command line, or poorly support a GUI interface • Such as most HPC systems • Many things can be accomplished only through the command line • Much systems administration & troubleshooting • You want to be cool cja 2014

11. The command shell • The command shell is an application that reads command lines from the keyboard and passes them to the Linux operating system to be executed • When you login to a remote Linux system using a tool like ssh, you will automatically be connected to a shell • On your desktop, laptop, or tablet, you may have to find and execute a terminal emulator application to bring up a shell in a window cja 2014

12. The command shell cja 2014 http://www.livefirelabs.com/208_global/images/unix-operating-system.gif

13. Types of shells • Standard Linux shell (used in this course) • bash “Bourne-again” shell • Other shells • sh Bourne shell, original AT&T Unix shell • csh “C shell”, common on older Unix systems • tcsh “Enhanced C shell”, better version of csh • What shell am I using? echo $SHELL cja 2014

14. Connecting via ssh • Terminal emulators • Linux and Mac OS X • Start Terminal • Use sshcommand • Windows • U-M PuTTY/WinSCP (U-M Blue Disc)https://www.itcs.umich.edu/bluedisc/ • PuTTYhttp://www.chiark.greenend.org.uk/~sgtatham/putty/ • SSH Secure Shell (deprecated) cja 2014

15. Lab 0 Task: Start a local shell on Mac OS X • Login to Mac OS X with your uniqname and Kerberos password • If there is a Terminal icon in the Dock , double-click itOtherwise: • Bring up a Finder window:Click on the desktop, type Command-N • Start the Terminal Application:In the Finder window, click on Applications on the left, scroll down on the right until you find the Utilities folder, double-click on the Utilities folder, scroll down on the right until you find the Terminal application, double-click it • This creates a Terminal window with a shell running inside it • From a Terminal window, Command-N will start a new Terminal cja 2014

16. Lab 0 cja 2014

17. The shell prompt • The “~$ “ is the shell prompt • This means the shell is waiting for you to type something • Format can vary, usually ends with “$” , “%” or “#” • If $ or %, you have a normal shell • This shell has your privileges • If #, you have a so-called “root shell” • This shell has administrator privileges • You can do a great deal of irreversible damage cja 2014

18. Typing into the shell • Basic input line editing commands • Backspace erases previous character • Left and right arrow move insertion point on the line • Control-U erases the line so you can start over • Enter executes the line you typed • Control-C interrupts whatever command you started and returns you to the shell prompt (usually) • Up and downarrow will access your command history • Type “exit” without the quotes to exit the shell • Click the red "close" icon at the top of the Terminal window to close it cja 2014

19. Lab 1 Task: Enter some basic commands ~$ date ~$ id ~$ ps ~$ df -kh ~$ who ~$ top # type Control-C to exit cja 2014

20. Interlude: Flux cja 2014

21. Flux • Flux is a university-wideshared computational discovery / high-performance computing service. • Interdisciplinary • Provided by Advanced Research Computing at U-M (ARC) • Operated by CAEN HPC • Hardware procurement, software licensing, billing support by U-M ITS • Used across campus • Collaborative since 2010 • Advanced Research Computing at U-M (ARC) • College of Engineering’s IT Group (CAEN) • Information and Technology Services • Medical School • College of Literature, Science, and the Arts • School of Information http://arc.research.umich.edu/resources-services/flux/ cja 2014

22. The Flux cluster Login nodes Compute nodes Data transfernode Storage … cja 2014

23. A Flux node 48 GB RAM 12 Intel cores Local disk Network cja 2014

24. Logging in to Flux • ssh flux-login.engin.umich.edu • MToken (or Software Token) required • You will be randomly connected a Flux login node • Currently flux-login1 or flux-login2 • Firewalls restrict access to flux-login.To connect successfully, either • Physically connect your ssh client platform to the U-M campus wired or MWireless network, or • Use VPN software on your client platform, or • Use ssh to login to an ITS login node (login.itd.umich.edu), and ssh to flux-login from there cja 2014

25. A shell on a login node • Now you have a shell on the login node • Your shell prompt has changed to something like[flux-login1 ~]$ • Try those simple commands here! • Flux is for HPC work • The login nodes are reserved for copying data, editing and compiling programs and scripts, and short test runs • For HPC computation, you should run on the compute nodes • A batch job will always run on the compute nodes • An interactive batch job will get you a shell on a compute node cja 2014

26. A shell on a compute node • Getting a shell on a compute node • Enter this command on the login node:[flux-login1 ~]$ ~cja/hpc100/getshell • After a short delay, you’ll get an interactive shell:qsub: waiting for job 11147004.nyx.engin.umich.edu to startqsub: job 11147004.nyx.engin.umich.edu ready[nyx5509 ~]$ • You can type commands into this shell for all of our in-class experiments • Type “exit” to exit this shell • After two hours, your interactive shell will be killed cja 2014

27. Navigating the filesystem cja 2014

28. Linux Filesystem Concepts • Files are stored in a directory • Directories may contain other directories as well as files • A hierarchy of these directories is called a directory tree • A directory tree (a connected graph with no cycles) has a single topmost root directory • A directory tree rooted at the system root directory “/” is called a filesystem cja 2014

29. cja 2014 http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html

30. Linux Filesystem Concepts • A file is accessed using its path name • Absolute path name • /dir1/dir2/…/dirn/filename • /usr/X11R6/bin • Relative path name • current-working-directory/filename • bin • Every shell maintains a notion of a current working directory • Initialized at login to your home directory • Changedvia cd command • Two special directories • . refers to the current directory • .. refers to the current directory’s parent directory cja 2014

31. Basic commands cja 2014

32. Prerequisites Some fundamental commands: ~$ file file # what kind of file is file? ~$ cat file # display contents of text file ~$ less file # paginate text file ~$ man command # get info about command Exercise: figure out how to make the date command display the date in Coordinated Universal Time (UTC) cja 2014

33. Lab 2 Task: navigate the file system Commands: ~$cd # make your home directory the current working directory ~$pwd # print working directory ~$ mkdirfoo # create directory foo ~$ cd foo # cd to the foo directory ~$ mkdir bar # create directory bar ~$ cd .. # cd back to the foo directory ~$ tree # display foo’s directory tree cja 2014

34. Navigating the filesystem Some fundamental commands: ~$ pwd # print working directory ~$ cd dir# makedirthe current working directory ~$ cd # cd to your home dir ~$ cd ~cja# cd to cja’s home dir ~$ mkdirdir# create directory dir ~$ rmdirdir# remove (empty) directory dir ~$ tree # display dir tree cja 2014

35. Listing info on files ls – list information about files ~$ ls # list contents of cur dir ~$ lsdir # list contents of dir ~$ ls –l # list details of files in cur dir including access, owner & group, size, and last-modified time ~$ ls–t # list newest files first ~$ ls –R dir # list all files in tree dir ~$ ls–ltdir # options can be combined cja 2014

36. Working with files These commands manipulate files ~$ mv foo bar # rename file foo to bar ~$ cp foo bar # copy file foo to bar ~$ cp–r dir1 dir2 # copy dir tree dir1 to dir2 ~$ cp f1 f2 dir # copy f1 and f2 to directory dir ~$ mkdirdir # create empty directory dir ~$ rmdirdir # remove empty directory dir ~$ rmfile # remove file file ~$rm –r dir # remove directory tree dir cja 2014

37. Lab 3 Exercise: Create a directory named tutorialin your home directory. In that directory, create a directory named sampleand a directory named test . Create a file named msgin directory testthat contains a copy of the file/etc/motd. Extra credit: Make the last-modified time of your copy identical to that of /etc/motd. Hint: look at the options of the copy command cja 2014

38. Compressing and archiving These commands compress and archive files ~$ gzip foo # compress foo to foo.gz ~$ gunzip foo # uncompressfoo.gz to foo ~$ bzip2 foo# better compress foo to foo.bz2 ~$ bunzip2 foo# uncompressfoo.bz2 to foo ~$ tar cffoo.tar bar # archive subtree bar in file foo.tar ~$tar xffoo.tar# restore archive from file foo.tar ~$tar zcffoo.tgz bar # archive and compress ~$tar jcffoo.tjzbar # archive and compress better cja 2014

39. Wildcards The shell accepts wildcarded arguments This is also called "shell globbing" Wildcards: ? Matches a single character * Matches zero or more characters [chars] Matches any of the chars [c1-c2] Matches chars ‘c1’ through ‘c2’ [^chars] Matches any but the chars ~$ ls foo.? # match files named foo.x, where x is any character ~$ echo *.[cs] # echo files that end in .c or .s ~$ mv [o-z]* save # move files starting with o through z to directory save ~$ echo [^A-Z]? # ??? cja 2014

40. Shell redirection & pipelining cja 2014

41. Shell redirection A Linux command can have its inputs and outputs redirected ~$ ls >foo # put list of files in current directory into file foo ~$ ls>>bar # add list of files in current directory to end of file foo ~$ sort <bar # sort lines from file bar ~$ sort <<EOF # sort lines entered at keyboardwhiskeybravotangoEOF ~$ wc -l </etc/motd >~/mycounts # count number of lines from file /etc/motd and put result in filemycounts in my home directory cja 2014

42. More Linux commands More useful Linux tool commands ~$ grepstring # show lines of input containing string ~$ tail # show last few lines of input ~$ head # show first few lines of input ~$ sort # sort the input cja 2014

43. Shell pipelining A Linux command can have its output connected to the input of another Linux command ~$ ls | wc –l # count files in current directory ~$ last | grep reboot# when did we reboot? Exercise: How many people are running MATLAB on your compute node? cja 2014

44. Editing text files cja 2014

45. Editing text files • Simple editor • nano or pico • "What you see is what you get” editor • Simple to learn if you want to get started quickly • Supported editors • vi or vim • emacs • Powerful but more complex • If you have time and inclination to become proficient, spend time here cja 2014

46. Text files • Watch out for source code or data files written on Windows systems • Use these tools to analyze and convert source files to Linux format • file • dos2unix cja 2014

47. Permissions cja 2014

48. File Permissions • Three permission bits, aka mode bits • Files: Read, Write, Execute • Directories: List, Modify, Search • Three user classes • User (File Owner), File Group, Other • man chmod cja 2014

49. File Permissions, examples -rw------- cjalsait 40 Oct 1 12:03 foo.bz2file readand write rights for the owner, no access for anyone else chmod u=rw,g=r,o= file -rw-r----- cjalsait 40 Oct 1 12:03 foo.bz2file read and write rights for the owner, read for members of the lsait group, no access and for others drwxr-x--x cjalsait 4096 Oct 1 12:15 barlist, modify, and search for the owner, list and search for group, and execute only for others cja 2014

50. Processes cja 2014