510 likes | 683 Views
Introduction to the Linux Command Line for High-Performance Computing. Dr. Charles J Antonelli LSAIT ARS October, 2013. Roadmap. Linux overview the command shell navigating the filesystem basic commands shell redirection permissions processes command environment. Poll.
E N D
Introduction to theLinux Command Line for High-Performance Computing Dr. Charles J Antonelli LSAIT ARSOctober, 2013
Roadmap • Linux overview • the command shell • navigating the filesystem • basic commands • shell redirection • permissions • processes • command environment cja 2013
Poll • How many here to learn the command line • For HPC, specifically Flux, work? • For other computing cluster work? • For desktops/other Linux/Unix/Mac OS X? • Other? cja 2013
Linux Overview cja 2013
cja 2013 http://linuxforums.org.uk/index.php?action=media;sa=item;in=262
The command shell cja 2013
The command line • A basic way of interacting with a Linux system • Execute commands • Create files and directories • Edit file content • Access the web • Copy files to and from other hosts • Run HPC jobs • … do things you can’t do from the conventional point-and-click Graphical User Interface (GUI) cja 2013
Why command line? • Linux was designed for the command line • You can create new Linux commands using the command line, without programming • Many systems provide only the command line, or poorly support a GUI interface • Such as most HPC systems • Many things can be accomplished only through the command line • Much systems administration & troubleshooting • You want to be cool cja 2013
The command shell • The command shell is an application that reads command lines from the keyboard and passes them to the Linux operating system to be executed • When you login to a remote Linux system using a tool like ssh, you will automatically be connected to a shell • If you login to a local Linux system using its GUI, you may have to find and execute a terminal emulator application to bring up a shell in a window • It will usually be named something like “Terminal” cja 2013
The command shell cja 2013 http://www.livefirelabs.com/208_global/images/unix-operating-system.gif
Lab 1 Task: Start a local shell on Mac OS X • Login to Mac OS X with your uniqname and Kerberos password • Bring up a Finder window:Click on the desktop, type Command-N • Start the Terminal Application:In the Finder window, click on Applications on the left, scroll down on the right until you find the Utilities folder, double-click on the Utilities folder, scroll down on the right until you find the Terminal application, double-click on itThis creates a Terminal window with a shell running inside it • From a Terminal window, Command-N will start a new Terminal Sometimes there will be a terminal icon in the Dock – just double-click it cja 2013
Lab 1 cja 2013
The shell prompt • The “~$ “ is the shell prompt • This means the shell is waiting for you to type something • Format can vary, but will usually end with “$” or “#” • If $, you have a normal shell • If #, you have a so-called “root shell” • This shell has administrator privileges • You can do a great deal of irreversible damage cja 2013
Typing into the shell • Basic input line editing commands • Backspace erases previous character • Left and right arrow move insertion point on the line • Control-U erases the line so you can start over • Enter executes the line you typed • Control-C interrupts whatever command you started and returns you to the shell prompt (usually) • Up and downarrow will access your command history • Type “exit” without the quotes to exit the shell • Click the red circle at upper left of the Terminal window to close it cja 2013
Lab 2 Task: Enter some basic commands ~$ date ~$ id ~$ ps ~$ df -kh ~$ who ~$ top cja 2013
Interlude: Flux cja 2013
Flux • Flux is a university-wideshared computational discovery / high-performance computing service. • Interdisciplinary • Provided by Advanced Research Computing at U-M (ARC) • Operated by CAEN HPC • Hardware procurement, software licensing, billing support by U-M ITS • Used across campus • Collaborative since 2010 • Advanced Research Computing at U-M (ARC) • College of Engineering’s IT Group (CAEN) • Information and Technology Services • Medical School • College of Literature, Science, and the Arts • School of Information http://arc.research.umich.edu/resources-services/flux/ cja 2013
The Flux cluster Login nodes Compute nodes Data transfernode Storage … cja 2013
A Flux node 48 GB RAM 12 Intel cores Local disk Network cja 2013
Logging in to Flux • ssh flux-login.engin.umich.edu • MToken (or Software Token) required • You will be randomly connected a Flux login node • Currently flux-login1 or flux-login2 • Firewalls restrict access to flux-login.To connect successfully, either • Physically connect your ssh client platform to the U-M campus wired or MWireless network, or • Use VPN software on your client platform, or • Use ssh to login to an ITS login node (login.itd.umich.edu), and ssh to flux-login from there cja 2013
Logging in to Flux • Now you have a shell on the login node • Your shell prompt has changed to something like[flux-login1 ~]$ • Try those simple commands here! • Flux is for HPC work • The login nodes are reserved for copying data, editing and compiling programs and scripts, and short test runs • For HPC computation, you should run on the compute nodes • A batch job will always run on the compute nodes • An interactive batch job will get you a shell on a compute node cja 2013
A shell on a compute node • Getting a shell on a compute node • Enter this command on the login node:[flux-login1 ~]$ ~cja/hpc100/getshell • After a short delay, you’ll get an interactive shell:qsub: waiting for job 11147004.nyx.engin.umich.edu to startqsub: job 11147004.nyx.engin.umich.edu ready[nyx5509 ~]$ • You can type commands into this shell for all of our in-class experiments • Type “exit” to exit this shell • After two hours, your interactive shell will be killed cja 2013
Navigating the filesystem cja 2013
Linux Filesystem Concepts • Files are stored in a directory • Directories may contain other directories as well as files • A hierarchy of these directories is called a directory tree • A directory tree (a connected graph with no cycles) has a single topmost root directory • A directory tree rooted at the system root directory “/” is called a filesystem cja 2012
Linux Filesystem Concepts • A file is accessed using its path name • Absolute path name • /dir1/dir2/…/dirn/filename • /home/test/tux/test.c • Relative path name • current-working-directory/filename • test.c • Every shell maintains a notion of a current working directory • Initialized at login to your home directory • Changedvia cd command • Two special directories • . refers to the current directory • .. refers to the current directory’s parent directory cja 2012
cja 2013 http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html
Lab 3 Task: navigate the file system Commands: ~$pwd # print working directory ~$ cd dir # makedirthe working dir ~$ cd # cd to your home dir ~$ cd ~cja # cd to cja’s home dir ~$ mkdirdir# create directory dir ~$ tree # display dir tree cja 2013
Basic commands cja 2013
Prerequisites Some fundamental commands: ~$ file file # what kind of file is file? ~$ cat file # display contents of text file ~$ less file # paginate text file ~$ man command # get info about command Exercise: figure out how to make the date command display the date in Coordinated Universal Time (UTC) cja 2013
Listing info on files ls – list information about files ~$ ls # list contents of cur dir ~$ lsdir # list contents of dir ~$ ls –l # list details of files in cur dir ~$ ls–t # list newest files first ~$ ls –R dir # list all files in tree dir ~$ ls–ltdir # options can be combined cja 2013
Lab 4 Exercise: Create a directory named tutorialin your home directory. In that directory, create a directory named sampleand a directory named test . Create a file named msgin directory testthat contains a copy of the file/etc/motd. cja 2013
Wildcards All Linux commands take wildcarded arguments Wildcards: ? Matches a single character * Matches zero or more characters [chars] Matches any of the chars [c1-c2] Matches chars ‘c1’ through ‘c2’ [^chars] Matches any but the chars ~$ ls foo.? # match files named foo.x, where x is any character ~$ echo *.[cs] # echo files that end in .c or .s ~$ mv [o-z]* save # move files starting with o through z to directory save ~$ echo [^A-Z]? # ??? cja 2013
Working with files These commands manipulate files ~$ mv foo bar # rename file foo to bar ~$ cp foo bar # copy file foo to bar ~$ cp–r dir1 dir2 # copy dir tree dir1 to dir2 ~$ cp f1 f2 dir # copy f1 and f2 to directory dir ~$ mkdirdir # create empty directory dir ~$ rmdirdir # remove empty directory dir ~$ rmfile # remove file file ~$rm –r dir # remove directory tree dir cja 2013
Shell redirection & pipelining cja 2013
Shell redirection A Linux command can have its inputs and outputs redirected ~$ ls >foo # put list of files in current directory into file foo ~$ ls>>bar # add list of files in current directory to end of file foo ~$ sort <bar # sort lines from file bar ~$ sort <<EOF # sort lines entered at keyboardwhiskeybravotangoEOF ~$ wc -l </etc/motd >~/mycounts # count number of lines from file /etc/motd and put result in filemycounts in my home directory cja 2013
Shell pipelining A Linux command can have its output connected to the input of another Linux command ~$ ls >foo # put list of files in current directory into file foo ~$ sort <bar # sort lines from file bar ~$ wc -l </etc/motd >~/mycounts # count number of lines from file # /etc/motd and put result in file # mycounts in my home directory ~$ rm –f * # ??? ~$ echo [^A-Z]? # ??? cja 2013
More Linux commands More useful Linux tool commands ~$ grepstring # show lines of input containing string ~$ tail # show last few lines of input ~$ head # show first few lines of input ~$ sort # sort the input ~$ sed # edit the input by matching patterns ~$ awk # transform the input by programming ~$ uniq # suppress duplicate lines in input cja 2013
Regular expressions All Linux tools that match strings accept regular expressions: . Matches a single character* Matches zero or more occurrences of the previous character + Matches oneor more occurrences of the previous character [chars] Matches any of the chars [c1-c2] Matches chars ‘c1’ through ‘c2’ [^chars] Matches any but the chars ^ Matches the beginning of a line $ Matches the end of a line [A-Z][a-z]+ # match uppercase letter followed by one or morelowercase letters cat /proc/cpuinfo | grep '^model name' cat /proc/cpuinfo | head -2 cja 2013
Text files • Watch out for source code or data files written on non-Linux systems • Use these tools to analyze and convert source files to Linux format • file • dos2unix, mac2unix cja 2013
Editing cja 2013
Editing text files • Two recommended editors • nano or pico • "What you see is what you get" editor with on-screen help • Simple to learn and use if you want to get started quickly • vi or vim • Powerful but more complex • If you have time and inclination to become proficient, spend time here cja 2012
Permissions cja 2013
File Permissions I • Three permission bits, aka mode bits • Files: Read, Write, Execute • Directories: List, Modify, Search • Three user classes • User (File Owner), File Group, Other • man chmod cja 2012
File Permissions, examples -rwxr-xr-x cjalsaitfile read, write, and execute rights for the owner, read and execute for others -rwxr-x--x cjalsaitsame permissions as above, except only the owner can read the file, while all can execute chmodu=rwx,g=rx,o=x file drwxr-x--x cjalsaitlist, modify, and search for the owner, listand search for group, and execute only for others cja 2012
Processes cja 2013
Processes • On Linux, every program runs in a process • You can examine these processes man ps ps ps ax top cja 2012
Processes • You can signal a running process • To stop it, or "kill" it man kill cja 2012
Lab 5 Task: compile and execute simple programs on the Flux compute node • Copy sample code to your login directory:cd cp~brockp/cac-intro-code.tar.gz. tar -xvzfcac-intro-code.tar.gz cd ./cac-intro-code • Examine, compile & execute helloworld.f90: ifort-O3 -ipo -no-prec-div -xHost -o f90hello helloworld.f90 ./f90hello • Examine, compile & execute helloworld.c: icc-O3 -ipo -no-prec-div -xHost -o chellohelloworld.c ./chello • Examine, compile & execute MPI parallel code: mpicc-O3 -ipo -no-prec-div -xHost -o c_ex01 c_ex01.c mpirun-np 2 ./c_ex01 cja 2013
Any Questions? • Charles J. AntonelliLSAIT Advocacy and Research Supportcja@umich.eduhttp://www.umich.edu/~cja734 763 0607 • Please evaluate our course: • http://umichlsa.qualtrics.com/SE/?SID=SV_3pzjrMdjGXtUDAh cja 2013