Introduction to Linux on Iceberg

1. Introduction to Linux on Iceberg • What are UNIX and Linux? • Getting Connected • Basic UNIX/Linux Commands • Help! • Working with Directories • Running Programs • Editors • The Shell • Exercise

2. History of UNIX/Linux? . • Unix operating system was developed around 1969 in the Bell Labs • Originally written using C • Around 1990 Linus Torvalds of Helsinki University started off a freely available academic version of Unix • Linux is the Antidote to a microsoft dominated future

3. What is UNIX/Linux ? • Multi-Tasking O/S • Multi-User O/S • Available on a range of Computers

4. Which UNIX? • SunOS Sun Microsystems • IRIX Silicon Graphics • HP-UX Hewlett Packard • Linux For IBM PC compatibles There are a number of certification bodies with published standards and test suites to ensure quality of products. Such as; • Posix: Portable Operating System Interface ( IEEE standard) • COE: The Common Operating Environment (COE) Platform certification.

5. UNIX Internals (Simplified) • Kernel • System Internals • Shell • Command Interpreter • Programming language • File System • Process Management

7. UNIX Shells • sh Bourne Shell (Original Shell) (Steven Bourne of AT&T) • bash Bourne Again Shell (GNU Improved Bourne Shell) • csh C-Shell (C-like Syntax)(Bill Joy of Univ. of California) • ksh Korn-Shell (Bourne+some C-shell)(David Korn of AT&T) • tcsh Turbo C-Shell (More User Friendly C-Shell). You can switch from one shell to another by just typing the name of the shell. exit return you back to previous shell.

8. Linux and GNU • Linux is an implementation of Unix • Linux/Unix operating system is written in ‘C’ • Linux is not part of the GNU project but uses the same licensing agreements • Many of the linux utilities and tools are taken from the GNU project. • There are many flavours of linux distributions. The mix of the kernel (linux) with the utilities (GNU and other) and the installation procedure determine the flavour. Some of these are; • Fedora • SuSE • Redhat • Debian • Mandrake • Knoppix

9. Terminal Emulation SoftwareUsed in Sheffield Univ. • ssh : Secure Shell Client (no XWindows) • Exceed : ( Xwindows support and also secure shell)

10. Graphical UNIX via Exceed • X Concepts • Using the Exceed Software • Starting an xterm session • Starting a ssh session

11. Basic X Concepts • X Server runs on local machine • PC Exceed, Cygwin, Xming • UNIX Workstation Included in OS • Apple Mac Exodus • X Client runs on remote machine • Graphical Application • xterm • xcalc • Modelling and visualisation packages etc.

12. ssh on Managed/XP workstations Secure-shell client can be accessed via; Start Unix_ConnectivityExceed3Dssh. When the program initialises you will be asked to enter hostname (default is iceberg) , username and password.

13. ssh on Managed/XP workstations When ssh starts you will be presented with the login panel as shown below; Default host name is iceberg.shef.ac.uk and can be changed if you want to login somewhere else. You may next be asked if you wish to save the public host-key into a local-database. This is not a crucial question and you may answer YES or NO. Also if asked to provide host response just press ENTER.

14. Exceed on the stand-alone PCsStarting an Exceed xterm session Start Exceed ->Xstart and fill in the panel as shown below Fill in host,username,p/w,command and click on the run icon to start xterm

15. Exceed on the stand-alone PCsAn Exceed xterm session

16. Multiple ssh or xterm shells There are no limits to the number of ssh or xterm windows one can start simultaneously by methods described in the previous slides. You may also start extra xterm windows from the host by simply typing xterm & On titania and iceberg we also have a local command named Xterm that starts up an xterm window with nicer to use parameters. On iceberg we strongly recommend that you use Qsh rather than xterm or Xterm command so as to make use of a free worker node. Qsh will act like Xterm but will make use of a worker node. Typing exit will terminate an xterm or ssh session neatly. This will also close the xterm window but not the ssh window. ssh windows can be closed via the file exit menu.

17. Starting gnome Gnome was the default graphical desktop interface that was installed on iceberg before delivery. We currently do not allow the use of gnome on the head-node ‘iceberg’ for efficiency reasons. You could however use the gnome panel while working on one of the worker nodes ( via qsh ). This provides the user with access to all the gnome commands available via the red-hat menus. To do this type qsh to start an interactive session and in that session type /usr/bin/gnome-panel & This is the nearest, one would get to using the common-desktop environment as was provided on titania but it is not worth it.

18. Some basic rules • Unix is case sensitive. • Commands are in lower case. • Backspace and/or Del Keys correct typing errors. If the terminal parameters are not correctly set; try Ctrl+H • Ctrl+C Aborts a program or command. • You can use the arrow keys to recall previous commands, optionally edit and execute them.

19. Key combinations in bash

20. passwords • In normal linux environment the passwd command can be used to change the user passwords. However, because we manage passwords centrally this command will not work on iceberg. • If you wish to change your iceberg password you will have to do this via a web interface at the following URL: http://www.shef.ac.uk/cics/password

21. Filenames • Filenames can comprise of: a-z, A-Z alphabetic characters 0-9 digits .-_+ special characters mon+tue_01.06-03-96 • Wildcards when referencing files * any character or sequence of characters ? any single character

22. Format of Unix commands • command [option ...] [filename ...] eg: ls ls -l tutorial more tutorial

23. List Directory • ls list directory • ls -l list directory in long format • ls -a list all (inc. hidden) files -rw------ l course01 57 Oct 18 11:05 hello.c Access Permissions Number ofbytes in file Date and time last modified

24. Displaying contents of a text_file • more filename This command will start listing the contents of filename on screen and pause after a screenfull of data. While pausing, use the following characters to control the output. Spacebar next screenful n Spacebar : next n lines Enter next line b back one screen n b : back n screen’s full q quit ? or h list commands where n is a whole number

25. Displaying contents of a text_file… continued • cat [options] filename [filename … ] This command will output the contents of filename[s] to standard-output ( normally screen) without pausing. Following options are useful; -v display non-printing characters -n display with lines numbered on the left • tail [-n] filename Thiscommand lists the last 10 lines of a text file. If a number is specified (.eg. -20 ) lists the last n (i.e 20) lines

26. Copying files • Copy files (optionally directories) cp fromfile tofile Some of the useful options are: -R or –r : Recursive copy ‘fromfile’ is a directory so the entire directory and its contents are copied. e.g. cp –r mydir newdir -p : preserve. Preserves all attributes of the file ,such as access rights and creation date. • Copy and concatenate files by using cat Cat command concatenates contents of list of files and directs the output to standard output (normally screen). When used with redirection ‘>’ it can be used to join files together. e.g. cat file1 file2 file3 > new_big_file

27. Renaming and deleting files • mv :This command will move a file or directory to a new location. It can thus be used to rename files/directories as well as change their locations in the global directory structure. Syntax: mv source destination Example: mv myfile mynewfile mv myfile subdirectory/myfile mv mysubdir mynewsubdir • rm : This command will delete a file (optionally a directory if used with –r option). Syntax: rm object_to_delete Example: rm myfile rm –r mydirectory

28. Finding files and information about them • find :Finds a file in the directory hierarchy find root_dir –name filename –print Example : find . –name “myprog.*” -print Note that wild-character containing string must be enclosed in quotas • which : Shows in which directory a command is located. Syntax: which command_name • file: Can be used to see what type of data a particular file contains. For example, script , program, library, executable binary etc… Syntax: find command_name

29. Searching in files • grep string file This command finds and prints out the lines in the file(s) containing the specified string string = word or phrase file = file or list of files (wild_card can be used) Note: We strongly advise that the string is quoted. Examples: grep ‘Green Man’ england.dat grep ‘Zodiac’ t*.dat grep ‘Zone[a-z]’ security.fil

30. Directory Structure / (root) home1 usr me cs ph cs4un1 cs4un2 Home directory of user cs4un1 : /home1/cs/cs4un1 When you log in you are positioned in your home directory. The environment variable $HOME is also set to contain this directory name.

31. Working with Directories • pwd print working directory • cd change directory cd move to home directory cd .. move up one level cd mydir move into a subdirectory cd /var/adm move to an absolute directory • mkdir directory_namecreate a new directory • rmdir directory_name delete an empty directory

32. Managing your filestore • Each user has an allocation of filestore (Disk Space) • Monitor filestore usage with the ‘quota’ command EG: quota Disk quotas for username (uid 20002) Filesystem usage quota limit grace files quota limit timeleft /home1 1200 2000 4000 153 0 0 /data 411 99600 100000 13 0 0 /scratch 0 99600 100000 0 0 0 Note: quota –s will display quota in sensible units. • The quota value can be exceeded for 7 days quota < usage < limit for 7 days • The limit value can not be exceeded at all. usage < limit always • If usage > quota for 7 days account is expired and files can not be created

33. Manual Pages and Info system It is possible that there are more than one manual pages with the same name. This is because the manual pages contain information about the system library routines as well as the usual unix commands. Manual pages for commands are contained in section 1 whereas the system library call related items are in section 3. To access manual pages within a specific section simply specify the section_number as the first parameter. Example: man 3 random • Note that man – k topic will list the relevant manual pages and their section numbers. • The –S parameter can also be specified. Example: man files –S 1:2:4

34. Manual Pages and Info system Man: Manual pages : Give text-based help on usage. Manual pages are grouped into sections (1,2,3,4 ..). There is usually one manual page per command which is located in one of the directories defined by the MANPATH environment variable. To access the Manual page for a command just type; • man command To get a list of manual pages that contain a ‘word’ type; • man – k topic Info : Similar to man but can scroll with cursor keys and link to other information.

35. Using whatis and apropos commands • whatis displays the short description of a system command ‘if you know the command name’ . • apropos searches the same ‘whatis-database` and reports back the names of all the commands that a key-word occurs. Example usage: Suppose we wish to create a temporary file with a unique name but do not know if there is a command for doing so. You could try : apropos temporary or apropos file or apropos create This will give ‘whatis’ output for a list of commands which contain the keyword. You can now use the man command for potentially useful commands to get the full details of how to use them. man mktemp

36. Running programs on iceberg • Iceberg is the gateway to the cluster of worker nodes and the only one where direct logging in is allowed. • Iceberg’s main purpose is to allow access to the worker nodes but NOT to run cpu intensive programs. • All cpu intensive computations must be performed on the worker nodes. This is achieved by the qsh command for the interactive jobs and qsub command for the batch jobs. • Once you log into iceberg, taking advantage of the power of a worker node for interactive work is done simply by typing qsh and working in the new shell window that is opened. This what appears to be a trivial task has would in fact have queried all the worker nodes for you and started a session on the least loaded worker in the cluster. • The next set of slides assume that you are already working on one of the worker nodes (qsh session).

37. Running programs • Two modes of operation foreground and background • Foreground Interact with program via keyboard/screen • Background No connection with keyboard/screen Submit to backbround by Appending ‘&’ EG: myprog >& myfile & The symbols ‘>&’ redirect output and any errors to the file myfile Although the above method of running jobs on the background is feasible, the prio we recommend that you submit your background into the batch queue via the qsub command.

38. Redirection Most unix commands are not aware of the source of their input or the destination of their output. They simply read/write from/to stdin/stdout. The shell takes care of these issues. • Standard Input (default=>keyboard) • Standard Output (default=>screen) • Redirection symbols <,>,>> can be used to specify files as the source/destination of the read/write operations to override the above defaults.

39. Redirection continued … Most unix commands are not aware of the source of their input or the destination of their output. They simply read/write from/to stdin/stdout. The shell takes care of these issues. • To redirect the output to a file use the ‘>‘ symbol. Example: ls -l > dirlist • The ‘>‘ symbol should be used with care as it may over-write an existing file. ‘>>’ symbol can be used instead if the output should be appended to the end of an existing file rather than over-writing it. Example: ls -l >> logfile • If nothing is directed to a file then a zero size file is created, or if the file already existed then the contents of the file is removed Example: > afile • The file /dev/null is a special symbol to indicate a ‘black-hole’ Example :ls –l > /dev/null

40. Redirection continued … If any program expects any of it’s input from the standard-input-channel , i.e. the keyboard, it can also read the same information from a file by redirection. • To read input from a file use the ‘<‘ symbol. Example: write cs1xyz < message.fil Here any text input {write} program expects from the keyboard will be simply read from a file named message.fil. Each end-of-line will be treated as an <ENTER> on the keyboard.

41. Piping • Feeding the output of one command into the input of another command • The symbol ‘|’ is called a pipe command | command • eg: ls -al | more ls -la | grep Nov

42. Examples of re-direction and piping • ls –l | grep ‘Jun’ • ls –l | grep `Jun` > june_files • ls –l | grep `Jun` | cut –c 57-80 > june_files • cut –c 1-10 < test_files • aspell –l < message.txt > report.txt • grep fluent < news.dat • Grep fluent < news.dat | cut –c 1-72 • (grep fluent < news.dat ) | cut –c 1-72 • (grep fluent < news.dat ) | cut –c 1-72 > fluent.news

43. Foreground Program Control • Kill a program Ctrl C • Stop a program Ctrl Z Note a stopped program still exists in the system and hence can be re-started.

44. Program control within current shell • jobs Lists jobs (programs) • bg %job_id Place a job in the background • fg %job_id Return a job to the foreground • stop %job_id Stop a job • kill %job_id Kill a job Process_id can be used in place of %job_id for more definitive way of identification. jobs [1] + Running time.sh > out stop %1 [1] + Stopped (signal) time.sh > out bg %1 [1] + time.sh > out & kill %1 Terminated

45. Program control using ‘ps’ and ‘kill’ • ps Report process status ps -f -u username UID PID PPID C STIME TTY TIME CMD username 24816 24585 0 16:23:04 pts/50 0:00 sleep 2 username 20169 19956 0 16:05:45 pts/50 0:01 -csh username 24585 20169 0 16:35:07 pts/50 0:00 /bin/sh time.sh • kill Terminate process • Find its process ID (PID) using the ps command • Kill the process using the kill command kill 24585 • Sometimes kill on its own does not work so try kill -KILL 24585 or kill -9 pid

46. Program Control using top • top is a separate utility and not part of the OS top -Uusername • k kill program (process) k PID, k -KILL PID • h help • q quit top

47. Editors There are a number of editors for Linux platforms most are not as easy to use as the Windows based editors. Our recommendation is; Use nedit if you are using XWindows ‘e.g. Exceed’, cygwin. Use vi or vim if you have a text terminal ‘telnet’. • pico Easy ‘graphical’ , rather basic. • nedit Easy ‘graphical’ , good. • vi, vim Require some knowledge, vim is like vi but more and includes help • emacs Have faithful following, good once learned.

48. Vi Text Editor • Very powerful • Available on all versions of UNIX • No in-built help • Man page gives detailed help • Vi has three modes • Command • Text entry • Last line

49. Vi Text Editor • Very powerful • Available on all versions of UNIX • No in-built help • Man page gives detailed help • Vi has three modes • Command • Text entry • Last line

50. T-Shell - Filename Completion • Complete a partially typed filename • Operation • Type enough characters to uniquely identify the name • Press the ‘Tab’ key (for C – shell use ‘Esc’ key) • If the response is a ‘bleep’ press Ctrl-d to list possible matches • Setup • Add the following to your .cshrc file • set filec