Introduction to the School of Earth Sciences UNIX computer network

1. Introduction to the School of Earth SciencesUNIX computer network Kevin Keay February 8 2010 Presented by Ailie Gallant

2. Outline • Course materials • What is UNIX? • Our UNIX network • A brief discussion of why we use UNIX • A taste of UNIX • A basic overview • A typical UNIX task • The University of Surrey, UNIX Tutorial for Beginners: Selected topics • An Introductory Tutorial on UNIX (a lab session)

3. Course materials • Handout: An Introductory Tutorial on UNIX which is also at: http://www.earthsci.unimelb.edu.au/~kevin/UNIX_Course/An_Intro_Tute_on_UNIX_Feb-2009.pdf • Notes: Introduction to the School of Earth Sciences UNIX computer network at: http://www.earthsci.unimelb.edu.au/~kevin/UNIX_Course/Intro_to_SES_UNIX_computer_network.pdf • A useful online resource from the University of Surrey, UNIX Tutorial for Beginners, is available at: http://www.ee.surrey.ac.uk/Teaching/Unix/

4. What is UNIX? • UNIX is a computer operating system • Well suited to multi-tasking and multiple users • Powerful command-line and scripting capabilities • Graphic user interface: X Windows • Easily accessed from Microsoft Windows (PC) or X Darwin (iMac) as well as a native UNIX machine • Important: UNIX is case-sensitive e.g. nedit is different from NEDIT or Nedit

5. What is UNIX? (2) • UNIX was developed initially by Bell Labs circa 1970 and has grown into an open software project with many world-wide collaborators. A history is given at: http://www.bell-labs.com/history/unix/ • There are many variants e.g. Linux, Mac OS X. • Cygwin for Windows is a UNIX emulation i.e. looks and feels just like UNIX.

6. What is UNIX? (3) • After three decades of use, the UNIX computer operating system from Bell Labs is still regarded as one of the most powerful, versatile, and flexible operating systems (OS) in the computer world. Its popularity is due to many factors, including its ability to run on a wide variety of machines, from PCs to supercomputers, and its portability - all of which led to its adoption by many manufacturers.

7. Our UNIX network • (1) Solaris OS:atlas, mawson, orthus, rossby • (2) Red Hat Linux OS:cove, gulf, gyre, tide, wave • (3) Debian Linux OS: vislab01, vislab02, vislab03, vislab04 • (4) Red Hat Enterprise Linux OS: abyss • (OS means operating system). The single machine in group 4 (abyss) is the newest and fastest. At this time abyss is the only machine you can access. The OS upgrade of the machines in group 3 to Ubuntu will occur soon. Groups 3-4 are recommended for most uses. Apart from atlas and orthus, the group 1-2 machines are relatively slow. So, normally you would select a Linux machine in group 3 or 4 unless you need to use the Solaris OS. Also, see section 1.1 of the notes Introduction to the School of Earth Sciences UNIX computer network.

8. A brief discussion of why we use UNIX • UNIX machines are generally better at multi-tasking than MS Windows • Very stable platform from a user standpoint: difficult to crash! • Compilers (Fortran, C) allow the user to development their own software or implement programs written by others • There are many sophisticated commercial packages available e.g. Matlab

9. A brief discussion of why we use UNIX (2) • Scripts can be used to automate complex sequences of tasks: leads to efficiency and reduction of user errors • Command-line arguments add flexibility and are very useful in scripts e.g. C-shell scripts • There is a global resource of software designed to work under UNIX, much of it is freely available: you don’t have to develop everything from scratch

10. A taste of UNIX • Imagine that you have connected to a UNIX machine e.g. abyss, via ssh from a Cygwin xterm window: ssh –Y STUDENT\\user@abyss e.g. ssh –Y STUDENT\\jsmith@abyss NOTE: Your username must be prefixed by the text STUDENT\\ (For staff the prefix is UNIMELB\\ but this is optional). • You will find yourself located in your UNIX home folder e.g. /home/keay

11. A taste of UNIX (2) • Change to another folder cd /work/keay/Project1 • /work is another disk and the user keay may create folders and files within it • Check that you are in the correct location (optional!) pwd /work/keay/Project1

12. A taste of UNIX (3) • List the contents (files) of this folder ls pmsl.cmp sortdata.f Test.dat test2.dat X.dat • List the files with ‘.dat’ as part of the filename ls *.dat Test.dat test2.dat X.dat

13. A taste of UNIX (4) • Edit the Fortran source file sortdata.f (we are making changes to the existing file) nedit sortdata.f

14. A taste of UNIX (5) • Compile the Fortran source program to create an executable i.e. a piece of software g77 –o sortdata sortdata.f Here the –ooption expects an argument, in this case the filename of the executable (an option could be standalone e.g. –j ) Note: An option usually starts with a ‘-’ • List the files in the folder (sortdata is a new file) ls pmsl.cmp sortdata sortdata.f Test.dat test2.dat X.dat

15. A taste of UNIX (6) • Run the executable (software) that you designed in Fortran sortdata • You might design your program to create an output data file or the results of a computation, even an image

16. A taste of UNIX (7) • Create a plot of a pressure file conmap –k iconkk –B pmsl.cmp < icon • The items with ‘-’ are called options and control the program called conmap. The –k option has an argument – a file called iconkk which contains some additional parameters to control the program • Additional control comes from the instruction file called icon • The output of this command is a file called gmeta that can be converted to a PNG image

17. A taste of UNIX (7)

18. A basic overview • When you connect to a remote UNIX machine on our network you will be working in the C-shell (actually tcsh which is aliased to csh) • C-shell scripts will begin with # or #!/bin/csh -f • There are other kinds of script e.g. sh (Bourne shell – bash under Cygwin is a relative), perl • After connection you will be located in: /home/user where ‘user’ is your UNIX username e.g. /home/keay • Remember: UNIX is case-sensitive

19. A basic overview (2) • Under MS Windows files and folders (directories) are usually relative to the C drive i.e. c: and under UNIX are relative to / Also \ is used to denote the file tree structure under Windows and / under UNIX Hence: c:\home\keay\test.dir\prog.f under Windows would be: /home/keay/test.dir/prog.f under UNIX • You will be working under /home or /work e.g. /home/keay, /work/keay

20. A basic overview (3) • ls is used to list files in a folder e.g. ls readme.txt ls –l pmsl.*.dat • Note that the ls command has many options. See: man ls for more information

21. A basic overview (4) • mkdir is used to create folders (directories) e.g. mkdir test.dir • rmdir is used to remove a folder e.g. rmdir test.dir Note: The folder must be empty i.e. use rm first, thenrmdir • chmod is used to change file access permissions e.g. chmod 755 readcmp or chmod u+rwx,g+rx,o+rx readcmp where u is user, g is group and o is others.

22. A basic overview (5) • rm is used to remove (delete) files e.g. rm test.dat By default rm will remove without warning. Your initial setup will alias rm such that rm == rm –i i.e. you will be prompted to continue with the removal • Prefixing with a \ will override the alias e.g. \rm –rf */* will delete all of your files starting with the current folder – BEWARE!

23. A basic overview (6) • cp is used to copy files to other locations e.g. cp readme.txt /work/keay/src • mv is used to rename files e.g. mv readme.txt Readme.txt Remember: UNIX is case-sensitive • pwd will tell you where you are i.e. your current location (folder)

24. A basic overview (7) • cd is used to change folders (directories) • Assume you are in: /home/keay cd test.dir will change the folder (directory) to test.dir i.e. you are now in: /home/keay/test.dir • cd .. will place you in the folder one level above i.e. in this case: /home/keay cd ~ will always take you ‘home’ i.e. to: /home/user

25. A basic overview (8) cd /home/keay/src will place you in the folder called src cd /work/keay/src/readgrib.dir will place you in the folder readgrib.dir under /work/keay/src • Combine .. and / to change folders e.g. cd ../test2.dir

26. A basic overview (9) • Wildcard characters *,?,- and , in conjunction with various brackets {} and [] are very useful in commands to represent complex sets of filenames e.g. cp pmsl.*.dat /work/keay/DATA This will copy all files of this format where the * stands for anything in that part of the filename.

27. A basic overview (10) ? represents a single character e.g. cp pmsl.{19[6-9]?,2000,2001}.dat /work/keay/DATA This will copy: pmsl.1960.dat,pmsl.1961.dat,…, pmsl.1999.dat,pmsl.2000.dat, pmsl.2001.dat i.e. {19[6-9]?,2000,2001} is expanded to: 1960,1961,1962,…,1999,2000,2001 We could also use: pmsl.{19[6-9]?,200[0,1]}.dat

28. A basic overview (11) • For text editing use nedit on atlas or orthus and nedit or emacs on the Linux machines e.g. nedit prog.f • To transfer files to and from a PC you can use the utility SSH Secure File Transfer Client on the Windows desktop • You can also use scp under Cygwin – see Appendix A of the Notes

29. A basic overview (12) • Some documentation is available with man or info, however it is often cryptic e.g. man csh • Often a Google web search will find more helpful information, examples and tutorials

30. A basic overview (13) Summary • The basic UNIX commands to understand are: ls cd mkdir rmdir rm chmod cp mv pwd csh • Learn how to use wildcards: * ? - , [] {} • Ensure that you can use a text editor: nedit (Solaris OS) neditemacs (Linux) • SSH Secure File Transfer Client (Windows) is useful for transferring files from a PC to a remote UNIX machine Also see: scp

31. A typical UNIX task • Task: Prepare average geographical maps of mean sea level pressure (MSLP) for the months of Jan and Aug using NCEP Reanalysis data and plot the results • The data is available in a widely-used format called NetCDF from the NOAA web site – this site has interactive data selection features • I manually downloaded the relevant files using a web browser (Mozilla) to my UNIX account e.g. /work/keay/mean_MSLP.dir

32. A typical UNIX task (2) • I have written a Fortran program (read_nc2cmp) to decode the NetCDF files to our internal conmap format • This program was based on an example from NCAR so I didn’t have to begin from scratch • The conmap files are then input to my Fortran program statconmap which produces the average maps • Note: The average is performed at each gridpoint of the map for the set of desired maps

33. A typical UNIX task (3) • The average maps (one for Jan, the other for Aug) may be plotted with various software e.g. NCAR Graphics, GrADS, Matlab • One method that is useful in scripts is: (1) run conmap (NCAR Graphics) on an average map; output is a file called gmeta (2) Translate gmeta to a Postscript file using the utility g2ps (3) Use the utility convert to transform the Postscript file to a PNG file – this may be pasted into a Word document

34. A typical UNIX task (4) • In a UNIX C-shell window you can type commands directly e.g. read_nc2cmp MSLP.*.Jan.nc out.cmp statconmap out.cmp ave.Jan.cmp sd.cmp var.cmp conmap –G ave.Jan.cmp • You can also place these commands in a C-shell script

35. A typical UNIX task (5) #!/bin/csh –f # Required first line of a C-shell script or simply # set mons = (Jan Aug) # Array of months @ nmons = $#mons # No. of months, in this case it will be 2 @ i = 1 # @ is used to define numbers as opposed to text (set) while ($i <= $nmons) # Perform loop (to end) while i <= 2 set m = $mons[$i] # m is mons(i) i.e. mons(1) = Jan => m = Jan # Decode NetCDF files for this month (m) # Note: out.cmp contains multiple maps read_nc2cmp MSLP.*.$m.nc out.cmp # Perform average of multiple conmap file out.cmp # Note: Outputs sd.cmp and var.cmp are not used but need to be specified statconmap out.cmp ave.$m.cmp sd.cmp var.cmp # Create global geographical plot of average map e.g. ave.Jan.cmp conmap –G ave.$m.cmp # Output is gmeta # Convert gmeta file to Postscript; creates g.ps (a hardwired output) g2ps gmeta # Convert Postscript file to PNG e.g. MSLP.Jan.png convert –trim –density 120 g.ps MSLP.$m.png # @ i ++ # i -> i + 1 end exit

36. A typical UNIX task (6) • We won’t look in detail at scripts in this introduction but will re-visit them later in the semester • The data processing and display sequence is not unique. Depending on your project you may use other tools e.g. GrADS, NCL, Python • Remember: Practice makes perfect! Experiment … play with the commands and see if things make sense

37. The University of Surrey, UNIX Tutorial for Beginners: Selected topics See: http://www.ee.surrey.ac.uk/Teaching/Unix/ Note: Tutorial 7 is a bit advanced for this introduction

38. An Introductory Tutorial on UNIX • Refer to the handout • Also see: Appendix C An Introductory Tutorial on UNIX of the notes

39. Direct connection to a UNIX machine From the console of a machine in the UNIX Lab or the VisLab: (1) Log on with your UNIX username and password (2) You should be in the KDE desktop. Left-click on the Shell-Konsole icon or Terminal Program icon in the toolbar at the bottom of the screen. This will activate a terminal window (xterm). You may now enter UNIX commands. Remote connection to a UNIX machine via Cygwin From a PC in the PC Lab or elsewhere: (1) Log on with your University username and password (2) Click on the Cygwin desktop icon. This will open a bash shell window. (3) Within this window type: startx & This will open a terminal window (xterm). (4) Within the terminal window type: xterm & to open a new window. From this new window type: ssh –Y STUDENT\\user@machine Note for ‘machine’ choose from: abyss,atlas,orthus,cove,gyre,tide,wave,vislab01,vislab02,vislab03 An Introductory Tutorial on UNIX (2)

40. An Introductory Tutorial on UNIX (3) Initial setup of a UNIX account • Connect to a remote UNIX machine – right now you only have access to abyss: ssh –Y STUDENT\\user@abyss • After logging on: % cp /home/keay/UNIX_Course/zunix_setup.zip . - Note the dot % unzip zunix_setup.zip % ls –l .cshrc.* % exit

41. An Introductory Tutorial on UNIX (4) • Owing to recent changes you can only use abyss: e.g. ssh –Y STUDENT\\jsmith@abyss • Hence run the tutorial on abyss