Introduction to Unix, GrADS , and the Snow cluster

1. Introduction to Unix, GrADS, and the Snow cluster ATM 419/563 Spring 2019 Fovell http://www.atmos.albany.edu/facstaff/rfovell/NWP/

2. Log onto the Snow cluster headnode • ssh -Y yournetid@headnode.rit.albany.edu

3. Some Unix basics(basis of Linux and Mac OS X) • The command prompt is $ (don’t type prompt) • Unix commands presented in Courier font $ ls [lists your “visible” files] $ ls -a [lists all files, including invisible] $ ls -al [lists all files, includes size & other info] $ pwd [print working directory - shows your location]

4. Some Unix basics(continued) $ mkdir TEST [makes a directory called “TEST”] $ cd TEST [changes directory to “TEST”] $ cd .. [moves UP in directory hierarchy] $ cd TeST [command should FAIL… Unix is usually CASE SENSITIVE – though it usually isn’t on Mac]

5. Some Unix basics(continued) $ touch crap.txt [makes an empty file] $ nedit crap.txt [invoked Notepad-like text editor. Another is gedit] [add some text, save file, and exit] $ ls -al crap.txt [see it’s there] $ cp crap.txt crap.txt.backup [make a copy with cp]

6. Some Unix basics(continued) $ rm -i crap.txt [the remove command. Asks for confirmation. Don’t type ‘y’.] $ \rm crap.txt [removes without confirmation. Think before hitting ENTER!] $ mkdir BACKUP [makes a new directory] $ cp crap.txt.backup BACKUP/. [type exactly as written - there is NO space before the period] [the period means “keep the same name”]

7. Some Unix basics(continued) $ ls -l BACKUP/ [see a copy is in the new location] $ mkdir BACKUP/more_backup [a new subdirectory in a directory] $ cd BACKUP/more_backup [move to new subdirectory] $ cd $ pwd [cdby itself always takes you to your home directory]

8. Some Unix basics(continued) $ cp /network/rit/lab/atm419lab/DATA/part* . [asterisk is wild card character. There IS a space before the period] [this copies all files matching text pattern to your directory] $ ls part* [lists files matching pattern] $ cat part1 part2 > both_parts [concatenates files] [greater-than sign is output redirect] $ more both_parts [more lists contents, one screen at a time] [hit space bar for next screenful, type “q” to quit]

9. ls vs. ls -a • By default, Unix hides files and directories that start with a period. Compare these commands: $ ls $ ls -a One of these hidden directories is .snapshot, which contains your automatic backups.

10. Your Unix PATH $ cp /network/rit/lab/atm419lab/DATA/execute_me . $ execute_me - if you receive execute_me: command not foundyour PATH needs changing - The PATH is an environment variable that tells Unix where to find stuff, and the order in which to look $ ./execute_me - this forces Unix to look in the current directory for the program to run

11. Fortran compiler • Try this command $ which ifort • If you get this… we definitely need to alter your Unix environment $ ifort: Command not found

12. Which shell? $ ps -p $$ PID TTY TIME CMD 16511 pts/32 00:00:00 tcsh Unix/Linux supports several “shells” (environments that interpret commands): bash, sh, csh, tcsh, ksh If your shell is NOTtcsh or bash, let me know.

13. Preparation for new tcsh users ONLY $ cd $ mv .cshrc .cshrc_old [mv is move command] [.cshrc is script file of commands, paths, etc., executed for every new terminal window] $ cp /network/rit/lab/atm419lab/SOFTWARE/.cshrc . [type exactly as shown. Final period is preceeded by a space.] $ source .cshrc [changes to script file in present window do not take effect until “sourced”] $ which ifort [see: /network/daes/linuxapps/intel17/bin/ifort] $ which grads [see /network/rit/lab/atm419lab/SOFTWARE/grads-2.1.0/bin/grads]

14. Preparation for new bashusers ONLY $ cd $ mv .bash_profile .bash_profile_old [mv is move command] [.bash_profile is script file of commands, paths, etc., executed for every new terminal window] $ cp /network/rit/lab/atm419lab/SOFTWARE/.bash_profile . [type exactly as shown. Final period is preceeded by a space.] $ source .bash_profile [changes to script file in present window do not take effect until “sourced”] $ which ifort [see: /network/daes/linuxapps/intel17/bin/ifort] $ which grads [see /network/rit/lab/atm419lab/SOFTWARE/grads-2.1.0/bin/grads]

15. Home and lab space Your home directory is /network/rit/home/yournetid Your lab directory will be /network/rit/lab/atm419lab/yourlastname

16. $ cd /network/rit/lab/atm419lab [cd $LAB] $ pwd $ mkdir yourlastname $ cd yourlastname $ pwd $ nedit ~/.cshrc OR ~/.bash_profile [tilde is shorthand for your home dirrectory] • The very first line is (no equals sign for .cshrc): alias lab="cd /network/rit/lab/atm419lab/” • Change this to (no equals sign for .cshrc) alias lab="cd /network/rit/lab/atm419lab/yourlastname” • Save file and quit [DO NOT change the second line...] $ resource [executes .cshrc/.bash_profile again] $ lab $ pwd please use lower case

17. Using GrADS[Grid Analysis and Display System] Example: WRF 2D sea-breeze simulation

18. GrADS • GrADS uses two files, a binary data file (*.dat or *.bin) and a control file (*.ctl) $ lab $ mkdir EXAMPLE $ cdEXAMPLE $ cp $LAB/EXAMPLE/sb*.* . [single line; note two wildcards being used] $ grads -l [starts up GrADS; ga-> is GrADS prompt] ga-> open sb_lat30 ga-> query file $LAB is defined as “/network/rit/lab/atm419lab/”

19. http://cola.gmu.edu/grads/gadoc/gadocindex.html Getting to know GrADS • Important commands open [file] open my_experiment q or query q file, q dims d or display d theta, d qvapor c or clear c set gxout contour, shaded, print, scatter, barb d u;v [vectors, barbs, or scatterplot set t 13 [sets to 13th time in file set cint 1.5 [contour interval set ccolor 4 [contour color blue gxprintout.png [makes a plot in PNG format set display white [makes white background quit [quits GrADS

20. GrADS control file ga-> q file File 1 : Descriptor: sb_lat30.ctl Binary: sb_lat30.dat Type = Gridded Xsize = 201 Ysize = 1 Zsize = 34 Tsize = 25 Esize = 1 Number of Variables = 50 u 34 0 U Compoment of wind v 34 0 V Component of wind w 34 0 W Component of wind theta 34 0 Theta tc 34 0 Temperature in C p 34 0 Pressure (HPa) z 34 0 Height (m) [more…] • partial contents of sb_lat30.ctl shown

21. ga-> set display whitega-> clearga-> set t 19ga-> d u

22. ga-> cga-> set lev 0 1.5ga-> set x 150 175ga-> d u heights, in km grid points

23. ga-> d w

24. ga-> sb_plotuw.gs • changed aspect ratio • shaded plot for w • plot color bar • relabeled x, z axes • provided a plot title

25. sb_plotuw.gs 'set display white' * set plot aspect ratio 'set vpage 0.5 11.0 0.5 5’ 'clear' 'set grads off' * get rid of map projection 'set mproj off’ * rgbset.gs defines new colors 'run rgbset.gs' * subset domain 'set lev 0 1.5' 'set x 150 175' * X label assumes dx=2 km - wrong otherwise 'set xaxis 300 350' 'set yaxis 0 1.5 0.25’ [continues…]

26. sb_plotuw.gs * shaded plot for W 'set gxout shaded' * set shading levels and assign colors 'set clevs -.15 -.12 -0.09 -0.06 -0.03 0 0.03 0.06 0.09 .12 .15' 'set ccols 49 47 45 43 41 0 61 62 63 65 67 69' 'd w' * draw a colorbar 'cbarn 1.0 0 6.0' * contour plot for U 'set gxout contour' 'set cthick 5' 'set cint 0.3' 'set ccolor 1' 'd u' 'draw ylab z(km)' 'draw title U and W'

27. ga-> sb_plotuw.gs ga-> set ccolor 4 ga-> d u;w*10

28. Stepping through timesga-> set x 1 201 ga-> step.gs u 2 25 1.0 Plots field u …for steps through times 2 through 25, inclusive …with contour interval 1.0 …hit return to proceed to the next time step.gs is a script I put in your GRADS library path via .cshrc/.bash_profile

29. ga-> cga-> set x 200ga-> set t 1 25ga-> set z 1ga-> d swdown Plotting a time series Model time is fake for idealized WRF (here, 00Z = sunrise) Page formatting has been preserved

30. ga-> draw title Solar radiation at surface, x = 200 ga-> draw xlab time ga-> draw ylab Downward SW (W/m^2)

31. ga-> cga-> d t2 2-m temperature not initialized @ first time, so it is zero.

32. ga-> cga-> set vrange 280 305ga-> d t2 More elegant approach: use “maskout”

33. Three important commands ga-> reset • resets GrADS environment to starting conditions… Time dimension set to 1st time, gxout to contour, removes page formatting, etc.. Does not close open files. ga-> reinit • resets AND closes all open files. “All but quit.” ga-> quit

34. ga-> resetga-> set mproj offga-> set z 1ga-> set t 1 lastga-> set xaxis 0 400 100ga-> d u Plotting a Hovmoller diagram coarse time resolution (hourly) noisy zero contours coast

35. From a file with higher temporal resolution Suppressed zero contour with set black 0 0 finer time resolution (10 min) coast

36. ga-> c ga-> set grads off ga-> d u;v U, V vectors at z=1

37. ga-> c ga-> set arrscl 0.5 6 ga-> d skip(u10,5,1);v10

38. ga-> set cmin 280 ga-> set cmax 320 ga-> set cint 2 ga-> set ccolor 2 ga-> d t2

39. GrADS default color sequence If display color=black 0 is black 1 is white If display color=white 0 is white 1 is black http://cola.gmu.edu/grads/gadoc/16colors.html

40. Dimensions of various fields(from the .ctl file) 34 height levels for atmospheric variables (z = 1 is lowest level) TD 340 Dewpoint Temperature (diagnostic) RH 340 Relative Humidity (diagnostic) TSLB 50 SOIL TEMPERATURE T2 00 TEMP at 2 M 5 depth levels for soil (z = 1 is highest level) “0” means field has one level

41. Soil layers in slab LSM model

42. ga-> resetga-> set mproj offga-> set z 1ga-> set t 1 25ga-> set x 200ga-> d tslb(z=1)ga-> d tslb(z=2)ga-> d tslb(z=3)ga-> d tslb(z=4)ga-> d tslb(z=5) Layer 1 (0.5 cm depth) Layer 5 (23 cm depth)

43. 0.5 cm Soil temperatures vs. depth in Nebraska during summer (Kuo 1968) 40 cm

44. Multiple files with GrADS ga-> reinit ga-> open sb_lat00 ga-> open sb_lat30 ga-> open sb_lat60 ga-> open sb_lat90 ga-> set mproj off When multiple files are open, append file number to variable name: e.g., d u.2

45. ga-> set z 1 ga-> set x 200 ga-> set t 1 25 ga-> d swdown.1 ga-> d swdown.2 ga-> d swdown.3 ga-> d swdown.4

46. ga-> set lev 0 1.5 ga-> set x 1 201 ga-> set t 13 ga-> d p.2-p.3 q files lists names of open files

47. horizontal velocity u at 6pm coast sb_plot4.gs uses vpage to put four plots on one page NOTE: Contour intervals vary among plots

48. More on GrADS image output ga-> gxprint image01.pngga-> gxprint image01.ps ga-> gxprint image01.pdf [Quit out of GrADS with “quit” command] (jpg, gif are no longer supported...)

49. Using srun $ hostname [by default, your commands headnode.rit.albany.edu run on headnode– not optimal] $ srun hostname [srun commands run on student cc1-12.rit.albany.edu cluster “cc”] $ srun –p snow hostname [run on general Snow cluster] snow-10.rit.albany.edu $ srun –p snow-nwp hostname [run on class subcluster] snow-16.rit.albany.edu Some information: https://www.rit.albany.edu/wiki/SLURM

50. System/job information $ sinfo snow-vis up 12:00:00 15 alloc snow-[01-05,16-19,24,27-31] snow-vis up 12:00:00 14 idle snow-[06-15,25-26,32-33] snow-dbg up 2-00:00:00 15 alloc snow-[01-05,16-19,24,27-31] snow-dbg up 2-00:00:00 14 idle snow-[06-15,25-26,32-33] snow up 12:00:00 15 alloc snow-[01-05,16-19,24,27-31] snow up 12:00:00 14 idle snow-[06-15,25-26,32-33] snow-1 up 4-00:00:00 15 alloc snow-[01-05,16-19,24,27-31] snow-1 up 4-00:00:00 14 idle snow-[06-15,25-26,32-33] snow-gst up 2-00:00:00 15 alloc snow-[01-05,16-19,24,27-31] snow-gst up 2-00:00:00 14 idle snow-[06-15,25-26,32-33] student* up 14-00:00:0 8 drain* cc1-[01-05,07,09,11] student* up 14-00:00:0 3 down* cc1-[10,14,16] student* up 14-00:00:0 2 idle cc1-[12-13] snow-nwp up infinite 1 alloc snow-20 snow-nwp up infinite 3 idle snow-[21-23] $squeue –u yournetid