2.0.0.3.3 Introduction to the Shell – Session 3

1. 2.0.0.3.3 Introduction to the Shell – Session 3 • Job control • Start, stop and monitor running processes • Monitoring disk usage • Command line tools • grep, cut, sort, uniq, paste, fold, tr, head, tail… • Command line Perl • When command line tools don’t quite do it Introduction to the Shell – Session 3

2. Seeing Running Processes: top • See a constantly updated list of what’s running with • Sorted by CPU use • Also shows available memory and processors • Processes can be controlled based on the PID (process ID) Overall CPU use Overall memory use Process CPU use Process memory use top I’m not running all that! Many processes are run by the OS (as user “root”) in the background. These processes generally have very low PIDs. Introduction to the Shell – Session 3

3. Seeing Running Processes: ps • See a complete list of what’s running with • By default only shows what you are running (no root or other user’s processes) • Can select which processes to show: eg. U by user p by PID r only running processes • See full list of everything running with ps auxw ps Introduction to the Shell – Session 3

4. Seeing Running Jobs: jobs • See currently running jobs with • These are processes that the current user is running in this terminal (does not show other user’s processes, or jobs started from other terminal windows) • Useful for job control…. jobs Introduction to the Shell – Session 3

5. Putting Processes in the Background: &, bg and fg • What if you’re running a long job (eg. BLAST), but you want to use your prompt at the same time? • Put the job in the background by starting it with <command> & • Or, once the job is running, stop it with Cntl-Z (interrupts but does not kill), then use • To put the job back in the foreground (and lose your prompt), use • To specify which job to fg or bg, use the PID or %<job id> bg fg Job ID Introduction to the Shell – Session 3

6. Ending Jobs: Cntl-C, kill • To end (without saving anything) a job currently running in the terminal, use Cntl-C • To kill a job that is running in the background or from another terminal, use • If it really won’t die, try kill -9 but only as a last resort! kill Killing applications If you are running applications (eg. word processors, web browsers) from the command line, do NOT kill them unless you cannot exit the application normally – this can corrupt files or cause application errors. Introduction to the Shell – Session 3

7. Sharing the CPU: nice • Jobs are assigned priority (the lower the number, the higher the priority) which determines sharing of CPU • Set the priority of your job lower when running intensive jobs so that others can use the CPU too with • Or, for running jobs, with nice renice Niced Priority Introduction to the Shell – Session 3

8. Memory and Swap • Nice does not control memory use • When running memory-intensive jobs, avoid requiring swap (virtual memory – actually disk space) as this will slow all jobs on that computer Introduction to the Shell – Session 3

9. Checking Disk Location and Usage: df • To confirm that the disk you are on has enough space for you to create large files: -h human-readable df Introduction to the Shell – Session 3

10. Determining Disk Space Usage: du • To check total space taken by a directory structure, use • Find out what is taking up all your disk space! -h human-readable -c produce total --max-depth only print totals for this deep in directory tree du Saving space Cut down on disk usage by using du and find to find large, old files, and use gzip to compress them. Introduction to the Shell – Session 3

11. Working with Text • Simple text processing is a large part of bioinformatics • Converting between program formats • Data processing • Command line tools are a fast and easy way to do a lot of text processing • Usually faster than Perl • Don’t require any coding! Introduction to the Shell – Session 3

12. Searching for Text: grep • Search for text in a file with • Prints matching lines • Can do regular expression matching (slightly different from Perl) • Useful options: -i case-insensitive -w text must appear as a “word” (surrounded by white space) -v find non-matching lines -c count matching lines -B <#> print # lines before matching lines -A <#> print # lines after matching lines -f <file> find lines matching patterns listed in <file> grep Introduction to the Shell – Session 3

13. Sorting Text: sort • Sort a file using • Useful options: -f case-insensitive -g sort numerically (i.e. 11<100) -r reverse sort -k <#> sort on column # -u only print one copy of unique lines sort Introduction to the Shell – Session 3

14. Working with Columns: cut and paste • Get particular columns from a file with • Have to specify which field (column) to get • Useful options: -f <#> fields to output -d field delimiter • Put columns back together with cut paste Introduction to the Shell – Session 3

15. Finding Unique or Repeated Lines: uniq • Determine unique lines with • Collapses repeated lines • Useful options: -i case insensitive -c count copies of each line -d print only duplicated lines -u print only unique lines -f <#> do not compare the first # fields uniq Pipe it! The UNIX text tools are particularly powerful in combination – use pipes to do a series of processing commands on your text. Introduction to the Shell – Session 3

16. Counting: wc • Count lines, words, and characters with • Good for checking if files have correct number of records • Useful in combination with other commands wc bytes (characters) lines words How many SAGE tags match more than one gene Introduction to the Shell – Session 3

17. Replacing Characters: tr Capitalize • Replace characters with • Like Perl’s tr/// • Useful options: -d delete -s replace repeats with single occurrence -c use complement (replace everything EXCEPT what is specified) tr Format comma-separated list Input to tr Instead of reading from a file, tr reads from standard input. If you want it to read from a file, redirect that file to standard input with “<“, or use cat and pipe the output to tr. Introduction to the Shell – Session 3

18. Formatting Text: fold • To wrap long lines in a file, use • Good for formatting before printing text files • Useful options: -w width to wrap to -s break at spaces fold Introduction to the Shell – Session 3

19. Getting Parts of Files: head and tail • Print the first 10 lines with • Print the last 10 lines with • Specify how many lines to get with head <#> eg. head -100 -> gets first 100 lines • Good for making smaller “test” files head tail Make a test file Divide a file into three Introduction to the Shell – Session 3

20. Comparing files: diff • Find differences between lines in files with • Can also compare directories • Useful options: -b ignore differences in white space -i ignore changes in case diff Confirm file copying Introduction to the Shell – Session 3

21. More advance processing: sed and awk • Simple programming languages • Designed for command-line use • Good for more advanced text processing • But I prefer…. Introduction to the Shell – Session 3

22. Command Line Perl • Perl can be run on a script, or directly on text input at the command line (a very short Perl script that does not have to be saved to disk) • Input file with cat or < • Options to use command line: -e find code on command line not in file -n run command line script on each line of input • Check out other Perl command line options with man perlrun “What?” ‘Quotes?’ When running Perl on the command line, use single quotes (‘) so that the shell does not try to interpret the Perl commands. Within these quotes, you can then use double quotes (“) for eg. print statements. Introduction to the Shell – Session 3

23. 2.0.0.3.3 Introduction to the Shell – Session 3 • Now you know…. • How to control your processes • How to keep up on system usage • How to play with text on the command line • How to run Perl from the command line Introduction to the Shell – Session 3

24. 2.0.0.3.3 Further Readings • Job control • Linux Cookbook Ch. 4.3 • Learning the UNIX Operating System Ch. 6 • Command line goodies • Linux Cookbook Ch.13.1, 15.1-15.4 • Command line Perl • Linux Cookbook Ch. 13.5, 15.5 Introduction to the Shell – Session 3