Unix Filters

1. Unix Filters Commands with a dual purpose

2. Objectives • Reinforce Directory and File Commands • Review Redirection • Introduce Filters: • head • tail • more • less • sort • uniq • wc • tee • cut • Introduce Pipes

3. Filters • Filter commands – Unix commands that serve dual purposes: • standalone • used with other commands and pipes • Why are filters important? • The output from a command may be overwhelming, e.g. recall the output from the: # command to list all of the files in the /home directory ls –l /home filters may be a way of reducing output to only the pertinent data • We may wish to organize the data presented. • We may wish to reformat the data presented.

4. Definition of a Filter • A filter is a program that: • receives input from the standard output (stdout) of a Unix command, typically as text; • transforms that output, according to a parameterized set of user guidelines; and, finally, • writes that output to the standard output (stdout) or possibly to another filter (through redirection).

5. Filter Commands • Filter commands include: • head • tail • more • less • sort • uniq • wc • tee • cut • grep* • awk* * to be covered later

6. Filters can be used with or without Pipes • Examples: • with: ls | wc –l counts the number of files in the directory • without wc –l abc.txt Counts the number of lines in the file abc.txt

7. Pipes and Filters pipe Unix Command filter output • Example: ls | wc –l • This particular command: • gets a list of the files in the directory • sends this list through the pipe • the filter counts the lines of the data received • The result is equivalent to: The number of files in the directory

8. What do pipes do? Using the example: ls | wc –l to do this without a pipe would require something like this: ls >tempfile # list contents of dir, redirect to tempfile # count the lines in tempfile (i.e. tell me the number of files) wc –l tempfile rmtempfile # delete the tempfile

9. Using cat with pipes • Avoid using cat with pipes unless the cat command will transform the data through the use of options! • Good: cat –n abc | more # the cat adds line numbers to the listing of file abc • Bad: catabc | more # the cat does not alter the listing at all. In this case simply use more as a command as in: moreabc

10. Redirection Examples cat <abc >def3 cat >abc cat >>abc cat abc1 cat <abc2 1.This assumes the file abc does exist. 2.This example is merely a demonstration of redirecting from a file. It is the equivalent of: cat abc 3. This is equivalent to: cp abc def

11. Redirection • Redirection deals with changing the source or destination of the standard input and output. • Standard (I/0): • input : the keyboard • redirect is the less than (<) symbol • output: the computer screen • redirect is: • greater than symbol (>) to create a new output file (if file exists writes over it) • two greater than symbols (>>) append to existing file or if not an existing file create a new one

12. Pipes versus Redirection • Pipes (|) send output from a Unix command to another Unix command (filter) • Redirection: • output (>, >>) send output of Unix command to a device (usually a file) other than the standard output (the screen) • input (<) provide input to a Unix command from a device (usually a file) other than the standard input (the keyboard)

13. Pipes versus Redirection std out std in no redirection Unix command < > redirection (<,>) device to/from a Unix command disk disk output input Unix command Unix command pipe (|) Unix Command to Unix Command

14. head • head outputs the first part of files • Format: head [options] filename • Selected options: • -cn : print the first n characters of the file • -n : print the first n lines the default is 10

15. head Examples • Command: head –c5 abc # displays first 5 characters (includes whitespace) headabc # displays first 10 lines of abc • Pipe ls | head -15 #displays first 15 lines of file listing

16. tail • tail outputs the last part of files • Format: tail [options] filename • Selected options: • -cn : print the last n characters of the file • -n : print the last n lines the default is 10

17. tail Examples • Command: tail –c5 abc # displays last 5 characters (includes whitespace) tail abc # displays last 10 lines of abc • Pipe ls | tail -15 #displays last 15 lines of file listing headabc | tail -5 # prints lines 6 through 10 of file abc

18. more • more displays files on a page-by-page basis • Format: more [options] filename • Options (selected): • -n : number of lines to show at a time • -d :prompt to continue • +/pattern : begin at pattern • Commands to use while in more: • space : advance a page at a time • return : advance a line at a time • q : quit more

19. more Examples • Command: moreabc # lists file abc one page/line at a time • Pipe: ls –l | more # lists directory contents one page/line at a time

20. less • less displays files on a page-by-page basis • Format: less [options] filename • Options (selected): • -E : quit at EOF (does not require the q) • -N : show line numbers • -ppattern : begin at pattern • Commands to use while in less: • h : help (try it, you’ll like it) • space : advance a page at a time • return : advance a line at a time • q : quit less

21. less Examples • Command: less abc # lists file abc one page/line at a time • Pipe: ls –l | less # lists directory contents one page/line at a time

22. sort • sort sorts lines of text [files] • usually (but not always) used with redirection • Format: sort [options] • Options (selected): • -b : ignore leading blanks • -f : ignore case • -r : sort descending rather than ascending • -M : sort using month order (i.e. Jan comes before Feb) • -n : sort numerical rather than by string • -u : unique, eliminate duplicate lines • -tchar : use char as field separator (instead of whitespace)

23. sort Anomaly • Caution: If multiple, adjacent delimiters are found, only the first delimiter is treated as a delimiter! This includes the default delimiter (whitespace) • For example:

24. sort • When using fields to sort, you may also use characters within a field. • Examples: • sort –t: k+5,5 /etc/passwd (uses colon as field delimiter and does the 5th field only) • sort –k 1.2 shuffled (sorts on 2nd character (and beyond) in first field) • sort –nu myfile (sorts starting with first field, values in numerical order, removing duplicates)

25. Sort Fields • Examples: • sort –k 2 filename # sorts on last name (Smith, Ye, etc…) • sort –t, -k 2 filename # sorts on first name, month, and day (field2 of comma delimited fields) • sort –t: -k 1.8 filename # sorts on 2nd letter of last name (this assumes that the 2nd letter of the last name is in the 8th column of the first field. There is no colon (:) delimiter in the data making the entire line the 1st field

26. sort • When sorting on multiple fields, you may also use the –k pos1[,pos2] options. • If you wish to indicate the format of the key to be sorted, place the format value adjacent to the field number. • Examples:

27. uniq • uniq removes duplicate adjacent lines from a file • To ensure that all of the data is unique you should sort the file first (recall that sort also can produce unique values with the –u option) • Format: uniq [options] filename • Options (selected): • -c : count the instances • -d : print only the duplicates (once) • -u : print only the unique lines • -n : ignore first n fields

28. uniq Examples • Command: uniq –u abc # only prints out the unique lines in file abc assuming duplicates are next to each other (without the –u, uniq will print out all of the lines, but only once, the 2nd or more duplicate lines will not print out) uniq –c abc # only prints out the count of unique lines assuming duplicates are next to each other • Pipe: sortabc | uniq –c # put abc in order and tell me how many lines are unique

29. wc • wcprints the number of newlines, words, and characters (bytes) in files • Format: wc [options] filename • Options: • -c : print the number of characters in the file • -w : print the number of words in a file • -l : print the number of lines in a file • If no options given, wc prints newlines, words, and bytes in files in that order

30. cut • cut removes columns or fields from a line • Format: cut [options] filename • Options: *lists may be specified using integers separated by commas, ranges are separated by hyphens (-) Examples: 1,5-7 means units 1,5,6,7 and 1,5- means units 1,5 and beyond to end of line

31. cut • Examples: cut -d: -f5 /etc/passwd (print only the 5th field, delimited by colon(:), of the /etc/passwd file) cut –d” “ –f2 shuffled (print only the 2nd field, delimited by a space, from file shuffled) cut –c4-8 shuffled (print only the columns 4 through 8 of each line in shuffled)

32. tees • A filter that permits redirection to a file and standard output (screen) • Requires a pipe • Format: cmd| tee outfile where: • cmd is any Unix command that produces standard output • outfile is the file receiving the output data

33. tee Examples • Filter only: ls –al ~ | tee dirlist # displays a long listing of all the files in your home directory and write the same to the dirlist file. ls –al ~ | tee dirlist | sort –r # displays a long listing of all the files in your home directory in reverse order, but saves the listing in order to the file dirlist.

34. Pipes and tee tee disk Unix Command pipe screen

35. Review

36. Unix Filters Commands with a dual purpose