Intermediate Unix

1. Intermediate Unix Presented July 29th, 2001 by: “Robin” R. Battey (zanfur@cs.washington.edu) Evgeny Roubinchtein (evgenyr@cs.washington.edu) with tips, suggestions, and corrections by:Hannah Tang (hctang@cs.washington.edu) http://www.cs.washington.edu/people/acm/tutorials/

2. The “File System” • Under UNIX, (almost) everything is a “file”: • Normal files • Directories • Hardware • Sockets • Pipes • Things that are not files: • Users • Groups • Processes

3. Ownership • Files have two owners • Every file has exactly one user owner • Every file has exactly one group owner • Everyone is a user • Users are in at least one group • Processes have owners, too (known as an “id”) • Every process has exactly one user id • Every process has at least on group id • Users and groups are really just numbers with names • Every username is mapped to a single numeric “uid” • Every groupname is mapped to a single numeric “gid”

4. Who am I? • Commands that tell you who you are: • whoamidisplays your username • iddisplays your username and groups • Commands that tell you who others are: • finger [<name>]displays info for <name> • id [<username>]displays info for <username> • Commands that change who you are: • su <username>“switch user” to <username> • login login as a different user

5. File Permissions • Every file has three access levels: • user (the user owner of the file) • group (the group owner of the file) • other (everyone else) • At each level, there are three access types: • read (looking at the contents) • write (altering the contents) • execute (executing the contents)

6. Strange Things • There are three “strange” permissions: • setuid (run program as user owner) • setgid (run program as group owner) • text (stay in swap after executing) • Directories act differently • “write” (creating/deleting files) • “execute” (cd-ing to that directory) • “setuid” (ignored) • “setgid” (created files have same group owner) • “text” (deletion restricted to user owned files)

7. Examining Permissions • A “long” ls listing shows file permissions: [zanfur@odin zanfur]$ id uid=8774(zanfur) gid=100(users) groups=100(users),101(mp3) [zanfur@odin zanfur]$ ls -l total 524 -rwxr-xr-x 1 zanfur users 512668 Jul 31 00:18 bash prw-r--r-- 1 zanfur users 0 Jul 31 00:24 fifo -rw-r--r-- 1 zanfur users 0 Jul 31 00:18 file drwxr-xr-x 2 zanfur users 4096 Jul 31 00:16 normal drwxrws--T 2 zanfur mp3 4096 Jul 31 00:14 shared drwxrwxrwt 2 zanfur users 4096 Jul 31 00:14 tmp drwxrwxr-x 2 zanfur www 4096 Jul 31 00:15 www [zanfur@odin zanfur]$ _

8. Permissions Listing Breakdown

9. What You Can Do With Permissions

10. Changing Ownership • Changing user ownership: • The command is “change owner”: chown <username> <filename> • but, you can only do it if you are root • so just copy it instead • Changing group ownership: • The command is “change group”: chgrp <groupname> <filename> • but, you can only do it if you are in group <groupname> • and you must be the user owner of the file

11. Changing Permissions • The “change mode” command: chmod <level><op><permissions>[,…] <filename> <level> string of: u, g, o, a (user, group, other, all) <op> one of +, -, = (gets, loses, equals) <permissions> string of: r, w, x, s, t, u, g, o (read, write, execute, set-id, text, same as user, same as group, same as other), • Examples: chmod u+rwx,go-w foobar chmod g=u,+t temp/ chmod u=rwx,g=rwxs,o= shared/

12. Process Management • What can you do with it? • Start programs in the background • Run more than one program per terminal • Kill bad and/or crashing programs • Suspend programs mid-execution • List all jobs running in a shell • Move foreground jobs to the background • More …

13. Three States of a Process • Foreground • Attached to keyboard • Outputs to the screen • Shell waits until the process ends • Background, running • Not attached to keyboard • Might output to the screen • Shell immediately gives you another prompt • Background, suspended • Paused mid-execution • Can be resumed in background or foreground

14. Background Processes • Listing jobs: • jobs lists background “jobs” and job #’s • ps lists processes and their process id (“pid”) • %<job#> expands to the process id of the job • Stopping foreground jobs • Press ^Z (Ctrl-Z) in the terminal window • Starting a process in the background • Append a & character to the command line • Examples: ls –lR > ls-lR.out & xemacs my_program.cc & • Resuming a stopped job • In the foreground: fg [<pid>] • In the background: bg [<pid>]

15. Killing Processes • The “kill” command: kill [-<signal>] <pid> Send <signal> to process <pid> • The “killall” command: killall [-<signal>] <command> Send <signal> to all processes that start with <command> • Useful signals (kill –l for the complete list): TERM the default, “terminate”, kills things nicely KILL will kill anything, but not nicely HUP “hangup”, used to reload configurations STOP stops (suspends) a running process

16. What are environment variables? • The environmentis an array of strings of the formNAME=VALUE • Each process (program) has its own copy of the environment • Processes started by the shell get a (“deep”) copy of the shell’s current environment • Each process can examine and modify its own (and only its own) environment • The “meaning” of environment variables is merely a matter of established conventions

17. Viewing your shell’s environment • To view a single environment variable’s value:echo $<name> • For example:echo $HOMEwill display/homes/iws/evgenyr • To view all environment variables at once: • tcsh/csh/bash: printenv • sh/ksh: set

18. Setting environment variables • tcsh/csh: • setenv <name> <value> • Example: setenv PRINTER ps329 • bash/ksh: • export <name>=<value> • There is no space on either side of ‘=’! • Example: export PRINTER=ps329 • sh: • <name>=<value>; export <name> !

19. Appending to environment variables • Appending /uns/bin/ to your PATH • bash/ksh: export PATH=$PATH:/uns/bin • tcsh/csh: setenv PATH=$PATH:/uns/bin • Prepending is similar: • bash/ksh: export INFOPATH=/uns/info:$INFOPATH • tcsh/csh: setenv INFOPATH /uns/info:$INFOPATH

20. Common environment variables • The “meaning” of environment variables is purely a matter of convention. • However, these environment variables are quite common: • PATH, INFOPATH, MANPATH, EDITOR, VISUAL, PAGER, HOME, MAIL, USER • The man page for a program will usually list the environment variables the program pays attention to.

21. Useful shell features (a preview of upcoming attractions) • Aliases • Redirecting input and output • Command substitution • Scripts

22. Shell as a work-saver: aliases • Aliases: textual substitution, similar to C-preprocessor macros. • Syntax: • bash/ksh: alias <text>=’<replacement>’ • tcsh/cshalias <text> ’<replacement>’ • When you type <text>, the shell “substitutes” <replacement> • Typing alias by itself lists all your current aliases • unalias <text> removes the alias • Check the csh/tcsh man page for extra features

23. Line continuation character, just like in C/C++! Alias examples • Always print postscript files double-sided • tcsh/csh:alias lpr ’lpr –Zduplex’ • bash/ksh:alias lpr=’lpr –Zduplex’ • Collect a few tree-friendly options to enscript: • tcsh/csh:alias print \ ’enscript –2rhB –SDuplex=DuplexNoTumble’

24. What is input/output redirection? • Normally, a program’s standard output is displayed on user’s terminal, and its standard input comes from the keyboard. • Redirecting the output of a program means asking the shell to put the program’s output (stdout [C++’s cout]) into a file. • Redirecting the input of a program means asking the shell to feed a file as the program’s standard input (stdin [C++’s cin]). • Note: redirection works with files.

25. Why redirect program’s output? • You may want to save output and examine it at your leisure: • if the program generates a lot of output • if the program takes a long time to run • You may want to present the output to another person • Having the program write to standard output may make the program simpler to write

26. Standard output vs Standard error • By convention, “normal” output of a program is sent to standard output (stdout [C++’s cout]), while debugging or error output is sent to standard error (stderr [C++’s cerr]).

27. How to redirect program’s output? • To redirect just the standard output:<program> > <FILE> • To redirect just the standard error:sh/ksh/bash: <program> 2> <FILE>csh/tcsh: ( <program> > STDOUT ) >& STDERR • To redirect both standard output and standard error: csh/tcsh/bash: <program> >& <FILE>sh/ksh/bash: <program> > <FILE> 2>&1

28. > vs. >> • Both > and >> will create the output file, if it doesn’t already exist • If the file does exist, then: • Using > to redirect output will overwrite the output file: • ls > newlisting • printenv > my_environment • Using >> to redirect output will append to the output file • cat ch1 ch2 ch3 > book • cat ch4 ch5 ch6 >> book

29. Why redirect program’s input? • To run the program repeatedly with the same (or similar input) • Having the program read from standard input may make the program simpler to write.

30. How to redirect program’s input? • Simple!<program> < <FILE> • Examplesort < my_grades.txthead < really_long_book.txt

31. Piping • Piping is connecting programs together by using the output of one program as the input to the next. • Syntax:<program1> | <program2> | … | <programN> • A simple example (view a sorted file-listing a page at a time):ls | sort | less • Note: piping deals with the input/output of programs (that is, stdin, stdout, stderr)

32. Why piping? • Because “the whole is bigger than the sum of its parts. • By combining Unix utilities in a pipeline, you can build tools “on-the-fly” as you need them.

33. Piping examples • How many .c files are in this directory?ls *.c | wc –l • What files were modified most recently?ls –t | head • What processes am I running?ps auxw | grep <mylogin> • Make an alias for the above, for other Linux boxen too:alias myps=’ ps auxw | grep `id –un`’

34. Thecatutility • Especially useful: cat • When used with pipes, “copies” stdin to stdout • cat can be used to redirect out of a pipe! • Example: make a file containing currently running processesps aux | grep evgenyr| cat > my_processes • When used with a file, “copies” the file into stdout • cat can be used to place a file’s contents into a pipe! • Example: list all the items in a list in alphabetical ordercat my_grocery_list.txt | sort | uniq

35. Command substitution(aka “what’s up with the backquotes?”) • Command substitution means that the shell substitutes a command’s output for the command itself. • To get the shell to perform command substitution, enclose the command in backquotes (`)

36. Shell as a work-saver: scripts • Instead of typing the same series of commands over and over again, put them in a file and have the shell execute the (commands in the) file! • The file must have execute permission • The first line of the file should be:#! <YOURSHELL> (e.g., /bin/bash) • There must be no space (or any other character) between ‘#’ and ‘!’. • Whether to put space after ‘!’ is a matter of style • Shell also has control flow, tests, etc. The shell tutorial on the ACM web page goes into much more detail.

37. Intermezzo: quoting • Problem: some characters are special to the shell (*, ?), but you would like to pass those characters to the programs the shell runs as-is. • So you quote the character(s): • A single character: by prefixing it with a \ (backslash) • A string of characters: by enclosing them in • Single quotes: no characters are special. None • Double quotes: some characters (notably $ and `) are still special. You’ll need to prefix those with a backslash to pass to the program as-is.

38. Quoting examples • List all the files in the current directory:ls * • List the file or directory called ‘*’ (or complain if it’s not there):ls \* • Print $HOME:echo ’$HOME’ • Print the path to your home directory:echo ”$HOME” • Print `id –un`: echo ’`id –un`’ • Print your login:echo ”`id –un`”

39. Next stop: utilities • (No, not electricity, water, and sewer) • diff and patch • grep • find and xargs

40. diff and patch • You have two versions of a file; how do you see what’s changed between the two versions? • diff shows the differences between two files; you’ll want to use the –u or –c option to diff to produce output in human-friendly format:diff –u my_file my_file.orig | less • patch applies differences to a filediff –u my_file my_file.orig > my_patchpatch my_file < mypatchmy_file is now the same as my_file.orig

41. grep – search for patterns • grep searches for patterns in texts:grep <string> <FILE> • grep uses regular expressions to describe the string(s) to search for • In a regular expression, most characters are treated as-is, but a few are magic • Ordinary characters just represent themselves • Magic characters do special things

42. grep’s magic characters • A few different kinds of magic characters: • Some characters “anchor” (parts of) a regular expressions to specific places in the string:^ - The beginning of a line$ - The end of a line • A dot (.) matches “any one character whasoever” (except for newline) • [ ]form a character class: • [aeiou] – any single vowel • [a-zA-z0-9] – Any single letter or digit • [^0-9] – Any single character that isn’t a digit Here, ‘^’means “not”

43. Even more of grep’s magic characters • Quantifiers say how many times the preceeding “thing” should be repeated: • * means “zero or more times” • ? Means “zero or one time”

44. Frequently used grep options -i : do case-insensitive search -n : print line numbers -v : print lines that do not match -l : only list the files from which output would have been printed

45. grep examples • Print all non-blank lines:grep –v ’^$’ my_file.txt • Print all the lines on which my_function is called (or declared):grep –n ’my_function *(’ my_code.c • Show all the xterms I am running:ps auxw | grep ”`id –un` .*xterm”

46. find find <PATHS> <OPTIONS> <EXPRESSION> • Traverse the tree(s) rooted at <PATHs>, and for each “thing” found, evaluate the <EXPRESSION> until the result of <EXPRESSION> is known. • The evaluation of <EXPRESSION> “short-circuits”, just like expressions in C/C++. • false && some_expression can never be true • Options apply to the entire find command, not the individual expression

47. Components of a find expression • An expression is zero or more primaries connected by operators • Two kinds of primaries: • Tests: just return true or false • Actions: do something, in addition to returning true or false • Operators work just as they do in C/C++. • ! / -not • -a / -and • -o / -or • , (comma) • Parentheses are used for grouping, just as in C/C++.

48. find examples • Print all the *.c* and *.h* files in and below the current directoryfind . –name ’*.[ch]*’ –a –print • Show line numbers myfunction calls in the above *.c* filesfind . –name ’*.c*’ | xargs grep –n ’myfunction.*(’ • Change permissions on files under your home directory so they’re inaccessible to everyone but you (except for files in the www directory)cd; find . –path ”./www*” –prune –o –exec chmod go-rwx {} \; • How big are my *.c* files?expr `find –name ’*.c*’ –printf ”%k + ”` 0 • Read the find manual (info find) for more examples

49. xargs: combine arguments xargs <OPTION> <COMMAND> <INITIAL_ARGS> • Feeds standard input as arguments to <COMMAND> • Often used with find(find … | xargs grep) • But it doesn’t have to be used with find:cat filelist | xargs cat {} > concatenated

50. Finding more information – at CSE • Use info and man • Peruse the “see also” section of the man pages • Peruse info’s i (search index) command • The uw-cs.lab-help group • Look at your .login, .cshrc, etc. to see how support has set things up