Lecture 9 Midterm Solutions

1. Lecture 9 Midterm Solutions ksh93 CGI (continued) Perl

2. Question 1 • b,c,d • b,c,f • c,d,f • a,c

3. Question 2 • same: both match everything • same side effect: both create /tmp/filesdifferent output: first one outputs number of lines, words and chars from ls; second one outputs 0 0 0 because wc gets empty stdin • different: first one outputs unique first fields; second one outputs first fields of unique lines • same: 6 is u=rw, 3 is g=wx, 1 is o=x

4. Question 2 (cont.) e) different: first command outputs the contents of foobar in uppercase; second command output the contents of foobar with a and z substituted by A and Z respectively, because sed translation command does not take ranges. e) different: first command also prints contents of directories in the current directory; second command only prints contents of current directory

5. Question 3 • find / -size +5000c –user elee –exec sed –n ’10,20p’ ‘{}’ \; • find / -size +5000c –user elee | xargs –L 1 sed –n ’10,20p’

6. Question 4 • cut –f1 –d\& | tr a-z A-Z • egrep –c ‘&1&[0-9](\.|$)’ • cut –f1 –d\& | egrep ‘(^|[[:space:]])(Beef|Chicken)$’ • sort –k3rn –t \& | cut –f1 –d\&

7. Question 5 • $FOO • ls a b • ls a b • * • a b • $BAR • \$BAR • ls a b • a b

8. Question 6 • 1 • 1-7 • 3,6 • 1,5 • 1,3,5

9. Question 7 s/^[^[:alnum:]]*// s/[^[:alnum:]]*$// y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/ /^$/c\ This line has been removed.

10. Question 8 { for (i=1; i<=NF; i++) { if ($i ~ /^[0-9]+$/) { x[i]++; } } nfield=NF } END { for (i=1; i<=nfield; i++) { print x[i]/NR*100 “%” } }

11. Question 9 #!/bin/sh linenum=1 while IFS= read line ; do for i in "$@" ; do if [ $i -eq $linenum ] ; then echo "$line" break fi done linenum=`expr $linenum + 1` done

12. Question 9 (alternate) #!/bin/sh trap 'rm –f /tmp/input$$' EXIT cat > /tmp/input$$ linetot=`wc –l < /tmp/input$$` linenum=1 while [ $linenum –le $linetot ] ; do for i in "$@" ; do if [ $i -eq $linenum ] ; then head -$linenum /tmp/input$$ | tail -1 break fi done linenum=`expr $linenum + 1` done

13. KornShell 93

14. Variable Attributes • By default attributes hold strings of unlimited length • Attributes can be set with typeset: • readonly (-r) – cannot be changed • export (-x) – value will be exported to env • upper (-u) – letters will be converted to upper case • lower (-l) – letters will be converted to lower case • ljust (-L width) – left justify to given width • rjust (-R width) – right justify to given width • zfill (-Z width) – justify, fill with leading zeros • integer (-I [base]) – value stored as integer • float (-E [prec]) – value stored as C double • nameref (-n) – a name reference

15. New Parameter Expansion • ${param/pattern/str} – Replace first pattern with str • ${param//pattern/str} – Replace all patterns with str • ${param:offset:len} – Substring with offset

16. Patterns Extended Regular Expressions Patterns • Additional pattern types so that shell patterns are equally expressive as regular expressions • Used for: • file expansion • [[ ]] • case statements • parameter expansion

17. ANSI C Quoting • $'…' Uses C escape sequences $'\t' $'Hello\nthere' • printf added that supports C like printing: printf "You have %d apples" $x • Extensions • %b – ANSI escape sequences • %q – Quote argument for reinput • \E – Escape character (033) • %P – convert ERE to shell pattern • %H – convert using HTML conventions • %T – date conversions using date formats

18. Associative Arrays • Arrays can be indexed by string • Declared with typeset –A • Set: name["foo"]="bar" • Reference ${name["foo"]} • Subscripts: ${!name[@]}

19. C Expressions • We have already seen built-in expressions with the [[ ]] operator: • [[ $var = *foo* ]] && print "contains foo" • New operator (( )) for C-like numeric expressions: • (( x > 10 )) && print "x=$x, greater than 10" • (( x ++ )) • Note variables don't have to be used with $ inside parens • Value of (( )) expression can be usedwith $(( )) • y=$(( x + 1 )) • print $(( x * y - sin(y) ))

20. Compound Variables • Variables can contain subfields (like structures or classes) • Syntax: variable name containing . • Example: cust=(name=Jeff zip=10003) cust.state=NY print ${cust.name} print ${!cust.*}

21. New for loop syntax • Regular syntax: for var in listdo …done • Additional syntax like C: for (( initialization; condition; increment )) do … done • Example: for (( i=0; i < $VAR; i++))

22. Example: Word Count #!/home/unixtool/bin/kshinteger l=0 w=0 c=0 while read –r LINE do (( l++ )) set -- $LINE (( w += $# )) (( c += ${#LINE}+1 )) done < $1 print "$l lines, $w words, $c characters"

23. Example: Word Count integer l=0 w=0 c=0 while read –r LINE do (( l++ )) set -- $LINE (( w += $# )) (( c += ${#LINE}+1 )) done < $1 print "$l lines, $w words, $c characters" • integer tag indicates variables will be used as integers • while loop is a command, so redirection works

24. Example: Word Count integer l=0 w=0 c=0 while read –r LINE do (( l++ )) set -- $LINE (( w += $# )) (( c += ${#LINE}+1 )) done < $1 print "$l lines, $w words, $c characters" • set -- $LINEturns LINE into positional parameters ($1, …), splitting up the value with IFS • $# is the number of positional parameters

25. Example: Word Count integer l=0 w=0 c=0 while read –r LINE do (( l++ )) set -- $LINE (( w += $# )) (( c += ${#LINE}+1 )) done < $1 print "$l lines, $w words, $c characters" • ${#LINE}returns the length of the value of LINE • We add 1 because the newline character is not part of LINE

26. Example: Spell a Phone Number Given a number, finds possible words that the number spells on a telephone. Example: $ phonespell 8643 void

27. Algorithm • Create function combo that prints all combinations of words. Check those against the dictionary. • function combo is recursive: • Pass in part of number, part of word spelled combo 8643 "" combo 643 t combo 643 u combo 643 v combo 43 vm combo 43 vn combo 43 vo … combo "" void

28. Example: Spell a Phone Number function combo { typeset num=$1 word=$2 if [[ $num = '' ]] then print $word else typeset -L1 digit=$num for letter in ${get_letter[digit]} do combo "${num#?}" "$word$letter" done fi } • functions defined in ksh take arguments as positional parameters, like commands • typeset makes a variable local

29. Example: Spell a Phone Number function combo { typeset num=$1 word=$2 if [[ $num = '' ]] then print $word else typeset -L1 digit=$num for letter in ${get_letter[digit]} do combo "${num#?}" "$word$letter" done fi } • End of recursion: If number is empty, just print the given word. Should end up happening for every combination

30. Example: Spell a Phone Number function combo { typeset num=$1 word=$2 if [[ $num = '' ]] then print $word else typeset -L1 digit=$num for letter in ${get_letter[digit]} do combo "${num#?}" "$word$letter" done fi } • Extract leftmost digit from num

31. Example: Spell a Phone Number function combo { typeset num=$1 word=$2 if [[ $num = '' ]] then print $word else typeset -L1 digit=$num for letter in ${get_letter[digit]} do combo "${num#?}" "$word$letter" done fi } • for loop goes through all letters that correspond to the number (stored in get_letter array, shown next slide) • Recursively calls itself for each letter, taking off one character from the left (using the # operator with pattern ?)

32. Spell a Phone Number (cont’) set -A get_letter o i "a b c" "d e f" "g h i" "j k l" \ "m n o" "p r s" "t u v" "w x y" # method 1 combo $1 | comm -12 /usr/dict/words - # method 2 trap 'rm -f /tmp/full$$' EXIT combo $1 > /tmp/full$$ spell < /tmp/full$$ | comm -13 - /tmp/full$$ • set –A arraynamevalue value ... • sets elements of an array all at once

33. Spell a Phone Number (cont’) set -A get_letter o i "a b c" "d e f" "g h i" "j k l" \ "m n o" "p r s" "t u v" "w x y" # method 1 combo $1 | comm -12 /usr/dict/words - # method 2 trap 'rm -f /tmp/full$$' EXIT combo $1 > /tmp/full$$ spell < /tmp/full$$ | comm -13 - /tmp/full$$ • Call function combo with first argument, pipe to comm • suppress fields 1 and 2 (show only matching lines) • comboemits sorted lines, and dictionary is sorted so comm works well

34. Spell a Phone Number (cont’) set -A get_letter o i "a b c" "d e f" "g h i" "j k l" \ "m n o" "p r s" "t u v" "w x y" # method 1 combo $1 | comm -12 /usr/dict/words - # method 2 trap 'rm -f /tmp/full$$' EXIT combo $1 > /tmp/full$$ spell < /tmp/full$$ | comm -13 - /tmp/full$$ • Another method: use spell command • Create temporary file storing combos • Run through spell, generating list of misspelled words • Pipe to comm, suppressing fields 1 and 3 (show correct words)

35. Example: Mortgage Calculator float rate=$1 principle=$2 payment integer months years=$3 [[ $1 ]] || read -r 'rate?rate in per cent: ' [[ $2 ]] || read -r 'principle?principle: ' [[ $3 ]] || read -r 'years?years to amoritization: ' print "\n\n\tprinciple\t$principle" print "\trate\t\t$rate" print "\tamortization\t$years" (( months = years*12 )) (( rate /= 1200. )) (( payment = (principle*rate)/(1.-pow(1.+rate,-months)) )) • Declare variables • Read in unspecified inputs

36. Example: Mortgage Calculator float rate=$1 principle=$2 payment integer months years=$3 [[ $1 ]] || read -r 'rate?rate in per cent: ' [[ $2 ]] || read -r 'principle?principle: ' [[ $3 ]] || read -r 'years?years to amortization: ' print "\n\n\tprinciple\t$principle" print "\trate\t\t$rate" print "\tamortization\t$years" (( months = years*12 )) (( rate /= 1200. )) (( payment = (principle*rate)/(1.-pow(1.+rate,-months)) )) • Initialize values • Uses built-in arithmetic (pow, floating point /)

37. Example: Mortgage Calculator printf "\tmonthly payment\t%8.2f\n\n" "$payment" print '\tYears Balance' print '\t====== =======' for (( months=0; principle > 0; months++)) do (( principle *= (1.+rate) )) (( principle -= payment )) if (( ((months+1)%12) == 0 )) then printf "\t%d\t%8.2f\n" months/12 "$principle" fi done • Print table header • Uses printf to format floating point number

38. Example: Mortgage Calculator printf "\tmonthly payment\t%8.2f\n\n" "$payment" print '\tYears Balance' print '\t====== =======' for (( months=0; principle > 0; months++)) do (( principle *= (1.+rate) )) (( principle -= payment )) if (( ((months+1)%12) == 0 )) then printf "\t%d\t%8.2f\n" months/12 "$principle" fi done • C-style for loop with numerical calculations

39. Documentation • Web version of Learning the KornShell documents ksh93. Good for learning ksh. • Glass documents ksh88 and bash • UNIX in a Nutshell has a chapter that is a great ksh93 reference. Documents: • Bourne shell compatible features • ksh88 compatible features • ksh93 features

40. Last Week…. • HTML forms are used to collect user input • Data sent via HTTP request • Server launches CGI script to process data • Browser sends form variable as name-value pairs • name1=value1&name2=value2&name3=value3 • Two methods (HTTP_METHOD) to send form input: • GET method: name-value pairs in URL (QUERY_STRING) • POST method: name-value pairs in body or HTTP request

41. CGI Script: Example

42. Part 1: HTML Form <html> <center> <H1>Anonymous Comment Submission</H1> </center> Please enter your comment below which will be sent anonymously to <tt>kornj@cs.nyu.edu</tt>. If you want to be extra cautious, access this page through <a href="http://www.anonymizer.com">Anonymizer</a>. <p> <form action=cgi-bin/comment.cgi method=post> <textarea name=comment rows=20 cols=80> </textarea> <input type=submit value="Submit Comment"> </form> </html>

43. Part 2: CGI Script (ksh) #!/home/unixtool/bin/ksh . cgi-lib.ksh # Read special functions to help parse ReadParse PrintHeader print -r -- "${Cgi.comment}" | /bin/mailx -s "COMMENT" kornj print "<H2>You submitted the comment</H2>" print "<pre>" print -r -- "${Cgi.comment}" print "</pre>"

44. Debugging • Debugging can be tricky, since error messages don't always print well as HTML • One method: run interactively $ QUERY_STRING='birthday=10/15/03'$ ./birthday.cgi Content-type: text/html <html>Your birthday is <tt>10/15/02</tt>.</html>

45. How to get your script run • This can vary by web server type http://www.cims.nyu.edu/systems/resources/webhosting/index.html • Typically, you give your script a name that ends with .cgi • Give the script execute permission • Specify the location of that script in the URL

46. CGI Security Risks • Sometimes CGI scripts run as owner of the scripts • Never trust user input - sanity-check everything • If a shell command contains user input, run without shell escapes • Always encode sensitive information, e.g. passwords • Also use HTTPS • Clean up - don’t leave sensitive data around

47. CGI Benefits • Simple • Language independent • UNIX tools are good for this because • Work well with text • Integrate programs well • Easy to prototype • No compilation (CGI scripts)

48. Example: Dump Some Info #!/home/unixtool/bin/ksh . ./cgi-lib.ksh PrintHeader ReadParse print "<h1> Date </h1>" print "<pre>" date print "</pre>" print "<h1> Form Variables </h1>" print "<pre>" set -s -- ${!Cgi.*} for var do nameref r=$var print "${var#Cgi.} = $r" unset r done print "</pre>" print "<h1> Environment </h1>" print "<pre>" env | sort print "</pre>"

49. Example: Find words in Dictionary <form action=dict.cgi> Regular expression: <input type=entry name=re value=".*"> <input type=submit> </form>

50. Example: Find words in Dictionary #!/home/unixtool/bin/ksh PATH=$PATH:. . cgi-lib.ksh ReadParse PrintHeader print "<H1> Words matching <tt>${Cgi.re}</tt> in the dictionary </H1>\n"; print "<OL>" grep "${Cgi.re}" /usr/dict/words | while read word do print "<LI> $word" done print "</OL>"