An Introduction to Sed & Awk

An Introduction to Sed & Awk Presented Tues, Jan 14th, 2003 Send any suggestions to Siobhan Quinn (squinn@cs.washington.edu)

Sed : a “Stream EDitor” What is Sed ? • A “non-interactive” text editor that is called from the unix command line. • Input text flows through the program, is modified, and is directed to standard output. An Example: The following sentence is input to the sed program: echo "Instrumental in ruining entire operation for a Midwest chain operation." | sed 's/ruining/running/‘ Instrumental in running entire operation for a Midwest chain operation.

Why Use Sed? • Eliminate the tedium of routine editing tasks! (find, replace, delete, append, insert) … but your word processor can already do that right? Wrong. Sed is extremely powerful AND comes with every Unix system in the world! • Sed is designed to be especially useful in three cases: • To edit files too large for comfortable interactive editing; • To edit any size file when the sequence of editing commands is too complicated to be comfortably typed in interactive mode. • To perform multiple `global' editing functions efficiently in one pass through the input.

How Sed Works While (read line){ 1 ) Sed reads an input line from STDIN or a given file, one line at a time, into the pattern space. Pattern Space = a data buffer - the “current text” as it’s being edited 2) For each line, sed executes a series of editing commands (written by the user, you) on the pattern space. 3) Writes the pattern space to STDOUT. }

How Sed Works (cont…) echo “amy enjoys hiking and ben enjoys skiing” | sed –e ‘s/skiing/hiking/g; s/hiking/biking/g’ 1 ) Sed read in the line “amy enjoys hiking and ben enjoys skiing” and executed the first ‘substitute’ command. The resulting line – in the pattern space: “amy enjoys hiking and ben enjoys hiking” 2) Then the second substitute command is executed on the line in the pattern space, and the result is : “amy enjoys biking and ben enjoys biking” 3) And the result is written to standard out.

$ sed [-e script] [-f script-file] [-n] [files...] -e an "in-line" script, i.e. a script to sed execute given on the command line. Multiple command line scripts can be given, each with an -e option. -n by default, sed writes each line to stdout when it reaches the end of the script (being whatever on the line) this option prevents that. i.e. no output unless there is a command to order SED specifically to do it -f read scripts from specified file, several -f options can appear files are the files to read, if a "-" appears, read from stdin,if no files are given, read also from stdin Different Ways to Invoke Sed: sed –e 'command;command;command' input_file see results sed –e 'command;command;command' input_file > output_file save results .... | sed –e 'command;command;command' | .... use in a pipeline sed -f sedcommandsinput_file > output_file commands are in file somewhere else Invoking Sed Commands

Invoking Sed (some notes) • sed commands are usually on one line • if we want more (multi-line commands), then we must end the first line with an `\' • if a command is one line only, it can be separated by a `;‘ • if it is a multi-line, then it must contain all of its line (except the first) by themselves • on command line, what follows a `-e' is like a whole line in a sed script

Sed uses regular expressions to match patterns in the input text, and then perform operations on those patterns. ^ matches the beginning of the line $ matches the end of the line . Matches any single character \ Escapes any metacharacter that follows, including itself. (character)* Match arbitrarily many occurences of (character) (character)? Match 0 or 1 instance of (character) (character)+ Match 1 or more instances of (character) [abcdef] Match any character enclosed in [ ] (in this instance, a b c d e or f) [^abcdef] Match any character NOT enclosed in [ ] (character)\{m,n\} Match m-n repetitions of (character) (character)\{m,\} Match m or more repetitions of (character) (character)\{,n\} Match n or less (possibly 0) repetitions of (character) (character)\{n\} Match exactly n repetitions of (character) \{n,m\} range of occurrences, n and m are integers $expression$ Group operator. expression1|expression2 Matches expression1 or expression 2. () groups regular expressions Regular Expressions

Regular Expressions (character classes) The following character classes are short-hand for matching special characters. [:alnum:] Printable characters (includes white space) [:alpha:] Alphabetic characters [:blank:] Space and tab characters [:cntrl:] Control characters [:digit:] Numeric characters [:graph:] Printable and visible (non-space) characters [:lower:] Lowercase characters [:print:] Alphanumeric characters [:punct:] Punctuation characters [:space:] Whitespace characters [:upper:] Uppercase characters [:xdigit:] Hexadecimal digits

/^M.*/ /..*/ /^$/ ab|cd a(b*|c*)d [[:space:][:alnum:]] Line begins with capital M, 0 or more chars follow At least 1 character long (/.+/ means the same thing) The empty line Either ‘ab’ or ‘cd’ matches any string beginning with a letter a, followed by either zeroor more of the letter b, or zero or more of the letter c, followed by the letter d. Matches any character that is either a white space character or alphanumeric. Regular Expressions (cont…) Note: Sed always tries to find the longest matching pattern in the input. How would you match a tag in an HTML document?

Line Addresses Each line read is counted, and one can use this information to absolutely select which lines commands should be applied to. 1 first line 2 second line ... $ last line i,j from i-th to j-th line, inclusive. j can be $ Examples : sed ’53!d’ prints through line number 52 sed –n ‘4,9p’ prints only lines 4 through 9

Context Addresses The second kind of addresses are context, or Regular Expression, addresses. Commands will be executed on all pattern spaces matched by that RE. Examples: sed ‘/^$/d’ will delete all empty lines sed ‘/./,$!d’ will delete all leading blank lines at the top of file Some Rules: • commands may take 0, 1 or 2 addresses • if no address is given, a command is applied to all pattern spaces • if 1 address is given, then it is applied to all pattern spaces that match that address • if 2 addresses are given, then it is applied to all formed pattern spaces between the pattern space that matched the first address, and the next pattern space matched by the second address. • If pattern spaces are all the time single lines, this can be said like, if 2 addrs are given, then the command will be executed on all lines between first addr and second (inclusive)

We will go over the only some basic sed commands. a append c change lines d delete lines i insert p print lines s substitute Sed Commands

Sed Commands (cont… ) APPEND [address]a\ text Append text following each line matched by address. In hello_world.script there is : From the command prompt type: /<body/i\ $ cat <an html file> | sed –f hello_world.script Hello World CHANGE [address1[,address2]]c\ text Replace (change) the lines selected by the address with text.. The append and insert commands can be applied only to a single line address, not a range of lines. The change command, however, can address a range of lines. In this case, it replaces all addressed lines with a single copy of the text. In other words, it deletes each line in the range but the supplied text is output only once. In change_mail.script there is: From the command prompt type: /^From /,/^$/c\ $ cat address_file | sed –f <Mail Header Removed> change_mail.script > new_mail.html

Sed Commands (cont…) DELETE [address1[,address2]]d Delete line(s) from pattern space. Thus, the line is not passed to standard output. A new line of input is read and editing resumes with first command in script. What do these do ? cat homework.html | sed -e '/[Hh]omework/d' >newhomework.html cat homework.html | sed –e ‘1,20d’ > newhomework2.html\ INSERT [address1]i\ text Insert text before each line matched by address. (See append example) PRINT [address1[,address2]]p Print the addressed line(s). Note that this can result in duplicate output unless default output is suppressed by using "#n" or the -n command-line option. sed –n ‘/regexp/,$p’ prints from regexp to the end of the file.

Sed Commands (cont…) SUBSTITUE s [address1[,address2]]s/pattern /replacement /[flags] Substitute replacement for pattern on each addressed line. If pattern addresses are used, the pattern // represents the last pattern address specified. The following flags can be specified: n Replace nth instance of /pattern/ on each addressed line. n is any number in the range 1 to 512, and the default is 1. g Replace all instances of /pattern/ on each addressed line, not just the first instance. p Print the line if a successful substitution is done. If several successful substitutions are done, multiple copies of the line will be printed. • Substitute bad grades for better ones Input File: CLASS GRADE cat grades.txt | CSE 321 2.4 sed 's/1\..\|2\..\|3\../4.0/g‘ old_transcript > CSE 341 3.0 new_transcriptSpeech Com 4.0 CSE 378 1.4 • Delete ONLY the regular expression and not the entire line sed 's/ <a bad word>//g' somefile > censured_file

Short Comings of Sed The 'sed' program takes the 'infile' A LINE AT A TIME and applies to that ONE LINE, ALL of the commands in 'cmdfile' in the order that they are written. If the problems with the file depend on what's on other lines (like LaTeX which creates environments), 'sed' will not be a quick fix. For example you cannot take 'times' (meaning math multiplication) in troff and substitute '\times' because the line-at-a-time substitution doesn't know when you are or are not in the equation environment. If you do you'll also get '\times' in the text (ditto with 'sum', 'over', etc.).

An Introduction to Sed & Awk