180 likes | 433 Views
Lex -- a Lexical Analyzer Generator (by M.E. Lesk and Eric. Schmidt) Given tokens specified as regular expressions, Lex automatically generates a routine (yylex) that recognizes the tokens (and performs corresponding actions). Lex source program {definition} %% {rules} %% {user subroutines}
E N D
Lex -- a Lexical Analyzer Generator (by M.E. Lesk and Eric. Schmidt) • Given tokens specified as regular expressions, Lex automatically generates a routine (yylex) that recognizes the tokens (and performs corresponding actions).
Lex source program {definition} %% {rules} %% {user subroutines} Rules: <regular expression> <action> Each regular expression specifies a token. Default action for anything that is not matched: copy to the output Action: C/C++ code fragment specifying what to do when a token is recognized.
lex program examples: ex1.l and ex2.l • ‘lex ex1.l’ produces the lex.yy.c file that contains a routine yylex(). • The int yylex() routine is the scanner that finds all the regular expressions specified. • yylex() returns a non-zero value (usually token id) normally. • yylex() returns 0 when end of file is reached. • Need a drive to test the routine. Main.cpp is an example. • You need to have a yywrap() function in the lex file (return 1), see the function in ex1.l. • Something to do with compiling multiple files.
Lex regular expression: contains text characters and operators. • Letters of alphabet and digits are always text characters. • Regular expression integer matches the string “integer” • Operators: “\[]^-?.*+|()$/{}%<> • When these characters happen in a regular expression, they have special meanings • Lex regular expressions cannot have space in them!!!
operators (characters that have special meanings): “\[]^-?.*+|()$/{}%<> • ‘*’, ‘+’, ‘|’, ‘(‘,’)’ -- used in regular expression • ‘ “ ‘ -- any character in between quote is a text character. • E.g.: “xyz++” == xyz”++” • ‘\’ -- escape character, • To get the operators back: “xyz++” == ?? • To specify special characters: \40 == “ “ • ‘[‘ and ‘]’ -- used to specify a set of characters • e.g: [a-z], [a-zA-Z], • Every character in it except ^, - and \ is a text character • [-+0-9], [\40-\176] • ‘^’ -- not, used as the first character after the left bracket • E.g [^abc] – any character except ‘a’, ‘b’ or ‘c’. • [^a-zA-Z] -- ??
operators (characters that have special meanings): “\[]^-?.*+|()$/{}%<> • ‘.’ -- every character • ‘?’ -- optional ab?c matches ‘ac’ or ‘abc’ • ‘/’ -- used in character lookahead: • e.g. ab/cd -- matches ab only if it is followed by cd • ‘{‘’}’ -- enclose a regular definition • ‘%’ -- has special meaning in lex • ‘$’ -- match the end of a line, ‘^’ -- match the beginning of a line • ab$ == ab/\n • ‘<‘ ‘>’: start condition (more context sensitivity support, see the paper for details).
Order of pattern matching: • Always matches the longest pattern. • When multiple patterns matches, use the first pattern. • To override, add “REJECT” in the action. ... %% Ab {printf(“rule 1\n”);} Abc {printf(“rule 2\n”);} {letter}{letter|digit}* {printf(“rule 3\n”);} %% Input: Abc What happened when at ‘.*’ as a pattern? Should regular expressions for reserved words happen before or after the regular expression for identifier?
Manipulate the lexeme and/or the input stream: • yytext -- a char pointer pointing to the matched C string • yyleng -- the length of the matched string • I/O routines to manipulate the input stream: • yyinput() -- get a character from the input character, return <=0 when reaching the end of the input stream, the character otherwise • yyinput() for c++, input() for c. • unput( c ) -- put c back onto the input stream • Deal with comments: (/* ….. */ • Why is pattern “/*”.*”*/” a problem? %% … “/*” {char c1; c2 = yyinput(); if (c2 <=0) {lex_error(“unfinished comment” …} else { c1 = c2; c2 = yyinput(); while (((c1!=‘*’) || (c2 != ‘/’)) && (c2 > 0)) {c1 = c2; c2 = yyinput();} if (c2 <= 0) {lex_error( ….) }
Reporting errors: • What kind of errors? Not too many. • Characters that cannot lead to a token • unended comments (can we do it in later phases?) • unended string constants.
Reporting errors: • How to keep track of current position (which line, which column)? • Use to global variable for this: yyline, yycolumn %{ int yyline = 1, yycolumn = 1; %} ... %% “\n” {yyline++; yycolumn = 0;} [ \t]+ {/* do nothing*/ yycolumn += yyleng;} If {yycolumn += yyleng; return (IFNumber);} “+” {yycolumn += 1; return (PLUSNumber);} {letter}{letter|digit}* {yylval = idtable_insert(yytext); yycolumn += yyleng; return(IDNumber);} ... %%
Dealing with identifiers, string constants. • Data structures: • Put the lexeme in a string table: e.g. vector of C strings. • See token.l • Recognizing constant strings with special characters • Assuming string cannot pass line boundary. • Use yymore() “[^”\n]* {char c; c = yyinput(); if (c != ‘”’) error else if (yytext[yyleng-1] == ‘\\’) { unput( c ); yymore(); } else {/* find the whole string, normal process*/}
Put it all together • Checkout token.l and main.cpp in lex2.tar