1 / 26

Using Lex

Using Lex. Flex – Lexical Analyzer Generator. A language for specifying lexical analyzers. lang.l. Flex compiler. lex.yy.c. C compiler -lfl. lex.yy.c. a.out. source code. a.out. tokens. The Structure of a Lex Program. ( Definition section) %% ( Rules section) %%

carriehawkins
Download Presentation

Using Lex

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Lex

  2. Flex – Lexical Analyzer Generator A language for specifying lexical analyzers lang.l Flex compiler lex.yy.c C compiler -lfl lex.yy.c a.out source code a.out tokens

  3. The Structure of a Lex Program (Definition section) %% (Rules section) %% (User subroutines section)

  4. Flex Programs %{auxiliary declarations%}regular definitions%%translation rules%%auxiliary procedures

  5. Example 1-1: Word recognizer ch1-02.l %{ /* * this sample demonstrates (very) simple recognition: * a verb/not a verb. */ %} %% [\t ]+ /* ignore white space */ ; is | am | are | were | was | be | being | been | do | does | did | will | would | should | can | could | has | have | had | go { printf("%s: is a verb\n", yytext); } [a-zA-Z]+ { printf("%s: is not a verb\n", yytext); } .|\n { ECHO; /* normal default anyway */ } %% main() { yylex(); }

  6. The definition section • Lex copies the material between “%{“ and “%}” directly to the generated C file, so you may write any valid C codes here

  7. Rules section • Each rule is made up of two parts • A pattern • An action • E.g. [\t ]+ /* ignore white space */ ;

  8. Rules section (Cont’d) • E.g. is | am | are | were | was | be | being | been | do | does | did | will | would | should | can | could | has | have | had | go { printf("%s: is a verb\n", yytext); }

  9. Rules section (Cont’d) • E.g. [a-zA-Z]+ { printf("%s: is not a verb\n", yytext); } .|\n { ECHO; /* normal default anyway */ } • Lex had a set of simple disambiguating rules: • Lex patterns only match a given input character or string once • Lex executes the action for the longest possible match for the current input

  10. User subroutines section • It can consists of any legal C code • Lex copies it to the C file after the end of the Lex generated code %% main() { yylex(); }

  11. Regular Expressions • Regular expressions used by Lex catenation . the class of all characters except newline. * 0 or more [] character class ^ not (left most) $ 以newline為跟隨字元 ab$ {} macroexpansion of a symbol ex: {Digit}+ \ escape character + once or more ? Zero time or once | or “…” … to be taken / 以甚麼為跟隨字元 ab/cd () used for more complex expressions - range

  12. Functions and Variables yylex() a function implementing the lexical analyzer and returning the token matched yytext a global pointer variable pointing to the lexeme matched yyleng a global variable giving the length of the lexeme matched yylval an external global variable storing the attribute of the token

  13. Functions and Variables • yywrap() • If yywrap() returns false (zero), then it is assumed that the function has gone ahead and set up yyin to point to another input file, and scanning continues. If it returns true (non-zero), then the scanner terminates, returning 0 to its caller. • If you do not supply your own version of yywrap(), then you must either use % option noyywrap or….

  14. Examples of Regular Expressions • [0-9] • [0-9]+ • [0-9]* • -?[0-9]+ • [0-9]*\.[0-9]+ • ([0-9]+)|([0-9]*\.[0-9]+) • -?(([0-9]+)|([0-9]*\.[0-9]+)) • [eE][-+]?[0-9]+ • -?(([0-9]+)|([0-9]*\.[0-9]+))([eE][-+]?[0-9]+)?)

  15. Example 2-1 %% [\n\t ] ; -?(([0-9]+)|([0-9]*\.[0-9]+)([eE][-+]?[0-9]+)?) { printf("number\n"); } . ECHO; %% main() { yylex(); }

  16. A Word Counting Program • The definition section %{ unsigned charCount = 0, wordCount = 0, lineCount = 0; %} word [^ \t\n]+ eol \n

  17. A Word Counting Program (Cont’d) • The rules section {word} { wordCount++; charCount += yyleng; } {eol} { charCount++; lineCount++; } . charCount++;

  18. A Word Counting Program (Cont’d) main(argc,argv) int argc; char **argv; { if (argc > 1) { FILE *file; file = fopen(argv[1], "r"); if (!file) { fprintf(stderr,"could not open %s\n",argv[1]); exit(1); } yyin = file; } yylex(); printf("%d %d %d\n",charCount, wordCount, lineCount); return 0; } • The user subroutines section

  19. stripquotes • No tossing mechanism provided. • int frompos, topos, numquotes=2; • For (frompos=1;frompos<yyleng;frompos++) { • yytext[topos++]=yytext[frompos]; • if(yytext[frompos]==‘”’ && yytext[frompos+1]==‘”’) • {frompos++;numqotes++;}} • yyleng-=numquotes; • yytext[yyleng]=‘\0’;

  20. Micro scanner • auxiliary declarations • %option noyywrap • %{ • #ifndef DEFINE • #define DEFINE 1 • #include <stdlib.h> • #include "token.h" • #endif • #define YYLMAX 33 • char token_buffer[YYLMAX]; • FILE *out_fd, *status_fd; • extern void check_id(char *); • extern void list_token_type(token); • extern void get_file_name(char *, char *, char *); • %}

  21. Micro scanner • regular definitions • letter [a-zA-Z] • digit [0-9] • KEYWORD "BEGIN"|"begin"|"END"|"end"|"READ"|"read"|"WRITE"|"write"|"SCANEOF" • literal {digit}+ • IDENTIFIER {letter}+[{letter}|{digit}|"_"]* • special_char "+"|"-"|"*"|":="|"("|")"|";"|"," • comment “--”.*[\n]

  22. Micro scanner • translation rules • %% • {KEYWORD} {list_token_type(check_reserved(yytext));} • {literal} {list_token_type(INTLITERAL) • {IDENTIFIER} {list_token_type(ID);} • {special_char}{list_token_type(check_special_char(yytext));} • [\n] • [ \t] • {comment} • . {lexical_error(yytext);}

  23. Micro scanner • auxiliary procedures • %% • main(int argc,char *argv[]) • { • if ((yyin=fopen(argv[1],"r"))==NULL) • { • printf("ERROR:file open error!!\n"); • exit(0); • } • yylex(); • fclose(yyin); • }

  24. Micro scanner • check_reserved • 確認是否為保留字 • check_special_char • 確認是否為+-*(),;:= • list_token_type • 顯示token型態

  25. Example • begin • read (A,B); • A := (B+3)*A; • B := A*3-B; • write (A,B,3*4); • end SCANEOF

  26. Example • MULTOP • ID • SEMICOLON • ID • ASSIGNOP • ID • MULTOP • INTLITERAL • MINUSOP • ID • SEMICOLON • WRITE • LPAREN • ID • COMMA • BeGIN • READ • LPAREN • ID • COMMA • ID • RPAREN • SEMICOLON • ID • ASSIGNOP • LPAREN • ID • PLUSOP • INTLITERAL • RPAREN • ID • COMMA • INTLITERAL • MULTOP • INTLITERAL • RPAREN • SEMICOLON • END • SCANEOF

More Related