1 / 35

MetaLexer : A Modular Lexical Specification Language

Andrew Casey Laurie Hendren McGill University. MetaLexer : A Modular Lexical Specification Language. www.sable.mcgill.ca/metalexer. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A A A A A A A A A A A A A A A A A A. Why MetaLexer ?

malha
Download Presentation

MetaLexer : A Modular Lexical Specification Language

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Andrew Casey • Laurie Hendren • McGill University MetaLexer: A Modular Lexical Specification Language www.sable.mcgill.ca/metalexer TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAAAAAAAAAA

  2. Why MetaLexer? • Why is it relevant to AOSD? • What are the challenges?

  3. Structure of a Compiler Front-End Regular Expressions + State (flex, jflex, ...) Context-free grammars + actions/attributes (yacc, bison, Polyglot, JastAdd, ...)

  4. Given a front-end specification for a language (i.e. Java), current method to implement a front-end for an extension of that language (i.e. AspectJ)? • Grammar rules for extension

  5. Desired Modular MetaLexer Approach • Grammar rules for extension • Lexical rules for extension

  6. We also want to be able to combine lexical specifications for diverse languages. • Java + HTML • Java + Aspects (AspectJ) • Java + SQL • MATLAB + Aspects (AspectMatlab)

  7. Would like to be able to reuse and extend lexical specification modules • Nested C-style comments • Javadoc comments • Floating-point constants • URL • regular expressions • …

  8. Scanning AspectJ

  9. Scanning Java/Javadoc

  10. First, let’s understand the traditional lexer tools (lex, flex, jflex). • programmer specifies regular expressions + actions • tools generate a finite automaton-based implementation • states are used to handle different language contexts

  11. Current (ugly) method for extending jflex specifications - copy&modify • Copy jflex specification. • Insert new scanner rules into copy. • Order of rules matters! • Introduce new states and action logic for converting between states. • Principled way of weaving new rules into existing rules. • Modular and abstract notion of state and changing between states.

  12. JflexLexing Structure Specificationin one file. • Lexing rules associated with a state. • Changing states associated with action code.

  13. Each component specified in its own file. • MetaLexer Structure Layout specified in its own file. • Components define lexing rules associated with a state and produce meta-tokens. • Layout defines transitions between components, state changes by meta-lexer.

  14. Structure of a MetaLexer Specification for Matlab

  15. Extending a MetaLexer Specification for Matlab

  16. Sharing component specifications with MetaLexer

  17. Scanning a properties file Properties Key Value Util_Patterns

  18. util_properties.mlc helper component

  19. key.mlc component

  20. value.mlc component

  21. properties.mll layout

  22. MetaLexeris implemented and available: • www.sable.mcgill.ca/metalexer properties.mll properties.jflex util_patterns.mlc MetaLexer key.mlc value.mlc

  23. Key problems to solve: • How to implement the meta-token lexer? • How to allow for insertion of new components, replacing of components, adding new embeddings (metalexer transitions). • How to insert new patterns into components as specific points.

  24. Implementing the meta-token lexer • Recognize a meta-pattern, i.e. when to go to a new component and when to return. • Recognize the matching suffix.

  25. Implementing inheritance (structured weaving).

  26. Implementing MetaLexer layout inheritance • Layouts can inherit other layouts • %inherit directive put at the location at which the inherited transition rules (embeddings) should be placed. • each %inherit directive can be followed by: • %unoption • %replace • %unembed • new embeddings

  27. Implementing MetaLexer component inheritance

  28. Weaving in inherited component Woven output O New Component adds some rules and inherits original component. Original Component

  29. Results: • Applied to three projects with complex scanners: • AspectJ (abc and extensions) • Matlab (Annotations and AspectMatlab extensions) • MetaLexer

  30. AspectJ and Extensions

  31. Using MetaLexer for an extensible front end for McLab PLDI 2011 Tutorial on McLab!!!!!

  32. MetaLexer scanner implemented in MetaLexer • 1st version of MetaLexerwritten in JFlex, one for components and one for layouts. • 2nd version implemented in MetaLexer, many shared components between the component lexer and the layout lexer.

  33. Related Work • Ad-hoc systems with separate scanner/ LALR parser • Polyglot • JastAdd • abc • Recursive-descent scanner/parser • ANTLR and systems using ANTLR • Scannerless systems • Rats! (PEGs) • Integrated systems • Copper (modified LALR parser which communicates with DFA-based scanner)

  34. Conclusions • MetaLexer allows one to specify modular and extensible scanners suitable for any system that works with JFlex. • Two main ideas: meta-lexing and component/layout inheritance. • Used in large projects such as abc, McLab and MetaLexer itself. • Available at: www.sable.mcgill.ca/metalexer

More Related