Macros

1. Macros Jonathan Bachrach MIT AI Lab

2. Macro • Syntactic extension to core language • Defines meaning of one construct in terms of other constructs • Their declarations are called macro definitions • Their uses are called macro calls

3. unless Example • Definition: (ds (unless ,test ,@body) `(if (not ,test) ,@body)) • Call: (unless done? (go)) • Expansion: (if (not done?) (go)) • Evaluation: …

4. Macro Motivation • “From now on, a main goal in designing a language should be to plan for growth.” • Guy Steele, “Growing a Language”, OOPSLA-98, invited talk • Power of abstraction where neither functional nor object abstraction will suffice, affording: • Clarity • Concision • Implementation hiding • People are lazy • do the right thing

5. Mathematical Notation • Continually inventing new abbreviations appropriate for expressing new concepts • Many mathematical concepts become usable only after someone invents a suitable notation to express them.

6. Proper Macros • Macros control the evaluation of their arguments • Four basic types of control: • Transformation pick apart before eval • Binding vars directly in code • Conditional Evaluation e.g., unless • Multiple Evaluation e.g., do

7. Useful Macros • With macros • (with-open-file (s “dump” :direction :output) (princ 99 s)) • Conditional evaluation • (nif x ‘p ‘z ‘n) • Iteration • (forever (yell)) • Anaphoric macros • (aif (big-long-calculation) (foo it))

8. Improper Macros • General Rule: It is inelegant to use a macro where a function would do. • Example: 1+ versus while • Specific Rule: don’t use macros for inlining. • Caveat: sometimes macros are used when function calls would be awkward (e.g., closures)

9. Quasiquote • WSYWIG specification of lists • Permits the writing a template of desired list with substitutions in place of the variable parts of the list • Evaluation • yields a list • turned off by default • turned on with unquote • Unquote - ,expr - inserts value of expr • Splicing - ,@expr - splices in value of expr • Example (let ((x 1) (y ‘(2 3))) `(,x ,@y)) ==> (1 2 3)

10. Pattern Matching • An intuitive and accessible way of expressing a parser • Permits the writing out of the general shape of the expected input, but with pattern bindings in place of the variable parts of the code • Proto uses quasiquote notation for patterns • Default is to match exactly and recursively • Unquote - ,name - matches one expr bound to name • Splicing - ,@name - matches zero or more exprs and binds them to name • Example • (mif ((,x ,@y) ‘(1 2 3)) (lst x y)) ==> (1 (2 3))

11. (for (e ‘(1 2 3)) (println out e)) ==> (let ((c ‘(1 2 3))) (rep loop ((s (ini-state c))) (unless (fin-state? S c) (let ((e (cur-elt s c))) (println out e) (rep (nxt-state s c))))))) (ds (for (,name ,collection) ,@body) `(let ((c ,collection)) (rep loop ((s (ini-state c))) (unless (fin-state? s c) (let ((,name (cur-elt s c))) ,@body (rep (nxt-state s c))))))) for Example

12. TestSuite Macros • Want test-suite that’s • Concise to write tests • runs to completion • records intelligent failure messages (test-group whales (test-eqv (moby) 37) …) (ds (test-eqv ,input ,output) `(do-test-eqv *group-name* ,(to-str input) (fun () ,input) (fun () ,output))) (dm do-test-eqv (group-name doc input-thunk output-thunk) (unless (= (input-thunk) (output-thunk)) (sig (cat group-name ": " doc))))

13. Declarative Data • Defining tables can be awkward • (deftab t (‘jb => ‘boy) (‘kk => ‘girl))

14. Declarative Data Examples f(list(x, y, z)); Studies { course Math101 { title “Mathematics 101”; 2 points fall term; } … exclusions { Math101 <> MathA; Math102 <> MathB; } prerequisites { (Math101, Math102) < (Math201, Math202, Math204); (CS101,CS102) < (CS201, CS203); } … }

15. Compilation • Parse into sexprs • Recursively expand macros top-down • Create IR • Optimize IR • Emit code

16. Scope of Macros • Macros usually are limited to known positions • In lisp, • Particular forms • Head named lists • Symbols • In particular contexts • Avoids macro calls in parameter specs, let bindings, …

17. Evaluation with Macros • Preparation for Macros • (run (prepare expression)) • Preparation can be split into many phases • Lexical e.g., read macros • Syntactic • Good design dictates that • run and prepare are strictly separated • Their relation is clarified • And most importantly that experiments can be repeated

18. Preparation • Multiple worlds • Preparation produces a result often stored in a file • Preparation and running typically execute in distinct processes • Expression being prepared is only means of communicating between preparer and evaluator • Unique world • All computations cohabit same memory • Communication can be neither limited nor controlled

19. Exogenous Mode • macroexpand is provided independently of expression to prepare, for example, by preparation directives: (dm prepare (expression directives) (let ((macroexpand (generate-macroexpand directives)) (really-prepare (macroexpand expression)))) • Macro expansion might either occur as • A cascade using piping • A synthesis of a new compiler • Ensures separation between macro expansion and evaluation • No connection between language of macro expander and the language that is being prepared

20. Endogenous Mode • All information necessary for expansion must be found in the expression to prepare: (dm prepare (expression) (really-prepare (macroexpand expression))) • Macro expansion algorithm preexists • Finds macro definitions as well as calls • Use eval rather than invent a special language • Complicated because requires an evaluator inside the macro expander • Must not confuse language for writing macros with the language that is being prepared • There are actually two different evaluators or at least evaluation contexts

21. Proto’s Endogenous Macros • Can side-effect macro evaluator with ct special form • (ct (dm map-boot-forms (f) …)) • ct forms can then be used during the expansion of subsequent macros • (ds (define-parents) • `(seq ,(map-boot-forms (fun (x) …)))) • ct forms can be stripped from resulting image if in compilation setting • Experiments can be more faithfully repeated because ct world is insulated from rt world

22. Macro Expansion • Classic Mode • Result returned by expander can include more macro calls • Result is re-expanded until it no longer contains any macro calls • Expansion Passing Style • Expander must return an expression without macro calls • Pass expander macroexpand function as argument • Easy to express local macros

23. Macro-Defining Macros • Patterns in code signal need for abstraction • Same applies to macros • Example: (ds (mvbind ,@args) `(multiple-value-bind ,@args)) (alias mvbind multiple-value-bind) (ds (mvbind ,@args) (let ((name ‘multiple-value-bind)) `(,name ,@args)) `(ds (,short ,@args) (let ((name ‘,long)) `(,name ,@args)) `(ds (,short ,@args) `(,’,long ,@args)) (ds (alias ,short ,long) `(ds (,short ,@args) `(,’,long ,@args)))

24. Self Generating Code Quote Fragment f = #{ #{ Fragment f = #{ ??f }; ?f; }; }; #{ Fragment f = #{ ??f }; ?f; }; Based on Mike McMahon’s self generating quasiquote solution published in Alan Bawden’s quasiquote paper. (let ((let '`(let ((let ',let)) ,let))) `(let ((let ',let)) ,let))

25. Unexpected Captures (ds (apair ,key ,value ,alist) `(let ((f pair)) (f (f ,key ,value) ,alist))) (let ((pair lst) (f #f)) (apair ‘false f ‘())) • Captures • Definition pair • Call f

26. Simple Hygiene Solutions • Renaming for call captures • Use gensym for bindings introduced by macro • Call Example: (ds (apair ,key ,value ,alist) (let ((f (gensym)) `(let ((,f pair)) (,f (,f ,key ,value) ,alist))) • Package prefixing for definition captures (doesn’t work though for local macros) • Definition Example: (ds (apair ,key ,value ,alist) `(let ((f proto::pair)) (f (f ,key ,value) ,alist)))

27. Semi-automatic Hygiene • Explicitly mention the words whose meaning is to be frozen: (ds (apair ,key ,value ,alist) (let ((f (gensym))) (with-aliases ((cons pair)) `(let ((,f ,cons)) (,f (,f ,key ,value) ,alist)))

28. Automatic Hygiene • Variables record definition (e.g., module) origin • Expansion variables record hygiene context which • Is unique number • Is created afresh during each macro expansion • After expansion • Names are renamed such that names connect if they have both same name and hygiene context • If free then use module origin to start lookup • Example (ds (apair ,key ,value ,alist) #[let ((f pair)) (f (f ,key ,value) ,alist)])

29. Hygiene Escapes • There are important macros that aren’t hygienic: there is a need to escape hygiene (ds (loop ,@body) #[lab (#=exit) (rep again () ,@body (again))]) (let ((i 0)) (loop (if (= i 10) (exit 37)) (set i (+ i 1))))

30. R5RS Macros Examples (define-syntax or (syntax-rules () ((or) #f) ((or test) test) ((or test1 test2 ...) (let ((x test1)) (if x x (or test2 ...)))))) (define-syntax let (syntax-rules () ((let ((name val) ...) body1 body2 ...) ((lambda (name ...) body1 body2 ...) val ...)) ((let tag ((name val) ...) body1 body2 ...) ((letrec ((tag (lambda (name ...) body1 body2 ...))) tag) val ...))))

31. R5RS Macros • syntax-rules is not procedural • Two environments • … is cute but brittle • Pattern variables are defaults • No hygiene escapes • Local syntax • Other Scheme macro systems exist

32. Code Walking • Think of macros as compiler extensions • Macros are still limited (no with-slots nor series) • Can’t access lexical context • Can’t build up form dependent information • Code walking is a framework that makes this possible • Current state of the art is AST/OO walkers • See LiSP

33. Conventional Syntax Macros • Power of macros has been limited to Lisp family of languages because Lisp has • a very regular syntax based on lists • a large library of list fabs and xforms • Has been some recent work towards remedying this situation

34. Grammar Extensions “Programmable Syntax Macros” by Weise and Crew and “Growing Languages with Metamorphic Syntax Macros” by Brabrand and Schwartzbach • User writes new rewrite rules • Example Syntax <stm> unless (<exp E>) <stm S1> ::= { if (!<E>) <S> }

35. Advantages Can ensure that expansions produce admissible code fragments Disadvantages Users need to know grammar (e.g., nonterminals) Limited to rewrite rules only and thus awkward to write more complicated macros Grammar Extensions Critique

36. Dylan Macros • Simple regular conventional syntax • Converts surface syntax to SST form • Offers constraint-based pattern matching and code quotes all with automatic hygiene • Rewrite rule only but allows auxiliary rules that take arguments • Simple Example: define macro unless { unless ?test:expression ?:body } => { if (~?test) ?body } end macro;

37. Sophisticated Dylan Example define macro iterate { iterate ?:name (?bindings:*) ?:body end } => { local method ?name (?@vars{ ?bindings }) ?body end; ?name(?@inits{ ?bindings }) } vars: { } => { } { ?:variable = ?:expression, ... } => { ?variable, ... } inits: { } => { } { ?:variable = ?:expression, ... } => { ?expression, ... } end macro;

38. Advantages High level Surface syntax Hygiene support Disadvantages Rewrite rule only No guarantees about producing admissible code fragments Dylan Macro Critique

39. Procedural Macro Motivation • Analysis and rewriting no longer constrained • Simplified pattern matching and rewrite rule engine • Can package and re-use syntax expansion utilities • Pattern matching engine is extensible

40. Java Syntactic Extender • Builds on Dylan macros having constraint-based pattern matching and code quotes • Adds procedural macros • Macros are classes • Uses CLASSPATH to find them • Example: public syntax unless { case #{ unless (?test:expression) ?:statement }: return #{ if (!?test) ?statement }; }

41. JSE Assert Example assert(moby() == 37); public syntax assert { case #{ assert(?:expression); } : return #{ if (?expression) System.out.println("Error"); } ; case #{ assert(?:expression, ?message:expression); } : return #{ if (?expression) System.out.println(?message); } ; } public class assertSExpander implements SyntaxExpander { … public Expansion expand (Fragment fragments) throws SyntaxMatchFailure { syntaxSwitch (fragments) { case #{ assert(?:expression); } : return #{ if (?expression) System.out.println("Error"); } ; case #{ assert(?:expression, ?message:expression); } : return #{ if (?expression) System.out.println(?message); } ; } } }

42. JSE Parallel forEach Example public syntax forEach { case #{ forEach (?clauses:*) ?:statement }: Fragment inits = #{ }; Fragment preds = #{ true }; Fragment nexts = #{ }; return #{ ?(loop(clauses, statement, inits, preds, nexts)) }; private Fragment loop (Fragment clauses, Fragment statement, Fragment inits, Fragment preds, Fragment nexts) throws MatchFailure { syntaxSwitch (clauses) { case #{ }: return #{ ?inits while (?preds) { ?nexts ?statement } }; case #{ ?:type ?:name in ?c:expression, ... }: Fragment newInits = #{ ?inits Iterator i = ?c.iterator(); }; Fragment newPreds = #{ ?preds & i.hasNext() }; Fragment newNexts = #{ ?nexts ?name = (?type)i.next(); }; return #{ ?(loop(..., statement, newInits, newPreds, newNexts)) }; } } }

43. Open Problems • Conventional syntax macros • More flexibility in macro shapes • Provable admissibility

44. Reading List • Bawden: Quasiquotation in Lisp • Queinnec: Lisp In Small Pieces • Graham: Advanced Techniques for Common Lisp • Kelsey, Clinger, and Rees: R5RS • Clinger: Hygienic Macros through Explicit Renaming. • Shalit: The Dylan Reference Manual • Braband and Schwartzbach: Growing Languages with Metamorphic Syntax Macros