A Theory of Hygienic Macros

1. A Theory ofHygienic Macros David Herman, Mitchell Wand Northeastern University

2. A Theory of Hygienic Macros • Introduction to macros and hygiene • The challenge of formalizing hygiene • mand its properties

3. In the beginning MACDEF SUM A,B,C LOAD A ADD B STORE C ENDDEF

4. The essence of macros • Derived (user-defined) syntactic constructs • Defined by rewriting to existing constructs • Translated at compile-time (“macro expansion”) repeat suntile  begins ; while (note) sendend

5. The essence of macros • Derived (user-defined) syntactic constructs • Defined by rewriting to existing constructs • Translated at compile-time (“macro expansion”) (repeat suntile)  (begins(while(note)s))

6. Example: short-circuit or (or (foo"bar") (baz42)) (define-macro (ore1e2) (with (te1) (iftte2))) pattern; matches subexpressions e1 and e2 evaluate first; evaluate only once evaluate only if (foo"bar") produces#f template; rewrites expression using e1 and e2

7. functions, recursion, iteration, primitive datatypes The power of macros • Abstractions for control, scope, program structure • Powerful meta-programming idioms • Simplifies language core (“extensible languages”) library writer short-circuit operators, local bindings, custom loop forms, test harnesses, data constructors, language designer conditionals, regularexpressions, little lang-uages, …

8. A brief history of hygiene

9. Naïve macro expansion (define-macro (ore1e2) (with (te1) (iftte2)))  (with (x1) (or#fx))  (with (x1) (with (t#f) (ifttx))) (with (t1) (or#ft))  (with (t1) (with (t#f) (ifttt)))

10. Naïve macros are leaky abstractions (define-macro (ore1e2) (with (te1) (iftte2))) A leaky abstraction is a brittle abstraction. Please don’t refer to any variablesnamed t in expression e2! Oh, and don’t use or in a contextthat shadows the names with or if!

11. Kohlbecker: This calls for hygiene For reasons of hygiene it will always be assumed that the bound variables that occur in a certain expression are different from the free ones. This can be fulfilled by renaming bound variables. —H. P. Barendregt

12. Hygienic macro expansion (define-macro (ore1e2) (with (te1) (iftte2))) (with (x1) (or#fx))  (with (x1) (with (t#f) (ifttx))) (with (t1) (or#ft))  (with (t1) (with (t′#f) (ift′t′t)))

13. functions, recursion, iteration, primitive datatypes functions, recursion, iteration, primitive datatypes Hygiene and syntactic abstraction libraries short-circuit operators, local bindings, custom loop forms, test harnesses, data constructors, language conditionals, regularexpressions, little lang-uages, …

14. local bindings, software contracts, custom loop forms, derived datatypes, first-class module systems, object-orientation, test harnesses, data constructors, laziness, short-circuit boolean operators, iteration protocols, coroutines, exceptions, functions, recursion, iteration, primitive datatypes threads, type checkers, static analyses, parser generators, partial evaluation, debuggers, introspection, etc… Hygiene and syntactic abstraction libraries language

15. What do we know about hygiene?

16. Hygiene by example (define-macro (ore1e2) (with (te1) (iftte2))) (with (x1) (or#fx))  (with (x1) (with (t#f) (ifttx))) (with (t1) (or#ft))  (with (t1) (with (t#f) (ifttt)))

17. Hygiene by informal rules Kohlbecker et al, ’86:Generated identifiers that become binding instances in the completely expanded program must only bind variables that are generated at the same transcription step. Clinger and Rees, ’91:1. It is impossible to write a macro that inserts a binding that can capture references other than those inserted by the macro.2. It is impossible to write a macro that inserts a reference that can be captured by bindings other than those inserted by the macro. SRFI 72:A binding for an identifier can only capture a reference to another if both were present in the source or introduced during a single evaluation of a syntax or quasisyntax form, with the understanding that the evaluation of any nested, unquoted syntax or quasisyntax forms counts as part of the evaluation of an enclosing quasisyntax.

18. Hygiene by implementation

19. But which implementation? Kohlbecker et al, ’86 Clinger & Rees ’91 Dybvig et al, ’93 Van Tonder, ’05 ?

20. Insight #1: Lexical scope (define-macro (ore1e2) (with (te1) (iftte2))) (with (x1) (or#fx))  (with (x1) (with (t#f) (ifttx))) (with (t1) (or#ft))  (with (t1) (with (t′#f) (ift′t′t))) = =

21. The essence of hygienic macros Hygienic macro expansionpreserves -equivalence.

22. What variables are bound by with? (define-macro (with (ae1) e2) ) … (with (x1) (or#fx)) (with (t1) (or#ft)) =

23. What is the binding structure? (define-macro (murkyae) (begin (set!ae) (lambda (a) e))) (lambda (foo) (murkyfoo (+foo1)))

24. What is the binding structure? (define-macro (turing-machinee) )

25. What is the binding structure? (define-macro (indecisiveae) … (set!ae) … (lambda (a) e)) (lambda (foo) (indecisivefoo (+foo1)))

26. What variables are bound by with? (define-macro (with (ae1) e2) …) ) Standard answer: look at results of expansion … (with (x1) (or#fx))  ((lambda (x) ((lambda (t) (ifttx)) #f) 1) (with (t1) (or#ft))  ((lambda (t) ((lambda (t′) (ift′t′t)) #f) 1)

27. Using expansion to define scope • To specify hygienic expansion, we need to know the binding structure (scope) of programs. • To specify the binding structure of programs, we need to know the results ofhygienic expansion. Oops.

28. Inside every large language is a small language struggling to get out… —Igarashi, Pierce, and Wadler

29. Insight #2: Specifying lexical scope

30. What variables are bound by with? (define-macro (with (ae1) e2) …) ) In practice, programmers write binding specifications: withtakes the form (with (ae1) e2) and produces an expression, where: • a is an identifier • e1 and e2 are expressions • a is bound in e2 …

31. A shape type for with (with (<a> expr) expra) →expr expression in scope of a binder expression in same environment

32. Explicitly type-annotated macros (define-macro (with (ae1) e2) :expr ((a:var) (e1:expr) (e2:expra)) ) … … (with (x1) … x …) letvalx=1in … x … end

33. Shape-directed -conversion (with (<a> expr) expra) →expr (with (x1) (or#fx)) (with (t1) (or#ft)) =

34. Shape-directed -conversion (with (<a> expr) expra) →expr (with (z1) (or#fz)) (with (t1) (or#ft)) =

35. Shape-directed -conversion (with (<a> expr) expra) →expr (with (z1) (or#fz)) (with (z1) (or#fz)) =

36. Quick recap • Macro shape types  binding structure • Binding structure  -equivalence • -equivalence  correctness of hygienic expansion So, what of hygienic expansion?

37. (x.x) 1 m: a model of hygienic macros (with(x1)x) S-expressions compiler’s core language

38. m: a model of hygienic macros core language e ::= var | var.e | ee | let syntaxvar=mineend | (ops …) source

39. Hygienic macro expansion let syntaxx = mineend→e[m/x] if BV(e) ∩FV(m) =  ((macrop=>e) s …)→(e) if (p) = s for some substitution  and BV(s) ∩FV(e) =  and BV(e) # s

40. Properties of m

41. Confluence (with(x1) (or#fx))   (with(x1) (with(t#f) (ifttx))) (x.(or#fx))1   (x.(with(t#f)(ifttx)))1

42. Hygiene For (well-typed) e0 =e′0 and fully-expanded e and e′, if e0→*eand e′0→*e′, then e =e′. * e e0   * e′ e′0

43. Hygiene…from confluence s   s1 s′1 * * s′

44. Hygiene…from confluence s0 s′0    s1 s′1 * *  s2 s′2

45. Contributions • Semantics for hygienic expansion, accommodating: • Lexically scoped macros • Binding forms of arbitrary shape • Formal definitions and proofs • -equivalence in presence of macros • Shape-directed -conversion • Confluence and hygiene

46. What we’ve gained • Formalized programmers’ mental model of the binding structure of macros. • Showed the model has good theoretical properties. • Validated intuition: hygienic macro expansion frees the programmer to -convert with impunity. • But—more features yet to model…

47. Moral: Hygienic macro expansionpreserves -equivalence. Thank you.