Macros and general code walkers in Lisp: how useful! or, how useful?

1. Macros and general code walkers in Lisp:how useful! or, how useful? Ernst van Waning evw@infometrics.nl

2. Overview • Short overview of macros in Lisp • Macros transform macro calls • Macros are not code transformers • In frustration I have sometimes thought macros have a certain smell • Lisp has many specific code walkers • Q: can we make general code walkers? Macros in Common Lisp

3. Macros in Lisp • Macros are replaced by their expansion (expansion time) • The original call evaporates • After expansion, their code may be evaluated (evaluation time) Macros in Common Lisp

4. Expansion and evaluation • Arguments of macro calls are not evaluated • At expansion time, appropriate error messages do make sense • Evaluation is done after the macro expanded Macros in Common Lisp

5. Debugging macros • Macroexpand-1 expands your macro once (i.e., applies the expander once) • Macroexpand expands your macro call until it is no longer a macro call • With a non-macro they just return (values form ’nil) • Expanding all macros in all subforms is a different game: code walker Macros in Common Lisp

6. Tracing expansions • Useful for debugging local macros • Useful for studying system macros (defmacro trexp (form &environment env) (let ((*print-pretty* t)) (multiple-value-bind (expansion expanded-p) (macroexpand-1 form env) (cond (expanded-p (format *trace-output* "~&~S~%-m1->~%“form) `(trexp ,expansion)) (t (format *trace-output* "~&~S~%~%" form) expansion))))) Macros in Common Lisp

7. A model for macros • Lisp is explained using Lisp, so we explain Lisp’s macros with Lisp (defmacro our-expander (name)`(get ,name 'expander)) • Our-expander is an accessor, i.e., it can be set with setf Macros in Common Lisp

8. A model for macros (defun our-macro-call? (xpr)(and (consp xpr) (our-expander (car xpr)))) (defun our-macroexpand-1 (xpr) (if (our-macro-call? xpr) (funcall (our-expander (car xpr)) xpr) xpr)) Macros in Common Lisp

9. A model for macros (defmacro our-defmacro (name args &body body)(let ((g (gensym)));unique symbol `(progn (setf (our-expander ',name);store (lambda (,g) (block ,name;for return-from (destructuring-bind ,args(cdr ,g) ,@body)))) ',name))) Macros in Common Lisp

10. A model for macros • The actual argument to an expander is the entire macro call • Macros do not evaluate the arguments of the macro call • The expander is really a function retaining its lexical environment • Macros are not first-class values Macros in Common Lisp

11. What are macros good for?

12. Macros control evaluation • Special syntax • (dotimes (i 10 i) (princ i)) • Implicit quoting • `(setf (our-expander ‘,name) …) • Design your own control structures • (aif xpr pos zero neg) • Multiple evaluations • (dotimes (i 10 i) (princ i)) Macros in Common Lisp

13. Macros compute during expansion (defmacro avg (&rest numbers) `(/ (+ ,@numbers) ,(length numbers))) • Definers also compute during expansion • Defun, defmacro, &c. do a lot of bookkeeping • Think of writing your own definers! Macros in Common Lisp

14. Macros as accessors (setf (our-expander name) new) ~m~> (something like) (let ((#:g452 name)(#:g453 'expander)(#:g455 new)) (inverse-get #:g452 #:g453 #:g455)) • Setf looks “inside” its first argument • There are many accessors with their own setf-expanders Macros in Common Lisp

15. Macros set up lexical bindings (dotimes (i 10 i) (princ i)) • With-macros • with-open-file, with-output-to-string &c. • Do-macros • do, dolist, dotimes, do-symbols &c. Macros in Common Lisp

16. What more do we want?

17. Macros as code transformers • Suppose we want to have type-strict expressions but without pain • A macro scalar is fine: • (scalar :e) –m-> 2.718281… • But what if I want this? • (scalar (exp 1))-m-> (exp (scalar 1))-m-> (exp 1d0)-m-> 2.718281… ;at expansion time Macros in Common Lisp

18. Code transformation • Sums of squares of real numbers are non-negative (at least, in theory): • (scalar (sqrt ssq)) -m*-> (the (scalar 0) (sqrt (the (scalar 0) ssq))) • Can we prove that ssq is non-negative by expanding a macro? • Macros do not allow arbitrary code transformations Macros in Common Lisp

19. Domain specific extensions • Lisp is an extensible programming language • Build applications as extensions to Lisp • Telescoping: a strategy to automatically generate highly optimized domain specific libraries • But how easy is it to write in Lisp • domain-specific semantic checks? • domain-specific optimizations? Macros in Common Lisp

20. Term Rewriting Systems (TRS) aka Code Walkers • Useful in automatic theorem proving • Lisp has many code walkers: • Read, eval, print, compile • Cross-referencers, and more… • Lisp makes it easy to write specific ones Macros in Common Lisp

21. A general code walker in Lisp? • What is a general code walker? • Minimizes effort to write specific ones • Traverses all forms, a fortiori special ones • …more…? • Integration with macros is possible • Change macroexpand-1 • Surrounding code can be made visible by means of a stack Macros in Common Lisp

22. A general code walker in Lisp? • The infamous 24 special forms of Lisp: • Special forms implemented as macro • Macros implemented as special form • Traversal of implementation-specific special forms • All special forms retrievable in any Lisp • We can check if they have a walker Macros in Common Lisp

23. A general code walker in Lisp? • What about macros? • What about system-macros? • What about the portability of your walkers? Macros in Common Lisp

24. A general code walker in Lisp? • Looks like a clear idea, but is it? • Even if the idea would be clear, there are real problems to solve • Would it really help you write your code walkers? • Specific code walkers, however, remain very useful Macros in Common Lisp