The Evolution of the StringTemplate Engine

1. The Evolution of the StringTemplate Engine Presented to the UC Berkeley Harmonia group15 December 2004 Terence Parr University of San Francisco parrt@cs.usfca.edu

2. Overview • Introduction • what is StringTemplate? • why did I build it? • the nature of text generation • Design goals • The evolution; 4 phases • Experience • Lessons

3. What is StringTemplate • Template engine designed with Tom Burns (CEO, jGuru.com) while building commercial sites over many years in response to JSP • small: 170k uncompressed binary (w/o test rig) • Evolved from simple “document with holes” into a functional language capable of generating large class of languages • Well suited to generative programming as well as dynamic page generation; being used in new ANTLR parser generator for code generation • Distinguishing characteristic: strictly enforces separation of model and view

4. Canonical Operations • Attribute reference:<type> • Template references (possibly recursive):<statementList()> • Apply template to multi-valued attribute:<decls:decl()>or<decls:{<it.type> <it.name>;}><items:red(), blue(), green()> • Conditional include:<if(superClass)>extends <superClass><endif><if(name)>Name: <name><else>Guest<endif>

5. Example* • Translated “Program Manipulation via Interactive Transformations” example to actual StringTemplate template definition: • Integration with existing code clean, easy generateMemSize(node) ::= << int mem_size() { int result = 0; <node.fields:{result += <it.name>.mem_size();}> return result; >> *shameless StringTemplate plug

6. Semantics • Side-effect free expressions • No “state” • No defined order of execution • Lazy evaluation • Template inheritance group to subgroup • Dynamically scoped; values inherited • Recursive template instantiation

7. Design Goals • Optimized for enforcement of separation not for Turing completeness nor expressive “one-liners” • Conceptual integrity • single-minded fanaticism against entanglement • stick to a few concepts: attributes and templates • consistency; no special cases; driven by design not implementation details • proper language formalisms, semantics, sane syntax • Simple as possible • useful to nonprogrammers • makes it fast, easy to build/maintain • Powerful as possible

8. Phase I, 1999-2001 • “Document with holes”; only tag refs • Could nest templates by setting a tag value to template; dynamic scoping (“value inheritance”) • Not intended for full page generation • File of template definitions loaded into hash table; code manually created template instances <template name="db.topic.insert"> INSERT INTO Topics VALUES ( <TID>, '<topicname>' ); </template>

9. Phase II, 2001-2003 • Rewrite of jGuru.com; learned entanglement allowed by JSP is horrible! • Decided to enhance StringTemplate, letting needs dictate feature set; would not be Turing-complete! • Moved to single template per file format with $$…$$ indicating attribute ref • Needed “include” • added template reference rather than #include, macro • subconciously supported recursion

10. Phase II, 2001-2003 (Cont’d) • Needed to walk multi-valued attributes; added “apply” syntax: • $names:bold(item=names)$ • $names:bold(item=names, separator=“, “)$ • Allowed anonymous templates • $names:”<b>$names$<b>”$ • Iterated value naming convention evolved • $names:bold(item=names[i])$ • Needed multiple array walk for relational DB • $a,b:foo(x=a[i], y=b[i])$ • Finally, needed conditionalinclusion $if(userName)$ Name: $userName$ $endif(userName)$

11. Phase III, 2003-Summer 2004 • Wanted to say $names:bold()$ w/o arg • defined default attribute attr • Cleaned up IF: $if(foo)$…$endif$ • Nested anonymous blocks: $names:{…}$ • Round-robin template application • $users: rowOdd(), rowEven()$ • Properties: $user.name$ calls user.getName() • Indirect template ref: $(tname)()$; added in context of “immediate evaluation” • Template inheritance via StringTemplateGroup

12. Phase IV, 2004 • Added group file format: mutually-referential templates with formal arguments method(type,name,args,body) ::= << public <type> <name>( <args:arg(); separator=“,”> ) { <body> } >> assign(lhs,expr) ::= “<lhs> = <expr>;” …

13. Future • Named args for anonymous blocks ala Smalltalk • $list:{x | <b>$x$</b>}$ • <node.fields:{f | result += <f.name>.mem_size();}> • iterator could also choose single argument from named templates • Add parallel array walking back in • Syntax for super group? • group sub : super; • Lots of little clean up; e.g., whitespace, …

14. Experience (ANTLR v3.0) • Tree walker (controller) collects data from AST (model), pushes data into templates (view) • Decouples order of computation from order of output (this is huge) • Enforced separation guarantees easy retargeting, no code duplication, … • no code in template • no output strings in code generator • Previous code generator hopelessly entangled • Group file format (output grammar) is great! “Executable documentation”

15. Practical Lessons • Separate pattern matching from templates! • don’t make templates also analyze model • lazy evaluation essential; decouple order of computation from order of output; build in any order • Isolates templates (“user code”) from model changes, which will surely evolve • Templates may be reusable w/o embedded logic • Recursion required for nested structures • Attributes • dynamic scoping really handy • push don’t pull attributes; attributes are inert • Simple language is fast, easy to learn, easy to use, promotes retargetability

16. Philosophical Lessons • Say what you mean! e.g., no FOR-loops • Focus on principles, conceptual integrity • focus on what not how • other tools either doc with holes or Turing complete • Difficult to stick to your principles and still create usable tool; grey areas appear • Language design is hard;implementation is relatively easy • Selection pressure results in StringTemplate • output conforms to a language, hence, a grammar • strict enforcement of separation leads to grammar

17. Demo • Generating Java / XML with same model