Using Java Tree Builder

1. Using Java Tree Builder Professor Yihjia Tsai Tamkang University

2. Lecture Outline • Introduction • Syntax Directed Translation • Java Virtual Machine • Examples • Administration

3. Introduction • The Java Tree Builder (JTB) is a tool used to automatically generate syntax trees with the Java Compiler Compiler (JavaCC) parser generator. • It’s based on the Visitor design pattern please see the section entitled • Why Visitors?

4. Why Visitors? • The Visitor pattern is one among many design patterns aimed at making object-oriented systems more flexible • The issue addressed by the Visitor pattern is the manipulation of composite objects. • Without visitors, such manipulation runs into several problems as illustrated by considering an implementation of integer lists, written in Java

5. Integer lists, written in Java(Without Generics) interface List {} class Nil implements List {} class Cons implements List { int head; List tail; } What happens when we write a program which computes the sum of all components of a given List object?

6. First Attempt: Instanceof and Type Casts List l; // The List-object we are working on. int sum = 0; // Contains the sum after the loop. boolean proceed = true; while (proceed) { if (l instanceof Nil) proceed = false; else if (l instanceof Cons) { sum = sum + ((Cons) l).head; // Type cast! l = ((Cons) l).tail; // Type cast! } } What are the problems here?

7. What are the problems here? • Type Casts? • We want static (compile time) type checking • Flexible? • Probably not well illustrated with this example

8. Second Attempt: Dedicated Methods interface List { int sum(); } class Nil implements List { public int sum() { return 0; } } class Cons implements List { int head; List tail; public int sum() { return head + tail.sum(); } }

9. Tradeoffs • Can compute the sum of all components of a given List-object l by writing l.sum(). • Advantage: type casts and instanceof operations have disappeared, and that the code can be written in a systematic way. • Disadvantage: Every time we want to perform a new operation on List-objects, say, compute the product of all integer parts, then new dedicated methods have to be written for all the classes, and the classes must be recompiled

10. Third Attempt: The Visitor Pattern. interface List { void accept(Visitor v); } class Nil implements List { public void accept(Visitor v) { v.visitNil(this); } } class Cons implements List { int head; List tail; public void accept(Visitor v) { v.visitCons(this); } }

11. Second Part of Visitor Idea interface Visitor { void visitNil(Nil x); void visitCons(Cons x); } class SumVisitor implements Visitor { int sum = 0; public void visitNil(Nil x) {} public void visitCons(Cons x){ sum = sum + x.head; x.tail.accept(this); } }

12. Summary • Each accept method takes a visitor as argument. • The interface Visitor has a header for each of the basic classes. • We can now compute and print the sum of all components of a given List-object l by writing SumVisitor sv = new SumVisitor(); l.accept(sv); System.out.println(sv.sum);

13. Summary Continued • The advantage is that one can write code that manipulates objects of existing classes without recompiling those classes. • The price is that all objects must have an accept method. • In summary, the Visitor pattern combines the advantages of the two other approaches

14. Frequent type casts? Frequent recompilation? Instanceof and type casts Yes No Dedicated methods No Yes The Visitor pattern No No Summary Table

15. Overview of Generated Files • To begin using JTB, simply run it using your grammar file as an argument • Run JTB without any argumentsfor list. • This will generate an augmented grammar file, as well as the needed classes

16. Details • jtb.out.jj, the original grammar file, now with syntax tree building actions inserted • The subdirectory/package syntaxtree which contains a java class for each production in the grammar • The subdirectory/package visitor which contains Visitor.java, the default visitor interface, also • DepthFirstVisitor.java, a default implementation which visits each node of the tree in depth-first order. • ObjectVisitor.java, another default visitor interface that supports return value and argument. • ObjectDepthFirst.java is a defualt implemetation of ObjectVisitor.java.

17. General Instructions • To generate your parser, simply run JavaCC using jtb.out.jj as the grammar file. • Let's take a look at all the files and directories JTB generates.

18. The grammar file • Named jtb.out.jj • This file is the same as the input grammar file except that it now contains code for building the syntax tree during parse. • Typically, this file can be left alone after generation. • The only thing that needs to be done to it is to run it through JavaCC to generate your parser

19. The syntax tree node classes • This directory contains syntax tree node classes generated based on the productions in your JavaCC grammar. • Each production will have its own class. If your grammar contains 42 productions, this directory will contain 42 classes (plus the special automatically generated nodes--these will be discussed later), with names corresponding to the left-hand side names of the productions. • Like jtb.out.jj, after generation these files don't need to be edited. Generate them once, compile them once, and forget about them

20. Example • Let's examine one of the classes generated from a production. Take, for example, the following production void ImportDeclaration() : {} {   "import" Name() [ "." "*" ] ";“ }

21. What gets produced?Part 1 // Generated by JTB 1.1.2 // package syntaxtree; /** * Grammar production:   * f0 -> "import"   * f1 -> Name()   * f2 -> [ "." "*" ]   * f3 -> ";"   */ public class ImportDeclaration implements Node {    public NodeToken f0;    public Name f1;    public NodeOptional f2;    public NodeToken f3; All parts of a production are represented in the tree, including tokens.

22. The Syntax Tree Classes • Notice the package "syntaxtree". • The purpose of separating the generated tree node classes into their own package is that it greatly simplifies file organization, particularly when the grammar contains a large number of productions. • It’s often not necessary to pay the syntax classes any more attention. All of the work is to done to the visitor classes. • Note that this class implements an interface named Node.

23. Automatically-Generated Tree Node Interface and Classes

24. Node • The interface Node is implemented by all syntax tree nodes. Node looks like this: public interface Node extends java.io.Serializable {    public void accept(visitor.Visitor v);    public Object accept(visitor.ObjectVisitor v, Object argu); }

25. Nodes and Accept • All tree node classes implement the accept() method. • In the case of all the automatically-generated classes, the accept() method simply calls the corresponding visit(XXXX n) (where XXXX is the name of the production) method of the visitor passed to it. • Note that the visit() methods are overloaded, i.e. the distinguishing feature is the argument each takes, as opposed to its name.

26. Two New Features • Two features presented in JTB 1.2 may be helpful • The first is that Node extends java.io.Serializable, meaning that you can now serialize your trees (or subtrees) to an output stream and read them back in. • Secondly, there is one accept() method that can take an extra argument and return a value.

27. What gets produced?Part 1 // Generated by JTB 1.1.2 // package syntaxtree; /** * Grammar production:   * f0 -> "import"   * f1 -> Name()   * f2 -> [ "." "*" ]   * f3 -> ";"   */ public class ImportDeclaration implements Node {    public NodeToken f0;    public Name f1;    public NodeOptional f2;    public NodeToken f3; All parts of a production are represented in the tree, including tokens.

28. NodeListInterface • The interface NodeListInterface is implemented by NodeList, NodeListOptional, and NodeSequence. NodeListInterface looks like this: public interface NodeListInterface extends Node { public void addNode(Node n);    public Node elementAt(int i);    public java.util.Enumeration elements();    public int size(); }

29. Details • Interface not generally needed but can be useful when writing code which only deals with the Vector-like functionality of any of the three classes listed above. • addNode() is used by the tree-building code to add nodes to the list. • elements() is similar to the method of the same name in Vector, returning an Enumeration of the elements in the list. • elementAt() returns the node at the ith position in the list (starting at 0, naturally). • size() returns the number of elements in the list

30. NodeChoice • NodeChoice is the class which JTB uses to represent choice points in a grammar. An example of this would be ( "abstract" | "final" | "public" ) • JTB would represent the production void ResultType() : {} {   "void" | Type() } • as a class ResultType with a single child of type NodeChoice.

31. Details • The type stored by this NodeChoice would not be determined until the file was actually parsed. • The node stored by a NodeChoice would then be accessible through the choice field. • Since the choice is of type Node, typecasts are sometimes necessary to access the fields of the node stored in a NodeChoice.

32. Implementation public class NodeChoice implements Node {    public NodeChoice(Node node, int whichChoice);    public void accept(visitor.Visitor v);    public Object accept(visitor.ObjectVisitor v, Object argu);    public Node choice;    public int which; }

33. Which One? • Another feature of NodeChoice is the field which for determining which of the choices was selected • The which field is used to see which choice was used • If the first choice is selected, which equals 0 (following the old programming custom to start counting at 0). • If the second choice is taken, which equals 1. The third choice would be 2, etc. • Note that your code could potentially break if the order of the choices is changed in the grammar.

34. NodeList • NodeList is the class used by JTB to represent lists. An example of a list would be ( "[" Expression() "]" )+ • JTB would represent the javacc production: void ArrayDimensions() : {} { ( "[" Expression() "]" )+ ( "[" "]" )* } • as a class ArrayDimensions() with children NodeList and NodeListOptional respectively.

35. Details • NodeLists use java.lang.Vectors to store the lists of nodes. • Like NodeChoice, typecasts may occasionally be necessary to access fields of nodes contained in the list.

36. Implementation public class NodeList implements NodeListInterface {    public NodeList();     public void addNode(Node n);    public Enumeration elements();    public Node elementAt(int i);     public int size();    public void accept(visitor.Visitor v);     public Object accept(visitor.ObjectVisitor v, Object argu);    public Vector nodes; }

37. NodeToken • This class is used by JTB to store all tokens into the tree, including JavaCC "special tokens" (if the -tk command-line option is used). • In addition, each NodeToken contains information about each token, including its starting and ending column and line numbers.

38. Implementation public class NodeToken implements Node {    public NodeToken(String s);    public NodeToken(String s, int kind, int beginLine,  int beginColumn, int endLine, int endColumn); public String toString();    public void accept(visitor.Visitor v);    public Object accept(visitor.ObjectVisitor v, Object argu); // -1 for these ints means no position info is available. // … }

39. Continued public class NodeToken implements Node { // …. public String tokenImage;      public int beginLine, beginColumn, endLine, endColumn; // -1 if not available.    // Equal to the JavaCC token "kind" integer.       public int kind;   // Special Token methods below    public NodeToken getSpecialAt(int i);    public int numSpecials();    public void addSpecial(NodeToken s);    public void trimSpecials();    public String withSpecials();   public Vector specialTokens; }

40. Token Details • The tokens are simply stored as strings. • The field tokenImage can be accessed directly, and the toString() method returns the same string. • Also available is the kind integer. • JavaCC assigns each type of token a unique integer to identify it. • This integer is now available in each JTB NodeToken. For more information on using the kind integer, see the JavaCC documentation.

41. Member Variables of Generated Classes • Next comes the member variables of the ImportDeclaration class. • These are generated based on the RHS of the production. Their type depends on the various items in the RHS and their names begin with f0 and work their way up. • Why are they public? • Visitors which must access these fields reside in a different package than the syntax tree nodes • Package visibility cannot be used. • Breaking encapsulation was a necessary evil in this case.

42. What gets produced?Part 2 public ImportDeclaration(NodeToken n0, Name n1, NodeOptional n2, NodeToken n3) {       f0 = n0; f1 = n1; f2 = n2; f3 = n3;    }   public ImportDeclaration(Name n0, NodeOptional n1) {       f0 = new NodeToken("import"); f1 = n0; f2 = n1; f3 = new NodeToken(";");    }  

43. Constructors • The next portion of the generated class is the standard constructor. It is called from the tree-building actions in the annotated grammar so you will probably not need to use it. • Following the first constructor is a convenience constructor with the constant tokens of the production already filled-in by the appropriate NodeToken. This constructor's purpose is to help in manual construction of syntax trees.

44. What gets produced?Part 3 public void accept(visitor.Visitor v) {       v.visit(this);    }    public Object accept(visitor.ObjectVisitor v, Object argu) {       return v.visit(this,argu);    } }

45. The Accept Methods • After the constructor are the accept() methods. • These methods are the way in which visitors interact with the class. void accept(visitor.Visitor v) • works with Visitor Object accept(visitor.ObjectVisitor v, Object argu) • works with ObjectVisitor

46. References • Java Tree Builder Documentation • Why Visitors? • Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995