Applications of Type Constraints in Software Engineering Tools

1. Applications of Type Constraints in Software Engineering Tools Frank Tip IBM T.J. Watson Research Center

2. This Presentation is Based on Joint Work With • Ittai Balaban (New York University) • Dirk Bäumer (IBM Zurich Research Center) • Bjorn De Sutter (Ghent University) • Julian Dolby (IBM T.J. Watson Research Center) • Robert Fuhrer (IBM T.J. Watson Research Center) • Adam Kieżun (MIT)

3. IBM Research • about 3000 people world-wide • 1600 at IBM T.J. Watson Research Center • other sites: Almaden, Austin, Zurich, Haifa, China, India • Software Technology Department • about 70 people, director Daniel Yellin • projects on: compiler optimization (JikesRVM), aspects, performance analysis, web services, refactoring, verification, XML, ... • www.research.ibm.com/compsci/plansoft/index.html • ARTIST project (Advanced Refactoring Tools for Improving Software archiTecture) • Robert Fuhrer, Mandana Vaziri, Tim Klinger, Adam Kiezun (intern), Frank Tip (project leader) • collaboration with Eclipse JDT team at IBM Zurich • collaboration with IBM Rational • academic collaborations with Bjorn De Sutter (Ghent University), Ittai Balaban (NYU)

4. Other Research Activities • change impact analysis • given an old and a new version of a program, and a test that fails in the new version, find the subset of the source code changes responsible for the failure • with Barbara Ryder and Xiaoxia Ren (Rutgers) and Julian Dolby (IBM), Max Stoerzer (University of Passau) • papers: PASTE’01, OOPSLA’04 • Jax: an application extractor for Java • apply static analysis techniques to eliminate redundant functionality from Java applications, and apply size-reducing transformations • with Peter Sweeney, Chris Laffra, Aldo Eisma, David Streeter • transferred to IBM product (WebSphere Studio Device Developer) • papers: CACM’03, TOPLAS’02, OOPSLA’00, FSE’00, OOPSLA’99

5. Outline • background • type constraints for Java programs • notation and terminology • constraint generation rules • applications • generalization-related refactorings (OOPSLA’03) • customization of library classes (ECOOP’04) • refactorings for introducing generics (work in progress) • related work • conclusions and future work

6. Outline • background • type constraints for Java programs • notation and terminology • constraint generation rules • applications • generalization-related refactorings (OOPSLA’03) • customization of library classes (ECOOP’04) • refactorings for introducing generics (work in progress) • related work • conclusions and future work

7. Scope of our Research • start with a type-correct Java program P • for a given transformation that transforms P into P’ • we would like to check/guarantee that P’ is type-correct • we would like to check/guarantee that P’ has the same behavior as P • (in some cases) compute “maximal” P’ for which the above properties hold • we use type constraints to establish these properties • formalism for expressing relationships between program expressions that must hold in order for a program to be type-correct • traditionally used for type checking and type inference • transformations under consideration • refactorings: well-known maintenance operations, usually aimed at making code more flexible/general; proposed by the programmer • driven by static/dynamic analysis in link-time optimizer

8. Refactoring • refactoring: the application of behavior-preserving transformations to a program in order to improve a program’s design • eliminating undesirable program characteristics • e.g., duplicated code, classes/methods that are too large,... • making existing classes/methods usable in new contexts • preparing for extensions • breaking up monolithic systems into components • introduction of design patterns • refactoring (noun): a specific program transformation. Usually identified by: • name (e.g., “Extract Method”, “Pull Up Members”, ...) • preconditions • a specific set of transformations to be performed by a programmer or by an automated tool

9. Refactoring • pioneered by Griswold [1991], Opdyke [1992] & Johnson, leading to Smalltalk Refactoring Browser [Roberts 1992] • recently popularized by continuous-refinement methodologies such as “Extreme Programming” [Beck 2000] • catalogues of common refactorings:[Fowler 1999], [Kerievsky 2003] • Fowler describes refactorings as a series of steps to be performed by the programmer • manual refactoring is very error-prone • renewed interest in automated refactoring support in IDEs • refactoring support featured in Eclipse, IntelliJ IDEA, OmniCore, ...

10. Categories of Refactorings (see Fowler’s book) • making method calls simpler • Rename Method, Add/Remove Parameter, ... • composing methods • Extract Method, Inline Method, Inline Local, ... • moving features between objects • Move Method, Move Field, Extract Class, ... • organizing data • Self-Encapsulate Field, Replace Data Value with Object, ... • simplifying/eliminating conditionals • Replace Conditional with Polymorphism, ... • dealing with generalization • Extract Interface, Pull Up Members, ...

11. Eclipse (www.eclipse.org) • open-source (CPL) development environment • implemented in Java, XML • basis for commercial offerings by IBM (WSAD, WSDD) and others • plugin-architecture • plugins contribute views/perspectives • plugins provide extension points • state-of-the-art development environment for Java • quick-fixes, refactoring, type hierarchy view, call hierarchy, search facilities • support for other languages (C, Smalltalk, AspectJ) • various IBM programs focused on Eclipse • Eclipse Innovation Grants for academics (2002, 2003) • Eclipse Technology Exchange meetings (ICSE, OOPSLA, ECOOP) • solid basis for research/education projects • Penumbra, Gild, Hipikat, ECESIS, ... • Continuous Testing, Java Traits, Ownership Types, ...

12. Demo: Eclipse Refactorings

13. Outline • background • type constraints for Java programs • notation and terminology • constraint generation rules • applications • generalization-related refactorings (OOPSLA’03) • customization of library classes (ECOOP’04) • refactorings for introducing generics (work in progress) • related work • conclusions and future work

14. Type Constraints • formalism developed in 1990s • captures relationships between types of program constructs • original purpose: type checking/inference • prove that certain kinds of errors cannot occur at run-time • e.g., no “message not understood” errors • we use a variation on the formalism from a book by Palsberg & Schwartzbach • adapted/extended to capture the semantics of Java

15. Type Constraints Notation [E] the type of expression E [M] the declared return type of method M [F] the declared type of field F Decl(M) the type that contains method M Param(M,i) the i-th parameter of method M  ,  subtype relation

16. Syntax of Type Constraints [E] = [E’] the type of expression E must be the same as the type of expression E’ [E]  [E’] the type of expression E is a proper subtype of the type of expression E’ [E]  [E’] either [E] = [E’] or [E]  [E’] [E]  T the type of expression E is defined to be T [E]  [E1] or ... or [E]  [Ek] disjunction: at least one of subconstraints [E]  [E1],...,[E]  [Ek] must hold

17. Generating Type Constraints

18. Virtual Method Calls for a call E.m(E1,...,En)to a virtual method M RootDefs(M) = { M’ | M overrides M’, and there exists no M’’ (M’’ M’) such that M’ overrides M’’ }

19. Dictionary Map Hashtable Constraints for Virtual Method Calls put() put() publicvoidfoo(Strings1,Strings2){ Hashtableh=newHashtable(); h.put(s1,s2); } [h]  Decl(Map.put(...)) or [h]  Decl(Dictionary.put(...)) Map put() [h]  Map or [h]  Dictionary

20. Constraints for Overriding & Hiding if method M’ overrides method M, M’ M if fieldF’hides fieldF

21. Casts for a cast (C)E • the latter constraint need not be generated if C or |E| is an interface • these constraints only capture the requirements for type-correctness (not necessarily program behavior) • it is possible to avoid generating disjunctions by preserving the “directionality” of the cast

22. Outline • background • type constraints for Java programs • notation and terminology • constraint generation rules • applications • generalization-related refactorings (OOPSLA’03) • customization of library classes (ECOOP’04) • refactorings for introducing generics (work in progress) • related work • conclusions and future work

23. Refactoring for Generalization • several refactorings are concerned with generalization • moving methods/fields to superclasses and subclasses • splitting & merging of classes • manipulating the types of declarations • Chapter 11 of Fowler’s book mentions: • Extract Interface • Pull Up Member(s) • Push Down Member(s) • Extract Subclass • Generalize Type

24. Extract Interface – Recipe • select class C • select subset M of C’s methods • create interface I containing declarations of the methods in M • add inheritance “C implements I” • “Adjust client type declarations to use the interface” [Fowler, p.342]

25. Extract Interface: An Example • List class with methods as follows: • add(Comparable) add an element • addAll(List) add contents of another List • iterator() iteration support • sort() sorts the list • ListIterator class • implements java.util.Iterator; methods hasNext(), next() • Client class • create List; add some elements • add contents of another List; sort the List • print contents of the List • extract an interface Bag from List • declares add(Comparable), addAll(List), iterator()

26. interfaceBag{ publicIteratoriterator(); publicListadd(Comparablee); publicListaddAll(Listv0); } classListimplements Bag { intsize=0;Comparable[]elems=newComparable[10]; publicIteratoriterator(){returnnewListIterator(this);} publicListadd(Comparablee){ if(this.size+1==this.elems.length){ Comparable[]newElems=newComparable[2*this.size]; System.arraycopy(this.elems,0,newElems,0,this.size); this.elems=newElems; } this.elems[this.size++]=e;returnthis; } publicListaddAll(Listv1){ java.util.Iteratori=v1.iterator(); for(;i.hasNext();this.add((Comparable)i.next())); returnthis; } publicvoidsort(){/* insertion sort */} } List/Bag Example (1)

27. classListIteratorimplementsjava.util.Iterator{ privateintcount=0;privateListv2; ListIterator(Listv3){v2=v3;} publicbooleanhasNext(){returnthis.count<this.v2.size;} publicObjectnext(){returnthis.v2.elems[this.count++];} } publicclassClient{ publicstaticvoidmain(String[]args){ Listv4=createList();populate(v4);update(v4); sortList(v4);print(v4); } staticListcreateList(){returnnewList();} staticvoidpopulate(Listv5){v5.add("foo").add("bar");} staticvoidupdate(Listv6){ Listv7=newList().add("zap").add("baz");v6.addAll(v7);} staticvoidsortList(Listv8){v8.sort();} staticvoidprint(Listv9){ for(Iteratoriter=v9.iterator();iter.hasNext();) System.out.println("Object: "+iter.next()); } } List List/Bag Example (2) List List List List

28. Problem Statement • identify all declarations that can be updated to make use of the newly extracted interface • want to be able to reason about: • correctness of the solution • maximality of the solution

29. Using Type Constraints • declared types of variables, fields, parameters constrained by: • field access, method calls • assignments, parameter-passing • several other invariants must be maintained to preserve type-correctness & program behavior • Observation: all these constraints can be stated succinctly and uniformly using type constraints

31. Observation • the constraints for the original program contain all the information we need • some declarations cannot be updated List  [v3]  [v2]  List [v4]  [v8]  List • other variables are less constrained [v1]  Bag

32. Algorithm for Determining “Updatable” Declarations • iterative algorithm for determining non-updatable declarations • first determine declarations that cannot be updated because of member access (e.g., [v2]  List, [v8]  List) • if x is non-updatable, and there is a type constraint [y]  [x], [y] = [x], or [y] < [x] then y is non-updatable • iterate until fixed-point is reached

33. Non-Updatable Declarations for the Example Program { v2, v3, v4, v8, Client.createList() } (consistent with earlier result)

34. Justification (Details in Paper) • type-correctness • updating the “updatable” declaration elements results in a program that satisfies all type constraints • preservation of behavior • argument based on the fact that method dispatch, cast/instanceof behavior do not depend on declared types • maximality • updating any non-updatable declarations will result in the violation of type constraints

35. ? Another Refactoring: Pull Up Members [this]  Decl(B.foo()) Decl(B.foo())  B [B.foo()]  B [this]  [B.foo()] classA{ ... } classB extends A{ public B foo(){returnthis;} }

36. Pull Up Members (2) classA{ public B foo(){returnthis;} } classB extends A{ ... } [this]  Decl(A.foo()) Decl(A.foo())  A [A.foo()]  B [this] ≤ [A.foo()]

37. Other Refactorings • Generalize Type • update the type of a declaration E • use type constraints to determine allowable supertypes/subtypes • may enable Pull Up Members in certain cases • Extract Subclass • splitting of a class • can be treated similarly as Extract Interface • Push Down Members • the “inverse” of Pull Up Members • similar issues

38. Perspective • infer from original program a system of ordering constraints between types of declaration elements • original program is just one possible solution • Extract Interface • declarations: variables • locations of members: constants • Pull Up Members • declarations: constants • locations of members: variables • Generalize Type • selected declaration: variable • all other declarations & locations of members: constants

39. Demo: Extract Interface & Generalize Type

40. Outline • background • type constraints for Java programs • notation and terminology • constraint generation rules • applications • generalization-related refactorings (OOPSLA’03) • customization of library classes (ECOOP’04) • refactorings for introducing generics (work in progress) • related work • conclusions and future work

41. Class Libraries • class libraries improve programmer productivity • programmers don’t have to waste time developing & debugging standard infrastructure • but... class libraries are often implemented with some typical/ average usage pattern in mind • for example: container class implementations assume that: • elements are accessed often & frequently • a large number of elements is stored • performance loss if the actual usage of a library class differs from this typical usage pattern • “MyHashTable”, “SmartHashtable”,... in various benchmarks

42. Our Approach • derive custom versions from library classes • rewrite application to use these custom versions • ship custom library classes with application • technical foundations: • use type constraints to determine where custom classes can be used • use profile information to determine where introducing custom classes is profitable • use static analysisandprofile information to decide how to customize

43. Object O S String Dictionary D M Map H Hashtable O S D M H Example Program class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H r2 = new H(); H r3 = new H(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } } class Example { void foo(Map m){ Hashtable r1 = new Hashtable(); JTree tree = new JTree(r1); Hashtable r2 = new Hashtable(); Hashtable r3 = new Hashtable(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(Object o){ Hashtable r4 = (Hashtable) o; if (r4.contains(“FOO”)) {…} } }

44. O S D M H H1 H2 class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H r2 = new H()H1(); H r3 = new H()H2(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } } How to customize? class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H r2 = new H(); H r3 = new H(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } }

45. class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H r2 = new H()H1(); H r3 = new H()H2(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } } H1 H2 class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H H1 r2 = new H()H1(); HH2 r3 = new H()H2(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } } H1 H2 O How to customize? S D M H

46. class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H H1 r2 = new H()H1(); HH1 r3 = new H()H1(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } } H1 H2 O How to customize? S D M H

47. class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H H1 r2 = new H()H1(); HH1 r3 = new H()H1(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } } H1 H2 class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H AH r2 = new H()H1(); HAH r3 = new H()H2(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } } H1 H2 AH O How to customize? S D M H • update allocations of library types • update declarations

48. O S D M H H1 H2 AH Restrictions? call to: javax.swing.JTree(Hashtable) class Example { void foo(M m){ H r1 = new H(); JTree tree = new JTree(r1); H r2 = new H(); H r3 = new H(); r2.put(“FOO”,“BAR”); bar(r3); r2 = r3; r2.putAll(m); bar(“HELLO”); } void bar(O o){ H r4 = (H) o; if (r4.contains(“FOO”)) {…} } } • type correctness • interface compatibility • preserve behavior of cast and instanceof operations

49. Outline of Approach • generate type constraints for program • additional constraints generated to ensure that behavior of cast/instanceof operations is preserved • constraint simplification • rewrite/replace all constraints to use “≤” only • solve the resulting constraint system • rewrite the program’s declarations and allocation sites to use the inferred types

50. Preserving the Behavior of Cast & instanceof • we want to change declarations and allocation sites • need to ensure that cast/instanceof operations succeed and fail in exactly the same cases as before • use points-to analysis to approximate the set of objects to which the cast/instanceof is applied • easily expressed using constraint (to be replaced with a ≤constraint) publicclass Example { void zip(){ zap(new Hashtable()); // A1 zap(new String()); // A2 } void zap(Object o){ Hashtable h = (Hashtable)o; // C } } A1 ≤ C A2  C