Refactoring Functional Programs

1. Refactoring Functional Programs Huiqing Li Claus Reinke Simon Thompson Computing Lab, University of Kent www.cs.kent.ac.uk/projects/refactor-fp/

2. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • format :: [String] -> String • format [] = [] • format [x] = [x] • format (x:xs) = (x ++ "\t") : fomrat xs

3. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • format :: [String] -> String • format [] = [] • format [x] = [x] • format (x:xs) = (x ++ "\t") : fomrat xs

4. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • format :: [String] -> String • format [] = [] • format [x] = [x] • format (x:xs) = (x ++ "\t") : format xs

5. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • format :: [String] -> String • format [] = [] • format [x] = [x] • format (x:xs) = (x ++ "\t") : format xs

6. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • format :: [String] -> [String] • format [] = [] • format [x] = [x] • format (x:xs) = (x ++ "\t") : format xs

7. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • format :: [String] -> [String] • format [] = [] • format [x] = [x] • format (x:xs) = (x ++ “\t") : format xs

8. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • format :: [String] -> [String] • format [] = [] • format [x] = [x] • format (x:xs) = (x ++ "\n") : format xs

9. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • appNL :: [String] -> [String] • appNL [] = [] • appNL [x] = [x] • appNL (x:xs) = (x ++ "\n") : appNL xs

10. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . format • appNL :: [String] -> [String] • appNL [] = [] • appNL [x] = [x] • appNL (x:xs) = (x ++ "\n") : appNL xs

11. Writing a program • -- appNL a list of Strings, one per line • table :: [String] -> String • table = concat . appNL • appNL :: [String] -> [String] • appNL [] = [] • appNL [x] = [x] • appNL (x:xs) = (x ++ "\n") : appNL xs

12. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . appNL • appNL :: [String] -> [String] • appNL [] = [] • appNL [x] = [x] • appNL (x:xs) = (x ++ "\n") : appNL xs

13. Writing a program • -- format a list of Strings, one per line • table :: [String] -> String • table = concat . appNL • where • appNL :: [String] -> [String] • appNL [] = [] • appNL [x] = [x] • appNL (x:xs) = (x ++ "\n") : appNL xs

14. Refactoring • Refactoring means changing the design of program … • … without changing its behaviour. • Refactoring comes in many forms • micro refactoring as a part of program development, • major refactoring as a preliminary to revision, • as a part of debugging, … • As programmers, we do it all the time.

15. Not just programming • Paper or presentation • moving sections about; amalgamate sections; move inline code to a figure; animation; … • Proof • introduce lemma; remove, amalgamate hypotheses, … • Program • the topic of the lecture

16. Overview of the talk • Example refactorings … what do we learn? • Refactoring functional programs • Generalities • Tooling: demo, rationale, design. • What comes next? • Conclusions

17. Refactoring Functional Programs • 3-year EPSRC-funded project • Explore the prospects of refactoring functional programs • Catalogue useful refactorings • Look into the difference between OO and FP refactoring • A real life refactoring tool for Haskell programming • A formal way to specify refactorings … and a set of proofs that the implemented refactorings are correct. • Currently mid-project: the latest HaRe release is module-aware and has module refactorings.

18. Refactoring functional programs • Semantics: can articulate preconditions and … • … verify transformations. • Absence of side effects makes big changes predictable and verifiable … … unlike OO. • Language support: expressive type system, abstraction mechanisms, HOFs, … • Opens up other possibilities … proof …

19. f x y = …  Name may be too specific, if the function is a candidate for reuse. findMaxVolume x y = …  Make the specific purpose of the function clearer. Rename Needsscopeinformation: just change thisfand not allfs(e.g. local definitions or variables). Needsmoduleinformation: change fwherever it is imported.

20. f x y = … h … where h = …  Hide a function which is clearly subsidiary to f; clear up the namespace. f x y = … (h y) … h y = …  Makes h accessible to the other functions in the module and beyond. Lift / demote Needsfree variableinformation: which of the parameters of f is used in the definition of h? Need h not to be defined at the top level, … , DMR.

21. Lessons from the first examples • Changes are not limited to a single point or even a single module: diffuse and bureaucratic … • … unlike traditional program transformation. • Many refactorings bidirectional … • … as there is never a unique correct design.

22. How to apply refactoring? • By hand, in a text editor • Tedious • Error-prone • Depends on extensive testing • With machine support • Reliable • Low cost: easy to make and un-make large changes. • Exploratory … a full part of the programmer’s toolkit.

23. Machine support invaluable • Current practice: editor + type checker (+ tests). • Our project: automated support for a repertoire of refactorings … • … integrated into the existing development process: Haskell IDEs such as vim and emacs.

24. Demonstration of HaRe, hosted in vim.

25. Proof of concept … • To show proof of concept it is enough to: • build a stand-alone tool, • work with a subset of the language, • ‘pretty print’ the refactored source code in a standard format.

26. … or a useful tool? • To make a tool that will be used we must: • integrate with existing program development tools: the program editors emacs and vim: only add to their capabilities; • work with the complete Haskell 98 language; • preserve the formatting and comments in the refactored source code; • allow users to extend and script the system.

27. Refactorings implemented in HaRe • Move def between modules • Delete /add to exports • Clean imports • Make imports explicit • Rename • Delete • Lift (top / one level) • Demote • Introduce definition • Remove definition • Unfold • Generalise • Add / remove params • All these refactorings are module aware.

28. The Implementation of Hare Information gathering Pre-condition checking Program transformation Program rendering

29. Information needed • Syntax: replace the function called sq, not the variable sq …… parse tree. • Static semantics: replace this function sq, not all the sq functions …… scope information. • Module information: what is the traffic between this module and its clients …… call graph. • Type information: replace this identifier when it is used at this type …… type annotations.

30. Infrastructure • To achieve this we chose to: • build a tool that can interoperate with emacs, vim, … yet act separately. • leverage existing libraries for processing Haskell 98, for tree transformation, yet … • … modify them as little as possible. • be as portable as possible, in the Haskell space.

31. The Haskell background • Libraries • parser: many • type checker: few • tree transformations: few • Difficulties • Haskell98 vs. Haskell extensions. • Libraries: proof of concept vs. distributable. • Source code regeneration. • Real project

32. Programatica • Project at OGI to build a Haskell system … • … with integral support for verification at various levels: assertion, testing, proof etc. • The Programatica project has built a Haskell front end in Haskell, supporting syntax, static, type and module analysis … • … freely available under BSD licence.

33. The Implementation of Hare Information gathering Pre-condition checking Program transformation Program rendering

34. First steps … lifting and friends • Use the Haddock parser … full Haskell given in 500 lines of data type definitions. • Work by hand over the Haskell syntax: 27 cases for expressions … • Code for finding free variables, for instance …

35. Finding free variables … 100 lines • instance FreeVbls HsExp where • freeVbls (HsVar v) = [v] • freeVbls (HsApp f e) • = freeVbls f ++ freeVbls e • freeVbls (HsLambda ps e) • = freeVbls e \\ concatMap paramNames ps • freeVbls (HsCase exp cases) • = freeVbls exp ++ concatMap freeVbls cases • freeVbls (HsTuple _ es) • = concatMap freeVbls es • … etc.

36. This approach • Boiler plate code … • … 1000 lines for 100 lines of significant code. • Error prone: significant code lost in the noise. • Want to generate the boiler plate and the tree traversals … • … DriFT: Winstanley, Wallace • … Strafunski: Lämmel and Visser

37. Strafunski • Strafunski allows a user to write general (read generic), type safe, tree traversing programs … • … with ad hoc behaviour at particular points. • Traverse through the tree accumulating free variables from component parts, except in the case of lambda abstraction, local scopes, … • Strafunski allows us to work within Haskell … other options are under development.

38. Rename an identifier • rename:: (Term t)=>PName->HsName->t->Maybe t • rename oldName newName = applyTP worker • where • worker = full_tdTP (idTP ‘adhocTP‘ idSite) • idSite :: PName -> Maybe PName • idSite v@(PN name orig) • | v == oldName = return (PN newName orig) • idSite pn = return pn

39. The coding effort • Transformations with Strafunski are straightforward … • … the chore is implementing conditions that guarantee that the transformation is meaning- preserving. • This is where much of our code lies.

40. The Implementation of Hare Information gathering Pre-condition checking Program transformation Program rendering

41. Program rendering example • -- This is an example • module Main where • sumSquares x y = sq x + sq y • where sq :: Int->Int • sq x = x ^ pow • pow = 2 :: Int • main = sumSquares 10 20 • Promote the definition of sq to top level

42. Program rendering example • module Main where • sumSquares x y • = sq powx + sq powy where pow = 2 :: Int • sq :: Int->Int->Int • sq pow x = x ^ pow • main = sumSquares 10 20 • Using a pretty printer: comments lost and layout quite different.

43. Program rendering example • -- This is an example • module Main where • sumSquares x y = sq x + sq y • where sq :: Int->Int • sq x = x ^ pow • pow = 2 :: Int • main = sumSquares 10 20 • Promote the definition of sq to top level

44. Program rendering example • -- This is an example • module Main where • sumSquares x y = sq powx + sq powy • where pow = 2 :: Int • sq :: Int->Int->Int • sq pow x = x ^ pow • main = sumSquares 10 20 • Layout and comments preserved.

45. Rendering: our approach • White space and comments in the token stream. • 2 views of the program: token stream and AST. • Modification of the AST guides the modification of the token stream. • After a refactoring, the program source is extracted from the token stream not the AST. • Use heuristics to associate comments with semantic entities.

46. Production tool (version 0) Programatica parser and type checker Refactor using a Strafunski engine Render code from the token stream and syntax tree.

47. Production tool (version 1) Programatica parser and type checker Refactor using a Strafunski engine Render code from the token stream and syntax tree. Pass lexical information to update the syntax tree and so avoid reparsing

48. Module awareness: example • Move a top-level definition ffrom module A to B. • -- Is fdefined at the top-level of B? • -- Are the free variables in f accessible within module B? • -- Will the move require recursive modules? • -- Remove the definition of f from module A. • -- Add the definition to module B. • -- Modify the import/export lists in module A, B and the client modules of A and B if necessary. • -- Change uses of A.f to B.f or f in all affected modules. • -- Resolve ambiguity.

49. What have we learned? • Emerging Haskell libraries make it a practical platform. • Efficiency issues … type checking large systems. • Limitations of IDE interactions in vim and emacs. • Reflections on Haskell itself.

50. Reflecting on Haskell • Cannot hide items in an export list (though you can on import). • The formal semantics of pattern matching is problematic. • ‘Ambiguity’ vs. name clash. • ‘Tab’ is a nightmare! • Correspondence principle fails …