Constraint Based Synthesis for Beginners

1. Constraint Based Synthesis for Beginners PSY 2012 Armando Solar-Lezama

2. CAP View of Synthesis Code Synthesis Methodology

3. A different approach Synthesis Methodology Code Domain Specific Tool

4. Exhibit A: Synthesis of SQL With Alvin Cheung and Sam Madden

5. Motivation • It turns out SQL is challenging to learn • Who would have thought? • Frameworks simplify program/DB interface • You can access the DB without using SQL • Which can lead to some interesting code...

6. Examples: Explicit Select public Set<Project> getUnfinishedProjects() {   Set<Project> unfinishedP = new HashSet<Project>();   List<Project> projects = this.projectDao.getAllProjects();   for (Project project : projects) { if (!(project.getIsFinished())) { unfinishedP.add(project);    }   }   return unfinishedP; } SELECT * FROM Projects WHERE isFinished=FALSE Get list of projects from the DB Select unfinished projects

7. Examples: Explicit Select public List<WilosUser> getRoleUser(){ List<WilosUser> listUser = new ArrayList<WilosUser>(); List<WilosUser> user= this.userDao.getUsers(); List<Role> role = this.roleDao.getRoles(); for(inti = 0; i < user.size(); i ++){ for(int a = 0; a < role.size(); a++){ if(user.get(i).getRole_id(). equals(role.get(a).getRole_id())){ WilosUseruserok = user.get(i); listUser.add(userok); } } return listUser; } SELECT u FROM users u, roles r WHERE u.roleId == r.id Start with users and roles Find users in the Roles list and add them to the output

8. Why is this so bad? • These can be performance bottlenecks • Where performance matters, people write SQL • More controversial arguments can be made

9. Is this a synthesis problem? • It is Domain knowledge about relational algebra Equivalent Formula in Relational Algebra Imperative code with loop nests Synthesizer

10. Framing the problem List getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; inti = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } } return results; } Precondition: true Postcondition:

11. Framing the problem • We want to synthesize a post-condition! • Challenges • Defining the language • Solving the synthesis problem • Generating code

12. Defining the language • Requirements • Should make reasoning about the program easy • Should have expressiveness comparable to SQL

13. Language • Just like relational algebra... • ...but with lists rather than sets • The program is written in terms of lists, not sets

14. Invariants List getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; inti = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } } return results; }

15. Verification • Easy if you know the invariants • Z3 can do it in seconds • All you need are a few axioms: : ...

16. Synthesis • Synthesis works best in a finite domain • Invariant generation can be framed as a sketch • Model operations in terms of their source code • Construct the verification condition leaving invariant and post condition as unknowns • Create a sketch for unknowns • Solve!

17. Results so far • We have tried this with a dozen loop nests from real open source projects • All solve in less than 7 minutes • Slow, but not very optimized.

18. Exhibit B: Automated Grading With Rishabh Singh and SumitGulwani

19. The real software problem • The Software Quality problem is a symptom • The real problem: The demand for software in our society far exceeds the supply of people skilled enough to produce it

20. Three pronged attack • Make programmers more productive • Make programming more accessible • Reduce the cost of training the next generation

21. Grading Programming Assignments • Test-cases based grading • No precise correctness correlation • No student tailored feedback • Manual grading by TAs • Error-prone • Time consuming • Expensive • Manual grading will not scale to 100K students

22. Buggy Program for Array Reverse 6:28::50 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } }

23. Buggy Program for Array Reverse 6:32::01 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } }

24. Buggy Program for Array Reverse 6:32::32 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } } No change! Sign of Frustation?

25. Buggy Program for Array Reverse 6:33::19 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i <= a.Length; i--) { b[count] = a[i]; count++; } return b; } }

26. Buggy Program for Array Reverse 6:33::55 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i < a.Length; i--) { Console.Writeline(i); b[count] = a[i]; count++; } return b; } } Same as initial attempt except Console.Writeline!

27. Buggy Program for Array Reverse 6:34::06 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i < a.Length; i--) { Console.Writeline(i); b[count] = a[i]; count++; } return b; } } No change! Sign of Frustation?

28. Buggy Program for Array Reverse 6:34::56 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i <= a.Length; i--) { Console.Writeline(i); b[count] = a[i]; count++; } return b; } } The student has tried this before!

29. Buggy Program for Array Reverse 6:36::24 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } } Same as initial attempt!

30. Buggy Program for Array Reverse 6:37::39 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } } The student has tried this before!

31. Buggy Program for Array Reverse 6:38::11 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i > 0; i--) { b[count] = a[i]; count++; } return b; } } Almost correct! (a[i-1] instead of a[i] in loop body)

32. Buggy Program for Array Reverse 6:38::44 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i >= 0; i--) { b[count] = a[i]; count++; } return b; } } Student going in wrong direction!

33. Buggy Program for Array Reverse 6:39::33 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } } Back to bigger error!

34. Buggy Program for Array Reverse 6:39::45 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } } No change! Frustation!

35. Buggy Program for Array Reverse 6:40::27 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } } No change! More Frustation!!

36. Buggy Program for Array Reverse 6:40::57 AM using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; intcount = 0; for(inti=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } } No change! Too Frustated now!!! Gives up.

37. AutoGrader • Automate grading • Find semantic errors • Feedback to fix them Students make similar mistakes

38. Is this a synthesis problem? • It is Error Model Reference Solution Corrections for student program Buggy implementation Synthesizer

39. Array Reverse i <= a.Length i = 1

40. Challenge 1: Different Algorithms

41. Challenge 2: Scalability 1010 different possible candidate corrections

42. Our Approach • Use data of previous student solutions • Correction rules based on observed errors • Create a set of candidate solutions using rules and find closest correct solution.

43. Aren’t rewrite systems hard? • Can teachers really write rewrite rules? • Angelic non-determinism helps • Ambiguities and redundancies no longer matter

44. Results: Problems Fixed

45. Results: Performance

46. Results: Generalization

47. Feedback for Tutoring System OK for Grading, But not ideal for teaching

48. Broad research agenda ahead • Transformative for students in under-funded schools • Reduce the resources required to support quality instruction • Enable “true” distance education for programming courses • Same technology could be used for automatic tutoring • Identify errors stemming from deep misconceptions(e.g. not understanding difference in values vs. references) • Synthesize small examples that make misconceptions explicit

49. Sketch Tutorial on Saturday http://bit.ly/sketch2012