Parallel Execution of Test Runs for Database Application Systems

1. Parallel Execution of Test Runs for Database Application Systems Donald Kossmann: ETH Zurich, i-TV-T AG Florian Haftmann: i-TV-T AG Eric Lo: ETH Zurich

2. Some Facts • Microsoft spends 50% of their development cost on testing • SAP product cycle = 18 months • 6 months to execute tests Testing is the most expensive phase of the software development cycle

3. Observation • The more test runs, the better • However, it takes more time! • Goal: Optimize Testing Time

4. Definition: Test Run Ti • A sequence of requests • Test Run “Login” (2 requests):

5. Expected Result

6. More Definitions • Failed Test Run: At least one request does not return the expected result • Test Database D: The state of an Application + Database at the beginning of each test • Database Reset R: Bring the database back to D

7. A Test Run Fails When: • The application has a real bug • Or the test database is in wrong state due to execution of test runs  Carry out resets to find real bugs

8. Resetting the Test Database? • P.O. Insertion • Count P.O. • … DatabasePurchaseOrder={P1}

9. Resetting the Test Database? TA: Insert Purchase Order P2 • P.O. Insertion • Count P.O. • … DatabasePurchaseOrder={P1}

10. Resetting the Test Database? TA: Insert Purchase Order P2 • P.O. Insertion • Count P.O. • … <TA Success> DatabasePurchaseOrder={P1 } P2

11. Resetting the Test Database? • P.O. Insertion • Count P.O. • … TB: Get Total Purchase Order Expected Result: 1 Actual Result: 1 <TB Success> DatabasePurchaseOrder={P1}

12. Resetting the Test Database? TA: Insert Purchase Order P2 • P.O. Insertion • Count P.O. • … TB: Get Total Purchase OrderExpected Result: 1 Actual Result: 2 <TB Fails> DatabasePurchaseOrder={P1 } P2

13. Reset DB TB: Get Total Purchase OrderExpected Result: 1 Database Reset is needed! TA: Insert Purchase Order P2 • P.O. Insertion • Count P.O. • … DatabasePurchaseOrder={P1 } P2

14. Database Reset • Resetting a database for a large scale application takes about 2 minutes! • Back-of-the-envelop calculation: • 10000 test runs = 10000 resets x 2 min = 2 weeks on DB resets for 1 complete test

15. TA: Insert Purchase Order P2 TB: Get Total Purchase OrderExpected Result: 1Actual Result: 2<TB Fails> Reordering Test Runs • P.O. Insertion • Count P.O. • … DatabasePurchaseOrder={ P1, P2 }

16. TB: Get Total Purchase OrderExpected Result: 1Actual Result: 1<TB Success> Reordering Test Runs • P.O. Insertion • Count P.O. • … DatabasePurchaseOrder={ P1 }

17. TA: Insert Purchase Order P2 <TA Success> TB: Get Total Purchase OrderExpected Result: 1Actual Result: 1<TB Success> Order Matters! • P.O. Insertion • Count P.O. • … DatabasePurchaseOrder={ P1, P2 }

18. Our Previous Work (CIDR 2005) • A test run depends on a correct state of a database • Control the database state • Reduce the number of database resets • Algorithms to optimize order of test runs • No parallelism in testing

19. Can we do better if we have > 1 machine?

20. Parallel Testing is a Two-dimensional Problem! • Fully utilize the available resources • Load Balancing! • Same as single machine, we still have to control the database state • Reduce the database resets!

21. More about the Problem • Regression test • Later stage of the development cycle • Minor changes between versions • Execute the same set of test runs • Version 1.1 • Execute test: T1 T2 T3 T4 • Version 1.2 (Bug fixed and/or minor changes) • Execute test: T1 T2 T3 T4

22. Parallel TestingShared-Nothing vs. Shared-Database

23. Shared-Nothing (SN) ... ... • If I work for IBM, I can install: • N applications • N databases • N machines • One more machine: • More admin. work! • More license fees! • Applications do not SHARE the database T4 T31 T12 T5 Application Application ... Database Database Machine 1 Machine N

24. If I work for PoorEric.com, I install: N threads (e.g., open N browsers) 1 database 1 machine The threads SHARE the database Test runs interference with each others Can’t scale as good as Shared-Nothing Shared-Database (SDB) Thread N Thread 1 ... ... T4 T31 ... T12 T5 Application Database

25. Parallel Testing Framework Application History T1 T2 M1 ... Database Reset? T5 MN ... Machine/Thread 1 Scheduler ... T6 T2 T5 T1 Reset? Application Conflicts DB Database Machine/Thread N

26. Parallel Testing is a Two-dimensional Problem! • Fully utilize the available resources • Load Balancing! • Same as single machine, we still have to control the database state • Reduce the database resets!

27. Execution Strategies • Optimistic Execution: • Reset the database only when it is a must • Example: R T1 T2 T3 T4 • Optimistic++ Execution: • Avoid to execute a test run twice, again • Example (Wk 1): R T1 T2 T3 T4 R T4 T5 • Example (Wk 2): R T1 T2 T3 (Next is T4 ?) • Slice Reordering Heuristics: • Slice: A sequence of test runs without conflicts • Example: R T1 T2 T3 T4 R T4 T5 • Collect <slice>s during each test • Graph Reordering Heuristics R T4 T5 - R T4T5 <T1 T2 T3>T4

28. Parallel TestingShared-Nothing (SN)

29. Shared-Nothing Application Reset Database Scheduler Machine 1 ... T6 T2 T5 T1 Test Run Input Queue Reset Application Conflicts DB Database Machine 2

30. Test 1 M1: R Scheduler T1 T5 T2 T6 T3 T7 T8 Test Run Input Queue M2: R Conflicts DB

31. Test 1 M1: R T1 Scheduler T5 T2 T6 T3 T7 T8 Test Run Input Queue M2: R Conflicts DB

32. Test 1 M1: R T1 Scheduler T2 T6 T3 T7 T8 Test Run Input Queue M2: R T5 Conflicts DB

33. Test 1 M1: R T1 T2 Scheduler T6 T3 T7 T8 Test Run Input Queue M2: R T5 Conflicts DB

34. Test 1 M1: R T1 T2 Scheduler T3 T7 T8 Test Run Input Queue M2: R T5 T6 Conflicts DB

35. Test 1 M1: R T1 T2 T3 Scheduler T7 T8 Test Run Input Queue M2: R T5 T6 Conflicts DB

36. Test 1 M1: R T1 T2 T3 Scheduler T7 T8 Test Run Input Queue M2: R T5 T6 R Conflicts DB

37. Test 1 M1: R T1 T2 T3 Scheduler T7 T8 Test Run Input Queue M2: R T5 T6 R T6 Conflicts DB T5T6

38. Test 1 M1: R T1 T2 T3 R Scheduler T7 T8 Test Run Input Queue M2: R T5 T6 R T6 Conflicts DB T5T6 T1T2T3

39. Test 1 M1: R T1 T2 T3 R Scheduler T8 Test Run Input Queue M2: R T5 T6 R T6 T7 Conflicts DB T5T6 T1T2T3

40. Test 1 M1: R T1 T2 T3 R Scheduler Test Run Input Queue M2: R T5 T6 R T6 T7 T8 Conflicts DB T5T6 T1T2T3

41. Test 1 M1: R T1 T2 T3 R T3 Scheduler Test Run Input Queue M2: R T5 T6 R T6 T7 T8 Conflicts DB T5T6 T1T2T3

42. Shared-Nothing - Slice 3 major principles: • The slices in the input queue are ordered by: • Reordering the slices on each machine locally • Merge the partial order • Executes all test runs of the same slice on the same machine • The scheduler makes sure conflicting slices are executed on different machines as much as possible

43. Collect Slices M1: R T1 T2 T3 R T3 Scheduler Test Run Input Queue M2: R T5 T6 R T6 T7 T8 Conflicts DB T5T6 T1T2T3

44. T1 T2 T3 T5 T6 T7 T8 Reordering Slices M1: R T3 R Local Order M1: Local Order M2: M2: R T6 R

45. T1 T2 T3 T5 T6 T7 T8 Merge Partial Order M1: R T3 R Local Order M1: Local Order M2: M2: R T6 R Test Run Input Queue

46. Shared-Nothing - Slice 3 major principles: • The slices in the input queue are ordered by: • Reordering the slices on each machine locally • Merge the partial order • Executes all test runs of the same slice on the same machine • The scheduler makes sure conflicting slices are executed on different machinesas much as possible

47. Test 10 M1: R Scheduler T3 T6 T7 T8 T1 T2 T5 Test Run Input Queue M2: R Conflicts DB T5T6 T1T2T3 T3T1

48. Test 10 M1: R T3 T3 Scheduler T6 T7 T8 T1 T2 T5 Test Run Input Queue M2: R Conflicts DB T5T6 T1T2T3 T3T1

49. Test 10 M1: R T3 T3 Scheduler T1 T2 T5 Test Run Input Queue M2: R T6 T7 T8 Conflicts DB T5T6 T1T2T3 T3T1

50. Test 10 M1: R T3 T3 Conflict? Scheduler T1 T2 T5 Test Run Input Queue M2: R T6 T7 T8 Conflicts DB T5T6 T1T2T3 T3T1