Performance Measurement

1. Performance Measurement

2. Performance Analysis Paper and pencil. Don’t need a working computer program or even a computer.

3. Some Uses Of Performance Analysis determine practicality of algorithm predict run time on large instance compare 2 algorithms that have different asymptotic complexity e.g.,O(n) and O(n^2)

4. Limitations of Analysis Doesn’t account for constant factors. but constant factor may dominate 1000n vs n^2 and we are interested only in n < 1000

5. Limitations of Analysis Modern computers have a hierarchical memory organization with different access time for memory at different levels of the hierarchy.

6. 8-32 1C 32KB 2C 256KB 10C 128MB 100C Memory Hierarchy MAIN L2 L1 ALU R

7. Limitations of Analysis Our analysis doesn’t account for this difference in memory access times. Programs that do more work may take less time than those that do less work.

8. Performance Measurement Measure actual time on an actual computer. Measure actual space on an actual computer. Not used here: Size of instructions and statically allocated data known at compile time. Can accurately estimate run-time stack using the analytical methods discussed earlier.

9. Performance Measurement Measure actual time on an actual computer. What do we need?

10. Performance Measurement Needs programming language working program computer compiler and options to use javac -o

11. Performance Measurement Needs • data to use for measurement worst-case data best-case data average case data

12. Performance Measurement Needs • How many data sets? • In theory, if know the big-oh time, need 3 data points to draw the curve. • In practice, only know the curve for large n and don’t necessarily know n0. • In practice, lower order terms will change the actual shape of the curve.

13. timing mechanism --- clock Performance Measurement Needs • Difficult to create data set that demonstrates average time complexity.

14. Timing In Java long startTime = System.currentTimeMillis(); // gives time in milliseconds since 1/1/1970 GMT // code to be timed comes here long elapsedTime = System.currentTimeMillis() - startTime;

15. Example: insertion sort

16. Example: insertion sort

17. Results: insertion sort

18. Short Comings Repeat work many times to bring total time to be >= 1 second (doesn’t really take 0 time to sort!)

19. Short Comings Clock accuracy assume 100 milliseconds (don’t really know) If reported time is t, actual time is between t-100ms and t+100ms Example: from fig 4-1, n=1000 takes 110ms. So actual time is between 10 and 210ms! elapsedTime should be at least 1000ms to get 10% accuracy.

20. Fix Need times to be at least 1000ms Must repeat sort several times for each value of n. Since sort changes array, must initialize array before each sort. Total time: time to sort + time to initialize array + while loop overhead.

21. Accurate Timing long startTime = System.currentTimeMillis(); long counter; do { counter++; doSomething(); } while (System.currentTimeMillis() - startTime < 1000) long elapsedTime = (System.currentTimeMillis() - startTime) / counter;

22. Accuracy Now accuracy is 10%. first reading may be just about to change to startTime + 100 second reading may have just changed to finishTime so finishTime - startTime is off by 100ms

23. Accuracy Examining remaining cases, we get trueElapsedTime = finishTime - startTime +- 100ms To ensure 10% accuracy, require elapsedTime = finishTime – startTime >= 1sec

24. What Went Wrong? long startTime = System.currentTimeMillis(); long counter; do { counter++; InsertionSort.insertionSort(a); } while (System.currentTimeMillis() - startTime < 1000) long elapsedTime = (System.currentTimeMillis() - startTime) / counter;

25. The Fix long startTime = System.currentTimeMillis(); long counter; do { counter++; // put code to initialize a[] here InsertionSort.insertionSort(a); } while (System.currentTimeMillis() - startTime < 1000)

26. Final Program

27. Final Program

28. Final Program

29. Final Program

30. Final Program

31. Problem • Have introduced overhead: do { counter++; // put code to initialize a[] here InsertionSort.insertionSort(a); } while (System.currentTimeMillis() - startTime < 1000)

32. Problem • To determine overhead take out sort and then calculate time again: do { counter++; code to initialize a[] here // InsertionSort.insertionSort(a); } while (System.currentTimeMillis() - startTime < 1000)

33. Accounting for overhead

34. Time Shared System UNIX time MyProgram problem: UNIX is a timeshared system Program may be swapped out between calls to System.currentTimeMillis( )

35. Bad Way To Time do { counter++; startTime = System.currentTimeMillis(); doSomething(); elapsedTime += System.currentTimeMillis() - startTime; } while (elapsedTime < 1000)