Analysis of Simulation Results

1. Analysis of Simulation Results Andy Wang CIS 5930-03 Computer Systems Performance Analysis

2. Analysis of Simulation Results • Check for correctness of implementation • Model verification • Check for representativeness of assumptions • Model validation • Handle initial observations • Decide how long to run the simulation

3. Model Verification Techniques • Verification is similar to debugging • Programmer’s responsibility • Validation • Modeling person’s responsibility

4. Top-Down Modular Design • Simulation models are large computer programs • Software engineering techniques apply • Modularity • Well-defined interfaces for pieces to coordinate • Top-down design • Hierarchical structure

5. Antibugging • Sanity checks • Probabilities of events should add up to 1 • No simulated entities should disappear • Packets sent = packets received + packets lost

6. Structured Walk-Through • Explaining the code to another person • Many bugs are discovered by reading the code carefully

7. Deterministic Models • Hard to verify simulation against random inputs • Should debug by specifying constant or deterministic distributions

8. Run Simplified Cases • Use only one packet, one source, one intermediary node • Can compare analyzed and simulated results

9. Trace • Time-ordered list of events • With associated variables • Should have levels of details • In terms of occurred events, procedure called, or variable updates • Properly indented to show levels • Should allow the traces to be turned on and off

10. On-line Graphic Displays • Important when viewing a large amount of data • Verifying a CPU scheduler preempt processes according to priorities and time budgets

11. Continuity Test • Run the simulation with slightly different values of input parameters • Δ change in input should lead to Δ change in output • If not, possibly bugs

12. Degeneracy Test • Test for extreme simulation input and configuration parameters • Routers with zero service time • Idle and peak load • Also unusual combinations • Single CPU without disk

13. Consistency Tests • Check results for input parameters with similar effects • Two sources with arrival rate of 100 packets per second • Four sources with arrival rate of 50 packets per second • If dissimilar, possibly bugs

14. Seed Independence • Different random seeds should yield statistically similar results

15. Model Validation Techniques • Should validate • Assumptions • Input values and distributions • Output values and conclusions • Against • Expert intuition • Real system measurements • Theoretical results

16. Model Validation Techniques • May not be possible to check all nine possibilities • Real system not available • May not be possible at all • The reason why the simulation was built • As the last resort • E.g., economic model

17. Expert Intuition • Should validate assumptions, input, and output separately and as early as possible Why would increased packet loss lead to better throughput? Throughput % packet loss

18. Real-System Measurements • Most reliable way for validation • Often not feasible • System may not exist • Too expensive to measure • Apply statistical techniques to compare model and measured data • Use multiple traces under different environments

19. Theoretical Results • Can apply queueing models • If too complex • Can validate a small subset of simulation parameters • E.g., compare analytical equations with CPU simulation models with one and two cores • Use validated simulation to simulate many cores • Can validate only the common scenarios

20. Transient Removal Transient Removal • In most cases, we care only about steady-state performance • We need to perform transient removal to remove initial data from analysis • Difficulty • Find out where transient state ends

21. Long Runs • Just run the simulation for a long time • Waste resources • Not sure if it’s long enough

22. Proper Initialization • Start the simulation in a state close to steady state • Pre-populate requests in various queues • Pre-load memory cache content • Reduce the length of transient periods

23. Truncation • Assume steady state variance < transient state variance • Algorithm • Measure variability in terms of range • Remove the first L observations, one at a time • Until the (L + 1)th observation is neither min nor max or the remaining observations

24. Truncation L

25. Initial Data Deletion • m replications • n data points for each replication • Xij = jth data point in ith replication

26. Initial Data Deletion • Step 1: average across replications

27. Initial Data Deletion • Step 2: compute grand mean µ • Step 3: compute µL = average last n – L values,  L

28. Initial Data Deletion • Step 4: offset µL by µ and normalize the result to µ by computing relative change ΔµL= (µL - µ)/µ Transient interval

29. Moving Average of Independent Replications • Similar to initial data deletion • Requires computing the mean over a sliding time window

30. Moving Average of Independent Replications • m replications • n data points for each replication • Xij = jth data point in ith replication

31. Moving Average of Independent Replications • Step 1: average across replications

32. Moving Average of Independent Replications • Step 2: pick a k, say 1; average (j – k)th data point to (j + k)th data point,  j; increase k as necessary Transient interval

33. Batch Means • Used for very long simulations • Divide N data points into m batches of n data points each • Step 1: pick n, say 1; compute the mean for each batch • Step 2: compute the mean of means • Step 3: compute the variance of means • Step 4: n++, go to Step 1

34. Batch Means • Rationale: as n approaches the transient size, the variance peaks • Does not work well with few data points Transient interval

35. Terminating Simulations • Terminating simulations: for systems that never reach a steady state • Network traffic consists of the transfer of small files • Transferring large files to reach steady state is not useful • System behavior changes with time • Cyclic behavior • Less need for transient removal

36. Final Conditions • Handling the end of simulations • Might need to exclude some final data points • E.g., Mean service time = total service time/n completed jobs

37. Stopping Criteria: Variance Estimation • If the simulation run is too short • Results highly variable • If too long • Wasting resources • Only need to run until the confidence interval is narrow enough • Since confidence interval is a function of variance, how do we estimate variance?

38. Independent Replications • m runs with different seed values • Each run has n + n0 data points • First n0 data points discarded due to transient phase • Step 1: compute mean for each replication based on n data points • Step 2: compute µ, mean of means • Step 3: compute 2, variance of means

39. Independent Replications • Confidence interval: µ ± z1-α/22 • Use t[1-α/2; m – 1], for m < 30 • This method needs to discard mn0 data points • A good idea to keep m small • Increase n to get narrower confidence

40. Batch Means • Given a long run of N + n0 data points • First n0 data points discarded due to transient phase • N data points are divided into m batches of n data points

41. Batch Means • Start with n = 1 • Step 1: compute the mean for each batch • Step 2: compute µ, mean of means • Step 3: compute 2, variance of means • Confidence interval: µ ± z1-α/22 • Use t[1-α/2; m – 1], for m < 30

42. Batch Means • Compared to independent replications • Only need to discard n0 data points • Problem with batch means • Autocorrelation if the batch size n is small • Can use the mean of ith batch to guess the mean of (i + 1)th batch • Need to find a batch size n

43. Batch Means • Plot batch size n vs. variance of means • Plot batch size n vs. autocovariance • Cov(batch_meani, batch_meani+1),  i

44. Method of Regeneration • Regeneration • Measured effects for a computational cycle are independent of the previous cycle Regeneration points Regeneration cycle

45. Method of Regeneration • m regeneration cycles with ni data points each • Step 1: compute yi, sum for each cycle • Step 2: compute grand mean, µ • Step 3: compute the difference between expected and observed sums wi = yi - niµ • Step 4: compute 2 based on wi

46. Method of Regeneration • Step 5: compute average cycle length, c • Confidence interval: µ ± z1-α/22/(cm) • Use t[1-α/2; m – 1], for m < 30

47. Method of Regeneration • Advantages • Does not require removing transient data points • Disadvantages • Can be hard to find regeneration points

48. White Slide