Performance Analysis and Monitoring Facilities in CPN Tools

1. Performance Analysis and Monitoring Facilities in CPN Tools Tutorial CPN’05 October 25, 2005 Lisa Wells

2. Plan • Observing and controlling simulations • Introduction to monitors • Example: queue system • Demo 1 • Performance facilities • Demo 2 Lisa Wells, CPN'05

3. Simulations • Simulations of CP-nets are run for many reasons: • Debugging • Analysis of system behavior (performance or functional) • Presentation of a model to project team • Communication with external programs • The usefulness of the simulations is heavily dependent on the flexibility and functionality of the simulator. • Access to simulation information • Stopping simulations Lisa Wells, CPN'05

4. Accessing simulation information • It is often useful to be able to exchange information between the CPN simulator and external processes or files. • Code segments can be added to transitions for: • Reading and writing in files • Calculating some performance measures • Sending and receiving information from external processes Lisa Wells, CPN'05

5. Controlling a simulation • Simulation stop criteria • Number of steps executed • Dependent on model time • Dependent on markings or transitions Lisa Wells, CPN'05

6. Common functionality • These controlling and accessing activities share a common pattern: If certain conditions are fulfilled, then do something. • If transition T occurs, then save information in a file. • If there are no tokens on place P, then stop the simulation. • If model time is greater than C, then calculate the average number of tokens on place P. • If transition T occurs, send the values of variables x, y, and z to the animation facilities. Lisa Wells, CPN'05

7. Current problems in CPN Tools • Problems controlling and accessing simulation information: • Cannot access marking information. • Net structure may have to be changed to obtain desired functionality. • If multiple transitions need to be inspected, then code segments must be duplicated. • Code segments cannot be disabled. • Only low-level support, which is difficult to use. Lisa Wells, CPN'05

8. Monitors in CPN Tools monitor (verb) to watch, keep track of, or check, usually for a special purpose Merriam-Webster’s Collegiate Dictionary • A monitor is a mechanism that is used to observe, inspect, control or modify a simulation of a CP-net. • Important characteristics of monitors: • They can inspect the statesand eventsof a simulation, and take appropriate actions based on the observations. • There is an explicit separation between monitoring the behavior of a net, and modeling the behavior of the system. Lisa Wells, CPN'05

9. Levels of monitors • Standard monitors • Very easy to define • Do not require users to write any code • Parameterized monitors • Similar to standard monitors, but slightly more flexible, and require some programming • User-defined monitors • Very flexible, but require more programming Lisa Wells, CPN'05

10. Kinds of monitors • Simulation breakpoint monitors • Write-in-file monitors • User-defined monitors • Data collector monitors Lisa Wells, CPN'05

11. Monitoring subnets • A monitor can inspect markings and occurring transitions, with variable bindings, during a simulation • Zero or more places • Zero or more transitions • Monitors are activated after simulation steps • After every simulation step, if no transitions are monitored • After every relevant transition has occurred Lisa Wells, CPN'05

12. Monitoring functions • A monitor typically has several functions: • Initialization function • Called once before a simulation • Predicate function • Called after simulation steps • Observation function • Called when predicate function returns true • Extracts relevant data from the model • Action function • Does something appropriate with the observed value • Stop function • Called once after a simulation Lisa Wells, CPN'05

13. Job arrival Queue Server Example: simple queue system • A single server processes two kinds of jobs. • If the server is busy when a job arrives, the job is added to a queue. • Job inter-arrival times and server processing times are exponentially distributed. • Queuing strategy is FIFO. Lisa Wells, CPN'05

14. Template code: Write-in-File • Monitor one place, and two transitions fun init (System'Queue_1_mark : Jobs ms) = "" fun pred (bindelem, System'Queue_1_mark : Jobs ms) = let fun predBindElem (Arrivals'Arrive (1, {jobs, job})) = true | predBindElem (Server'Start (1, {jobs, job})) = true | predBindElem _ = false in predBindElem bindelem end Lisa Wells, CPN'05

15. Demo 1 • Monitoring tools, monitor index entries, marking menus, syntax checking, … Lisa Wells, CPN'05

16. Performance analysis using CPN • CP-nets have the potential to be used to model and analyze the performance, i.e. quantitative aspects, of systems. • In practice, there are very few studies using CPN for performance analysis. • This is due, in part, to lack of tool support. Lisa Wells, CPN'05

17. Performance facilities for CPN Tools • High-level support for simulation-based performance analysis is currently being implemented for CPN Tools. • Monitors • Data collection during simulations • Net-specific simulation breakpoints • Simulation output • Managing data files • Running multiple simulations • Generating a status file for multiple simulations • Collecting data from independent, terminating simulations • Calculating confidence intervals for performance measures Lisa Wells, CPN'05

18. Performance analysis of the queue system • Data collection monitors for calculating performance measures, such as: • Expected average delay in queue • Expected average queue length • Expected utilization of the server • Breakpoint monitor: • Stop a simulation when 100jobs have passed through the queue • The system can be modeled in CPN Tools, but until now it has been very difficult to collect and process data, and to create net-specific breakpoints. Lisa Wells, CPN'05

19. Data collector monitors • The data that is collected is numerical data, i.e. integers, infinite integers, reals. • Statistics are calculated. • Data can be saved in log files. • Data can be post-processed, e.g. plotted in graphs, after a simulation has completed. #data counter step time 0 1 0 0 0 2 1 0 0 3 4 548 1 4 5 579 0 5 7 595 1 6 8 697 2 7 9 710 3 8 10 848 Lisa Wells, CPN'05

20. Estimating average queue delay • Wish to calculate average queue delay for the first n jobs in the queue. • When a job arrives in the system, the time of its arrival must be noted and saved. • When a job leaves the queue, its queue delay is (current time – arrival time). Lisa Wells, CPN'05

21. Estimating average queue length • Wish to calculate the time-average queue length until n jobs have left the queue. • When calculating statistics, the length of the queue is weighted with a time interval. • Measure the length of the queue when the length changes. • The time-average queue length is equal to the area under an appropriate curve divided by the appropriate interval of time. Lisa Wells, CPN'05

22. Estimating server utilization • Wish to calculate the utilization of the server until n jobs have left the queue. • Use an indicator variable: server is busy => 1, server is idle => 0. • Weight the values with appropriate intervals of time. • The utilization of the server is then equal to the area under an appropriate curve divided by the appropriate interval of time. Lisa Wells, CPN'05

23. Demo • Performance analysis of the queue system… Lisa Wells, CPN'05