مراجع درس: 1-“Introduction to computer system performance Evaluation”, k.kant, Mcgraw-Hill/1992.

1. مراجع درس: 1-“Introduction to computer system performance Evaluation”, k.kant, Mcgraw-Hill/1992. ( كتاب اصلي درس است). 2-“Performance modeling of communication Networks and computer Architecture”, Peter G. Harison and Naresh M. pattle, Addison- wesley pub 1td/1993. 3-“Computer systems performance evaluation”, O. ferrari, 1978/prentice- Hall Inc. 4-“Computer systems performance Modeling”, C.H. sauer and K.M. chandy, prentice- Hall/1981. 5-“Reversibility and stochastic Networks”, F.P Kelly, John-willey/1979. 6. "Performance by Design: Computer capacity planning by example." Prentice Hall, 2004 (NEW 7. " Capacity Planning for web Services: metrics, models, and methods," Prentice Hall. 2002 (translated into Russian and Portuguese). 8. "Scaling for E-Business: technologies, models, performance, and capacity planning." Prentice Hall, 2000 (translated into Korean)

2. 10. " Capacity Planning and performance Modeling: form mainframes to client-server System", Prentice- Hall, 1998. • 9. " Capacity Planning for Web Performance: metrics, models, and methods", Prentice Hall, 1998. 1- ميان ترم 30% نمره 2- آخر ترم 40% نمره 3- تعدادي برنامه و مقاله تحقيقي و سمينار 30% نمره

3. Overview of Performance Analysis • Introduction • purpose of evaluation • applications of performance evaluation • performance evaluation techniques • criteria for selecting an evaluation technique • applicability of evaluation techniques • steps for a performance evaluation study • performance evaluation metrics • capacity of a system • performance evaluation study example • common mistakes in performance evaluation

4. Introduction Computer system users, administrators, and designers are all interested in performance evaluation since the goal is to obtain or to provide the highest performance at the lowest cost. A system could be any collection of H/W, S/W and firmware components; e.g., CPU, DB system, Network. Computer performance evaluation is of vital importance in the selection of computer systems, the design of applications and equipment, and analysis of existing systems.

5. Purpose of Evaluation threegeneral purposes of performance evaluation: • selection evaluation - system exists elsewhere • performance projection - system does not yet exist • performance monitoring - system in operation

6. Selection Evaluation • evaluate plans to include performance as a major criterion in the decision to obtain a particular system from a vendor is the most frequent case • to determine among the various alternatives which are available and suitable for a given application • to choose according to some specified selection criteria • at least one prototype of the proposed system must exist

7. Performance Projection • orientated towards designing a new system • to estimate the performance of a system that does not yet exist • secondary goal - projection of a given system on a new workload, i.e.modifying existing system in order to increase its performance or decrease it costs or both (tuning therapy) • upgrading of a system - replacement or addition of one or more hardware components

8. Performance Monitoring • usually performed for a substantial portion of the lifetime of an existing running system • performance monitoring is done: • to detect bottlenecks • to predict future capacity shortcomings • to determine most cost-effective way to upgrade the system • to overcome performance problems, and • to cope with increasing workload demands

9. Applications of Performance Evaluation 1. procurement 2. system upgrade 3. capacity planning : process of predicting when future load levels will saturatethe system and of determining the most cost-effective way of delaying system saturation as much as possible. 4. system design

10. Performance Measures And Evaluation Techniques 1.1Evaluation Metrics A computer system, like any other engineering machine, can be measured and evaluated in terms of how well it meets the needs and expectations of its users. It is desirable to evaluate the performance of a computer system because we want to make sure that it is suitable for its intended applications, and that it satisfies the given efficiency and reliability requirements. We also want to operate the computer system near its optimal level of processing power under the given resource constraints.

11. All performance measures deal with threebasic issues: -How quickly a given task can be accomplished, -How well the system can deal with failures and other unusual situations, and -How effectively the system uses the available resources. We can categorize the performance measures as follows. Responsiveness: These measures are intended to evaluate how quickly a given task can be accomplished by the system. Possible measures are: waiting time. Processing time.

12. Conditional waiting time(waiting time for tasks requiring a specified amount of processing time), Queue length. etc. Usage Level: These measures are intended to evaluate how well the various components of the system are being used. Possible measures are: Throughput and utilization of various resources. Missionability: These measures indicate if the system would remain continuously operational for the duration of a mission.

13. Possible measures are: the distribution of the work accomplished during the mission time, interval availability (Probability that the system will keep performing satisfactorily throughout the mission time), life-time (time when the probability of unacceptable behavior increases beyond some threshold). Dependability: These measures indicate how reliable the system is over the long run. Possible measures are: number of failures/day. MTTF(mean time to failure). MTTR(mean time to repair).

14. Long-tern availability, and cost of a failure. These measures are useful when repairs are possible and failures are tolerable. Productivity: These measures indicate how effectively a user can get his or her work accomplished. Possible measures are: user friendliness. Maintainability. And understandability.

15. The relative importance of various measures Because these measures are difficult to quantify, we shall not consider them. The relative importance of various measures depends on the application involved. In the following, we provide a broad classification of computer systems according to the application domains, indicating which measures are most relevant: • General purpose computing: These systems are designed for general purpose problem solving. Relevant measures are: responsiveness, usage level. and productivity. Dependability requirements are modest, especially for benign failures.

16. High availability: Such systems are designed for transaction processing environments: (bank, Airline. Or telephone databases. Switching systems. etc.). The most important measures are responsiveness and dependability. Both of these requirements are more severe than for general purpose computing systems, moreover, any data corruption or destruction is unacceptable. Productivity is also an important factor. .3Real-time control: Such systems must respond to both periodic and randomly occurring events within some(possibly hard) timing constraints.

17. They require high levels of responsiveness and dependability for most workloads and failure types and are therefore significantly over-designed. Note that the utilization and throughput play little role in such systems. .4Mission Oriented: These systems require extremely high levels of reliability over a short period, called the mission time. Little or no repair/tuning is possible during the mission. Such systems include fly-by-wire airplanes, battlefield systems, and spacecrafts. Responsiveness is also important, but usually not difficult to achieve. Such systems may try to achieve high reliability during the short term at the expense of poor reliability beyond the mission period.

18. Long-life: Systems like the ones used for unmanned spaceships need long life without provision for manual diagnostics and repairs. Thus. In addition to being highly dependable, they should have considerable intelligence built in to do diagnostics and repair either automatically or by remote control from aground station. Responsiveness is important but not difficult to achieve. .1.2Techniques of Performance Evaluation There are three basic techniques for performance evaluation: (1) measurement. (2) simulation. and (3) analytic modeling. The latter two techniques can also be combined to get what is usually known as hybrid modeling.

19. In the following, we discuss these briefly and point out their comparative advantages and disadvantages. 1.2.1 Measurement Measurement is the most fundamental technique and is needed even in analysis and simulation to calibrate the models. Some measurements are best done in hardware, some in software, and some in a hybrid manner. 1.2.2 Simulation Modeling Simulation involves constructing a model for the behavior of the system and driving it with an appropriate abstraction of the workload. The major advantage of simulation is its generality and flexibility, almost any behavior can be easily simulated.

20. However, there are many important issues that must be considered in simulation: .1It must be decided what not to simulate and at what level of detail. Simply duplicating the detailed behavior of the system is usually unnecessary and prohibitively expensive. .2Simulation, like measurement. Generates much raw data. which must be analyzed using statistical techniques. .3Similar to measurements. A careful experiment design is essential to keep the simulation cost down. Both measurement and simulation involve careful experiment design, data gathering, and data analysis. These steps could be tedious;

21. moreover, the final results obtained from the data analysis only characterize the system behavior for the range of input parameters covered. 1.2.3 Analytic Modeling Analytic modeling involves constructing a mathematical model of the system behavior(at the desired level of detail) and solving it. The main difficulty here is that the domain of tractable models is rather limited. Thus, analytic modeling will fail if the objective is to study the behavior in freat detail. However. For an overall behavior characterization, analytic modeling is an excellent tool.

22. The major advantages of analytic modeling over the other two techniques are (a) it generates good insight into the workings of the system that is valuable even if the model is too difficult to solve, (b) simple analytic models can usually be solved easily, yet provide surprisingly accurate results, and (c) results from analysis have better predictive value than those obtained from measurement or simulation. 1.2.4 Hybrid Modeling A complex model may consist of several submodels, each representing certain aspect of the system.

23. Only some of these submodels may be analytically tractable, the others must be simulated. We can take the hybrid approach. Which will proceed as follows: • Solve the analytic model assuming no fragmentation of memory and determine the distribution of memory holding time. • Simulate only memory allocation. Holding. And deallocation. And determine the average fraction of memory that could not be used because of fragmentation. • Recalibrate the analytic model of step 1 with reduced memory and solve it. (It may be necessary to repeat these steps a few times to get convergence.)

24. 1.3 Applications of Performance Evaluation Performance modeling and evaluation may be needed for a variety of purposes; However, the following needs stand out: • System design: In designing a new system. One typically starts out with certain performance/ reliability objectives and a basic system architecture. And then decides how to choose various parameters to achieve the objectives. This involves constructing a model of the system behavior at the appropriate level of detail, and evaluating it to choose the parameters.

25. At higher levels of design, simple analytic reasoning may be adequate to eliminate bad choice, But simulation becomes an indispensable tool for making detailed design decisions and avoiding costly mistakes .2System selection: Here the problem is to select the “best” system from among a group of systems that are under consideration for reasons of cost, availability, compatibility, etc. Although direct measurement is the ideal technique to use here.

26. There might be practical difficulties in doing so (e.g., not being able to use them under realistic workloads, or not having the system available locally). Therefore, it may be necessary to make projections based on available data and some simple modeling. 3. System upgrade: This involves replacing either the entire system or parts there of with a newer but compatible unit. The compatibility and cost considerations may dictate the vendor, So the only remaining problem is to choose quantity, speed, and the like. Often, analytic modeling is adequate here; however, in large systems involving complex interactions between subsystems. Simulation modeling may be essential.

27. Note that a direct experimentation would require installing the new unit first, and thus is not practical. .4System tuning: The purpose of tune up is to optimize the performance by appropriately changing the various resource management policies. Some examples are process scheduling mechanism. Context switching, buffer allocation schemes, cluster size for paging, and contiguity in file space allocation. It is necessary to decide which parameters to consider changing and how to change them to get maximum potential benefit. Direct experimentation is the simplest technique to use here, but may not be feasible in a production environment.

28. Since the tuning often involves changes to aspects that cannot be easily represented in analytic models, simulation is indispensable in this application. 5. System analysis: Suppose that we find a system to be unacceptably sluggish. The reason could be either inadequate hardware resources(CPU, memory, disk, etc.) Or poor system management. In the former case, we need system upgrade, and in the latter, a system tuneup. Nevertheless, the first task is to determine which of the two cases applies. This involves monitoring the system and examining the behavior of various resource management policies under different loading conditions.

29. Experimentation coupled with simple analytic reasoning is usually adequate to identify the trouble spots; however, in some cases, complex interactions may make a simulation study essential. .1.4Workload Characterization Irrespective of which of the three basic performance evaluation techniques we use. We need to provide certain inputs to the real system or its model. There are two categories of relevant inputs. The first category includes parameters that can be controlled by the system designer/administrator, e.g. the scheduling disciplines. Interconnections between devices, and resource allocation policies.

30. The second category includes inputs generated by the environment in which the system is used ,e.g., The interarrival times and service demands of incoming jobs, and are usually not under the control of the system designer/administrator. These inputs can be used for driving the real system(as in easurement) or/its Simulation model, and for determining distributions for analytic/simulation modeling. We refer to these inputs as the workload. Workload characterization is one of the central issues in performance evaluation because it is not always clear what aspects of the workload are important. In how much detail the workload should be recorded, and how the workload should be represented and used.

31. .1.5Benchmarking Computer Systems When selecting a machine for a given application environment, it is useful to have some comparative benchmarks available that can be used to either narrow down the choices or make a final selection. In a selection scenario, it is usually not possible to actually access the machine and do a detailed evaluation; instead, one must depend on the published data on the machine. The benchmarks are primarily intended to provide an overall assessment of various machines on the market. Benchmarks are usually run by vendors or third parties for “typical” configurations and workloads, and not by the user interested in the selection process. Although this is a necessary limitation for practical reasons,

32. It also leaves a lot of room for misinterpretation and misuse of the measure both by the vendors and the users. It is therefore important to examine the advantages and disadvantages of commonly used measures.

33. Steps for a Performance Evaluation Study 1. State the goals of the study and define the system boundaries. 2. List system services and possible outcomes. 3. Select performance metrics. 4. List system and workload parameters. 5. Select factors and their values. 6. Select evaluation techniques. 7. Select the workload. 8. Design the experiments. 9. Analyze and interpret the data. 10. Present the results. Start over, if necessary.

34. Performance Evaluation Metrics • Different performance metrics are used depending upon the services offered by the system: • correctly performed • turnaround time - the time between the submission of a batch job and the completion of its output • response time - the interval between a user's request and the system response • throughput (or productivity) - the rate (requests per unit time) at which requests are serviced by the system • utilization of a resource - is a measured as the fraction of time the resource is busy servicing requests

35. Performance Evaluation Metrics • incorrectly performed • reliability - measured by the probability of errors or the meantime between errors • unable to do • availability - the fraction of total time during which the system is at the disposal of the users MTTF - mean time to failure, MTTR - mean time to repair

36. Performance Evaluation Metrics • Performance metrics can be categorised into three classes based on their utility function: • Higher is Better or HB • Lower is Better or LB • Nominal is Best or NB LB NB HB utility utility utility metric (e.g., response time) metric (e.g., throughput) metric (e.g., utilisation)

37. Cost/performance Ratio • A commonly used metric for comparing two or more systems in system procurement application • cost includes h/w and s/w licensing, installation and maintenance costs • performance is measured in terms of throughput under a given response time constraint

38. Response Time • defined as the time interval between the instant the inputting command to an interactive system terminates and the corresponding reply begins to appear at the terminal Keying in of command Wait in output queue Wait in input queue Transmission time Transmission time Wait Wait Wait Output CPU CPU or I/O I/O CPU Send command Start output Stand-alone response time Response time

39. Response Time • response time depends very much on the type of the command the system executes: • light commands - requires less than one quantum of CPU time (e.g. text editing commands - insert, delete, modify; requests for information such as date and time, etc. • heavy commands - requires more than one quantum time to be executed (e.g. compilation, execution, sort, etc.) • The mean response time Rm does not provide complete information about an interactive system's performance, i.e. variability of performance. The standard deviation of response time, , is commonly used: where Ri is the response time of the ith command.

40. Throughput • defined as the rate (requests per unit time) at which requests are serviced by the system • examples: batch systems - jobs per second; interactive systems - requests per second; CPUs - MIPS or MFLOPS; networks - packets per second (pps) or bits per second (bps); transaction processing systems - TPS

41. Throughput • Depending on the system, throughput is influenced by many factors: • e.g., in a computer network, good routing combined with congestion control schemes can improve throughput routing 1 routing 2 Mean response time Throughput ideal with congestion control Without congestion control Load Load throughputs

42. Throughput Response time Load Load Capacity of a System • Nominal capacity: maximum achievable throughput under ideal workload conditions (e.g., bandwidth (in bps) of a n/w) • Usable capacity: maximum throughput achievable without exceeding a prespecified response time limit • Knee capacity: throughput at the knee (point beyond which response time increases rapidly with load but increase in throughput is small) Nominal Usable Knee

43. Client Network Server System Performance Evaluation Study Example 1. System Definition: The goal of the case study is to compare the performance of data transfer using remote pipes to those of data transfer using remote procedure calls.

44. Example 2. Services: The services offered by the system are the two types of channel calls - remote procedure call and remote pipe. 3. Metrics: The resources are the local computer (client), the remote computer (server), and the network link. Restricting the study to only correct operation, we have the following performance metrics: (a) Elapsed time per call (b) Maximum call rate per unit of time, or equivalently, the time required to complete a block of n successive calls (c) Local CPU time per call (d) Remote CPU time per call (e) Number of bytes sent on the link per call

45. Example 4. Parameters: The system parameters that affect the performance of a given application and data size are the following: (a) Speed of the local CPU (b) Speed of the remote CPU (c) Speed of the network (d) Operating system overhead for interfacing with the channels (e) Operating system overhead for interfacing with the networks (f) Reliability of the network affecting the number of retransmissions required

46. Example The workload parameters that affect the performance are the following: (a) Time between successive calls (b) Number and sizes of the call parameters (c) Number and sizes of the results (d) Type of channel (e) Other loads on the local and remote CPUs (f) Other loads on the network

47. Example 5. Factors: The key factors chosen for this study are the following: (a) Type of channel. Two types-remote pipes and remote procedure calls-will be compared. (b) Speed of the network. Two locations of the remote hosts will be used-short distance (in the campus) and long distance (across the country). (c) Sizes of the call parameters to be transferred. Two levels will be used-small and large. (d) Number n of consecutive calls. Eleven different values of n -1, 2, 4, 8, 16, 32,…, 512, 1024, - will be used. The factors have been selected based on resource availability and the interest of the sponsors.

48. Example 6. Evaluation Technique: Since prototypes of both types of channels have already been implemented, measurements will be used for evaluation. Analytical modeling will be used to justify the consistency of measured values for different parameters. 7. Workload: The workload will consist of a synthetic program generating the specified types of channel requests. This program will also monitor the resources consumed and log the measured results. Null channel requests with no actual work but with monitoring and logging activated will be used to determine the resources consumed in monitoring and logging.

49. Example 8. Experimental Design: A full factorial experimental design with 23 x 11 = 88 experiments will be used for the initial study. 9. Data Analysis: Analysis of Variance will be used to quantify the effects of the first three factors and regression will be used to quantify the effects of the number n of successive calls. 10. Data Presentation: The final results will be plotted as a function of the block size n.

50. Common Mistakes in Performance Evaluation 1. Is the system correctly defined and the goals clearly stated? 2. Are the goals stated in an unbiased manner? 3. Have all the steps of the analysis followed systematically? 4. Is the problem clearly understood before analyzing it? 5. Are the performance metrics relevant for this problem? 6. Is the workload correct for this problem? 7. Is the evaluation technique appropriate? 8. Is the list of parameters that affect performance complete?