Performance Modelling of Complex Hardware/Software Systems

1. Performance Modelling of Complex Hardware/Software Systems B.D. Theelen www.ics.ele.tue.nl/~btheelen

2. Overview • Introduction • System-Level Design • Software/Hardware Engineering • Case Studies • Modelling and Validation • Performance Evaluation • Accuracy Analysis • Summary www.ics.ele.tue.nl/~btheelen

3. Concepts Requirements System-Level Design Design Time Component-Level Design (Automatic)System Implementation Implementation Introduction • Designing a complex system within limited time involves taking important decisions in an early phase of the design process, which may have a deep impact on the performance of the system • Early assessment of the impact of design decisions involves using system-level design methods and tools www.ics.ele.tue.nl/~btheelen

4. Three Phases • Formulation • Informal identification and specification of system concepts and design issues • Formalisation • Formal specification of behaviour and architecture • Validation of adequacy of formal system specification • Formal specification of the properties to evaluate • Evaluation • Property analysis and design decisions www.ics.ele.tue.nl/~btheelen

5. Software/Hardware Engineering (SHE) • Formulation using(stereotyped) UML diagrams • Performed case studies: documents with plain texts and various diagrams • Formalisation based on system-level modelling language POOSL • Intended for complex real-time distributed hardware/software systems • Small set of very expressive language primitives • Formal semantics • Defines unambiguously how to handle concurrency, time and probabilism • Evaluation • Performance evaluation founded onMarkov chain analysis • Analytic computation or estimation by simulation • Formal verification of correctness properties based on model checking • Tools • SHESim: Interactive creation and validation of POOSL models • Rotalumis: Fast execution of POOSL models for analysis www.ics.ele.tue.nl/~btheelen

6. Case Study: Network Processor • (Deep) packet processing in telecommunication systems • Utilisation of shared busses • Identify possible bottlenecks • Comparison POOSL - SystemC www.ics.ele.tue.nl/~btheelen

7. Case Study: Dataflow System • Part of Alcatel’s Internet Router • Prepares/processes packets for correct routing • Arbitration of memory accesses from 3 components • Evaluate alternative arbitration mechanisms www.ics.ele.tue.nl/~btheelen

8. Sources Input Buffers Switch Core Output Buffers Sinks 1 1 1 1 1 N Outputs Inputs 1 N N N N N Case Study: Internet Router Is the specified flow-control mechanismsuitable for various product variants? www.ics.ele.tue.nl/~btheelen

9. Executable Model • Modelling guidelines (validation) • Readability (aggregate data objects), postpone optimising for execution speed • Log considerations on the adequacy of abstractions • Modelling patterns • Point abstractions, abstraction from scheduling mechanisms, stream-based modelling, scalability of model, model refinement www.ics.ele.tue.nl/~btheelen

10. Validation • Validation guidelines • Perform step-by-step inspections • Analyse easy-to-check properties • Investigate impact of rare events • Consider symmetries in system • Recall considerations on adequacy of abstractions www.ics.ele.tue.nl/~btheelen

11. Understandable POOSL Model + Extensions for Performance Evaluation Formal Semantics Markov Decision Process + Reward Structure External Scheduler Discrete-Time Markov Chain + Reward Structure Property Formalisation • Two options for expressing performance metrics • Performance analysis based on external observers: temporal rewards • Reflexive performance analysis: POOSL • Extend model with additional behaviour to evaluate performance metrics • Guidelines for reflexive performance analysis • Define extensions in subclasses • Use method overriding capabilities of POOSL www.ics.ele.tue.nl/~btheelen

12. Performance Evaluation • Analytical computation: Ergodic Theorem • Estimation by simulation: Central Limit Theorem • Example: packet-loss probability • Enables accuracy analysis with confidence intervals • Enables automatic termination of simulation when results are accurate www.ics.ele.tue.nl/~btheelen

13. Accuracy Analysis • For Internet Router example: a 99% confidence interval [0.001046, 0.001052] is obtained for the average latency • Estimation of average latency is 0.001049 seconds • We are 99% sure that the real average latency lies in [0.001046, 0.001052] • An upper bound for the relative error is 1% • Takes 0.0072 seconds of simulated real-time • Takes 11.9 hours of simulation time • Central Limit Theorem only applicable for simple averages • Latency, packet loss probability • Complex averages and variances: algebra of confidence intervals • Defines set operations on confidence intervals • Enables accuracy analysis for range of different averages and variances • Jitter, buffer occupancy, processor utilisation, throughput, burstiness • POOSL library classes for analysing different long-run averages www.ics.ele.tue.nl/~btheelen

14. Summary • Three large case studies in corporation with industrial partners • SHE method distinguishes three phases • Formulation based on informal UML diagrams • Formalisation with formal modelling language POOSL • Guidelines for modelling and validation • Modelling patterns • Guidelines for reflexive performance analysis • Performance evaluation based on Markov chain analysis • POOSL model implicitly defines Markov chain • Extensions for performance analysis define reward structure • Analytical computation or estimation by simulation • Techniques for performance analysis • Confidence intervals allow automatic termination of simulation • Algebra of confidence intervals for accuracy analysis of complex metrics • Library classes for accuracy analysis www.ics.ele.tue.nl/~btheelen