Challenges in Modeling

1. Challenges in Modeling

2. COMPLEXITIES OF MODELS • Large State Space (e.g. Bedrock, Wireless handoff) • Model construction problem • Model solution problem • Model Stiffness. Fast and slow rates acting together • Failure And Recovery/Repair (HSP Markov model in Bedrock) • Performance and failure (Wireless handoff)

3. COMPLEXITIES OF MODELS(Continued) • Modeling Non-Exponential Distributions (e.g. N+1 problem) • Believability/Understandability/Usability • What about software?

4. Potential Solutions • Largeness • Largeness Tolerance • Largeness Avoidance

5. LARGENESS TOLERANCE • Automated Model Construction • Loops in the specification of CTMC (SHARPE) • Stochastic Petri nets (SPNP, SHARPE) • High level languages (SAVE, QNAP, ASSIST, SDM) • Fault-Tree + Recovery Info (HARP) • Object-Oriented Approaches (TANGRAM)

6. LARGENESS TOLERANCE (Continued) • Efficient numerical solution techniques • Sparse Storage • Accurate and Efficient Solution Methods We have Generated and Solved Models with 1,000,000 states (has gone up considerably recently) Steady-State : NEAR-Optimal SOR Transient: Modified Jensen's method

7. MODEL SPECIFICATION LANGUAGES • Different languages can be used to specify a single model type: SAVE, QNAP, SPNP all appear very different; underlying model type is Markov • Same language can be used to specify different model types:SPNP input language used for Markovian SPN analytic numeric solution or non-Markovian SPN simulation solution

8. MODEL SPECIFICATION LANGUAGES (Continued) • Languages can be domain specific: • Reliability: HARP, SDM • Availability: SAVE • Performance: RESQ, QNAP • Language can be domain independent: • SHARPE, SPNP

9. LARGENESS AVOIDANCE • Non-State-Space methods • Reliability block diagrams • Fault-trees • Product-Form Queuing Networks • Approximate solutions • State Truncation SAVE, SPNP (Kantz and Trivedi: PNPM91)

10. Case Study: JPL REE System Availability Modeling in Spacecraft Architecture

11. LARGENESS AVOIDANCE (Cont.) • Stochastic Petri Nets (State-space-based modeling) • State truncation by introducing guard function Guard g is defined as If (mark(“…_dn”) >= K) return (0); else return (1);

12. SPN MODELING

13. AVAILABILITY MEASURES

14. LARGENESS AVOIDANCE (Continued) • Approximate solutions • Hierarchical Decomposition and Fixed-Point Iteration amongsubmodels: • Heidelberger and Trivedi; IEEE-TC,1983 (Queueing Models) • Ciardo and Trivedi; PNPM91 (SPN Models) • Tomek and Trivedi (Availability Models) • Lanus, Liang & Trivedi: (Bedrock) • Wireless handoff work: Ma, Han & Trivedi

15. LARGENESS AVOIDANCE (Continued) • Approximate solutions • Performability: Multiprocessor example • Fluid Approximation: Mitra; Kulkarni; Ciardo; Nicol, and Trivedi; FSPN

16. Difficulties in Modeling Using MRMs • Stiffness Causes numerical difficulties in solution • Stiffness Tolerance Develop stiffness tolerant numerical solution methods • Stiffness Avoidance Avoid generating stiff models through decomposition

17. Potential Solutions (Continued) • Stiffness • Stiffness Tolerance • Stiffness Avoidance • Modeling Non-Exponential Distributions • Stage-type expansion, MRGP, NHCTMC, DES

18. STIFFNESS TOLERANCE • Automatic Detection of Stiffness (HARP) • Special Stable ODE Solver Reibman and Trivedi (TR-BDF2) Computers and Operations Research, 1988. Malhotra and Trivedi (Pade, Implicit RK)

19. STIFFNESS TOLERANCE (Continued) • Uniformization for Stiff Markov Chains Muppala and Trivedi We can solve models with rate ratios of 108 or higher Implemented in SHARPE & SPNP

20. STIFFNESS AVOIDANCE • Model-level decomposition • Hierarchical Composition (SHARPE) Composition of Submodel solutions without generating a single one-level overall model (Bedrock example) • Fixed-Point Iteration (Wireless handoff example)

21. STIFFNESS AVOIDANCE (Continued) • Importance Sampling (simulation) • Lewis, Goyal, Heidelberger, Shahbuddin, Geist, Nicola • Can also apply to analytic-numeric methods (Heidelberger, Muppala, and Trivedi; Performance 93) • Importance splitting (Simulation) • Tuffin and Trivedi; Tools’ 00

22. Non-Exponential Behavior • Non state space models: Fault Trees, Reliability Graphs, RBDs; no problem

23. Non-Exponential Behaviorin State Space Models

25. NON-EXPONENTIAL DISTRIBUTIONS • Phase-Type Expansions • N+1 example • Non-Homogeneous Markov Chains CARE III, HARP Soft Rel model with imperfect repairs solved using SHARPE

26. NON-EXPONENTIAL DISTRIBUTIONS (Continued) • Semi-Markov Chains N+1 example • Markov Regenerative Processes: Choi, Logothetis, Kulkarni, Trivedi • DSPN and MRSPN: Choi, Kulkarni, Trivedi • Discrete-Event Simulation Now in SPNP (FSPN and Non-Markovian SPN Simulation), RESQ, QNAP, Bones, SES workbench

27. CASE STUDY: AT & T • GSHARPE: • A Preprocessor to SHARPE developed at Bell Labs by a Duke Student. • User can specify Weibull Failure times and lognormal and other repair time distributions. • GSHARPE fits these to phase type distributions and produces a Markov model that is generated for processing by SHARPE

28. Potential Solutions (Continued) • Believability/Understandability/Usability • GUI, many practical examples, short-courses, tools, Boeing SDM project • Incorporation in the design process • VHDL  Availability Model, • C Program  Perf. Model • Ada Program  SPN Perf. Model (SPC) • Connection between measurements & models

29. BELIEVABILITYUNDERSTANDABILITY • Integration of Measurements and Models • Measurements Provide Parameters to Models • Models Provide Guidelines For Measurements • Models Validated Against Measurements • Integration of Different Modeling Tools • Boeing SDM project

30. BELIEVABILITY/UNDERSTANDABILITY (Continued) • Many Case-Studies of Validations Needed • Vaxcluster Availability Model: Wein & Sathaye • Hsueh, Iyer and Trivedi; IEEE-TC, Apr. 1988 • Lucent Validation of ESS; Veena Mendiratta • Technology Transfer • Short courses • Development and Dissemination of Tools (SHARPE, SPNP)

31. BELIEVABILITY/UNDERSTANDABILITY (Continued) • Application of the Techniques and Tools • Motorola • Cisco • 3Com • HP • Sun

32. CASE STUDY: BOEING • An Integrated Reliability Environment • A working prototype • Developed a high-level modeling language (SDM) • Designed and implemented an intelligent interpreter

33. CASE STUDY: BOEING(Continued) • Interpreter determines which solution method is applicable • Translator translates the SDM input file into an input file of any of the engines down below • Five different modeling engines are integrated: • CAFTA, SETS, EHARP, SHARPE and SPNP.

34. MODELING AND MEASUREMENTS: INTERFACES • Measurements supply Input Parameters to Models (Model Calibration or Parameterization) Confidence Intervals should be obtained Boeing, Draper, Union Switch projects • Model Sensitivity Analysis can suggest which Parameters to Measure More Accurately: Blake, Reibman and Trivedi: SIGMETRICS 1988; Fricks and Trivedi: 1997

35. MODEL CALIBRATION What is ? • Fault Model for Each Component • Design,Manufacturing: Heisenbugs, Bohrbugs • Operational: Permanent, Intermittent,Transient • Human • Fault Arrival Processes (PP,Weibull,NHPP) • Failure Rates (Sources:MIL-STD)

36. MODEL CALIBRATION (Continued) What is c ? • Field Data • Fault/Error Injection (FIAT,MESSALINE) • Analytic Coverage Model What is  ? • Maintenance Model Corrective; dispatch , travel, repair time, dead on arrival, imperfect repair Preventive

37. MODEL CALIBRATION (Continued) What is r ? • Binary: Up & Down • Capacity-Oriented: Number of Operational Resources in Each State • Performance-Oriented: Evaluate Perf. in Each Degraded Level of Syst. Config. 1. Measurements 2. Simulation Model 3. Analytic Model -- SHARPE, SPNP

38. VALIDATION&VERIFICATION • Validation: Does the conceptual model faithfully reflect the behavior of the system? • Verification: Has the conceptual model been correctly implemented?

39. MODEL VALIDATION (Continued) • Three step process outlined by Naylor and Finger • Face validation: Discussion with the experts • Input-Output validation: Compare results obtained from model with those from measurements • Validation of model assumptions: Either prove that the assumptions are correct or do statistical testing

40. MODEL ASSUMPTIONS/ERRORS • Errors in Model Structure • Missing or Extra Arcs • Missing or Extra States • Use Face Validation to avoid these errors. • Errors Due to Non-Independence • Distributional Errors • Parametric Errors

41. MODEL ASSUMPTIONS/ ERRORS(Continued) • Errors Due Approximations • Decomposition/Aggregation/Iteration • State Truncation • Numerical Solution Errors • Discretization Errors • Round-Off Errors

42. Model Verification • Programming Errors • Approximation errors: Tight bounds due to approximations are desirable • Numerical: Errors in numerical algorithms should be bounded

43. What about software? • Testing phase • Software reliability estimation • Black-box based approach • Architecture-based approach • Operational phase • Fault tolerance coverage (c in Markov model) • Countering software aging • Symptom-based fault management

44. Conclusions: • Availability evaluation is very important in characterizing systems • Evaluation can be performed either through measurements, simulation or analytical modeling • Model verification and validation should form an integral part of the modeling process