370 likes | 456 Views
How Fast and Fat Is Your Probabilistic Model Checker?. an experimental performance comparison David N. Jansen 3,1 , Joost-Pieter Katoen 1,2 , Marcel Oldenkamp 2 , Mari ëlle Stoelinga 2 , Ivan Zapreev 1,2 1 MOVES Group, RWTH Aachen University 2 FMT Group, University of Twente, Enschede
E N D
How Fast and Fat IsYour Probabilistic Model Checker? an experimental performance comparison David N. Jansen3,1, Joost-Pieter Katoen1,2, Marcel Oldenkamp2,Mariëlle Stoelinga2, Ivan Zapreev1,2 1 MOVES Group, RWTH Aachen University 2 FMT Group, University of Twente, Enschede 3 ICIS, Radboud University, Nijmegen
ProbabilisticModel Checking Probabilistic System Probabilistic Requirement PRISM (hybrid) PRISM (sparse) MRMC VESTA ETMCC Probabilistic Model Probabilistic Formula YMER ≤ Probabilistic Model Checker Yes No Probability
Why Are Probabilities Useful? • system performance • uncertainty in the environment • randomized (networking) algorithms • abstract from large populations
ProbabilisticModel Checking... • What is inside? • temporal logics + model checking • numerical and optimisation techniquesfrom performance and operations research • Where is it used? • powerful tools • applications: distributed systems, security, biology, quantum computing... • Problem: Which tool to choose?
Probabilistic Models Discrete time Markov chains Continuous time Markov chains automata transitions are probabilistic timing for CTMC: Prob(wait time ≤ t) = 1 – e–t Probabilistic System Probabilistic Requirement Probabilistic Model Probabilistic Formula Probabilistic Model Checker Yes No Probability
SynchronousLeader Election • nodes in a ring elect a leader • each node selects random number as id • passes it around the ring (synchronously) • if unique id,node with maximum unique id is leader • [Itai & Rodeh 1990]
1 4 2 2 5 3 1 5 SynchronousLeader Election
1 4 4 5 1 5 2 5 1 4 5 1 2 2 1 1 3 5 3 5 2 2 2 3 SynchronousLeader Election 1 4 2 2 5 3 1 5
Probabilistic Formulas Reachability Bounded Reachability Steady-State Property extensions of CTL Probabilistic System Probabilistic Requirement Probabilistic Model Probabilistic Formula Probabilistic Model Checker Yes No Probability
ProbabilisticModel Checkers Probabilistic System Probabilistic Requirement Choices made Three examples Probabilistic Model Probabilistic Formula Overall evaluation Probabilistic Model Checker Yes No Probability
Modelling informal description PRISM model adapt syntax VESTA model .tra format model YMER model ETMCC MRMC PRISM YMER VESTA
Experiment Relevance • Repeatable • Verifiable • Significant • Encapsulated
Experiment 1Reachability • Cyclic Polling Server:server cycles over n stationsand serves each one in turn • e.g. teacher walks through class,each pupil may ask a question • busy1P≥1(trueUpoll1)If station 1 is busy,the server will poll it eventually
PRISM: MTBDD Size • Multi-Terminal BDD =data structure for transition matrix • size heavily depends on model • large MTBDD slow
CPS versus SLE runtime 458.847 states 1.131.806 MTBDD nodes 7.077.888 states 2.745 MTBDD nodes
VESTA:simulation problem • actual probability close to bound P≥p(...) • estimate is almost always in [p–,p+] • some irregularity stops the simulation • 0.95 Prob(yes actual Prob≥p) Prob(actual Prob≥p yes)
Result Overview: Timing depends heavily on MTBDD size depends heavily on MTBDD size depends heavily on MTBDD size
Result Overview: Memory MTBDD size varies heavily almost independent from model size
Experiment 2Bounded Reachability • Tandem Queueing Network • two queues after each other • P<0.01(trueU ≤2full)Is the probabilitythat the system gets full in 2 time unitssmall? checkin counter security check
Experiment 3Steady State Property • Tandem Queuing Network • S>0.2( P>0.1(X2nd queue full) )In equilibrium,the probability to satisfy is > 0.2 P>0.1(X2nd queue full) P>0.1(X ...)
Simulating Steady State? • simulation of bounded reachabilityhas clear stopping criterion • simulation of unbounded reachability reachability with very large bound • simulation of steady state? never stops
Nested Formulas • we also checked nested propertiesP≥0.8(P≥0.9(trueU ≤100n70) Un50) • not detailed here
Result Overview: Timing based on a single property only: did not terminate