How Fast and Fat Is Your Probabilistic Model Checker?

1. How Fast and Fat IsYour Probabilistic Model Checker? an experimental performance comparison David N. Jansen3,1, Joost-Pieter Katoen1,2, Marcel Oldenkamp2,Mariëlle Stoelinga2, Ivan Zapreev1,2 1 MOVES Group, RWTH Aachen University 2 FMT Group, University of Twente, Enschede 3 ICIS, Radboud University, Nijmegen

2. ProbabilisticModel Checking Probabilistic System Probabilistic Requirement PRISM (hybrid) PRISM (sparse) MRMC VESTA ETMCC Probabilistic Model Probabilistic Formula YMER ≤ Probabilistic Model Checker Yes No Probability

3. Why This Work? • Used more often • applications: distributed systems, security, biology, quantum computing... • Powerful tools • Problem: Which tool to choose?

4. ProbabilisticModel Checkers Probabilistic System Probabilistic Requirement Choices made Four examples Probabilistic Model Probabilistic Formula Overall evaluation Probabilistic Model Checker Yes No Probability

5. Tools

6. Experiment Relevance • Repeatable • Verifiable • Significant • Encapsulated

7. Selected Benchmarks

8. SynchronousLeader Election • nodes in a ring elect a leader • each node selects random number as id • passes it around the ring (synchronously) • if  unique id,node with maximum unique id is leader • [Itai & Rodeh, 1990]

9. 1 4 2 2 5 3 1 5 SynchronousLeader Election

10. 1 4 4 5 1 5 2 5 1 4 5 1 2 2 1 1 3 5 3 5 2 2 2 3 SynchronousLeader Election 1 4 2 2 5 3 1 5

11. RandomizedDining Philosophers • Dining Philosophers • pick up chopsticks in random order • Deadlocks resolved • if there is no second chopstick,give up eating • [Pnueli & Zuck, 1986]

12. Birth–Death Process • Models a waiting queue • Standard modelin performance evaluation • Limit queue size to get finite model

13. Tandem Queueing Network • Two queues after each other • [Hermanns, Meyer-Kayser & Siegle, 1999] checkin counter two-phase security check exponential

14. Cyclic Polling System • server cycles over n stationsand serves each one in turn • e.g. teacher walks through class,each pupil may ask a question • [Ibe & Trivedi, 1990]

15. Modelling informal description PRISM model adapt syntax VESTA model .tra format model YMER model ETMCC MRMC PRISM YMER VESTA

16. Experiment 1Qualitative Properties • unbounded reachability with prob 1 • Cyclic Polling System: busy1P≥1(trueUpoll1)If station 1 is busy,the server will poll it eventually

18. PRISM: MTBDD Size • Multi-Terminal BDD =data structure for transition matrix • size heavily depends on model • large MTBDD  slow

19. CPS versus SLE runtime 458.847 states 1.131.806 MTBDD nodes 7.077.888 states 2.745 MTBDD nodes

20. VESTA:simulation problem • actual probability close to bound P≥p(...) • estimate is almost always in [p–,p+] • some irregularity stops the simulation • 0.95 Prob(yes  actual Prob≥p) Prob(actual Prob≥p  yes)

21. P≥1(... U ...)Timing Overview

22. Analysis

23. Result Overview: Timing depends heavily on MTBDD size depends heavily on MTBDD size depends heavily on MTBDD size

24. Result Overview: Memory MTBDD size varies heavily almost independent from model size

25. Experiment 2Bounded Reachability • Tandem Queueing Network:P<0.01(trueU ≤2full)Is the probabilitythat the system gets full in 2 time unitssmall?

27. Analysis

28. Result Overview: Timing

29. Result Overview: Memory

30. Experiment 3Steady State Property • Tandem Queuing NetworkS>0.2( P>0.1(X2nd queue full) )In equilibrium,the probability to satisfy is > 0.2 P>0.1(X2nd queue full) P>0.1(X ...)

32. Simulating Steady State? • simulation of bounded reachabilityhas clear stopping criterion • simulation of unbounded reachability reachability with very large bound • simulation of steady state? never stops

33. Analysis

34. Result Overview: Timing

35. Result Overview: Memory

36. Nested Formulas • Birth–Death Process:P≥0.8(P≥0.9(trueU ≤100n70) Un50)The probability to reach n50 (while the probability to reach n70in 100 steps never drops <0.9)is ≥0.8

38. Result Overview: Timing did not terminate

39. Result Overview: Memory

40. Conclusions