Venkita Subramonian and Christopher Gill* {venkita,cdgill}@cse.wustl

1. Reusable Models for Timing and Liveness Analysis of Middleware for Distributed Real-Time and Embedded Systems César Sánchez and Henny Sipma+ {cesar, henny}@cs.stanford.edu Department of Computer Science Stanford University, Stanford, CA EMSOFT 2006 October 23-25, 2006, Seoul, Korea Venkita Subramonian and Christopher Gill* {venkita,cdgill}@cse.wustl.edu Dept. of Computer Science and Engineering Washington University, St. Louis, MO *Research supported in part by NSF CAREER award CCF-0448562 +Research supported in part by NSF grants CCR-01-21403, CCR-02-20134, CCR-02-09237, CNS-0411363, and CCF-0430102, ARO grant DAAD19-01-1-0723, and NAVY/ONR contract N00014-03-1-0939 Thanks also to Joseph Sifakis, Marius Bozga and Iulian Ober for valuable advice about the IF tool set, and to Doug Schmidt and the hundreds of ACE+TAO+CIAO developers/users world-wide

2. client server image side side source displays virtual folder, images transmission adaptation middleware middleware low (and variable) bandwidth radio link A Motivating Example: Real-Time Image Transmission • Soft real-time distributed system (over wireless) • Many end-to-end activities must be managed at once • image (de-)compression, transmission, analysis, display • Needs adaptive enforcement • Task scheduling, rate adaptation, feedback control • Many applications are like this • Is correct reuse of middleware possible? Console [Gill et al., “Integrated Adaptive QoS Management in Middleware: An Empirical Case Study” (RTAS ‘04)] [Wang et al., “CAMRIT: Control-based Adaptive Middleware for Real-time Image Transmission” (RTAS ‘04)] Camera

3. Motivation for Formal Models of Middleware …a main obstacle to the application of rigorous developmenttechniques is the lack of methodology for relating application software and functional design to physical architecture and implementation… Joseph Sifakis, “Modeling real-time systems-challenges and work directions”, EMSOFT 2001 • The formal verification community needs accurate middleware models ORB configurations have a substantial impact on performance, e.g., TAO's fastest concurrency model is thread-per-connection, rather than thread pool… • The real-time embedded systems community needs formal ways to evaluate the semantics of different middleware configurations …what's ultimately most important is to determine how things perform for applications that are representative of actual user configurations, rather than micro-benchmarks alone… Doug Schmidt, ICE/CORBA newsgroup discussion 3/29/05

4. A Brief Survey of Other Related Work • Vanderbilt University and UC Irvine • GME, CoSMIC, PICML, DREAM • UC Berkeley • Ptolemy, E-machine, Giotto • Kansas State University and University of Nebraska • Bogor, Cadena • University of Pennsylvania • VERSA • University of Brest • Cheddar

5. TAO+CIAO ACE ACE+TAO+CIAO COTS Middleware Architecture CODES+ISSS (Tuesday, session A10) system software engineering focus A “Distributed System Software Stack” EMSOFT (Monday, session B1) protocol design and proof focus EMSOFT (Wednesday, session B9) modeling and verification focus

6. Bridging the Gap: composable, reusable and executable timed automata models of ACE building blocks Goal: Correct (Re-)use of DRE Middleware Thesis: formal models of fine-grain middleware building blocks provide a principled approach to analyze timing and liveness properties of middleware-based DRE systems System Implementation System Modeling CADENA DREAM RMA Ptolemy GME CoSMIC UPPAAL IF-toolkit SPIN Bogor Formal approach to composition Reusable software building blocks

7. Another Gap: Between High Level Models and Lower-Level Consequences of their Configuration Choices Application Models Implementation Platform CORBA CCM Collocation strategies Buffer flushing strategies Communication strategies Connection strategies Concurrency strategies • Modeling at a high-level of abstraction is helpful • Focus on nuances of the application • Low-level design choices matter too • May invalidate (implicit) assumptions • We focus on low-level frameworks • Capture fundamental concurrency and timing details of the middleware EJB Pattern-based frameworks OS

8. Model Architecture in IF for ACE • Network/OS layer: interprocess communication abstractions • Middleware layer: ACE pattern-oriented abstractions • Application layer: application-specific semantics within ACE event handlers

9. Modeling Threads • Challenge • No native constructs for threads in model checkers that currently support timed automata • Option 1: model all thread actions as a single automaton • Suitable for high level modeling of application semantics • Option 2: model a thread as multiple interacting automata • Interactions model the flow of control • This option better abstracts the nuances of ACE-level mechanisms Foo Bar input method_request output method_request input method_result output method_result

10. 2 automata per thread Modeling Thread Scheduling Semantics (1/4) Activity1 Activity2 Update Display Control Flow Rate • Easy to achieve with one automaton per thread • Specify to model checker directly • E.g., using IF priority rules • More difficult with more than one automaton per thread • Thread of control spans interactions among automata • Need to express thread scheduling in terms of execution control primitives provided by the model checker 1 automaton per thread prio_rule: pid1 < pid2 if pid1 instanceof Activity1 and pid2 instanceof Activity2 Foo Bar input m_req output m_req input m_result output m_result

11. Modeling Thread Scheduling Semantics (2/4) Resulting Behavior Bar1 Foo1 Foo1 Bar1 1 2 Bar2 Foo2 Bar1 Foo1 Prio=5 Prio=8 • Solution • Introduce a thread id that is propagated along automata interactions • Thread id acts as an index to a storage area which holds each thread’s scheduling parameters Foo2 Bar2 2 1 thread_schedule: pid1 < pid2 if pid1 instanceof Foo1 and pid2 instanceof Bar1 and ({Foo1}pid1).threadid <> ({Bar1}pid2).threadid and ({Thread}(({Foo1}pid1).threadid)).prio < ({Thread}(({Bar1}pid2).threadid)).prio ) Hint to the model checker Give higher preference to the automaton whose “thread” (pointed to by thread id) has higher priority

12. Modeling Thread Scheduling Semantics (3/4) Bar1 Foo1 Foo2 Foo3 Foo1 Bar2 Foo2 Bar1 Foo1 • What if two threads have the same priority? • In an actual implementation, run-to-completion (SCHED_FIFO) may control the possible interleavings • How can we model run-to-completion? Bar3 Foo3 Bar1 Bar2 Bar3 Bar1 Foo1 Bar1 How do we prune out this space? Bar2 Foo2 Foo3 Bar3

13. Modeling Thread Scheduling Semantics (4/4) • Solution • Record id of currently executing thread • Update when executing actions in each automaton Current=nil Foo1 Bar1 Current=1 Current=2 Foo2 Bar1 Current=1 Foo3 Bar1 Current=1 Bar1 Hint to the model checker Give higher preference to the automaton whose thread is the currently running thread. Non-deterministic choice if Current is nil Current=2 Bar2 Current=2 Bar3 Current=2

14. Problem: Over-constraining Concurrency Hint to the model checker Current=nil Give higher preference to the automaton whose thread is the currently running thread. Non-deterministic choice if Current is nil Foo1 Bar1 Current=1 Current=2 Foo2 Bar1 Current=1 Foo3 Bar1 Current=1 Bar1 Current=2 Bar2 Current=2 Bar3 always chosen to run Bar3 Bar3 Current=2 Current=2 Time progresses Foo3

15. Solution: Idle Catcher Automaton Foo3, Bar3 blocked at this point Current=2 Idle catcher runs Current=nil Time progress Current=nil Foo3 Bar3 • Key idea: lowest priority “catcher” runs when all others are blocked • E.g., catcher threadin middleware group scheduling (RTAS ‘05) • Here, idle catcher automaton • runs when all other automata are idle (not enabled), but before time progresses • Resets value of current id to nil Foo3 or Bar3 could be chosen to run. Over-constraining eliminated

16. Problem: Tractability “right away” “maybe tomorrow?” “go for an espresso” “get coffee” “in a minute” • Model checking can suffer from state space explosion • State space reduction, live variable analysis can help • But even good model checkers don’t fully solve this • Need to think of modeling as a design issue, too • Does the model represent what it needs to represent? • Can the model be re-factored to help the checker? • Can domain specific information help avoid unnecessary checking?

17. Optimization 1: Leader Election Token to access the reactor is available T1 T3 T2 T2 T3 T3 • Leader/Followers concurrency • Threads in a reactor thread pool take turns waiting on the reactor • One thread gets the “token” to access the reactor - leader • All other threads wait for the token – followers • It does not matter which thread gets selected as leader in a threadpool • Model checker not aware of this domain specific semantics • For BASIC-P protocol example, saved factor of ~50 in state space, and factor of ~20 in time Prune this out

18. Optimization 2: System Initialization S1 creates R S2 creates R 1 1 S1 R S2 R S2 creates R S1 creates R 1 1 S1 R S2 R • Similar idea, but different technique • Iff ok to establish initial object relations in any order, can optimize away • E.g., 2 server automata, each of which creates a reactor automaton • Useful when modeling object systems in model checkers with dynamic automaton creation capability (e.g., IF) • State space reduction depends on application • Factor of ~250 for a deadlock scenario with 2 reactors and 3 threads in each reactor 2 2 S2 R S1 R Prune this out

19. Verification of a Real-Time Gateway Consumer1 Supplier1 Consumer2 Gateway Consumer3 Supplier2 Consumer4 • An exemplar of many realistic ACE-based applications • We modified the Gateway example to add new capabilities • E.g., Real time, Reliability, Control-Push-Data-Pull • Value added service in Gateway before forwarding a to consumer • E.g. Using consumer specific information to customize data stream • Different design, configuration choices become important • E.g., number of threads, dispatch lanes, reply wait strategies

20. Model Checking/Experiment Configuration C1 100ms Gateway 20 C2 100ms Period 20 100ms S1 10 C3 50ms 10 50ms S2 C4 50ms Relative Deadline Value-added execution (and its cost) • Gateway is theoretically schedulable under RMA • Utilization = 80% • Schedulable utilization = 100% for harmonic periods • Assumption – Messages from 50ms supplier is given higher preference than 100ms supplier • ACE models let us verify scheduling enforcement • IN THE ACTUAL SYSTEM IMPLEMENTATION

21. Real-time Gateway – Single Thread Gateway ConsumerHandler SupplierHandler ConsumerHandler SupplierHandler Consumer Supplier ConsumerHandler ConsumerHandler • Single reactor thread dispatches incoming events • I/O (reactor) thread same as dispatch thread • I/O thread responsible for value added service Reactor

22. Real-time Gateway – Dispatch Lanes Gateway ConsumerHandler SupplierHandler ConsumerHandler SupplierHandler Consumer Supplier ConsumerHandler ConsumerHandler • Single reactor thread again dispatches events to gateway handlers • I/O (reactor) thread puts message into dispatch lanes • Lane threads perform value added service, dispatch to consumers • DO QUEUES HELP OR HURT TIMING PREDICTABILITY? Reactor

23. Model/Actual Traces for Real-time Gateway Execution in the context of lane threads Execution in the context of reactor thread Single threaded Gateway Gateway with dispatch lanes S1,S2 S2 S1,S2 S2 Deadline miss for Consumer4 because of blocking delay at reactor C1 C2 C3 C4 C3 C4 C1 C2 C3 C4 C2 Model C1 C2 C3 C4 Actual C3 C4 C1 C2 C3 C4 C2 Time Time 10 20 30 40 50 60 10 20 30 40 50 60 70 80 90 100 Expected execution timeline with RMS C3, C4 C1, C2, C3, C4 C3 C4 C1 C2 C3 C4 C2 10 20 30 40 50 60 70 80 90 100

24. Concluding Remarks • Timed automata models of middleware building blocks • Can be used to verify middleware concurrency and timing semantics • Domain specific model checking optimizations • Can achieve significant reductions in state space • We have run multiple case studies using these models • Allow comparison of model checking & software execution traces • Demonstrate reusability and fidelity of our models • Future work • Continue to develop reusable models and optimizations • Incorporate domain information directly into model checking tools • Extend our approach beyond real-time concerns • Model mechanism faults and failure modes • Power usage (yet another ~real valued “clock”)

25. Questions?

26. Backup Slides

27. Timing Traces for BASIC-P Protocol EH11 EH21 R1 R2 EH31 Flow1 EH12 EH22 R1 R2 EH32 Flow2 Model checking & actual timing traces show BASIC-P protocol’s regulation of threads’ use of resources (no deadlock) EH13 EH23 R1 R2 EH33 Flow3

28. BASIC-P Blocking Delay Comparison Model Execution Actual Execution Blocking delay for Client2 Blocking delay for Client3

29. Overhead of ACE TP/DA reactor with BASIC-P Negligible overhead with no DA protocol Overhead increases linearly with # of event handlers due suspend/resume actions on handlers at BASIC-P entry/exit