Time and Event Based System Software Construction

1. Time and Event Based System Software Construction Christopher D. Gill cdgill@cse.wustl.edu Department of Computer Science and Engineering Washington University, St. Louis EECS Lecture Series Vanderbilt University, Nashville, TN Thursday October 13th, 2005 Research supported in part by NSF CAREER award CCF-0448562 Thanks: Venkita Subramonian, Huang-Ming Huang, Jacob Perkins, Robert Pless (WUSTL)César Sánchez, Henny Sipma, Zohar Manna (Stanford)

2. My Research Focus Boeing/DARPA F33615-01-C-1898 (NEST) • Middleware: system software spanning networked hosts • Hides complexity of underlying operating system and network • Location/language transparency, event and message scheduling • Significant potential to reduce system development time/cost • Research challenges: advancing the state of the art • Fine-grain customization for space/time constrained applications • Adaptation for real-time assurances in variable environments • Formal specification and checking for high-assurance systems

3. Motivation: Model/Software Chasm Boeing/AFRL F33615-97-D-1155/0005 (WSOA) • Increasing distance for system software between • What development tools can produce • What analysis tools can check for correctness • Distributed real-time embedded systems also face • Memory and power resource constraints • Stringent timing requirements end-to-end • Distribution across heterogeneous networks/platforms

4. Expert Opinions ORB configurations have a substantial impact on performance, e.g., TAO's fastest concurrency model is thread-per-connection, rather than thread pool … …what's ultimately most important is to determine how things perform for applications that are representative of actual user configurations, rather than micro-benchmarks alone… Dr. Douglas Schmidt ICE / CORBA newsgroup discussion 3/29/05 …a main obstacle to the application of rigorous development techniques is the lack of methodology for relating application software and functional design to physical architecture and implementation… Dr. Joseph Sifakis, “Modeling real-time systems-challenges and work directions”, EMSOFT 2001

5. It’s All About Abstraction • Abstraction Can Help • Location Transparent Remote Invocation • Abstraction Can Hurt • Semantic Mis-matches (especially real-time)

6. The Issue: Complexity at Each Level Component facet upcall servant Client lookup thread pool stub POA ORB core • CIAO – QoS Aware Component Model • Component, package, assembly, deployment • TAO/nORB – Object Request Brokers • Reference, stub, skeleton, ORB core, POA • ACE – Portable C++ Framework • Reactor, handlers, thread pools, timers • POSIX – OS interface • Threads, sockets timers ORB core recv reactor send IIOP message socket

7. Key Questions • (1) Does it really have to be so complicated ? • “But, I just need real-time remoteinvocations...” • I n many cases, complexity can’t be eliminated • (2) What problems have been addressed already ? • A lot of progress on functional properties • Frameworks, components, design patterns, static analysis, types • A lot of progress on individual QoS properties • Real-time theory, OS/middleware scheduling, RTSJ, RT-CORBA • Some promising steps towards bringing these together • DSLs, modeling environments, hybrid control, QoS aspects • (3) What problems still need to be addressed ? • General, viable foundation for software construction • Detailed models of software abstractions used by developers • Composition of models and software side-by-side • Tractable analysis techniques, tools for round-trip engineering

8. Goal: Low-Level Software/Model “Bridge” • Model primitive building blocks of well-designed software • Compose model elements & software elements side-by-side • Check and/or enforce properties in a tractable way

9. Research Problem I: Level of Abstraction • Higher levels of abstraction better for developers • Promote design consistency and quality • Hide tedious and error prone details • Lower levels of abstraction better for analysis • Sufficient detail to evaluate properties rigorously • Can model concurrency, timeliness explicitly

10. Solution Approach • Hypothesis: ACE/POSA2 level strikes good balance • Developers work with higher level abstractions • Models of those abstractions sufficient for real-time • Key abstractions • Reactor, Event Handler, IPC Service Access Point, ... • Run model and empirical evaluations side-by-side

11. Timed Automata Models (UPPAAL) Communication (Event) Guard (Value) Invariant (Value) Action (Value) State Variable Clock Variable

12. Selection of Modeling Tools • SPIN • Widely known, automata are untimed • UPPAAL • Nice graphical interface, timed automata • No message parameters, instances defined statically • IF (what we’ve chosen to work with) • Timed automata, program models in an input language • Connects to a variety of back-end toolsets (e.g. CADP) • ADT support to abstract details (C/C++ implementation) • Bogor (what we plan to help extend) • Highly extensible model checker, open-source • Explicit representations of objects, concurrency • Automata are untimed (we’re interested in adding time)

13. Example: Service Access Point Model in IF

14. Research Problem II: Composition • Developers must preserve QoS properties • As various software elements are integrated • Could preclude undesired semantics for a property • Time triggered architecture, timed languages • Can address a restricted domain very well • Could enforce desired semantics • Model integrated computing, real-time scheduling • Can address a wider domain at a cost of some complexity

15. Solution Approach • Hypothesis: enforcement is possible and sufficient • Extend from base of low-level mechanism models • Study issues that arise when they’re composed • Use model checking to unify different analyses

16. Example: Reactor Thread Management Environment Middleware Application De-multiplexing & Dispatching Application logic Event sources Event Handlers Reactor Synchronous wait-enable Serial dispatching Asynchronous event arrival

17. Two Different Reply Wait Strategies • Wait on connection strategy • Gives “non-interfering” calls • But may lead to deadlock • Wait on reactor strategy • Re-use thread while waiting • Introduces new blocking factors • Analysis depends on low-level middleware implementation • E.g., blocking strategy, kind of reactor, method continuation DARPA F33615-03-C-4111 (PCES)

18. Multi-Agent Deadlock Call Sequence • Wait on connection strategy + TP reactor + RMI graph • Can create “deadly embrace” scenarios • Even if single threaded case won’t deadlock

19. Wait-on-Connection : Static Dependency DAG reactor R2 k2 = 1 dependency m6 m2 threads m0 methods m3 m4 • Model needs abstractions from multiple layers • Application layer: method invocation dependency DAG • ACE layer: reactors, threads, reply wait strategy • Deadlock can occur w/ WOC • Iff ik > Ki calls into reactor Ri on a path in dependency DAG • Where Ki is the number of threads in reactor Ri • E.g., <m0,m2,m4,m6s> • Graph node colors designate to which reactor they belong • Count color changes along a path • Models wait-on-connection, but not wait-on-reactor (yet) m5 m1 m7 k1 = 2 reactor R1

20. Wait-on-Reactor : Dynamic Call Tree m0 m1 may interleave m7 m2 m3 m5 m4 m3 m6 m5 may be permutable • Reformulate dependency graph • As a dynamic method call tree • Edges denote method calls • Nodes represent stack frames • Tree traversal is ordered • E.g., left to right, post-order • E.g., deadline & blocking analysis • Child order may be permutable • Subject to additional constraints • Can now analyze wait-on-reactor • Distinct trees’ method calls may interleave • Upcalls serialized, nested per reactor thread (blocking factors) • Reactor threads concurrent

21. Recent Collaborations On Composition m2 m3 m4 m6 m5 Distributive reduction across m4 • WUGLE User Guided Lexical Engine • Jacob Perkins and Robert Pless • Expression DAG transformation tool • Distributive expansion/reduction • Maintain functional equivalence • Remove deadlocks • Reactor Thread Allocation Protocols • César Sánchez, Henny Sipma, Zohar Manna • Decorate call graph with “thread height” • Protocol: allocate thread iff safe • Blocking factor trade-offs (again) 2|3 m4 1|2 1|2 m2 m2 1 1 1 1 m3 m5 m3 m6

22. f2 f5 f3 f6 f5reply queued Blocking factor for f2 f3 returns f5 reply processed f2 returns Timed Automata Compose + Unify Analysis f5 reply processed f2 returns

23. Research Problem III: Tractability “right away” “maybe tomorrow?” “go for an espresso” “get coffee” “in a minute” • Model checking can produce a state space explosion • State space reduction, live variable analysis can help • But even good model checkers don’t fully solve this • Need to think of modeling as a design issue, too • Does the model represent what it needs to represent? • Can the model be re-factored to help the checker?

24. Solution Approach Theorem 3 (Annotation). Given a system S and annotation α, if the global call graph GS does not contain any cyclic dependencies, then Efficient-P used with α guarantees absence of deadlock for S. From: Sánchez, Sipma, Subramonian, Gill, and Manna, “Thread Allocation Protocols for Distributed Real-Time and Embedded Systems”, FORTE ‘05 • Hypothesis: • Some properties can be proven invariant • Other properties can be enforced at run-time • Remaining properties can be checked tractably • Model/software co-design • Add functionality to model to improve checking • Add functionality to software to enforce properties • But watch for whether this makes things better or worse!

25. Helping the Model Checker: Idle Catcher

26. Reactor with Deadlock Avoidance • Co-design • DA Reactor • IF Model • Can execute the model • Can run the software

27. Deadlock Avoidance Protocol Overhead

28. A Sampling of Other Related Work • Vanderbilt University • GME, CIAO, CoSMIC, PICML • UC Berkeley • E-machine, Giotto, Ptolemy • Washington University • Temporal types, schedulability using C++ type system • MIT • Concurrency type systems • Utah • Task/scheduling first-order logic

29. Concluding Remarks: ~5 Year Research Agenda • Model broad substrate of primitives • Via checkable formal models (timed automata) • Empirical evaluation (esp. of timing, concurrency) • Possibly extend beyond ACE to other domains • Define composition algebras, type systems • Minimize model checking to the extent possible • Capture/exploit domain knowledge/abstractions • Extend approach beyond real-time concerns • Model mechanism faults and failure modes • Power usage (yet another ~real valued “clock”?)

30. Christopher D. GillTime and Event Based System Software Construction Questions ?

31. Christopher D. GillTime and Event Based System Software Construction Thanks!

32. Backup Slides

33. IF Deadline Miss Extract

34. IF Priority Rules Extract

35. Unidirectional IPC Model in IF

36. IF Service Handler Extract

37. IF Select Extract

38. Single Agent Deadlock Scenario

39. Simple Blocking Factors Composition

40. Multi-Agent Blocking Factors Composition