Specification-Based Techniques for Validation at Run-time and Design-time*

1. Specification-Based Techniques for Validation at Run-time and Design-time* Insup Lee • SDRL (Systems Design Research Lab) • RTG (Real-Time Systems Group) • Department of Computer and Information Science • University of Pennsylvania, Philadelphia, PA * Joint work with H.S. Hong, S. Kannan, M. Kim, O. Sokolsky, M. Viswanathan VU '02

2. Embedded Systems • Difficulties • Increasing complexity • Decentralized • Safety critical • Resource constrained • Non-functional: power, size, etc. • Development of reliable and robust embedded software VU '02

3. Software Development Process • Requirements capture and analysis • Informal to formal • Consistency and completeness • Assumptions and interfaces between system components • Application-specific properties • Design specifications and analysis • Formal modeling notations • Abstractions • Analysis techniques (simulation, model checking, equivalence checking, testing, etc.) • Implementation • Manual/automatic code generation • Validation (testing, model extraction, etc.) • Run-time monitoring and checking Requirements Design specification Implementation VU '02

4. Goals of the HASTEN Project • High Assurance Systems Tools and ENvironments (HASTEN) • Develop tools for “end-to-end” software engineering • Requirements capture • Specification, analysis, simulation • Implementation testing • Deployed system monitoring and checking • Integrated use of tools • Vertical integration • Horizontal integration • Case studies • automotive controllers, mobile robots, medical devices, real-time Java, embedded Linux • www.cis.upenn.edu/hasten/ VU '02

5. Testing Monitoring Test Generation Instrumentation Running System/ Filter Test Suites Event Recognizer Testing Abstract Events Checker Test Results Checking Output Prototyping Evaluator Abstraction/ Reengineering Evaluation Report Analysis: - model checking - equiv. checking Rapid Prototyping/ Simulation Diagnostic Requirements Artifacts Informal Requirements Requirements Engineering Formal Requirements System Artifacts Informal Design Diagrams (UML) Verification Prototype Formal Specification Implementation VU '02

6. Java-MaC: a Run-time Validation Tool for Java Programs M. Kim, S. Kannan, I. Lee, O. Sokolsky University of Pennsylvania M. Viswanathan UIUC VU '02

7. Motivation • Weaknesses of formal verification • gap between an abstract model and the implementation • scalability challenge (software size and complexity) • Two approaches to implementation validation • Testing • widely used • limited coverage • lack of formal guarantee • run-time correctness is not guaranteed • Run-time monitoring and checking w.r.t. formal specification (also known as run-time verification) • complementary methodology to formal verification and program testing • validate properties on the current execution VU '02

8. Outline • Overview of the MaC architecture • The MaC languages • Java-MaC: the MaC prototype for Java programs • Example: stock client • Conclusion and future work VU '02

9. Human Automatic Instrumentation Automatic Translation Automatic Translation Program Filter Event Recognizer Run-time Checker Overview of the MaC Architecture Program Informal Requirement Spec Input Formal Requirement Spec Low-level Specification High-level Specification Static Phase low-level behavior high-level behavior Run-time Phase VU '02

10. Design of the MaC Languages • Must be able to reason about both time instants and information that holds for a duration of time in a program execution. • Events and conditions are a natural division, which is also found in other formalisms such as SCR. • Conditions, which are true or false for a finite duration of time (e.g., is variable x >5?) • Events, which are either present or absent at some instant of time (e.g., is the control right now at the end of method f?). • Need temporal operators combining events and conditions in order to reason about traces. raiseGate start(position==100) end(position==100) position == 100 1:00:10 1:00:15 1:00:30 Time VU '02

11. Logical Foundation • Conditions interpreted over 3 values: true, false and undefined. • [., .) pairs a couple of events to define an interval. • start and end define the events corresponding to the instant when conditions change their value. VU '02

12. The MaC Languages • Meta Event Definition Language(MEDL) • Describes the safety requirements of the system, in terms of conditions that must always be true, and alarms (events) that must never be raised. • Target program implementation independent. • Primitive Event Definition Language (PEDL) • Maps the low-level state information of the system to high-level events. • Provides primitives to refer to values of variables and to certain points in the execution of the program. • PEDL is defined so that events can be recognized in time linear to the size of the PEDL specification • Depends on target program implementation VU '02

13. Meta Event Definition Language (MEDL) • Expresses requirements using the events and conditions, sent by event recognizer. • Expresses the subset of safety properties. • Describes the safety requirements of the system, in terms of conditions that must always be true, and alarms (events) that must never be raised. • propertysafeRRC = IC -> GD; • alarm violation = start (!safeRRC); • Auxilliary variables may be used to store history. • endIC-> { num_train_pass’ = num_train_pass + 1; } ReqSpec <spec_name> /* Import section */ import event <e>; import condition <c>; /*Auxiliary variable */ var int <aux_v>; /*Event and condition */ event <e> = ...; condition <c>= ...; /*Property and violation */ property <c> = ...; alarm <e> = ...; /*Auxiliary variable update*/ <e> -> { <aux_v'> := ... ; } End VU '02

14. Java-MaC Prototype Requirement Specification Program (Java source code) Requirements (MEDL) Monitoring Script (PEDL) Program (Java byte code) MEDL Compiler PEDL Compiler Instrumentation Information Filter Generator (JTREK) Instrumented Code Event Recognizer Run-time Checker VU '02

15. PEDL for Java MonScr <spec_name> /* Export section */ export event <e>; export condition <c>; /* Overhead reduction */ [timestamp;] [valueabstract;] [deltaabstract;] [multithread;] /* Monitored entities */ monobj <var>; monmeth <meth>; /* Event and condition*/ event <e> = ...; condition <c>= ...; End • The language maps the low-level state information of the system to high-levelevents and conditions used in describing the requirements. • Provides primitives to refer to • primitive variables • beginnings/endings of methods • Primitive conditions are constructed from • boolean-valued expressions over the monitored variables • Ex: condition IC = (position == 100); • Primitive events are constructed from • update(x) • startM(f)/endM(f) • Ex: event raiseGate= startM(Gate.gu()); VU '02

16. PEDL for Java (cont.) • Events can have two attributes - time and value • time(e) gives the time of the last occurrence of event e • used for expressing temporal properties • value(e,i) gives the ith value in the tuple of values of e • value of update(var) : a tuple containing the current value of var • value of startM(f) : a tuple containing parameters of the method f • value of endM(f) : a tuple containing parameters and a return value of the method f VU '02

17. Monitoring Objects • Java handles an object via a reference pointing to the object • Object aliasing: a monitored object can be updated by several references • all references (ex: a.b1.b’, a.b2) which possibly point to the monitored object need to be tested whether they are actually pointing to monitored objects at run-time. • this testing may not be feasible due to Java scoping rules • Ex: b2 is declared as private in the class A. We cannot test whether a.b1 == a.b2 outside of class A • We need a globally visible table containing the addresses of monitored objects VU '02

18. Address Var Name 8200 a.b2 Address Table Monitoring Objects (cont.) • Address table contains addresses of monitored objects and monitored object names to test references • Ex: Suppose that we monitor a.b2 and the address of a.b2 is 8200. If a.b1.b’ is 8400, we know that a.b1.b’ is not pointing to the monitored variable • Java-MaC assumes that the reference of the monitored variable do not change • The update of the address table can cause too much overhead • one reference change can cause the large substitution of all descendent nodes already in the address table VU '02

19. Instrumentation • Java-MaC instruments Java executable code • Java-MaC instrumentor detects instructions • variable updates • putstatic/putfield for global variable updates • <T>store and iinc for local variable updates • execution points • instruction located at the beginning of method definition • return of method definition • At the each detected instruction, Java-MaC instrumentor inserts a probe invoking • sendObjMethod(Object parentAddress, <T> value, String varName) VU '02

20. Overview of Java-MaC VU '02

21. Run-time Components of Java-MaC (1) • Filter • A filter consists of • a communication channel • probes inserted into the target system • a filterthread which flushes the communication buffers to the event recognizer • Event recognizer • evaluates the abstract syntax tree generated from a PEDL specification whenever it receives snapshots from the filter. • If an event or a condition changing its value is detected, the event recognizer sends the event or the condition to the run-time checker. VU '02

22. Run-time Components of Java-MaC (2) • Run-time checker • evaluates the abstract syntax tree generated from a MEDL specification whenever it receives events and conditions from the event recognizer. • Detects a violation defined as alarm or property and raises a signal. • Connection among run-time components • TCP socket connection • FIFO file connection • User implemented connection using InputStream and OutputStream obtained by Java-MaC API VU '02

23. Specifications for Stock Clients MonScr StockClient export event startPgm, periodStart, conFail, queryResend, oldDataUsed; monmethvoid Client.main(String[]); monmeth void Client.run(); monmeth void Client.failConnection(ConnectTry); monmeth Object Client.retryGetData(int); monmeth Object Client.processOldData(); event startPgm = startM(Client.main(String[])); event periodStart = startM(Client.run()); event conFail = startM(Client.failConnection(ConnectTry)); event queryResend = startM(Client.retryGetData(int)); event oldDataUsed = startM(Client.processOldData()); End ReqSpec StockClient import event startPgm, periodStart, conFail, queryResend, oldDataUsed; var long periodTime; var long lastPeriodStart; var int numRetried; var int numConFail; alarm violatedPeriod = end((perioidTime’ >= 900) && (periodTime’ <= 1100)); alarm wrongFT = oldDataUsed when ( (numRetries’ < 4)|| (numConFail’ < 3)); startPgm -> {periodTime’ = 1000; lastPeriodStart’ = time(startPgm) -1000; numRetries’ = 0; numConFail’ = 0;} periodStart ->{ numREtries’ = 0; numConFail’ = 0; periodTime’ =time(periodStart)-lastPeriodStart; lastPeriodStart’ = time(periodStart);} ... End VU '02

24. Conclusion and Future Work • The MaC architecture provides a lightweight formal methodology for assuring of the correct execution of a target program at run-time • Rigorous analysis, Flexible, Automatic • Reduction of monitoring overhead • Systematic extension of the MaC architecture to platforms other than Java • http://www.cis.upenn.edu/~rtg/mac VU '02

25. The MaCS (MaC with Steering) system VU '02

26. A Temporal Logic Based Theory of Test Coverage and Generation Hyoung S. Hong1,3, Insup Lee1, Oleg Sokolsky1,Hasan Ural2 • 1SDRL (Systems Design Research Lab) • RTG (Real-Time Systems Group) • Department of Computer and Information Science • University of Pennsylvania, Philadelphia, PA 2School of Information Technology and Engineering University of Ottawa, Ottawa, Canada 3 Department of Computer Science KAIST, Korea VU '02

27. Specification-Based Testing • Determines whether an implementation conforms to its specification • Hardware and protocol conformance testing • Widely-used specifications • Finite state machines and labeled transition systems • Two main steps • Test generation from specifications • What to test, how to generate test • Test execution of implementations • Applies tests to implementations and validates the observed behaviors VU '02

28. Overall Structure Specification Test Suite Test Generation input Implementation Test Execution output Test Output Test Evaluator VU '02

29. Input to model checker Specification A set of tests Model checker Coverage criterion A set of formulas Our Approach • The problem: automatic test generation from specifications • Specifications: deterministic EFSMs • Coverage criteria • Control flow: all-states, all-transitions • Data flow: all-defs, all-uses, all-inputs, all-outputs • Test generation VU '02

30. Outline • EFSM • Testing coverage criteria in WCTL • Test generation using model checker • Test suite optimization problems • Current and future work VU '02

31. t1: insert[m+x<=5] /m:=m+x t2: coffee[m>1] /m:=m-1 IDLE BUSY t3: done t5: display/y:=m t4: display/y:=m Specifications: EFSM • S: a set of states • S0: initial state • E: a set of input/output events • V: variables are manipulated by transitions • T: a set of transitions VU '02

32. Coverage criteria in WCTL • Each coverage criterion is represented by a set of temporal logic formulas • WCTL: a subset of CTL • Atomic propositions p1,…,pn • Temporal operators EX, EU, EF • Conjunctions have at most one non-atomic conjuncts • Negations can be applied only to atomic propositions • Unrestricted disjunctions • E.g.: EF(p1 & EFp2) • WCTL formulas have linear witnesses VU '02

33. t1: insert[m+x<=5] /m:=m+x EF(idle & EFexit) EF(busy & EFexit) t2: coffee[m>1] /m:=m-1 IDLE IDLE BUSY BUSY t3: done t5: display/y:=m t4: display/y:=m All-states coverage criterion • Requires every state be covered at least once • With every state s, associate EF(s & EFexit) VU '02

34. t1: insert[m+x<=5] /m:=m+x t2: coffee[m>1] /m:=m-1 t3: done t5: display/y:=m t4: display/y:=m EF(t1 & EFexit) EF(t2 & EFexit) EF(t3 & EFexit) EF(t4 & EFexit) EF(t5 & EFexit) t1: insert[m+x<=5] /m:=m+x t2: coffee[m>1] /m:=m-1 IDLE BUSY t3: done t5: display/y:=m t4: display/y:=m All-transitions coverage criterion • Requires every transition be covered at least once • With every transition t, associate EF(t & EFexit) VU '02

35. Data flow: definitions and uses • Central notions in the data-flow analysis • Definition: a value is assigned to a variable • Use: a value of a variable is used in an expression • Variables are defined and used in transitions • Definition-use pair: (v,t,t’) • v is defined by t • v is used by t’ • There is a path from t to t’ free from other definitions of v VU '02

36. t1: insert[m+x<=5] /m:=m+x t2: coffee[m>1] /m:=m-1 t1: insert[m+x<=5] /m:=m+x EF(t1 & EXE[!(t1 | t2) U (t2 & EFexit)]) t2: coffee[m>1] /m:=m-1 IDLE BUSY t3: done t5: display/y:=m t4: display/y:=m Covering a definition-use pair • With a definition-use pair (v, t, t’), associate • EF(t & EXE[!def(v) U (t’ & EFexit)]) • def(v) : disjunction of all transitions that define v VU '02

37. t1: insert[m+x<=5] /m:=m+x t2: coffee[m>1] /m:=m-1 IDLE BUSY t3: done t5: display/y:=m t4: display/y:=m Data-flow coverage criteria • All-defs coverage criterion • Requires a definition-clear path from every definition to some use be covered at least once • All-uses coverage criterion • Requires a definition-clear path from every definition to every use be covered at least once All-uses coverage criterion EF(t1 & EXE[!(t1 | t2) U (t1 & EFexit)]) EF(t1 & EXE[!(t1 | t2) U (t2 & EFexit)]) EF(t1 & EXE[!(t1 | t2) U (t4 & EFexit)]) EF(t1 & EXE[!(t1 | t2) U (t5 & EFexit)]) EF(t2 & EXE[!(t1 | t2) U (t1 & EFexit)]) EF(t2 & EXE[!(t1 | t2) U (t2 & EFexit)]) EF(t2 & EXE[!(t1 | t2) U (t4 & EFexit)]) EF(t2 & EXE[!(t1 | t2) U (t5 & EFexit)]) VU '02

38. System model True or false Model checker Logic formula Witness or counterexample System model Model checker Coverage criterion A set of logic formulas A set of witnesses Test Generation VU '02

39. E[ U ] Witness Generation • Generating a witness for a formula • Cost: the length of a witness • A minimal-cost witness for a formula • Existing model checkers generate a minimal-cost witness by breadth-first search of state space VU '02

40. E[ U ] E[ U ] E[ U ] E[ U ] Test Generation • A set of witnesses for a set of formulas • Costs • The total length of witnesses or • The number of witnesses (reset operation is expensive) • Both optimization problems are NP-hard (Hitting Set Problem) VU '02

41. Test Generation • Heuristics • Generate a witness for each formula and remove redundant witnesses • Use model checker to construct witnesses • Generate all witnesses for each formula and intersect them • Need to extend model checker to construct all witnesses for a given formula • A greedy algorithm described in the paper. VU '02

42. Related Work • J. Callahan, F. Schneider, S. Easterbrook, “Specification-based testing using model checking”, 1996. • D. Geist, et al., “Coverage-directed test generation using symbolic techniques,” 1996. • A. Gargantini and C. Heitmeyer, “Using model checking to generate tests from requirements specifications,” 1999. • Many more … (see the paper) VU '02

43. Conclusions • Test generation from specifications using model checking • Basic idea • Express a test coverage criterion as a set of CTL formulas • Generate a set of witnesses for the formula set using a model checker • Advantages • Language-independent • Can be applied to other EFSM-based specifications • Executable • Only executable tests are generated • Limitations • Finite state specifications • Deterministic specifications VU '02

44. Current Research • Test generation from formal specifications • Reactive systems • Extended finite state machines (EFSMs) • FSM + data variables • Hierarchical reactive modules • EFSM + hierarchy + concurrency • Hybrid systems • CHARON • EFSM + hierarchy + concurrency + differential equations • Effectiveness evaluation & tool support • Case studies: • electronic throttle controller, infusion pump system VU '02

45. Specifications Abstract interpretation, Predicate abstraction, …… Fully syntactic Fully semantic Flow graphs Reachability graphs Abstraction graphs Challenge Problem • Understanding test coverage criteria as abstraction VU '02

46. Testing Monitoring Test Generation Instrumentation Running System/ Filter Test Suites Event Recognizer Testing Abstract Events Checker Test Results Checking Output Prototyping Evaluator Abstraction/ Reengineering Evaluation Report Analysis: - model checking - equiv. checking Rapid Prototyping/ Simulation Diagnostic Requirements Artifacts Informal Requirements Requirements Engineering Formal Requirements System Artifacts Informal Design Diagrams (UML) Verification Prototype Formal Specification Implementation VU '02

47. Q & A VU '02

48. Properties of embedded systems • Adherence to safety-critical properties • Meeting timing constraints • Satisfaction of resource constraints • Confinement of resource accesses • Supporting fault tolerance • Domain specific requirements • Mobility • Software configuration VU '02