Verification – Setting the Stage

1. Verification – Setting the Stage Presented by Doron Drusinsky Joint work with Bret Michael, Man-Tak Shing, and Tom Otani Naval Postgraduate School Monterey, CA NASA IV&V Facility Workshop on Verification Morgantown, WV

2. Validation – getting the right product Verification – getting the product right Verification – Setting the Stage A. Validation overview Role of assertions Sources of assertions Purpose of validation What can be wrong with my assertion? Validation testing patterns SRM and its role B. Verification – discussion NASA IV&V Facility Workshop on Verification Morgantown, WV

3. Assertion vs. implementation: its not about the language (e.g. LTL vs Java) Example: Implementation: Role of Correctness Property Assertions NASA IV&V Facility Workshop on Verification Morgantown, WV

4. Assertions vs. implementation: its not about the language (e.g. LTL vs Java) Assertions: Role of Correctness Property Assertions NASA IV&V Facility Workshop on Verification Morgantown, WV

5. f … output(s) inputs Time Our focus: assertion for reactive (sub)systems – Assertions for sequencing, time-constraints, and time-series constraints Role of Correctness Property Assertions Counter example: the accuracy of (fixed-point function) f should be 0.01 Transformational system: in out – done! NASA IV&V Facility Workshop on Verification Morgantown, WV

6. R … output(s) inputs Time Our focus: assertion for reactive (sub)systems – Assertions for sequencing, time-constraints, and time-series constraints Role of Correctness Property Assertions Example: when cruise control is active speed should be 98% stable Reactive system: Never done … NASA IV&V Facility Workshop on Verification Morgantown, WV

7. Test-vector generation (e.g., T-VEC) == for transformational components Verification of Transformational Systems Assertion (e.g. Java / C Assertion/condition) ?? T-VEC – for functions / transformational sub-systems OR? Transformational component inputs Expected output NASA IV&V Facility Workshop on Verification Morgantown, WV

8. Assertion vs. implementation: a Venn diagram view Role of Correctness Property Assertions NASA IV&V Facility Workshop on Verification Morgantown, WV

9. Role of Correctness Property Assertions NASA IV&V Facility Workshop on Verification Morgantown, WV

10. An assertion is not an implementation: Role of Correctness Property Assertions NASA IV&V Facility Workshop on Verification Morgantown, WV

11. 1. Specification documents (NL) – if created by contractor they should be treated with a grain of salt. Sources of Requirements for Assertions Req.in NL Issues: 1. Does contractor have other interest that play a role defining requirements? 2. Sequencing requirements are often not considered by spec. writers NASA IV&V Facility Workshop on Verification Morgantown, WV

12. 2. Sequencing requirements are often not considered by spec. writers Sources of Requirements for Assertions • Sequencing/temporal considerations: • Can a track toggle between DataStores? DS1DS2DS1DS2… If so, how often? • If toggling is expected then is there an allowed period of overlap (e.g., being in both DS1 and DS2?) • How soon should thread be published in DS? NASA IV&V Facility Workshop on Verification Morgantown, WV

13. 2. From Activity-Diagrams/MSC’s to requirements based on concerns. See SoSE-2008 paper Sources of Requirements for Assertions Req. in NL NASA IV&V Facility Workshop on Verification Morgantown, WV

14. To assure the representative assertion is a good representative of this and/or this The Purpose of Validation NASA IV&V Facility Workshop on Verification Morgantown, WV

15. No…, we need to validate their correctness w.r.t to cognitive and/or NL intent • Ambiguities in cognitive or NL specification • Insufficient detail in specification • Bugs in assertion/formal-specification • Pilot errors: errors in test-case Aren’t Assertions Correct by Construction? NASA IV&V Facility Workshop on Verification Morgantown, WV

16. $50 Black Red Ambiguous definition for the beginning time of the one year constraint 2 weeks • Ambiguities in cognitive or NL specification An overdraft account should, within 2 weeks, gain a balance of $50 or more and stay black for a whole year thereafter Issues Validation can Discover/Resolve NASA IV&V Facility Workshop on Verification Morgantown, WV

17. Ambiguities in cognitive or NL specification (cont.) An overdraft account should, within 2 weeks, gain a balance of $50 or more and stay black for a whole year thereafter Issues Validation can Discover/Resolve $50 Black Red If this is the definition of the beginning of one year stability, then is the account allowed to be red more than once within the two week period? 2 weeks NASA IV&V Facility Workshop on Verification Morgantown, WV

18. Ambiguities in cognitive or NL specification • Event P must never occur between events Q and R. • Is Q.R.P.Q.Rok? Issues Validation can Discover/Resolve (A traffic-light) must show yellow between a green and a red. Must it also show yellow when going red  green? NASA IV&V Facility Workshop on Verification Morgantown, WV

19. 2. Insufficient detail in specification Issues Validation can Discover/Resolve Every sensor log-request must be acknowledged within 30 seconds. Ignores that fact that a sensor has multiple log-requests, does each need a unique acknowledgement? NASA IV&V Facility Workshop on Verification Morgantown, WV

20. 3. Bugs in assertion/formal-specification Whenever event p occurs then event q must occur within 30 seconds Issues Validation can Discover/Resolve RIGHT NASA IV&V Facility Workshop on Verification Morgantown, WV

21. p p p 30 TestR1b_4: r1b.p(); r1b.incrTime(2); r1b.p(); r1b.incrTime(11); r1b.p(); r1b.incrTime(11); assertFalse(r1b.isSuccess()); 4. Pilot errors: errors in driving validation test-case Whenever event p occurs then event q must occur within 30 seconds Issues Validation can Discover/Resolve Expecting a failure but without waiting full 30 second period NASA IV&V Facility Workshop on Verification Morgantown, WV

22. p q q q q q q 30 30 Whenever event p occurs then event q must occur within 30 seconds Validation patterns: Validation Patterns p p p p 30 See SSIRI 2008 paper p p p 30 p p p 30 30 NASA IV&V Facility Workshop on Verification Morgantown, WV

23. Validation Coverage Can use animation or some batch/command-line technique States that were visited and transitions that were traversed by the validation suite are green i.e., validation didn’t test for timeout first, then publish NASA IV&V Facility Workshop on Verification Morgantown, WV

24. SRM in Eclipse Role of SRM ExecutableSRM NASA IV&V Facility Workshop on Verification Morgantown, WV

25. Executable SRM = Domain Model + Assertion Repository + Validation test suite + bridge Role of SRM NASA IV&V Facility Workshop on Verification Morgantown, WV

26. Using Code Coverage to find Missing Assertions NASA IV&V Facility Workshop on Verification Morgantown, WV

27. Computer-based name space sanity checking • Some requirements may depend on data model • Identifying missing assertions Role of SRM NASA IV&V Facility Workshop on Verification Morgantown, WV

28. Computer-based name space sanity checking Role of SRM class Library { … /** * verb/method/event */ publicvoidreqPublish() { // connect to External assertions bundle fireExternalAssertions("reqPublish"); } } NASA IV&V Facility Workshop on Verification Morgantown, WV

29. 2. Some requirements may depend on data model Role of SRM NASA IV&V Facility Workshop on Verification Morgantown, WV

30. 3. Identifying missing assertions  see technique presented in validation workshop Role of SRM NASA IV&V Facility Workshop on Verification Morgantown, WV

31. So now we have assertions … • Benefits: • Finding specification errors/issues early on. • For computer-aided verification. • Lets use them for verification … Verification NASA IV&V Facility Workshop on Verification Morgantown, WV

32. Verification NASA IV&V Facility Workshop on Verification Morgantown, WV

33. Possible testing architecture T-VEC for transformational behavior Verification – Lets have a discussion Assertion repository (RV) (including simple assertions for transformational systems) 3. Output events and data isSuccess System Under Test (SUT) 3. Clock tick 1.Use observations from repository? Manual + Automatic Test Generator (ATG) 2. Generated events 2b. Generated data Physical world data (e.g. from Matlab) 2a. Gen data command NASA IV&V Facility Workshop on Verification Morgantown, WV