Model-Checking in the Early Lifecycle

1. Model-Checking in the Early Lifecycle Zoë Stephenson

2. Acknowledgements • The work presented here was conducted under the MOSAIC project, project number TP/3/DSM/6/I/15780, a collaboration between the University of York, the University of Sheffield, Rolls-Royce plc., Goodrich Engine Control Systems and Jaguar Cars Limited. • This project is co-funded by the Technology Strategy Board's Collaborative Research and Development programme, following an open competition. • The Technology Strategy Board is an executive body established by the Government to drive innovation. It promotes and invests in research, development and the exploitation of science, technology and new ideas for the benefit of business --- increasing sustainable economic growth in the UK and improving quality of life.

3. V&V activities roughly 50% Motivation Proprietary industrial data, 2000

4. Motivation • Reducing V&V effort should be very effective in reducing project cost • Especially by addressing issues earlier in the lifecycle • We use model-checking to address V&V issues • Questions from industrial parties: • Is it applicable to our systems? • Does it scale? • How much effort does it save?

5. Outline • Early lifecycle characteristics • Eliciting individual properties • Demonstrating applicability

6. santi/MOCpages.com the LEGO Company the MathWorks LTSA Early vs. Late

7. Early Lifecycle Few stable requirements Some experience of previous systems Assessment targeted at technical risks – novelty and concurrent design Late lifecycle Complete requirements Full system available to evaluate Assessment targeted at properties of complete system Early vs. Late

8. Property Elicitation • Early lifecycle issues are: • Individual aspects • Not necessarily captured in requirements • Where do they come from? • Which ones are important? • How do we know we’ve found all the important ones? • How should we write them down?

9. Current draft requirements Sketches Documentation and standards Existing systems Domain experts Technical risk assessment Project priorities Properties that the system ought to satisfy Property Source

10. Technology-Independence • System needs to satisfy properties regardless of design, implementation, V&V technologies • Different properties will be amenable to different types of V&V technology • Properties will need to be translated into various different forms • At the point of elicitation: • Those forms are not guaranteed to be known • Those participating in elicitation may not be comfortable with those forms

11. Bounds elicitationand helps to focusdiscussion Should ideally be concise and precise Explicit Elicitation Process

12. Elicitation Process • Facilitated discussion • Can use a separate scribe and chair • Our experiments combined these roles • Series of prompts: • stimulate discussion • explore different kinds of behaviour • start with general issues and lead on to specific issues • e.g. from “expectations” to specific invariants

13. Typical Prompt Areas • Boundary properties • “is that all that the system needs to do?” • Completeness • “Did we cover every value/every state?” • Dynamic completeness • “…every value/state change?” • Feasibility • “…every high-risk area?” • Interaction • “…every sequence of interactions?” • Expectation • “Does it behave reasonably?”

14. Pilot Studies • Fixed-priority scheduler implementation • 23 properties elicited • Around 15 checkable by model-checking • Model-checking study ongoing • New implementation of surface friction estimator • 12 properties, 6 checkable • Small model-checking study conducted

15. Observations • Elicited properties not always easily checkable in a model-checker • “Detection time plus adjustment time to correct for the change in surface friction ought to be within 100ms.” • Elicited properties not always easily checkable in early lifecycle • “The error in estimation for lateral friction is approximately the same across the whole friction range” • Skills of a property elicitation specialist different to those of a domain expert • “Strategic pedantry”

16. Studies Performed

17. Requirement Underlying Error extract Test Case Throttle Resolver Study • Based on three problem reports

18. Study Aims • Given latest system specification, error definition, requirement and test case: • Produce a “damaged” version of the system that contains the error • Express the requirement as a property to check • Model-check the latest system and the damaged system against the property • Current system should satisfy property • Damaged system should violate property • Generated counterexample should be similar to test case

19. Requirements

20. endeec: do :: sync_eec?_; atomic { /* account for bus fault */ mtpafault1 = (tpafault1 || busfault); mtpafault2 = (tpafault2 || busfault); mtpafault3 = (tpafault3 || busfault); ntpafault = mtpafault1 + mtpafault2 + mtpafault3; /* Detect movement */ /* BEGIN TRACE 24 CSS-197-149-2.3 */ /* movTPA1 = | otpa1_u - tpa1_u | > tpamovt */ if :: otpa1_u == tpa1_u -> movTPA1 = 0 :: otpa1_u - tpa1_u - 1 >= tpamovt -> movTPA1 = 1 by hand endeec: do :: sync_eec?_; atomic { /* account for bus fault */ mtpafault1 = (tpafault1 || busfault); mtpafault2 = (tpafault2 || busfault); mtpafault3 = (tpafault3 || busfault); ntpafault = mtpafault1 + mtpafault2 + mtpafault3; /* Detect movement */ /* BEGIN TRACE 24 CSS-197-149-2.3 */ /* movTPA1 = | otpa1_u - tpa1_u | > tpamovt */ if :: otpa1_u == tpa1_u -> movTPA1 = 0 :: otpa1_u - tpa1_u - 1 >= tpamovt -> movTPA1 = 1 assert assert damage check visualise Counterexample Study Setup for spin Latest Specification endeec: do :: sync_eec?_; atomic { /* account for bus fault */ mtpafault1 = (tpafault1 || busfault); mtpafault2 = (tpafault2 || busfault); mtpafault3 = (tpafault3 || busfault); ntpafault = mtpafault1 + mtpafault2 + mtpafault3; /* Detect movement */ /* BEGIN TRACE 24 CSS-197-149-2.3 */ /* movTPA1 = | otpa1_u - tpa1_u | > tpamovt */ if :: otpa1_u == tpa1_u -> movTPA1 = 0 :: otpa1_u - tpa1_u - 1 >= tpamovt -> movTPA1 = 1 Requirement Underlying Error Test Case compare

21. Results • All properties satisfied by current model • All properties give counterexamples in damaged models • All counterexamples comparable with actual test-cases used

22. Effort Profile

23. Simulink Design Verifier • Extension to “Verification and Validation” toolbox • Integrates Prover Technology “Prover plug-in” • Adds a new library • Contains blocks that you use to specify assumptions and properties • Adds a new Tools sub-menu • Check model compatibility • Generate tests (coverage) • Prove properties • Options

24. Simulink Model

25. extract assert assert damage check Counterexample Study Setup for SDV Latest Specification Requirement Underlying Error Test Case compare

26. Results • All properties satisfied by current model • All properties give counterexamples in damaged models • All counterexamples comparable with actual test-cases used

27. Effort Profile

28. spin: Manual translation No model extraction required Test cases can be complex Test cases in terms of Promela, not original specification SDV: Automatic translation Model extraction required Test cases relatively simple Test cases in terms of original specification Observations

29. Results from Other Studies • Active roll control • Examined for mode conflict • Much easier in spin than SDV • Surface friction estimation • Very difficult to check due to arithmetic involved • Fixed-priority scheduler • Challenging to model preemption accurately in Promela • Discovered one interrupt-masking issue that we fed back into development cycle

30. Summary • Used property elicitation to determine most relevant properties in the early lifecycle • Investigated model-checking tools using case-studies from industrial parties: • Applicable to some parts of examined systems • Submodel extraction needed to address scalability • Effort profile results to be fed into cost-benefit analysis task next year