Forrest Shull: Fraunhofer CESE, USA Marcus Ciolkowski: Fraunhofer IESE, Germany

1. “Balancing” Practices Forrest Shull: Fraunhofer CESE, USA Marcus Ciolkowski: Fraunhofer IESE, Germany Tony Cowling, University of Sheffield Masa Katahira, Japanese Space Agency Lesley Land, University of New South Wales Jose’ Maldonado, University of São Paulo at São Carlos Bernard Wong, University of Technology Sydney

2. Session Goals • We hope to: • Collect & organize the available knowledge and experience of ISERN participants • Identify directions for future collaboration and research among members • Suggest some necessary empirical studies!

3. Motivation • An important open research question: • Given finite resources (for construction and analysis) • And desired end-product qualities (maintenance, safety, security, …) • What combinations of QA techniques can be recommended with high confidence to achieve those desired qualities?

4. The Research Goal(Cartoon Version) • Current model: • [Insert your favorite QA technique] catches many important defects… Therefore, use it! • Economic analysis says that using it earlier is better than later. • Desired model: Would be based on heuristics such as [hypothetical examples!]: • Security issues should be caught when the initial requirements are being specified, but they require sophisticated modeling. • Safety issues are not reflected at all in design diagrams. • Important performance issues can be avoided during design, but you can’t automate the checking. • Reliability requires code to undergo a combination of static and dynamic checking.

5. This session’s focus How does empiricism support decision support? • “Folklore”: Belief based on experience • Testable hypotheses • Empirically-tested knowledge

6. QA techniques:Inspections • Inspections / Reviews • Includes any kind of human-based, static checking from • Formal Fagan inspections to • Walkthroughs • Deliberately exclude less formal approaches such as desk checking, peer review, etc. • May include some kind of reading or checklist-based checking

7. QA Techniques:Testing • Basic Testing Methods • Base the generation of test cases on one of these models • Black-box testing uses the specification model • White-box testing uses the implementation model • Hybrid Testing Methods • Combine the approaches – eg: • Black-box methods to generate the test sets • White-box methods to measure their coverage • May provide more effective testing than individual basic methods • At least, according to some papers • State-based Testing • Uses state-machine models for specification and implementation • Extended models (eg the X-machine) allow powerful results: • absence of faults up to some bounds, • under some assumptions, complete absence of faults

8. QA Techniques:Formal Methods • Model Checking • Requires state-based specification models • Shows whether required properties hold for the models • Can handle very large systems (10^20 states) • Machine Model Verification • Uses state-based specification and implementation models (eg B) • Can verify that implementation is consistent with specification • Refinement • Typically uses relational models (eg Z, VDM) • Refinement steps produce correct-by-construction implementations • Discontinuities in the models need to be accommodated • Retrenchment has been proposed for this

9. Agenda • Today’s question: • For different business scenarios, how much and how can • Inspections • Formal methods • Testing • …make a contribution? • Does a group of informed individuals agree on the general contributions of each approach? • Can we develop guidelines for situations in which one or the other is more appropriate? • Can we summarize what has already been empirically validated?

10. Agenda • Today’s agenda: • Introduction (this talk) • A concrete scenario and QA techniques for brainstorming • Brainstorming: 2 groups • What QA techniques would you recommend to use for the scenario, and why? • What would you need to know? • What are the advantages and disadvantages of each tech? • Consolidating: • Each group summarizes their answers (1-2 slides) • An example of a solution to “balancing” techniques at Japanese Space Agency • Wrap-up • Did participants agree with each other? With JAXA? • Can we start to formulate some heuristics? • If not, what do we need to do so? What to measure?

11. What QA techniques would you recommend to use for the scenario, and why? • What would you need to know? • What are the advantages and disadvantages of each tech? • Need to know: • What is at risk? Manned or not? Nuclear powered? Or just data? • Safety-critical: Higher risk = higher justified cost • Need to segment by which functions are the most risky • Do I have the right info in the process? E.g. formal methods require: • Detailed design document (highly classified) • Are the documents even produced in the development process? Can be reverse engineered from source code?

12. Successful mission = good business case • Many formal methods require that they be integrated with the process – can’t be discussed independently of development lifecycle • Assuming formal methods – role for isnpection? • Better inspect specs • Prioritize the top 3 risks • Risks identified require many different alternative mitigations to be defined – no clear reln. Between risks and QA techniques • Strawman: • For the safety-critical functions – assume I apply formal methods upfront – have to apply inspections to get immediate feedback about defect-proneness – would still do testing in the end • Formal methods does not include a representation of what actually could go wrong in operation. • Evidence of use? NASA doesn’t use formal methods extensively… Do they simply not know the benefits? (Simulation approaches instead – more in line with testing under various circumstances). Hypothesis: Formal models are good when you have a good sense of the likely conditions of use. • Need to test the formal tools – put bugs into model checkers

13. Evidence from JAXA: For at least some defects inspection and formal methods find the same defects. • Anecdotal evidence: Inspection – requires reading line-by-line – more time consuming but finds more defects than formal methods • Defect taxonomies such as ODC can help identify which QA techniques are helpful for which subclasses • Fault detection and recovery • Determinant is how much rework of the development process is allowed • Parastoo & Conradi: Inspections capture more serious defects IF you have access to design and modeling guidelines about how software is developed

14. Summary • Risk is a central determinant • However path from identifying risks to choosing QA techniques is not straightforward • Need to

15. Scenario:JAXA spacecraft on-board software • JAXA develops the software to be embedded in the spacecraft such as satellites, launch vehicles, and a space station to explore the space and create a new business in the space. • A goal involves successful missions without any accidents and keeping crews in the ship/people on the ground and any propriety safe (without loss), then getting new customers in space business. • Currently the functionality to upload modified software to the spacecraft from the ground is installed. • There are safety-(time-)critical software functions that must not fail such as a critical operational scenario to transition into the orbit to the moon, the rendezvous docking to the space station, and FDIR (Failure Detection Isolation Recovery). • Other important concerns: • Schedule and budget limitation are important concerns. • Large number of defects are introduced mostly at the Requirement Phase or the Early Design Phase.

16. BACKUP

17. Scenario 1:XYZ cell phone software • XYZ company creates software to be embedded in the cell phones that they sell. A central business goal involves getting innovations to market as quickly as possible, to capture new customers and build a customer base. • At the same time a high level of reliability in the software is important so that customers have a pleasant experience – and remain loyal customers. • Other notes: • Security and safety are not particularly emphasized in this context. • Maintainability is important – of course new software is often produced based on previous versions of the model – but in the embedded environment performance issues are more important, if there is a conflict.

18. Scenario 3:Science code for ABC Satellite • A coding team at a space agency is developing code to control a scientific instrument that will be included on a new satellite. • Since this instrument is of general usefulness in astronomy research, the team hopes to leverage the code on many future missions. Therefore a primary goal is to make it easily parameterizable and highly maintainable. • The software should be well documented – Future mission scientists need to understand if the software will be useful to them. And, it may not always be the current team of developers that adapts the software for new satellites. • Cost and schedule are tight but not overly so. • Accuracy of calculations is of the highest importance. Without accurate measurements, the scientific value of the entire satellite mission is degraded.

19. “Brainstorming Rules” • Consider one of the business scenarios • Take a card • Write down: • What QA technique you think should be applied • Formal methods, testing, inspection… • In a few words, how should it be applied? • (In what phase? Targeting what issues?) • In a few words, why this would help? • Is this based on • STUDY RESULTS or • FOLKLORE (heard from customers, read about in papers)? • Stick the card on the appropriate poster • Repeat until time runs out

20. “Working Group Rules” • Construct a matrix: e.g. • Discuss which are the most important points – the key variables • Try to combine similar ideas • Highlight which entries are empirically-based and which are speculative • The team can decide to eliminate entries that are “excessively speculative”… • Color code to emphasize areas that need future research