Cleanroom Software Engineering

1. Cleanroom Software Engineering By Derek B. Larson

2. Cleanroom Software Engineering • What is Cleanroom Software Engineering? • Cleanroom Process • Waterfall Model into a Cleanroom • Case Studies

3. Introduction • Developed IBM • The stated goal of Cleanroom is software that exhibit a zero defect rate. • mathematical and statistical methods • IBM developed a device controller product using Cleanroom Software Engineering that had zero failures in three years used at 300+ locations. [1]

4. Intro Cont. Hardware cleanrooms keep problems out by keeping potential contaminating factors from reaching the product.

5. Why Cleanroom SE • The reason to use Cleanroom Software Engineering is simple: quality • Quality improvements of 10 to 20 times have been reported when the Cleanroom process was demonstrated in industry • If defects can cause loss of life or critical financial loss • Increases in productivity

6. Why Cont. • Increases in productivity? • Cleanroom is a uniquely defined process that decreases required testing, error correction, and maintenance. • These savings offset any additional overhead needed for the quality control • “Errorless Software Development”. If no errors enter development, no errors need to be tested or debugged. Elimination of testing and debugging means faster product development.

7. Reqs for Cleanroom SE • Most spend a lot of time and money at the start of the project for preventing defects. • Most use statistical methods to ensure quality. • Most formally state and “prove” requirement needs. (YEAH ‘Z’) [2]

8. Cleanroom Functions • Cleanroom uses three types of functions • All code that is developed will follow the basic structure of these functions. • Because the functions are sound, the code will likewise be sound. • These functions are called “Boxes” [1]

9. The Three Boxes • Black Box • State Box • Clear Box

10. Black Box • Black box is a view of an object that hides the implementation process and data . • By modeling code as a series of black boxes, we can ensure its quality verses our specification by ensuring that the actual black box performs according to the black box definition. [1]

11. Black Box Cont. • It will describe how that system will respond to stimuli; this is done usually in a formal specification language. • Z (Zed) Picture from http://uebb.cs.tu-berlin.de/zeta/zetadistdoc/Z-img13.gif

12. State Box • State box is where the view of an object shows the data implementation, but does not show the implementation process • It describes how state information is being transformed • In essence, the “history” of the black box is replaced by an existing “state”. [1]

13. State Box Cont. • This is an abstraction of the history that allows us to take a higher-level view of the system • Must ensure that there is no “history” case that is unaccounted for. [1]

14. Clear Box • Clear box shows both the data and process implementation • The goal is to stepwise refine functions and prove them as being correct. • Clear boxes show what is actually necessary to go between the old state and the new state. [1]

15. Clear Box Cont. • Sometimes there are multiple paths or multiple states that can result from a state box – the clear box lets us examine and design these transitions with flow control. • In the clear box, the procedures required to implement the state box transition function are defined explicitly. [1]

16. Cleanroom Approach • Requirements Analysis • Producing and reviewing “informal specifications.” • High-level Design • Converting the requirements into state machines and functions • Detailed Design • Further refinement of functions [3]

17. Cleanroom Approach Cont. • Coding by increment • Developing code and verifying it using formal methods. • Compiling code or unit testing is prohibited • Pretest by increment • Generation of test cases • Statistical Testing by Increment • The code is compiled, linked, and tested. Then the results are validated. [3]

18. Cleanroom Approach Cont. • Cleanroom Software Engineering prohibits unit testing • In a Cleanroom development, correctness verification replaces unit testing and debugging • After coding is complete, the software immediately enters system test with no debugging. [3]

19. Cleanroom Approach Cont. • All test errors are accounted for from the first execution of the program with no private testing allowed • The role of system testing in the Cleanroom process is to certify the quality of the software with the systems specifications in mind. • Not doing unit testing can only be done if the above requirements are followed, that way many of the defects are already found and fixed, so when the system is done coding it should be close to no defects. [3]

22. Correctness Verification • Once a piece of code is developed it goes through the Correctness Verification process. • correctness verification phase takes the developed code and compares it against the specification to see if it really does what is outlined in the specification. • The specifications define the conditions that code must meet in order to fulfill the function for which it was developed. [1]

23. Correctness Verification Cont. • Correctness verification uses function-theoretic static code analysis to do just that. • The term ‘function theoretic’ implies that there is a one-to-one mapping between the code and the specifications.

24. Correctness Verification Example function isNumeric(char c) { if ((c >= ‘0’) and (c <= ‘9’)) return true; else return false; }

25. Example Explanation • If the character passed in is in the set [0,1,2,3,4,5,6,7,8,9] we expect the function to return true. • Based on what we know about character sets and the language used to develop the code if the character is in the set then the logic ((c >= ‘0’) and (c <= ‘9’)) will return true. • When the character is not in the set then the logic will return false. • Both cases result in the expected behavior for the entire method, and so the code passes the correctness verification. [1]

26. Statistical Usage Testing • In conjunction with the box structure specifications in the pre-development phase, usage specifications are created for the statistical testing phase. • Usage specifications are simply descriptions of how the system will be eventually used. • Usage models need to be defined for all possible scenarios for a given piece of code along with the probabilities of each scenario occurring. [1]

27. Statistical Usage Testing Cont. • Use Markov Chains • Markov chains are essentially directed graphs with nodes as states of use and arcs as the stimuli that cause state transitions. • This is the most efficient way to test software, since the most destructive problems will be eliminated first, and money will not be spent on potentially harmless problems if it is not available. [1]

29. Waterfall Model into a Cleanroom • Waterfall model…we all know what that is • We want to take the Cleanroom model and add some milestones to the model.

31. Waterfall Model into a Cleanroom Milestones • Software Specification Review • (Historical) Addresses requirements and external interfaces • (Cleanroom) Remains the same with increment plan-mapping requirements to increment. • Preliminary Design Review • (H) Top-level architecture • (Cr) Top-level architecture updated as needed in later increments

32. Waterfall Model into a Cleanroom MilestonesCont. • Critical Design Review • (H) Detailed Design • (Cr) Detailed Design of functionality for particular increment • Qualification Test • (H) Verify requirements • (Cr) Verify requirements. Performed on final increment. Earlier increments have informal QT

33. Case Studies • STARS • Tank-automotive and Armaments Command

34. STARS • Developed by the Department of Defense • STARS receives radar data and flight plan information and presents the information to air traffic controllers on high resolution, 20" x 20" color displays allowing the controller to monitor, control, and accept hand-off of air traffic [4]

35. STARS Cont. STARS is capable of tracking up to 1350 airborne aircraft simultaneously within a terminal area.  The system interfaces with multiple radars (up to 16 short and long range), 128 controller positions, 20 remote towers, and a 400 by 400 mile area of coverage. [4]

36. STARS Cont. • The “STARS” program emphasized on three main points • Process-driven • Re-use based • Integrated software engineering environment • STARS evaluated current "traditional" processes. Then determined that a quality management philosophy (putting decision making in the hands of workers, focusing on processes, quantitative measurements) is critical and that Cleanroom Software Engineering follows this philosophy [4]

37. STARS Cont. • Cleanroom was combined with the TRW (spiral) Ada Process Model • Produced software at a rate of $30-40 per line of code versus industry averages of $130 per line for software of similar complexity and development timeframe (the size of the application is greater than 300 KLOC) [4]

38. STARS Savings • $130 - $30 = $100 per line of code • The project is around 300K lines of code • So, $100 * (around)300K = (around) $30,000,000 • Could buy 30, million dollarhouses • Or about one month of Alex Rodriguez playing baseball [4]

39. Tank-automotive and Armaments Command • TACOM generates, provides, and sustains mobility, lethality, and survivability for soldiers, other U.S. Armed Services, and our allies - all to ensure Army readiness today, tomorrow, and beyond

40. Tank-automotive and Armaments Command Cont. • After seven project increments (approximately 90K lines of code) • 4.2:1 productivity increase • 20:1 return on investment has been documented • Projects experienced an overall testing error rate (represents all errors found in all testing) of 0.5 errors/KLOC [4]

41. Summary • Cleanroom software development may be a wonderful advance in the process of software development or may just be a downright weird approach, most likely a little of both. • Looking at Cleanroom from a theorists’ point of view Cleanroom provides a theoretical foundation to software development in its use of mathematically based software development and statistical quality control.

42. Summary • By not introducing errors into the development phase there should be no testing requirement in the process. • Statistical testing provides the benchmark as to performance and failure rate and helps verify the inputs to the development process by checking its outputs.

43. On the other Hand • Many people sees Cleanroom as too radical a departure from conventional software development. • The level of experience and training required to have a functional team of Cleanroom developers may not cost effective • Developers develop code, and Cleanroom is more about specifications and statistical models than coding.

44. In the End • One has to understand the appropriate use of Cleanroom software development • Cleanroom is suitable for very particular types of software where the human and financial risks of having errors are too great to be left to chance • This generally does not fit the mold of mainstream software development, in which the concentration is often on getting the best price in the best time period. • From what has been shown, Cleanroom is anything but a mainstream development process.

45. References • Cleanroom Software Engineering. http://www.criticaljunction.com/werbicki/SENG623/Group/SENG623W03_Cleanroom.pdf • Uta.edu, Cleanroom Software Engineering. Found at: www.uta.edu/cse/levine/fall99/cse5324/cr/clean/page.html • DACS, The Data and Analysis Center for Software. Found at: www.dacs.dtic.mil/databases/url/key.php?keycode=64 • Carnegie Mellon, Software Engineering Institute. Found at: www.sei.cmu.edu/str/descriptions/cleanroom.html