PROCESS … The GLUE that holds Software Projects Together A Focus on Software Testing

1. PROCESS … The GLUE that holds Software Projects TogetherA Focus on Software Testing Dennis A. Morrison Senior Principal Engineer DD(X) System Architecture Dennis_morrison@raytheon.com

2. Presentation Outline • Software Process Overview • Software Testing Overview • Source Code Control • Trouble Reporting and Bug tracking • Summary

3. Conceive Design Code Build Test Deploy Deliver Support Software Phases(Waterfall Model) Requirements

4. Software Phases(Process Glue) Requirements Review – Are the Requirements testable Artifacts – Review documentation; Requirements placed under documentation control Requirements Design Review – Does the design support requirements? Artifacts – Review documentation Design Code Review – Does the code support requirements and implement the design?? Artifacts – Review documentation; Code under source code control Code Test Review – Test plans Test Results – Did the tests pass? Artifacts – Review documentation; Test plans; System Trouble Reports (STR) Test Deploy Test Review – Customer Acceptance Test Test Results – Did the tests pass? Artifacts – Review documentation; STRs Support Customer Survey – Is the system meeting customer expectations Artifacts - Software Updates; STRs

5. New Feature Request Requirement Clarification or changes Design Errors Software Bugs Software Bugs Software Phases and Loops Requirements Design Code Test Phases where most rework occurs Deploy Support New feature requests or feature refinement

6. Software TestingAn Art or a Science ? • Testing is a phase of software development that can become, or seem, endless • Classically, the most expensive part of the project • Schedule overruns are often attributed to the testing cycle • Testers are usually squeezed into smaller and smaller test cycles • How much testing is enough? • How do you know when you are finished testing? • How good is the product when you are finished?

7. White Box Testing Unit Test Grey Box Testing Integration Test Black Box Testing System Test Subset of Black Box Testing Acceptance Test Testing Has Many Stages Test

8. Unit Test = White Box Testing • Phase = Unit Test • Testing is focused on internal operations of the software component • Inputs/outputs • Logical operations • Limit testing • Limited functional testing with stubs • Testing methodologies • Formal testing with test plan and review • Ad-hoc random • Typically, tests are manually executed

9. Integration Test = Grey Box Testing • Phase = Integration Test • Testing is focused on functional aspects of an integrated set of software components • Inputs/outputs • Logical operations from outside of the software components • Limit testing (induced error paths from external stimuli) • Stubs are replaced by real software as they become available • Testing methodologies • Formal testing with test plan and review • Semi-automated and manually executed • Automation yields regression test capability

10. System Test = Black Box Testing • Phase = System Test • Testing is focused on system level requirement verification • User Interface (Human to Computer Interface) • Requirement verification • Limit and normal operational testing • Stress and longevity testing • Tests are executed on target hardware and software • Testing methodologies • Formal testing with test plan and review • Typically, tests are semi automated and manually executed • Automation yields regression test capability

11. Acceptance Test = Subset of Black Box Testing • Phase = Acceptance Test • Testing is focused on externally visible features • User Interface (Human to Computer Interface) • Operational focus • Interfaces with external systems in customer domain • Other collaborative systems • Customer network, etc. • Tests are executed on target hardware and software usually at the customer’s site • Testing methodologies • Formal testing with test plan and review • Typically, tests are manually executed

12. Some Software Testing Models • Developer level testing • Applicable to small teams • Community testing • Beta versions of a product are released to a volunteer community for functionality and user level testing • Used by gaming community prior to product releases • Microsoft uses this approach as one of their test phases • Collaborative testing • Developers and customers test together • Project level testing • Used by most large projects

13. When Should Software Testing Begin? • The software test cycle should begin during the project requirements phase • Requirements review should result in the first level of test specifications • Similar to the “Agile” software development process, test cases should be written and reviewed before the software is developed

14. When is Testing Finished? • When the schedule says you’re finished? • When you run out of money? • When the boss says to release it? • When the tester’s say you’re done? • When the customer says you’re done? • When the law suits are filed?

15. Requirements Verification • Raytheon uses requirement verification and statistical mechanisms to determine when testing is complete • Requirements are decomposed and allocated to software components • Requirements are “sold off” with during formal testing events with stakeholders (customers) present

16. Req 1.1 Req 1.2 Req 1.3 Token Reader Pin Reader Biometric Reader Req 1.1.1 Req 1.1.2 Req 1.2.1 Req 1.3.1 Req 1.3.2 Something you have Something you know Something you are Requirement Decomposition Parent requirement decomposes into three child requirements Operational Requirement The system shall provide three factor user authentication Req 1

17. Requirements Role in Testing • Software is tested against requirements • Before a parent requirement can be “sold off”, all children requirements must be fulfilled • Before a child requirement can be “sold off”, all of its siblings must themselves be “sold off” • The failure of any requirements will prevent the parent from being fulfilled

18. Req 1 Req 1.1 Req 1.2 Req 1.3 Token Reader Pin Reader Biometric Reader Req 1.1.1 Req 1.1.2 Req 1.2.1 Req 1.3.1 Req 1.3.2 Something you have Something you know Something you are Requirement Decomposition Parent requirement decomposes into three child requirements Operational Requirement The system shall provide three factor user authentication Req 1 Req 1

19. Requirements Traceability • Software test cases are built from system use cases • Scenarios (rainy day and sunny day) are used to test various threads through the software • Every scenario is traced to parent requirements • A scenario may have more than one parent requirement • Test cases must demonstrate interface level testing • All system and software level interfaces are documented and tracked • These are known as IRLLs, IRLEs, and IRLMs • IRLL = Interface Requirement Logical Level • IRLE = Interface Requirements Electrical Level • IRLM = Interface Requirements Mechanical Level • Test cases and requirements are stored in DOORS

20. Software Source Code Control • The ability to save and retrieve versions of software is a key component to success in medium to large software projects • Software can be “versioned” for new features or software load lines • Changes made to software modules due to bug fixes can be isolated • Enables developers to work on next release while maintaining current versions • Software builds can include or exclude changes

21. Using Source Control Systems Source Code Version 1 SW Build Specification Mod A-V1.1 Mod X-V7 Mod Z-V2 Put (ModA V1) Source Code Control System Source Code Version 1 get(ModA V1) New Feature put(ModA V2) Source Code Version 2 Software Build get(ModA V1) Source Code Version 1 put(ModA V1.1) Bug Fix Source Code Version 1.1

22. Trouble Reporting and Bug Tracking • Software bugs are documented using formal mechanisms • CMTS – Change Management Tracking System (MRs) • WSTR – Web System Trouble Report (STRs) • Bugs are considered addressed when software has been modified, tested, and submitted to the load-line • Bugs are considered fixed when they have been verified • Bug fixes are pulled into the software load via the source code control system • Bug reports and time based metrics guide management in software release decisions

24. Test Methods • Manual • Usually applied during unit test • Semi Automated • Applied during integration test • Fully Automated • System test

25. Test Automation • Test automation can provide a project with huge dividends • Test scripts • Drive test automation engines • Can be used in unit testing • Can be reused by multiple test organizations • Regression tests can be constructed from reusable test scripts

26. Metrics Drive Behavior • An organization or individual’s behavior is directly related to the metrics they are measured by • Defects found and fixed • On time delivery • Productivity • SLOC counts • Defect rate at delivery • Complexity indices • Customer satisfaction

27. Best Practices? • No real silver bullets • Process is the glue that holds a project together • The right solution depends on the project but is often a combination of methods/processes and tools • Just remember that customer satisfaction trumps all!!

28. BACKUP SLIDES

29. ABSTRACT In the world of software development, process is the “glue” that holds the project and team together. Too little, the pieces fall apart; too much and it can become a sticky quagmire resulting wasted effort, time, and money. The challenge facing many projects is finding just the right balance point on the process continuum. As a project progresses through its life cycle, the balance point will shift and the project will need to continuously review and adjust to maintain optimal performance. A typical software project life cycle is comprised of six distinct phases: Requirements, Design, Code, Test, Deployment, and Support. Most projects place a significant amount of emphasis on the front-end phases of the life cycle e.g. requirements, and design. However, industry data clearly demonstrates that the largest percentage of a projects budget is spent during the final phases, namely test, deployment, and support. These phases encapsulate much of the “rework” experienced by the project. This talk will describe the process philosophies and tools used by Raytheon to tackle large-scale multisided software development projects.

30. Tools • The right tool can result in huge pay backs • Source Code Control Tools • Trouble Report tracking • Stress testing • Performance testing

31. Testing Methodologies • Ad-hoc • Non structured testing with or without test artifacts • No test plans • Non repeatable tests • Formal testing • Test plans • Test Artifacts • Defect tracking

32. Software Phases(Spiral Model) Spiral Development:Experience, Principles, and Refinements Spiral Development WorkshopFebruary 9, 2000 Barry Boehm, edited by Wilfred J. Hansen

33. The NATO Sea Sparrow Surface Missile System (NSSMS) is a quick reaction, point defense system designed to protect the ship it is on. It is very, very effective against enemy aircraft, low flying sea skimming missile and even bomb laden inflatable boats. It was first developed by Raytheon in the '70's and installed on over 150 ships world-wide. In the mid '90's the Navy contracted Raytheon to re-architect the system so that it would be capable of firing the Navy's newest missile the Evolved Sea Sparrow Missile (ESSM). The Navy's brand new nuclear powered aircraft carrier , the USS Ronald Reagan, has 6 Seasparrow radars & 3 Seasparrow missile launchers. The movie shows the Navy doing Qualification Testing on these systems.

34. Software Phases(Process Glue) Requirements Review – Do the requirements support the customer’s intent? Are they testable? Are they complete (sunny day/ rainy day) Artifacts – Review documentation; Requirements placed under documentation control Requirements Design Review – Does the design support requirements? Artifacts – Review documentation Design Code Review – Does the code support requirements and implement the design?? Artifacts – Review documentation; Code under source code control Code Test Review – Test plans – Do the tests adequately exercise the requirements? Test Results – Did the tests pass? Artifacts – Review documentation; Test plans; System Trouble Reports (STR) or Modification Requests (MR) Test Deploy Test Review – Customer Acceptance Test Test Results – Did the tests pass? Artifacts – Review documentation; System Trouble Reports (STR) or Modification Requests (MR) Support Customer Survey – Is the system meeting customer expectations Artifacts - Software Updates; STRs