Secure Software Development: A Security Programmer’s Guide

1. Secure Software Development: A Security Programmer’s Guide Chapter 10 Testing for Quality and Security

2. Objectives • Understand the basic concepts and definitions of testing • Work with dynamic testing techniques • Use static testing techniques • Test the resiliency of the development process Secure Software Development: A Security Programmer’s Guide, First Edition

3. Testing in the Industry: Concepts and Definitions • Testing software consists of running a battery of test cases using multiple techniques against a specific use case and evaluating the results for pass or fail marks • As a developer, you will play a key role in helping the management team create, execute, and evaluate test cases • The following subsections will help you understand what testing is: • What testing and assurance are • What testing is not • The test plan • The problems with testing Secure Software Development: A Security Programmer’s Guide, First Edition

4. Testing in the Industry: Concepts and Definitions (continued) • What Testing and Assurance Are • Software testing is the discipline of examining the functionality of application software to determine whether it meets the requirements (verification) and satisfies the need (validation) • Software assurance (SwA) is how effective security is tested and implemented within the software (Redwine, et al. 2007) • During the software assurance process, the development team members must demonstrate their competency with software security testing through the use of artifacts, reports, and tools Secure Software Development: A Security Programmer’s Guide, First Edition

5. Testing in the Industry: Concepts and Definitions (continued) • What Testing Is Not • It is not a security measure • The application will never be 100 percent secure • It is not a quality measure • For the same reason that testing cannot guarantee security, it cannot prove quality • Testing does not generate error-free software; testing generates software that is acceptable for the user community and that meets the needs of the business • It is not always automated • Executing a bunch of shell scripts to emulate users is not enough • The earlier testing is done, the cheaper it is to fix Secure Software Development: A Security Programmer’s Guide, First Edition

6. Testing in the Industry: Concepts and Definitions (continued) • The Test Plan • A test plan is created by the project management team and specifies how testing should be performed, who executes the testing, when testing starts and ends, as well as any and all other pertinent data deemed necessary by the project management team • Define test scripts • Define the user community • Identify the showstoppers • Identify internal resources • Identify external resources Secure Software Development: A Security Programmer’s Guide, First Edition

7. Testing in the Industry: Concepts and Definitions (continued) • The Test Plan (continued) • Define test scripts • Test cases are what the developers use to run their tests • Scripts are very detailed, logical steps of instructions that tell a person or tool what to do during the testing • Test cases define the goals of each feature that needs to be executed • Functional testing scripts • Secure testing scripts Secure Software Development: A Security Programmer’s Guide, First Edition

8. Testing in the Industry: Concepts and Definitions (continued) • The Test Plan (continued) • Defining the User Community • Defining the user community helps testers identify acceptable levels of failures and risks Secure Software Development: A Security Programmer’s Guide, First Edition

9. Testing in the Industry: Concepts and Definitions (continued) • Identifying the Showstoppers • What does the application need to fulfill the user’s request? • The test plan should include the must-haves so that the developers can pull the plug on certain entities to see how the application responds • Identifying Resources • Far too often, the people who do most of the testing are the same ones who coded the programs • Developers are so focused on getting the functionality working, they forget to test different security features such as roles, privileges, and hard-coded comments Secure Software Development: A Security Programmer’s Guide, First Edition

10. Testing in the Industry: Concepts and Definitions (continued) • Identifying Resources (continued) • There are two types of resources needed to execute the test case: • Internal: Internal resources come from the company’s organization, including developers, analysts, software tools, and sometimes even project managers • External: Tools or people who are hired on a temporary basis to come into a project, test the application, and report findings Secure Software Development: A Security Programmer’s Guide, First Edition

11. Testing in the Industry: Concepts and Definitions (continued) • Problems with Testing in the Industry • Testing activity is the one phase that is shortened because of project overruns, loss of budgets, or lack of resources • Testing for security and quality are faced with the following problems: • Too much reliance on individual skills • Unpredictable users • Inconsistent definition of success Secure Software Development: A Security Programmer’s Guide, First Edition

12. Dynamic Testing in the Organization • Dynamic testingimplies executing the current use case using valid inputs and then comparing the real results with expected results for pass or failures (Redwine, et al. 2007) • Every method and technique requires the tester to analyze the results and perform some validation and verification (V&V) on the software and report the findings • Technique within the three phases of testing: • Unit testing • Integration testing • System testing Secure Software Development: A Security Programmer’s Guide, First Edition

13. Dynamic Testing in the Organization (continued) • Unit Testing • Unit testing is the lowest level of testing a developer can conduct • Compilers will stop the developer from making syntax mistakes or typos • Unit testing catches errors that compilers won’t find • Unit testing done right includes the following techniques: • Testing exception handling: Manually going into the source code and forcing an error or attack • Reviewing error logs: Every error that the application manages needs to be logged so that developers can analyze those logs for trends and abuses Secure Software Development: A Security Programmer’s Guide, First Edition

14. Dynamic Testing in the Organization (continued) • Integration Testing • Integration testing is when two or more modules, Web services, or platforms are linked together and tested • Integration testing requires testing of the whole use case rather than an individual piece of software • System Testing • A system test involves a complete front-to-back execution of the entire application • Each new use case that is added to the system must be tested for quality features, such as reliability and usability, and for security Secure Software Development: A Security Programmer’s Guide, First Edition

15. Dynamic Testing in the Organization (continued) • Fuzz Testing • Fuzz testing, or fuzzing, is a technique that feeds random input data into applications just to see what happens to the results • Fuzzing applications can be done using tools or human methods that simply write garbage data into enterable fields, XML files, or any other inputs to the application • To carry out a fuzz test: • Enter random data or spaces to some part of the input file • Execute the application with that input file • Evaluate results • Number each test case and report findings Secure Software Development: A Security Programmer’s Guide, First Edition

16. Dynamic Testing in the Organization (continued) • Reliability Testing • Reliability of an application is when the application produces correct results despite being under attack (Redwine, et al. 2007) or under extreme use • Testing the reliability of the application requires simulation tools that can replicate the number of expected uses, time of day, and software environment as in the real world • Some tools that can be used during reliability testing include the following: • Holodeck: Holodeck is a unique testing tool that “uses fault simulation • LoadRunner: LoadRunner is a simulation tool used to test concurrent users, database connections, and other application resources Secure Software Development: A Security Programmer’s Guide, First Edition

17. Static Testing in the Organization • Static testinginvolves executing methods and techniques to test the code without actually running the application • Static testing does not evaluate the application’s performance • Evaluates what the actual code looks like on paper • Static testing includes the following techniques: • Static code analysis • Code reviews and inspections • System documentation Secure Software Development: A Security Programmer’s Guide, First Edition

18. Static Testing in the Organization (continued) • Static Code Analysis • Static code analyzersscan (parse) through static code and analyze the code base for security vulnerabilities (such as input fields and buffer overflows) • A number of static code analyzers are available on the market • Tools come with sophisticated code parsers, but they also have reporting features Secure Software Development: A Security Programmer’s Guide, First Edition

19. Static Testing in the Organization (continued) • Code Reviews and Inspections • Code reviewsare people-intensive verification techniques that are conducted either formally or informally that allow peers to read code statements and look for common security vulnerabilities, such as hard-coded IDs or passwords, and general quality features • Code reviews, if conducted right, should have a very positive effect on the team • Reviewing code as a team is actually a great opportunity for the strongest skilled developers to share knowledge, rationale, and guidance with the entry/mid-level developers Secure Software Development: A Security Programmer’s Guide, First Edition

20. Static Testing in the Organization (continued) • Code Reviews and Inspections (continued) • The following are the steps to a successful code review: • The developer of the use case prints out all the code and makes copies for each participant • At the meeting, each participant walks through the use case, step-by-step, and the programmer talks about each activity • Every code statement is analyzed and reviewed by the participants against a checklist of rules and guidelines • As issues or concerns come up during the meeting, the code is highlighted on the hard copy and notes or instructions are taken • After the meeting, it is determined if another walk-through is needed or if the business analyst needs to be consulted for any questions Secure Software Development: A Security Programmer’s Guide, First Edition

21. Static Testing in the Organization (continued) • Verifying System Documentation • Before you put the test case to rest and go on to the next one, verify that all system documentation is finalized and up to date • System documentation is a lot like life insurance—you might never need to use it after it is finalized, but the one time you need it to be available and accurate, it will be there for you • The following activities will keep the software documentation up to date: • Verify that the use case documentation says what the software does • Verify traceability on the current use case throughout the whole life cycle • Verify the current version • Update the Application Guide Secure Software Development: A Security Programmer’s Guide, First Edition

22. Testing the Resiliency of the Development Process • Testing will give you a good measuring stick of how good your software is and where the weak areas are, but testing will not tell you if the process in which you created code worked • Despite all the pent testing, code review, static analysis, and fuzz testing efforts, disasters will happen, and how you respond and react will make or break your career • Coding for resiliency means accepting the fact that something bad will happen and that when it does, you will be ready • Knowing how to bounce back • Knowing how to make something bad happen Secure Software Development: A Security Programmer’s Guide, First Edition

23. Testing the Resiliency of the Development Process (continued) • Knowing How to Bounce Back • Delete a database table, steal and decompile the application code, and take whatever it is the software is supposed to protect and get rid of it • Policies and procedures need to be tested just as much as the software itself • What to do when this happens, who to call when that happens • If a fellow developer was caught e-mailing or copying code from one device to another, how fast can you change the program’s encryption logic or the values of the CONSTANT variables? Secure Software Development: A Security Programmer’s Guide, First Edition

24. Testing the Resiliency of the Development Process (continued) • Making Something Bad Happen • Making something bad happen actually tests the dependencies and resiliency of the software • What must be up and ready for the software application to work • The following are some bad things you can make happen to test the recovery time: • Slam a machine: How long does it take for a developer to reinstall the OS? • Stop all Web services that the application calls or offers: What is the desired effect on application? • Take out pieces of architecture:What happens when one of those layers is unavailable? Secure Software Development: A Security Programmer’s Guide, First Edition

25. Summary • Testing software requires a battery of test cases • Use multiple techniques, tools, and analyses to test secure code • Software assurance will guide how well security is tested within the testing phase • Testing software does not prove quality or security; it just proves that it passed the software test case • Testing generates software that is acceptable for the user community and that meets the needs of the business • Test plan is created by the project management team Secure Software Development: A Security Programmer’s Guide, First Edition

26. Summary (continued) • A test plan specifies how testing should be performed, who executes the testing, when testing starts and ends, as well as any and all other pertinent data deemed necessary for the testing process • Scripts are very detailed, logical steps of instructions that tell a person or tool what to do • Functional testing scripts are step-by-step instructions that depict a specific scenario or situation that the use case will encounter as well as the expected result • Functional testing scripts are created for both human (manual) interfaces and for automated testing Secure Software Development: A Security Programmer’s Guide, First Edition

27. Summary (continued) • Defining the user community will help testers identify acceptable levels of failures and risks • Software tests are conducted by two resources: internal and external • Unit testing is the lowest level of testing a developer can conduct • Integration testing is when two or more modules, services, or platforms are linked together and tested • A system test involves a complete front-to-back execution of the entire application Secure Software Development: A Security Programmer’s Guide, First Edition

28. Summary (continued) • Fuzz testing, or fuzzing, is a technique that feeds random input data into applications just to see what happens to the results • Static code analyzers scan (parse) through static code and analyze the code base for security vulnerabilities (such as input fields and buffer overflows) • Code reviews are people-intensive verification techniques that are conducted either formally or informally and allow peers to read code statements and look for common security vulnerabilities, such as hard-coded IDs or passwords, and general quality features Secure Software Development: A Security Programmer’s Guide, First Edition