Verification Methods CS2110: SW Development Methods

1. Verification MethodsCS2110: SW Development Methods Reading: Study Section 1.2.5 in the MSD text These slides supplement the textbook These touch on other important methods, but focus on execution-based testing

2. Some Definitions • Verification: showing that something meets specifications • We verify “against” something • Important: if you don’t have requirements specifications documented, then you can’t verify! • If behavior is not defined, then what counts as incorrect behavior? • Verification can happen on any kind of software work-product • designs, code, etc.

3. Verification Techniques At least four approaches to verifying code quality: • Execution-based Testing (running tests) • Inspections • People examine the work product by eye • Static Analysis by a tool • For C programs, an old tool called “lint” • For Java programs, some nice tools integrate with Eclipse: CheckStyle and PMD • Formal Correctness Proofs • Use mathematics to demonstrate the desired property

4. Execution-based Testing Some Basic Definitions: • Fault: What we think of as a bug • Something in the code that’s wrong • Failure: Observed incorrect behavior of a system resulting from a fault • Error: Mistake made by a developer A possible definition of execution-based testing: • Process of inferring certain behavioral properties of a product based, in part, on results of executing it in a known environment with selected inputs.

5. Famous and Important Quote • Program testing can effectively show the presence of bugs but is hopeless for showing their absence. • Edsger Dijkstra • Famous computer scientist • Turing Award winner • Wrote "Go To StatementConsidered Harmful“ (1968) • Shortest-path algorithm

6. Types of Testing • Unit testing • An individual class or function is tested thoroughly • Done as part of coding process • Done by the programmer him/herself • Usually by a test harness (or driver) or using a test-class • Integration testing • Multiple parts of a system are combined and tested to see if they work together • Acceptance testing • The customer is involved. (Alpha or beta testing)

7. Approaches to Unit Testing • Two approaches • Test only based on inputs, outputs (i.e. the specification) without looking at the code • Test based on the logic of the code (must study it) • The 1st of these is called black-box testing • The 2nd of these is glass-box or white-box testing • Some principles of glass-box testing • Make sure every line (or “case”) gets executed • Make sure loops get done zero times, >zero times • Etc.

8. Limitations of Glass-Box Testing • Why not fully exercise our code? • Logic of a piece of code defines a set of possible execution paths, “flow-paths,” through the function • Think of a flow-chart • The inputs control which path is taken • A good set of test data might make sure every possible path is followed • This tests every possible behavior • Problem: for even small programs with loops and conditionals, the total number of paths becomes too large too quickly • See MSD text, p. 24-25

9. Black-Box Testing • This means testing based on inputs to the code-module without basing tests on code’s internals • You can’t see inside a black-box • Note: MSD’s definition on p. 26 isn’t the usual one • Choose test cases for common problems • Boundary values, boundary conditions • Off-by-one errors • Incorrect inputs • Important: You have to know in advance what the expected result of any test should be!

10. Lab on Monday/Tuesday • The JUnit (v. 3) framework is a tool to help people generate and format unit tests in a systematic way. • Stand-alone and alsobuilt into Eclipse • Example JUnit test • Class contains alltests • setUp() performspreconditions • “test…” is anindividual test case • “assert” checks aprogram condition publicclass CoffeeMakerTestextends TestCase { private CoffeeMaker cm; private Recipe recipe1; publicvoid setUp() { cm = new CoffeeMaker(); recipe1 = new Recipe(); recipe1.setName("Coffee"); recipe1.setPrice(50); recipe1.setAmtCoffee(6); recipe1.setAmtMilk(1); } publicvoid testAddRecipe() { assertTrue(cm.addRecipe(recipe1)); } }

12. When we test a class’ method to see if its loop goes to size-1 and not size, is this clearly an example of: • A black box test • A glass-box test • A unit test • Both 1 and 3 • Both 2 and 3

13. Planning Your Tests • How do we plan a “test case”?Define three parts first: • A purpose (why are we doing this particular test?) • An input • An expected result • We execute the test to see if observed output matched the expected result • It’s really just like a scientific experiment • When testing, have a plan (at least in your head) • Don’t test blindly or randomly

14. Planning Your Tests • Poorly specified test cases…

15. Planning Your Tests • Why are these test cases better than the previous ones?

16. Unit Testing: Boundary values • Often programs misbehave because we make mistakes dealing with the “extremes” • Examples: • The last item or first item in a collection • The start or end of a range of values • Equivalent classes of test cases • Don’t have test cases or data that test the same situation (e.g. the middle of a range) • Boundary or extreme conditions • An empty list, an empty file • One item in a list or file

17. Unit Testing: Off-by-one errors • Related to boundary conditions • Using the nth value instead of n-1 • Using 0 instead of 1 • Executing a loop body once when it the body should not be executed at all • Wait – is this one a black-box testing method? Why or why not? • Etc.

18. Unit Testing: Invalid Inputs • Rely on pre-conditions defined for your functions • They specify what the agreement is about what you are responsible for when its time to handle mis-use • If a function is mis-used in this way • Return an error condition if the interface provides a way to do this • Otherwise, halt program! Better than producing unexpected results • Use assertions (more later!)

19. Example: Program with Student and Course classes • Homework 1 this term • Define class Student • methods include: • Define class Course • methods include: • Focus on defining: • Black-box test-cases for the methodboolean drop(Course c) in class Student • Let’s make a list of possible test-cases for this method!

20. Example of Full Test-Case Description • Test Case #1: Drop last course in list with >1 item • Purpose:  Drop Course c for Student s where that course is the last course stored in the Student object's courses list.  This tests the boundary condition of last item in list for Student.drop(c). • Input: Student s is enrolled in three courses, and the course to be dropped is the last of these stored in the list. • Expected output:   The Student s is unchanged except that only two courses are stored for that student. Also, the list of students stored for that course is altered by having that student removed.

21. Other Test Cases for this Program • There are a lot! • Boundary conditions? • for Student.drop(): end of list, start of list • same for Course.drop() • Question: affected by whether you used ArrayList instead of Java arrays? • Note: also test “typical” case, item in middle of list • Empty lists: Course.cancel(), Student.dropAll() • Off-by-one errors? • Really included in boundary condition testing here

22. Other Test Cases for this Program • Invalid inputs? • See the specifications! • Test return values • For boolean, test both success and failure • Remember: make it fail! • Test of equals() methods? • If you get this wrong, trouble! • Easy to test. (Easy to forget to.)

23. Good or Bad Test? Student s adds a course and the course appears in the schedule • Good Test? • Poor Test?

24. Good or Bad Test? Student s currently is inthe system and currentlyhas no courses. Student s adds a course number 394 and it appearsat the end of the schedule. • Good Test? • Poor Test?

25. Good or Bad Test? Student s has 4 courses,drops 3, adds 1. Check if the student is then deletedfrom the system. • Good Test? • Poor Test?

26. Extra Slides

27. #2: Inspections: We Might See These Later • Many different “styles” or approaches • Walk-throughs, Inspections • Advantages: • You can inspect all of a product but usually it is impossible to test all of a product. • Applicable to all work products (e.g. code, designs, requirements documents, etc.) • Fairly easy: training, application, follow-up • Disadvantages: • Staff time is expensive • Works best if follows a rigorous process