Evolving an Elective Software Testing Course: Lessons Learned Edward L. Jones

1. 3rd Workshop on Teaching Software Testing Evolving an Elective Software Testing Course: Lessons Learned Edward L. Jones Florida A&M University Tallahassee, FL USA

2. Agenda • Course Overview • Student Background • Driving Principles • Overview of Assignments • Course Reflection • Improvements • Assignment Walkthroughs

3. Course Overview DESCRIPTION: The purpose of this course is to build skills necessary to perform software testing at the function, class and application level. Students will be taught concepts of black-box (functional and boundary) and white-box (coverage-based) testing, and will apply these concepts to small programs and components (functions and classes). Students will also be taught evaluative techniques such as coverage and mutation testing (error seeding). This course introduces the software engineering discipline of software quality engineering and the legal and societal issues of software quality.

4. Programming Focus AUDIENCE: Not software testers but software developers. What distinguishes the course approach is that is stresses the programming aspect of software testing. A goal is to enhance and expand students’ programming skills to support activities across the testing lifecycle: C++ programming and Unix shell script programming to automate aspects of software testing. Students just needed a course to take ...

5. Conceptual Objectives • The student shall understand • The software testing lifecycle • The relationship between testing, V&V, SQA • Theoretical/practical limits of software testing • The SPRAE testing framework • Concepts and techniques for black-/white-box testing • Test case design from behavioral model • Design patterns for test automation • Test coverage criteria • Issues of software testing management

6. Performance Objectives • The student shall be able to: • Use the Unix development environment • Write simple Unix shell scripts • Design functional and boundary test cases • Develop manual test scripts • Conduct tests and document results • Write test drivers to automate function, object and application testing • Evaluate test session results; write problem reports

7. Learning/Evaluation Activities • 80% practice / 20% concepts • Lectures (no text) • Laboratory assignments • Unix commands and tools • Testing tasks • Examinations • 2 Online tests • Final (online) • Amnesty period (1 test / 2 labs)

8. Student Background • Reality: • 20 students • Not particularly interested in testing • Low programming skill/experience • Ideal: • An interest in software testing • Strong programming skills • Scientific method (observation, hypothesis forming) • Sophomore or junior standing • Desire for internship in software testing

9. My Perspective on Teaching Testing • Testing is not just for testers! • In ideal world, fewer testers required • Developers have tester’s skills/mentality • Testing overlays development process • No silver bullet … just bricks • Simple things provide leverage • No one-size-fits-all • Be driven by a few sound principles

10. Driving Principles • Testing for Software Developers • Duality of developer and tester • Few Basic Concepts • Testing lifecycle • Philosophy / Attitudes (SPRAE) • Learn By Doing • Different jobs across the lifecycle

11. Analysis Specification Test Strategy/Plan Design Test Cases Implementation Test Script, Data, Driver Execution Defect Data Problem Reports Test Results Evaluation A Testing Lifecycle

12. Experience Objectives • Student gains experience at each lifecycle stage • Student uses/enhances existing skills • Student applies different testing competencies • Competencies distinguish novices from the experienced

13. A Framework for Practicing Software Testing • Specification the basis for testing • Premeditation (forethought, techniques) • Repeatability of test design, execution, and evaluation (equivalence v. replication) • Accountability via testing artifacts • Economy (efficacy) of human, time and computing resources

14. Key Test Practices • Practitioner -- performs defined test • Builder -- constructs test “machinery” • Designer -- designs test cases • Analyst -- sets test goals, strategy • Inspector -- verifies process/results • Environmentalist -- maintains test tools & environment • Specialist -- performs test life cycle.

15. Test Products • Test Report (informal) of manual testing • Test Scripts for manual testing • Test Log (semi-formal) • Application Test Driver (Unix shell script) • Unit/Class Test Driver (C++ program) • Test Data Files • Test Results (automated) • Bug Fix Log (informal)

16. Specification Products • Narrative specification • Specification Diagrams • Specification Worksheet (pre/post conditions) • Decision Tables • Control Flow Graphs

17. Assignments Target Skills • Observation Skills • Systematic exploration of software behavior • Specification Skills • Describe expected or actual behavior • Programming Skills • Coding for development of test machinery • Test Design Skills • Derive test cases from specification using technique • Team Skills • Work with other testers

18. Course Reflection • Testing is programming intensive • Testing requires analytical skills and facility with mathematical tools • Testing generates data management problem that is amenable to automation • Testing gives students advantage in entry-level positions • Students take this course too late

19. Failed Course Expectations • Students test at all levels • No “in-the-large” application (e.g., web-based) • Students develop intuitive testing skills • On largest project, concepts did not transfer • 1 in 3 students show “knack” for testing • Impact of balance of concept and experience • Poor performance on exams with problems like those in labs • Test case design skills low • Homework needed v. labs (programming) • Mentoring (timely feedback) did not occur • Students left to own devices too much

20. Why These Outcomes? • Formalisms important, but difficult • Provide the behavior model (e.g., decision table) • Basis for systematic test case design, automation • Lack of textbook • Students need concepts + lots of examples • Poor availability when students were working • Students worked at last minute • Not always around • Automated grading lacked 1-1 feedback • Standards-rich/tool-poor environment a distraction • Assigned work too simple??

21. Proposed Changes • Improve lecture notes and example bank • Find and refine • Resources and workbook • Outside-in: testing in-the-large before in-the-small • Recitation/laboratory for discussion and feedback • Increase use of testing tools (no-cost) • Increase use of collection of code/applications • Examination testbank for practice, learning

22. Assignment Walkthroughs • (see paper)

23. Assignment Walkthroughs • Blind Testing • Test Documentation • Specification • Test Automation via Shell Scripts • Unit Test Automation (Driver) • White-Box Unit Testing • Class Testing

24. Blind Testing I • Objective: Explore behavior of software without the benefit of a specification • Given: Executables + general description • Results: Students not systematic in exploration or in generalizing observed behavior • Hello  output based on length of input • Add  1-digit modulus 10 adder, input exception • Pay  pay calculation with upper bound pay amount

25. Blind Testing II • Programming Objective: Student writes program that matches the observed behavior of Blind Testing I • Test Objective: Observations on Blind Testing I used as “test cases” for reverse-engineered program. • Results: Students did not see the connection; • Did not replicate the recorded behavior • Did not recognize (via testing) failure to replicate

26. SUPPLEMENTAL SLIDES • Student work

27. SCALING UP The heart of the approach is to use a decision table as a thinking tool. The most critical task in this process is to identify all the stimuli and responses. When there are many logical combinations of stimuli, the decision table can become large, indicating that the unit is complex and hard to test.

28. IDENTIFYING BEHAVIORApproaches • Work backwards • Identify each response • Identify conditions that provoke response • Identify separate stimuli • Work forward • Identify stimuli • Identify how each stimulus influences what unit does • Specify the response

29. IDENTIFYING STIMULI • Arguments passed upon invocation • Interactive user inputs • Internal, secondary data • global or class variables • External data (sources) • file or database status variables • file or database data • Exceptions

30. IT PAYS TO BE A GOOD STIMULUS DETECTIVE • Failure to identify stimuli results in an incomplete, possibly misleading test case • The search for stimuli exposes • interface assumptions -- a major source of integration problems • incomplete design of unit • inadequate provision for exception handling

31. IDENTIFYING RESPONSES • Arguments/Results passed back on exit • Interactive user outputs • Internal, secondary data • updated global or class variables • External data (sinks) • output file or database status variables • output file or database data • Exceptions

32. IT PAYS TO BE A GOOD RESPONSE DETECTIVE • Failure to identify responses results in • incomplete understanding of the software under test • shallow test cases • incomplete expected results • incomplete test "success" verification -- certain effects not checked • To test, one must know all the effects

33. A SKETCHING TOOL Black-Box Schematic Stimulus Type Response Type Argument Argument Inputs Outputs Software under Test Globals Globals Database Database Exception Exception

34. BEFORE CONTINUTING Much of the discussion so far involves how to identify what software does. We have introduced thinking tools for systematically capturing our findings. These thought processes and tools can be used anywhere in the lifecycle, e.g., in software design! One Stone for Two Birds!!

35. Specialist I - Competencies 1 2 3 4 5 ... Test Practitioner Practitioner 3 1 2 4 5 ... Test Builder 2 1 3 4 5 ... Test Designer 1 2 3 4 5 ... Test Analyst 1 2 3 4 5 ... Test Inspector 1 2 3 4 5 ... Test Environmentalist 1 2 3 4 5 ... Test SPECIALIST

36. BOUNDARY TESTING DESIGN METHODOLOGY • Specification • Identify elementary boundary conditions • Identify boundary points • Generate boundary test cases • Update test script (add boundary cases).

37. EXAMPLE: Pay Calculation(1) Specification • Compute pay for employee, given the number of hours worked and the hourly pay rate. For hourly employees (rate < 30), compute overtime at 1.5 times hourly rate for hours in excess of 40. Salaried employees (rate >= 30) are paid for exactly 40 hours.

38. EXAMPLE B(2) Identify Behaviors • Case 1: Hourly AND No overtime • (Rate < 30) & (Hours <= 40) • Expect Pay = Hours * Rate • Case 2: Hourly AND Overtime • (Rate < 30) & (Hours > 40) • Expect Pay = 40*Rate+1.5*Rate*(Hours - 40) • Case 3: Salaried (Rate >= 30) • Expect Pay = 40 * Rate

39. Condition c1: Rate < 30 | Y Y N N c2: Hours <= 40|Y N Y N Action a1: Pay = Straight time | X a2: Pay = Overtime | X a3: Pay = Professional | X X DECISION TABLE Columns define Behaviors

40. EXAMPLE B(3) Create Test Cases • One test case per column of decision table • Case 1: Hourly, No Overtime • Case 2: Hourly, Overtime • Case 3: Salaried, No Extra Hours • Case 4: Salaried, Extra Hours • Order the test cases by column

41. Stimuli Expected Response Step Hours Rate Pay = 1 30 10 300 2 50 10 550 3 30 40 1600 4 50 40 1600 EXAMPLE B(4) Write Test Script

42. Test set Data External Effects Driver Unit Test Set Results Arguments Results Testing Modules -- Drivers A test driver executes a unit with test case data and captures the results.

43. Implementing Test Drivers • Complexity • Arguments/Results only • Special set-up required to execute unit • External effects capture/inquiry • Oracle announcing "PASS"/"FAIL" • Major Benefits • Automated, repeatable test script • Documented evidence of testing • Universal design pattern

44. Driver D_pay uses unit_environment E; { declare Hrs, Rate, expected; testcase_no = 0; open tdi_file("tdi-pay.txt"); open trs_file("trs-pay.txt"); while (more data in tdi_file) { read(tdi_file, Hrs, Rate); read(tdi_file, expected); testresult = pay(Hrs, Rate); write (trs_file, testcase_no++, Hrs, Rate, expected, testresult); }//while close tdi_file, trs_file; }//driver Test Driver for Unit Pay

45. Test Data File File name: tdi-pay.txt Format: (test cases only) rate hours expected-pay File content: 10 40 400 10 50 550 10 0 0 ------ Note: No environment setup. Test Results File File name: trs-pay.txt Format: case# rate hours exp-pay act-pay File content: 1 10 40 400 400 2 10 50 550 500 3 10 0 0 0 ------ Note: Results file must be inspected for failures. Pass Test Script!! Fail Pass Test Driver Files (Pay)

46. Class Test set Data Method Args /Results Class Test Driver Method(s) Test Set Results Class-state Testing Classes -- Drivers(Black-Box)

47. class Stack { public: Stack(); void push(int n); int pop(); int top(); bool empty(); private: int Size; int Top; int Values[100]; }; Notes: (1) Class state -- variables Size, Top and the first 'Size' values in array Values. (2) Methods push and pop modify class state; top and empty inquire about the state. (3) Stack does not require any test environment of its own. (4) Class state HIDDEN from test, i.e., black box. Example -- Stack Class

48. Test Data File (tdi-stack.txt) File content: ----- 1 8 1 7 3 7 2 7 2 8 4 true ------ Note: No test environment setup. Methods: 1-push, 2-pop, 3-top, 4-empty Test Results File (trs-stack.txt) File content: ----- 1 1 8 2 1 7 3 3 7 8 4 2 7 8 5 2 8 7 6 4 1 ------ Note: Results file must be inspected for pass/fails. Fail Fail Fail Pass --- Push . --- Top should be 7. --- Pop, should be 8. --- Stack should be empty. Test Driver Files (Stack class)