Further Adventures in Session-based Testing

1. Further Adventures in Session-based Testing James Lyndsay Workroom Productions jdl@workroom-productions.com www.workroom-productions.com

2. ? What?

3. Session-Based Testing? • Manages and controls unscripted testing • Limited Duration • Directed Exploration • Recorded Activities • Supports ‘Agile’ Process • Allows quick metrics • Has wider uses

4. Adventures? • Two case studies • Permanent team - • 20 coders, 4 testers • Inexperienced and under-motivated • Internet-based product, ASP model • Part-time team • 2-6 coders, 0-3 testers bought in from pool of 6-8 • No permanent test team • Windows client / server product around known application • Shared business context • Rapid change • Commercial constraints; budget, time-to-market

5. Further? • Paper - Adventures in Session-based Testing • Contains detailed methods • Contains problems and (some) solutions • Now a year old • STARWest 2002 -Further Adventures . . . • Methods • Tools • Training, Coaching

6. Strategy • Stay within business constraints • Deal with change rather than enforce stasis • Rapidly transfer knowledge to allow independence • Four goals • Control the scope • Control the work • Measure risk and coverage • Promote ongoing improvement

7. 1 Methods

8. Methods: Overview • Control and recording • Control Scope • Control Work • Recording a Session • Assessment and Measurement • Assessing Risk • Assessing Coverage • Infrastructure • Real-time Metrics • Social techniques

9. Example: Test Points (Excel-based tool)

10. Methods: Control • Scope - made of Test Points • Test Point - piece of work • Risk, Estimated cost • Four groups • New tests • Regression tests • Retests for fixes • Investigatory tests • Work done - made of Test Sessions • Test Session - piece of time • Actual cost, Coverage

11. Example: A Test Session

12. Methods: Control • Recording a Test Session • Plan: Approach, Scope, Expectations • Record: Conditions, Actions, Observations • Bug details • Actual Cost • How done are we?  Coverage • Allows • Review, coaching • Forgetting • Retrospectives

13. Methods: Assessment • Risk • Coverage • Ask the Experts - the testers! • How done are you? How much is left? • Collect many subjective  few objective assessments • Overall figures can go down as well as up • Ongoing learning - estimates improved

14. Example: Metrics (Excel-based tool)

15. Methods: Support • Real-time metrics • Immediate feedback • State of testing, not of system • Fast, frequent, informed re-planning • Soft skills/Social Techniques • Experimental Approach • Ongoing learning • Communication • Empowerment

16. 2 Tools

17. Tools • Freely available - under GPL • Excel spreadsheet • Single-user, single location • Download from: • Web: PHP/MySQL • Multiple user • Download from:

18. 3 Training and coaching

19. Ongoing learning and development • Training - from unskilled to skilled • Skilled involvement - pair expert / novice • Show - practice - challenge • Careful choice of sessions • Coaching - from good to great! • Peer involvement, neutral assessment • Challenge - innovate • Discussion, metrics, personal investigation

20. Supporting the learning process • Session-based testing • Introduces discipline and framework • Helps overlap training and testing • Helps match challenge to individual • Experts, managers - get involved! • Sometimes compromise: • Good bugs found quickly, or • Better testers long-term • Feedback is integral • Before testing - techniques, planning, charter • After testing - session log, bug log • Stats and metrics - not just for management

21. Encouraging the team • Involve the test team • Allocate responsibility / ownership for tests • Allow direct influence on clear, open metrics • Relate metrics to business decisions • Share successes, failures • Reward discovery • Discourage blind faith, boredom • Step away - look at the wider picture

22. Changes: Case Study 1 • Active learning process is encouraged • Feedback has improved estimating skills • Session review has improved test techniques • Test Point ownership has improved planning and scoping skills • Weekly 2-hour self-training session as a team • Greater interest in testing, greater motivation • Useful, visible, up-to-date metrics • Increase openness, reduce surprise, blame • Joint ownership of decisions, successes, failures

23. ! Conclusion

24. Conclusion • Communication • Sessions gave us the tools to communicate • Better communication improved trust and effectiveness • Ongoing improvement • Sessions allow directed and disciplined approach • Feedback improved techniques and estimation • Empowerment increased morale and confidence • Openness • All figures available, all of the time, always current • Encouraged trust and interest • Allowed neutral feedback

25. Conclusion • Test sessions provide: • Control of unscripted testing • Framework for ongoing improvement • Overall lessons • Simple measures are the best • Favour effective communication and involvement over knee-jerk documentation • Unobtrusive, immediate metrics allow real-time control and feedback

26. More • Further details: • Paper, updates, references etc. • www.workroom-productions.com • Contact: • James Lyndsay • Jdl@workroom-productions.com