Test Process Improvement

1. Test Process Improvement The Model – Audit Methods – Lessons Learned Ver. 1.2 1

2. My Experience in Process Audit CIS Technologies HTE London Bridge Intersil Duval County School Board Parts.com TransTegrity • ISO 9000 training – TickIT Lead Auditor • Process audits and improvements • Building test teams & configuration management Ver. 1.2 3

3. Assessing & Auditing • The Hunt for Facts • Face Time Techniques • The TPI Framework • TPI Scoring • On Stage Reporting Ver. 1.2 5

4. The Steve Irwin Model In search of the phenomenon. Ver. 1.2 6

5. Assessing & Auditing • Self Assessment – assess your own group • Internal Assessment – assess another COF group • Formal Assessment – assessment by outside entity Ver. 1.2 7

6. Assessing & Auditing Internal versus External Audit • Fluid definitions, relative terms • External – outside the organization Today’s discussion addresses both. Ver. 1.2 8

7. Assessing & Auditing • Personal Observation • Documentation • Interviews Ver. 1.2 10

8. The Tom Brokaw Model Separate Fact From Perception & Feelings. Ver. 1.2 11

9. Audit Interviews? Why should self assessment involve interviews? • One viewpoint is not as robust. • “Reading between the lines” Compliance is through consistent, uniform practice! Ver. 1.2 12

10. What does a successfulaudit interview look like? ? Ver. 1.2 14

11. Audit Interview What things are done • Exchange information – auditor about the audit, audited about the content • Set boundaries – topics, interruptions, time-consuming fact hunts • Respect the audited, be cordial • Stay on the topic, with minor casual exceptions Ver. 1.2 15

12. Audit Interview • Develop working hypotheses. • Prove right or wrong. • Validate. • Follow-up with questions & calls. Ver. 1.2 16

13. The Sherlock Holmes Model Solve the mystery. Ver. 1.2 17

14. Audit Interview What shouldn’t happen • Lots of casual unrelated conversations • Educating the audited extensively on procedures, TPI • Stating conclusions as auditor Ver. 1.2 18

15. Audit Interview Feelings • Auditing is all about listening! People like to be heard. • Defensiveness can occur. Ver. 1.2 19

16. The Sigmund Freud Model What personal experiences has brought this person(/team) to this point? Ver. 1.2 20

17. Audit Interview Outputs: • Set of notes – absolutely necessary! • Other pertinent contacts – full names & roles • New documents & their locations • Action items – both sides Ver. 1.2 21

18. TPI as an Audit Model • Audit for Compliance Findings – benchmarking, not strict adherence to regulations & policies • Make practical recommendations to improve work life • CapitalOne culture readily accepts Ver. 1.2 23

19. Interview Techniques Ver. 1.2 30

20. Interviews – Etiquette & Personalities • Pre-interview – Provide an agenda & background material. • Dress for success. • Be punctual. • Address with respect. • Frame open-ended questions with charity. • Avoid confrontational questioning. Ver. 1.2 32

21. Interviews – Putting the Interviewee at Ease • Agenda spelled out. • Use appropriate humor. • Listen Actively. Ver. 1.2 33

22. Interview Difficulties Misunderstandings sometimes occur. • Listen carefully & actively. • Address concerns fully & repeat for feedback. • Stay grounded in facts. Ver. 1.2 36

23. Validation of Responses • Review and complete your notes soon after the interview. • Refer to documentation. • Chat informally with others in the group. • Set up additional interviews. • Phone or stop by for follow-up questions. Ver. 1.2 37

24. Pacing the Assessment • Two weeks includes interviews, research, and reporting. • Total effort should be less than 40 hours. Don’t forget the delay for scheduling! Schedule all interviews and the end report immediately! Ver. 1.2 38

25. The Stephen Covey Model Begin with the End in Mind. First Things First. Think Win-Win. Ver. 1.2 39

26. Scoring the Assessment Ver. 1.2 40

27. Develop the Score Ver. 1.2 42

28. Develop the Score KA 1.A.1 “A motivated consideration of the product risks takes place, for which knowledge of the system, its use and its operational management is required.” Question: “What is your test strategy, and how did you formulate it?” Ver. 1.2 43

29. Develop the Score Develop this on your own, as needed! Ver. 1.2 44

30. Develop the Score Ver. 1.2 45

31. Develop the Score Ver. 1.2 46

32. Develop the Score Ver. 1.2 47

33. Develop the Score Ver. 1.2 48

34. Develop the Score • Y,N • X or blank = don’t know • Any X, blank or N answers = “no pass for level” • Only all Y answers = pass Ver. 1.2 49

35. Develop the Score Ver. 1.2 50

36. Develop the Score Ver. 1.2 51

37. Develop the Score Ver. 1.2 52

38. Develop the Score Ver. 1.2 53

39. Reporting Findings Ver. 1.2 54

40. The Peter Senge Model Systems Thinking – People respond to the systems in which they are immersed. Ver. 1.2 55

41. Finalizing the Presentation • Remember your audience. • Should be a mere formality. • Tell them more than they know. • Provide actionable plan Ver. 1.2 58

42. Reporting Findings • Management team receives report • Written as well as verbal presentation • No surprises Ver. 1.2 61

43. Presentation – Focused Clarity One page summary • Key Area map • Brief explanations only where necessary Ver. 1.2 66

44. The Follow-Up • Provide further information sharing as required. • Avoid betraying trusts. • Provide engagement planning. Ver. 1.2 69

45. The Take-Home What’s the take-home? ? Ver. 1.2 75

47. Contents Agenda TPI Assessor’s Tools Chapter Selections Exercises References The 4Q03 Plan Ver. 1.2 77 TPI Assessments

48. The Successful Audit CapitalOne Environment LB Audit Experiences Interview Techniques Scoring the Assessment Reporting Findings Report Presentation Postscriptum About TPI Chapter Selections Ver. 1.2 78 TPI Assessments

49. Exercises The TPI Interview Scoring TPI Results The TPI Results Presentation Ver. 1.2 79 TPI Assessments