Developing Effective Performance Measurements

1. Developing Effective Performance Measurements

2. Begin to Think Differently About Performance Measurement Using Why You Need to Measure How You Can Measure with a Purpose What Risks You Need to Consider Introduction- The Theme

3. The Red Bead Experiment: Is an interactive exercise to expose everyone to the advantages of a systems approach to measuring It will demonstrate that traditional management practices will not affect or produce the desired outcome Emphasize the importance of top management decisions to ensuring safety as an outcome Show how statistical analysis can identify inherent variation Introduction- Agenda

4. How to link statistical analysis and performance measurement Overview of sequential steps Identifying your organizational risks Recognize Leading and Lagging performance indicators Making the links to your management System, goals and objectives Introduction- Agenda

5. Some Real-Life Examples from our aviation community Focussing on examples of analysis (accident rates or hazard frequencies) Counting and trending-Power of control charting Using Statistical control to recognize opportunities for continuous improvement Introduction- Agenda

6. Interactive Exercise using a Process Example Aligning your Performance Measures to Your Risks How it fits into the mgmt review When do you measure What are your risks How will you change what you do How do you monitor How do you feedback Introduction- Agenda

7. Summary and Closure Discuss the Main Take-Aways from the session Introduction- Agenda

8. Deming’s Red Bead Experiment: With a Twist

9. American Statistician – best known in Japan Taught Management in Post-War Japan to Improve Design and Quality using Stats The Red Bead Experiment A Teaching Tool Historically used to demonstrate quality Management System principals Can this be applied to Safety? Who is Edward Deming?

10. 6 Volunteers! 4 “Willing” Workers 2 Safety Inspectors What we Need:

11. TO PRODUCE WHITE BEADS, SAFELY! OUR MISSION

12. The “Production” of each White Bead follows a Process Processes inherently have “risks” (or Red Beads) that can impact on Safety The White Beads can be a metaphor for anything – they are an output (Flight to X, Repair of Y, Painting of Z, Storage of W, etc) THE EXPERIMENT:

13. OBJECTIVE:Produce Something (White Bead), following a pre-determined Process, safely (ie. Without Red Beads) THE EXPERIMENT:

14. Each “willing worker,” dips the paddle into the box of beads for his/her first day of production Safety inspectors identify and count the number of risks in “Producing” Chief Safety Inspector of the company announces the result, records and dismisses TRAINING

15. PRODUCTION STAGE 1

16. Production Standard: No more than 3 Red Beads today per worker! PRODUCTION STAGE 2

17. AWARDS CEREMONY

18. Motivation! PRODUCTION STAGE 3

19. PRODUCTION STAGE 4

20. Chart THE RESULTS

21. The system is at fault, not the “Workers” Management must change the system Performance Measures need to look at the system and process, not just at the people WHAT DOES THIS TELL US

22. It's the system, not the workers. If you want to improve performance, you must work on the system. WHAT MESSAGES ARE IN THIS EXPERIMENT?

23. Safety is made at the top. Safety is an outcome of the system. Top management owns the system. WHAT MESSAGES ARE IN THIS EXPERIMENT?

24. Numerical goals and production standards can become meaningless. The number of red beads produced is determined by the process, not by the standard. WHAT MESSAGES ARE IN THIS EXPERIMENT?

25. Rewarding or punishing the Willing Workers had no effect on the outcome. Extrinsic motivation is not effective. WHAT MESSAGES ARE IN THIS EXPERIMENT?

26. Rigid and precise procedures are not sufficient to produce the desired Safety. WHAT MESSAGES ARE IN THIS EXPERIMENT?

27. People are not always the dominant source of variability . WHAT MESSAGES ARE IN THIS EXPERIMENT?

28. Slogans, Exhortations and Posters Are Not Always Useful To The Willing Worker. WHAT MESSAGES ARE IN THIS EXPERIMENT?

29. Be Cautious about misusing performance data > Needs to focus on the process IN SUMMARY

30. WHEN YOU NEED A FUNDAMENTAL CHANGE TO YOUR SYSTEM: HOW CAN YOU USE STATISTICAL ANALYSIS TO SHOW IF YOU ARE ACHIEVING WHAT YOU WANT? NOW WHAT?

31. BREAK (15 Minutes)

32. Linking Performance Measures & Statistical Analysis

33. 1.0Safety Management Plan 1.1 Safety Policy 1.2 Non-punitive Policy 1.3 Roles, Responsibilities & Employees 1.4 Communication 1.5 Safety Planning – Objectives & Goals 1.6 Performance Measurement 1.7 Management Review 2.0 Document Management 2.1 Identification & Maintenance of Regulations 2.2 SMS Documentation 2.3 Records Management 3.0 Safety Oversight 3.1 Reactive Processes 3.2 Proactive Processes 3.3 Investigation and Analysis 3.4 Risk Management 4.0 Training 4.1 Training, Awareness & Competence 5.0 Quality Assurance 5.1 Operational Quality Assurance 6.0 Emergency Preparedness 6.1 Emergency Preparedness and Response Appendix A

34. SMS Operational Measurement Elements 1.0Safety Management Plan 1.1 Safety Policy 1.5 Safety Planning – Objectives & Goals 1.6 Performance Measurement 1.7 Management Review 3.0 Safety Oversight 3.1 Reactive Processes 3.2 Proactive Processes 3.3 Investigation and Analysis 3.4 Risk Management 5.0 Quality Assurance 5.1 Operational Quality Assurance

35. Essential SMS Measurement Elements 1.0Safety Management Plan 1.1 Safety Policy 1.5 Safety Planning – Objectives & Goals 1.6 Performance Measurement 1.7 Management Review 3.0 Safety Oversight 3.1 Reactive Processes 3.2 Proactive Processes 3.3 Investigation and Analysis 3.4 Risk Management

36. Where to look to find measurable • Safety policy objectives • Process purpose statement • Management decision duties • Hazard registry – risks • Control graphs • Aviation publications • Safety case • Industry associations • Intuition • Feedback

37. SP Objectives Examples 1. Ensure that aviation safety is our number one priority. 2. Document, implement and maintain and effective SMS 3. Meet or exceed all applicable regulatory requirements. 4. Establish and sustain a continual improvement philosophy. 5. Encourage employee participation and support their efforts. 6. Reduce and eradicate the real and potential causes of incidents or accidents. 7. Identify and control all known and potential hazards to aviation safety. 8. Implement a non-punitive policy for the security of all employees.

38. Purpose Measurable • On-time scheduling • Qty of recalls • Qty of emergency repairs • Duration of repairs • Spare parts inventory value • Reported hazards 1) To define the method and process to be used for the scheduling and implementing of aircraft maintenance services to ensure consistent quality and safety of the service. • Preventive actions • Qty of reported hazards • CADORS reduction • Incidents / accident rate • Positive impacts on processes • Cost reductions • Improved margins • Efficiency 2) To establish a method that allows for continual improvement, including preventive actions, throughout all aspects of the organization that have an affect on quality, safety, service and price

39. Purpose Measurable • Qty of reports • Cost of damages • Down time for A/C • Types of damage – how • Types of maintenance • Competency • Procedure impacts To review/analyze reports and provide feedback to maintenance managers so that they can make change to procedures/training to reduce damage to aircraft by employees conducting maintenance. This above example is an actual purpose statement currently in use today

40. Example of an actual measurement practiced in industry to reduce or eliminate hanger rash Purpose Statement: To review/analyze reports and provide feedback to maintenance managers so that they can make change to procedures/training to reduce damage to aircraft by employees conducting maintenance.

41. Portion of the Excel spreadsheet and the capturing of data

42. Captured data displayed in a bar graph

43. Action Taken 1st Qtr -09 • Observations • corrective actions to employee filed reports may not be effective. • potential trend existed • Corrective Actions • mined reports from the previous years and plotted them in the graph • specific training was provided • heightened awareness

44. Action Taken 4th Qtr -09 • Observations • Were reports involving damage while transitioning around aircraft on an elevating device? • Is the use of spotters effective • Actions • 4) Training department will review the “Working at Height” presentation

45. Action Taken • Actions • The working at height training presentation has been amended to better focus on clearance and the use of spotters • Center console covers have been completed for each type of aircraft • The training department will modify the human factors training program to more effectively provide a clear and concise message to employees. • Comment; Seven (7) CAPs / changes were discussed, reviewed and implemented to control and reduce damage to aircraft

46. Reports Not Usually Submitted Trend line beginning in year 2006 – 1st Qtr

47. Trend line beginning in year 2008 – 1st Qtr

48. Exercise

49. List several safety critical processes Securing Cargo Loading and securing of cargo Identify and control all known and potential hazards to aviation safety. Number of events Log the qty. of of incidents 10% reduction Per quarter

50. 1) Identify a process(s) Safety critical processes 2) Identify a measureable for each identify KPI identify continual improvementitem and unit of measure 3) Note the unit of measure How and what will you measure to demonstrate improvement 4) Establish a Target Can be a percent increase / decrease, SizeNbr of pcsQtyLbs 5) Link processes to objectives Note an objective from earlier slide