Measuring Regulatory Excellence: Key Performance Indicators

1. 2007 Annual Conference Measuring Regulatory Excellence: Key Performance Indicators Bruce G. Matthews, P.Eng. Professional Engineers Ontario

2. Overview • Project Definition and Scope • What is a Key Performance Indicator? • Project Process • Challenges • List of Potential KPIs • Survey re: Relevance and Importance • Recommended KPIs • Aftermath Council on Licensure, Enforcement and Regulation

3. Project Definition and Scope • Project arose from a Strategic Plan Objective: “Increase the public’s confidence in PEO as a regulator” • Associated Strategic Initiative: “Develop and implement key performance indicators and targets of regulatory effectiveness, benchmarked against other comparable regulatory bodies.” Council on Licensure, Enforcement and Regulation

4. Project Definition and Scope con’t • Literature review re: best practices • Review of other regulators’ performance measurement practices • Survey of stakeholders re: perceptions and expectations • Assess availability of data for ongoing tracking / reporting of KPIs Council on Licensure, Enforcement and Regulation

5. What is a Key Performance Indicator? • A metric used to reflect the critical success factors of an organization • Can be financial or non-financial • Nature of KPIs will differ depending upon the nature of an organization • A KPI is a key part of a measurable objectives (which includes a direction, KPI, benchmark, target, timeframe) Council on Licensure, Enforcement and Regulation

6. Effective KPIs… • Should help define: • The What – Are we doing the right things? What is our outcome in terms of impact and accomplishments? • The How – Are we doing things right? Do we have the right inputs in terms of resources and processes? Council on Licensure, Enforcement and Regulation

7. Project Process • Literature Review • Environmental Scan - Review performance measurement practices of other professional regulators • Identify potential KPIs • Survey stakeholders re: relevance and importance of potential KPIs • Review pros and cons of potential KPIs Council on Licensure, Enforcement and Regulation

8. Literature Review re: Best Practices • Performance measurement best practices • Balanced scorecard • UK National Consumer Council • “The Regulatory Craft” • “Red Tape to Smart Tape” International Conference (2002) Council on Licensure, Enforcement and Regulation

9. Balanced Scorecard • Developed by Harvard professors Kaplan and Norton • Four categories • Customers • Internal Business Processes • Financial • Organizational Capabilities • Weakness re: definition of “customer” Council on Licensure, Enforcement and Regulation

10. UK National Consumer Council • Three steps to credible regulation • Objectives • Content • Structure and Governance • www.ncc.org.uk Council on Licensure, Enforcement and Regulation

11. “The Regulatory Craft” • Book by Malcolm K. Sparrow (Brookings Institution Press, 2000) • Controlling risks, solving problems and managing compliance • Matrix of “Regulatory Competencies” and “Business Results” • Acknowledges “internal” versus “external” measures Council on Licensure, Enforcement and Regulation

12. Red Tape to Smart Tape Conference • Sponsored by the Ontario Government in association with the Institute of Public Administration of Canada • Core concept – Traditional business results measurements do not address public welfare issues and are not suitable for regulatory purposes • www.smarttape.ca Council on Licensure, Enforcement and Regulation

13. Environmental Scan • Other engineering regulators in Canada • Other professional regulatory bodies in Ontario and across Canada • Most report key transactional statistics (e.g., number of licence applications) • Little evidence of outcome measurement Council on Licensure, Enforcement and Regulation

14. Challenges • What does it mean to be an effective regulator? • Inability to benchmark due to lack of reporting by other “comparable regulatory bodies” • Can’t use measures typical for professional associations • KPIs must be evaluated in context Council on Licensure, Enforcement and Regulation

15. List of Potential KPIs • Competence programs • Intrusions on regulatory mandate • Complaints about licence holders or unlicensed individuals • Licensing turnaround times • Compliance action statistics • Council evaluation Council on Licensure, Enforcement and Regulation

16. List of Potential KPIs con’t • Fairness & dignity survey • Complaints about PEO • Employee engagement • Percentage of licence holders with no reported violations • Public confidence survey score • Ratio of administrative overhead to program delivery cost Council on Licensure, Enforcement and Regulation

17. Survey re: Relevance and Importance • PEO Council, Senior Management and selected external stakeholders • Relevance – the applicability of a KPI as a measure of PEO’s effectiveness as a regulator • Importance – the priority that should be given to a KPI as a measure of PEO’s effectiveness as a regulator Council on Licensure, Enforcement and Regulation

18. Survey re: Relevance and Importance • Survey also included proposed short-term measures for each KPI, intended to cover the introductory transitional period • Survey results were assessed in view of the best practices review • Need balance of internal and external measures) • Need to ensure key regulatory competencies are represented Council on Licensure, Enforcement and Regulation

19. Recommended KPIs • Competence programs • Complaints about licence holders or unlicensed individuals • Compliance action statistics • Employee engagement • Public confidence survey score Council on Licensure, Enforcement and Regulation

20. Competence Programs • Pros • Competent members reflect a well regulated profession • Competency is fundamental to professional regulation and easily understood by the public • Cons • PEO currently has no competence programs in place and none planned in the near term Council on Licensure, Enforcement and Regulation

21. Complaints about Licence Holders etc. • Pros • Easily measured (transaction statistics already reported) • Cons • Frivolous or vexatious complaints could skew the data • Does complaint volume really speak to regulatory effectiveness? Council on Licensure, Enforcement and Regulation

22. Compliance Action Statistics • Pros • Clear statement of enforcement activity • Some statistics already reported • Cons • Need to define what qualifies as a “compliance action” • Of greater interest to licence holders than to the public Council on Licensure, Enforcement and Regulation

23. Employee Engagement • Pros • Regular survey of employees already in place • Survey is widely recognized and used • Cons • Is PEO willing to take action in deficient areas? • Are the survey areas relevant to a professional regulatory organization? Council on Licensure, Enforcement and Regulation

24. Public Confidence Survey Score • Pros • Solid external measure of how PEO is perceived • Survey would be conducted by an independent research firm • Cons • Frequency of measurement • Who is “the public”? Council on Licensure, Enforcement and Regulation

25. Implementation Issues • Currently no competence programs in place at PEO • Measurement data exists for only three of the recommended KPIs • Public Confidence Survey still in development • Issues of KPI reporting (timing, frequency, format) still to be determined Council on Licensure, Enforcement and Regulation

26. Aftermath • Project report received and considered by PEO Council in November 2006 • Concern that recommended KPIs were all “lagging indicators” – there were no “leading indicators” • Concern that recommended KPIs would drive policy and program development • Concern about lack of competency programs Council on Licensure, Enforcement and Regulation

27. Aftermath con’t • PEO Council struck a task force to further examine the question of KPIs for PEO • Task force report (June 2007) recommends a new project with more research work re: critical success factors before defining KPIs • Status as of September 2007: awaiting new project charter / terms of reference Council on Licensure, Enforcement and Regulation

28. Speaker Contact Information Bruce G. Matthews, P.Eng. Manager, Complaints & Discipline Professional Engineers Ontario 25 Sheppard Avenue West, Suite 1000 Toronto, Ontario M2N 6S9 T: 416-840-1076 F: 416-224-9974 E: bmatthews@peo.on.ca Council on Licensure, Enforcement and Regulation

29. 2007 Annual Conference Measuring Regulatory Excellence – A Self Audit Tool Focused on Entry to Practice Jan Robinson Registrar & CEO College of Physiotherapists of Ontario September 8, 2007

30. Interesting times • accountability • transparency • movement of internationally educated professionals • public perspective Council on Licensure, Enforcement and Regulation

31. Politics in Ontario Council on Licensure, Enforcement and Regulation

32. Ontario Regulators for Access (ORA) • Info Sharing • Best practices • Collaboration Council on Licensure, Enforcement and Regulation

33. “measures” ➪ government driven Council on Licensure, Enforcement and Regulation

34. report card to progress report Council on Licensure, Enforcement and Regulation

35. Key Factors Included • # of internationally trained applicants (annual comparisons) • top five source countries • types of license granted • project initiatives Focus on “10” Council on Licensure, Enforcement and Regulation

36. Thomson review Council on Licensure, Enforcement and Regulation

37. Key Recommendations • fair registration practice code • information for applicants • criteria for decisions • reasonable costs Council on Licensure, Enforcement and Regulation

38. alternative evidence for unattainable documents • training • third party review • self audits with reporting • appeal procedures Council on Licensure, Enforcement and Regulation

39. 2002 – ISO 17024 General requirements for bodies operating certification schemes for persons Council on Licensure, Enforcement and Regulation

40. ANSI SCC Council on Licensure, Enforcement and Regulation

41. 2006 Fair Access to Regulated Professions Act Council on Licensure, Enforcement and Regulation

42. Key features ➪ access centre ➪ Fairness Commissioner ➪ audit Council on Licensure, Enforcement and Regulation

43. Consider self audit tool • gap analysis • demonstrate commitment • continuous improvement • achieve best practice Council on Licensure, Enforcement and Regulation

44. Focus on entry to practice Council on Licensure, Enforcement and Regulation

45. Conversion to standards 38 – registration 10 - organization Council on Licensure, Enforcement and Regulation

46. Examples • The College has policies and procedures related to confidentiality and privacy • The College has clear processes for reviewing, evaluating and making changes to registration requirements Council on Licensure, Enforcement and Regulation

47. Examples • The College has uniform competence requirements • The College ensures that individuals with a conflict of interest are not involved in the decision-making process related to an applicant. Council on Licensure, Enforcement and Regulation

48. Format • standard definition • evidence • analysis • recommendation • ISO reference Council on Licensure, Enforcement and Regulation

49. Key Findings • identified no major gaps or concerns • links between College and third party hard to evaluate • Identified new areas for policy work activity Council on Licensure, Enforcement and Regulation

50. identified areas where evidence weak • tedious • how often necessary? Council on Licensure, Enforcement and Regulation