CORE Evidence-/based Regulation: A Regulatory Performance Measurement System

1. COREEvidence-/based Regulation: A Regulatory Performance Measurement System

2. Goal To help State Boards of Nursing improve their management and service delivery

3. Need for Study • Multiple stakeholders were demanding accountability • Lack of clarity among stakeholders about Board roles and responsibilities • Trend toward outcome measurement at state level

4. Purpose of the Study Establishment of a Performance Measurement System • Incorporating data from internal and external sources • Using benchmarking strategies • Identifying best practices

5. History In 1998, the NCSBN Board of Directors appointed a project advisory group to provide oversight and guide development of a performance measurement system that incorporates data collection from internal and external sources, identification of best practices, and the use of benchmarking strategies.

6. History Twelve dedicated member boards of nursing (Kentucky, Louisiana-RN, Maryland, Missouri, Nebraska, New Mexico, North Carolina, North Dakota, Ohio, Tennessee, Texas-RN and West Virginia-PN) contributed time and resources to the development of 10 instruments used to gather data in five areas: discipline, licensure, practice, education and governance.

7. Phase 1 of the Project • Validation of Board Roles Study conducted by The Urban Institute

8. Methods to Gather Information • Interview • Focus Groups • Individuals • Review of Secondary Documents

9. Results of Phase 1: • Roles Identified • Establish scope of practice for nurses • Issue licenses to qualified nurses • Assure continued competence • Investigate complaints and impose disciplinary sanctions as appropriate

10. Phase 2 of the Project • Validation of Board Roles • Identification of Performance Indicators

11. Results of Phase 2: • Technical Work Group developed • Performance indicators • Outcome Indicators • Output Indicators • Efficiency Indicators

12. Examples • Performance Indicator • Timeliness of complaint handling • Outcome Indicator • Average time for complaint resolution

13. Examples continued • Output Indicator • Number of complaints resolved in FY • Efficiency Indicator • Average cost per completed complaint

14. Phase 3 of the Project • Validation of Board Roles • Identification of Performance Indicators • Tool Development

15. Tool Development Process • Original tools developed and piloted • Because the science of regulation was not formed enough to guide their efforts, they did not know which data would prove to be the crucial evidence and so they included all data. • Original tools revised and further tools developed • 6 data collection tools for boards of nursing • Surveys to collect data from 6 stakeholder groups

16. Board Surveys • Discipline • Licensure • Education Program Approval • Practice • Governance (Executive Staff) • Governance (Board President)

17. Board Surveys Included • Processes used, e.g.: • Investigator caseloads • Use of site visits or self-reports for education programs • Timeliness issues, e.g.: • Days needed to processes license request

18. Board Surveys Included • Outcomes achieved, e.g.: • Number of discipline cases closed • Opinions, e.g.: • From Executive Staff and Board President

19. Stakeholder Groups • Licensed nurses • Health care employers • Nurses who had been the subjects of complaints • Persons who had lodged complaints • Nursing associations • Nursing education programs

20. Stakeholder Surveys Included • Perceptions of board’s • Timeliness, • Fairness, • Adequacy of regulation, etc. • Satisfaction with board’s • Communication with stakeholder group, • Nursing program approval process, etc.

21. Phase 4 of the Project • Validation of Board Roles • Identification of Performance Indicators • Tool Development • Data Collection

22. Data Collection • Stakeholder contact information submitted by boards • Random samples selected from those submitted • 6 data collection tools sent to boards of nursing

23. Data Collection Wisdom • Not all boards routinely collected the data asked for • Many boards used this as an opportunity to improve/modify amount and types of data collected • Language/definitions (i.e., financial data, board processes) differed among boards

24. Phase 5 of the Project • Validation of Board Roles • Identification of Performance Indicators • Tool development • Data Collections • Reports of Findings

25. Report Format • Aggregate findings • Data results • From board surveys • From stakeholder surveys • Relationships among variables • State-specific findings • Comparison of state with all states • Comparison of state with “like” boards

26. Example of Comparison with Aggregate Data, i.e., Ed. Program Perceptions

27. Comparisons with Similar Boards • Boards evidenced a wide variety of resources, structures & processes • Boards were compared to other boards similar in a number a variables

28. Comparison Variables • Size of staff • Staff assigned to specific functions • Numbers of investigators • Whether or not state mandated reporting of errors • Processes related to complaint review

29. Comparison Variables continued • Board structure • Standard of proof • Staff autonomy • Number of board meeting per year • Timeliness of discipline processes • Timeliness of licensure processes

30. Comparisons of Sample Board with Other Boards with Similar Numbers of Staff

31. Phase 6 of the Project • Validation of Board Roles • Identification of Performance Indicators • Tool development • Data Collections • Reports of Findings • Search for “best practices”

32. The Search for Best Practices • Data were used to identify boards with consistently high ratings in • Outputs • Effectiveness • Ratings were explored in 5 functional areas • Discipline, licensure, education program approval, practice and governance

33. The Search for Best Practices • Selected boards were interviewed to discover • Practices common among boards with consistently high ratings • Differences from boards with lower ratings

34. Discipline Best Practices • Boards with the highest ratings on discipline outcomes • Delegated authority to board staff • Communicated well with stakeholders • Hired investigators and attorneys & actively managed discipline process • Trained and mentored investigative staff • Applied discipline sanctions consistently

35. Licensure Best Practices • Boards with the highest ratings on licensure outcomes • Secured essential human and other resources • Made an aggressive commitment to customer service

36. Education Best Practices • Boards with the highest ratings on education outcomes • Provided consultative, as well as evaluative services to education programs • Took a leadership role in establishing congruence between education and regulation

37. Practice Best Practices • Boards with the highest ratings on practice outcomes • Facilitated understanding of legal scope of practice • Made an aggressive commitment to customer service • Established a high level of involvement with the statewide nursing community • Delegated authority to board staff

38. Governance Best Practices • Boards with the highest ratings on governance outcomes • Promoted an understanding of the respective roles of staff and board members • Built an effective working relationship and a high level of trust between board and staff • Facilitated an effective working relationship among board members • Demonstrated a commitment to board member development

39. Phase 7 of the Project • Validation of Board Roles • Identification of Performance Indicators • Tool development • Data Collections • Reports of Findings • Search for “best practices” • Development of Ongoing System of Performance Measurement

40. CORE • Nursing Boards educated on CORE • Manuals prepared, distributed and explained • Ongoing presentations and publications • Best Practice “Tool Kit” • Submissions by boards of systems and processes that have facilitated best practice

41. CORE • Ongoing improvement of data collection system • All survey items linked to outcome and/or best practice • Data collection streamlined • Additional tools created and piloted • Information Technology • Finance • Board Member

42. CORE Timeline 2006 1998 1999 2000 2001 2002 2003 2004 2005 Advisory Committee Formed Stakeholder Surveys Board Surveys Revised Questionnaires Stakeholder Surveys Urban Institute Study

43. Jan Jan Feb Feb March March April April May May June June July July Aug Aug Sept Sept Oct Oct Nov Nov Dec Dec Schedule Conduct Stakeholder Surveys X X X X Analyze Data X Report Results in Aggregate & by State Conduct Board Surveys X X X X Analyze Data Report Results in Aggregate & by State X

44. Persons Who Made a Complaint About a Nurse Nurses Subject of a Complaint Education Programs Nurses Associations Employers Board Participation2006 Stakeholder Surveys 30 24 17 20 25 26 Total = 33 different boards participated in 2006 Stakeholder Surveys

45. Utilization of Data by Boards of Nursing Data has been used to: • Support decision-making • Develop mandated reports • Provide information to legislators • Change data management processes • Improve stakeholder satisfaction • Streamline processes • Determine priorities

46. Benchmarking • "How are we doing?" • "Are we tracking the right measures?" • "How do we compare with others?" • "Are we making progress fast enough?" • "Are we using the best practices?"

47. CORE Performance Benchmarking Process Benchmarking Identifying best practices Adopting best practices

48. Performance benchmarking The collection of (generally numerical) performance information and making comparisons with other compatible organizations.

49. Benchmarking of Board Performance Quality Efficiency (cost effectiveness & timeliness) Stakeholder survey Stakeholder survey Board survey (?) Board survey

50. Criteria of Item Selection • Measurable • Comparable between states • Objective, if possible