Evaluating Business Impact in Massachusetts Healthcare Sector Strategies

1. Massachusetts Sector Strategies EvaluationExperience Measuring Business Impact in Healthcare For NGA Sector Strategy Evaluation Call June 19, 2009 Navjeet Singh nsingh@commcorp.org 617-727-8158

2. Purpose & Agenda Purpose • To share the Massachusetts experience with measuring Business Impact in Sector Strategy Projects especially in the Healthcare Sector Agenda • Sector Strategy Initiatives in Massachusetts • Evaluation of Sector Strategy Initiatives in Massachusetts • ECCLI –Measuring Business Impact • BEST, BayStateWorks, WCTF –Measuring Business Impact –an evolving approach • Lessons learned

3. Sector Strategy Initiatives in Massachusetts Single Sector Targeted Initiative • Extended Care Career Ladder Initiative (ECCLI) • Targeting Nursing Homes and Home Health Care • 2000 – Present • About $10 million since 2000 Multiple Sector Initiatives • BEST • BayStateWorks • Workforce Competitiveness Trust Fund (WCTF) 2007 • Currently over 30 projects underway, 14 in Healthcare including 9 implementation grants

4. Evaluation of Sector Strategy Initiatives

5. Evaluation and Data Collection Methods Used

6. ECCLI –Business Impact MeasuresData collected from projects • Data collected from projects • Turnover or retention of Certified Nurse Assistants • Vacancy rates • Agency fee (high cost of contract workers hired through an ‘agency’) • Overtime cost • Advantages • Data available readily • Reported consistently for each project • Disadvantages • Self-reported data • Internal consistency errors • No comparison or counterfactual (compared with Industry averages)

7. ECCLI Evaluation Findings 2000-2003: Agency Fees Using data as reported by projects and long-term care facilities in six monthly reports: • For all early rounds agency fees declined • For Round 1 Agency fee generally remained flat

8. ECCLI Evaluation Findings 2000-2003:Quality of Care Indicators When compared to 161 similar homes not in ECCLI (the ‘comparison’ group), among the 69 nursing homes with ECCLI… A resident in an ECCLI facility was likely to have 12% fewer cases of per year exhibiting worsening behavioral symptoms • Tested other Quality of Care indicators also – only one with statistically significant result

9. ECCLI Evaluation Findings 2000-2003:Costs --CNA Purchased Services When compared to 161 similar homes not in ECCLI (the ‘comparison’ group), among the 69 nursing homes with ECCLI…using cost data submitted to Department of Health for Medicare/Medicaid reimbursement • Nursing Homes with ECCLI were likely to spend about $0.05 per patient day less than those with no ECCLI • Nursing homes who trained more employees had higher savings

10. ECCLI Qualitative Evaluation Findings, 2005-2007 Positive outcomes associated with ECCLI-funded trainings and programs included changes in organizational practice, as well as improvements in resident and client outcomes. Strongest outcomes areas:  Improved & Increased: • Clinical Skills • Staff Retention & Recruitment • Wages & Benefits • Quality of care and quality of life outcomes for residents/clients • Overall Organizational Practice Changes • More Teamwork • Better Communication • Increased Respect & Self-Confidence (among CNAs and HHAs) • Permanent instead of rotating resident/client assignments • Increased job autonomy & responsibility for direct care staff • Increased direct care staff participation in resident care planning

11. ECCLI Qualitative Evaluation Findings-2 • Most organizations reported some organizational changes – such as implementation of permanent assignments, increased flexibility and flattened staff hierarchy. • Some family members, nursing home residents, and home health agency clients interviewed noted recent improvements in CNA/HHA language proficiency and clinical skills, as well as increased resident/client autonomy and inclusion in care decisions. Sustainability and Systems Change: •  One of the three sites that no longer received funding continued its ECCLI activities by tailoring the training more to their specific organizational needs and absorbing the costs into their operating budget.

12. BayStateWorks –Business Impact Measurement Highlights • Reporting on Business Impacts was voluntary • CommCorp staff met with projects; technical assistance through contractors • 11 projects reported business impact measurements Lessons Learned • Project coordinators & program staff • Not comfortable with measuring business impact • Often not well engaged with employers • Working with contractors meant that there was very limited gain of knowledge and experience for CommCorp staff • Most employers involved in projects were small, lacked culture of managing by data

13. BayStateWorks: Selected Business Impact Measures in Healthcare • ‘Traditional’ measures’(incumbent worker measures) • Employment retention • Promotion/advancement rates • Wage increases • Recruitment related measures • Time to fill openings • Qualified candidates as a percentage of all candidates • Orientation time required to acclimate new hire • New hires from among trainees (compared to new hires from non-trainees) • Incumbent worker measures • No. of safety incidents (in unit) • Customer satisfaction scores, improvement in • Reduction in health and safety scores

14. Workforce Competitiveness Trust Fund (WCTF): Measuring Business Impact Highlights • Measuring business impacts required in proposal • All projects informed about key concepts at the beginning • Technical assistance offered through CommCorp research staff • using simple tools, planning templates • telephone calls, visits • presentations and discussions at project coordinator meetings • Plan to do 2-3 case studies to demonstrate how selected employers measured business impact Lessons learned: • Many project coordinators rose to the occasion – developed measures or surveys specific to their projects • Healthcare projects – usually focus on addressing labor shortages –key measure is ‘cost of vacancy’

15. WCTF: Selected Business Impact Measures in Healthcare • ‘Traditional‘ measures’ (incumbent worker measures) • Employment retention • Promotion/advancement rates • Wage increases • Increase in diversity • Recruitment related measures • Time to ‘competency’ • Qualified candidates as a percentage of all candidates • Cost of orientation (decrease) • Incumbent worker measures • Employee/supervisor satisfaction • Patient/ customer satisfaction

16. Conclusions and Lessons Learned-1 • Ideally measuring business impact is part of a more thoughtful and rigorous process of designing and implementing a workforce development project that includes early engagement with employers about what problem they intend/hope to address through the project and changes expected as a result of the intervention • Project coordinators (who implement the project) and the applicants (who design the project and seek funding) are usually different individuals. The project coordinators may not • Feel empowered to engage with the employer and ask questions about the underlying assumptions about the program • Have a firm grasp on Business Impact • Be knowledgeable about the industry sector, practices, etc. • The employer representative to the project partnership may not have deep knowledge about the business operations and may have to seek the engagement of others

17. Conclusions and Lessons Learned -2 • The employer may not be collecting a relevant data and may view the request as burdensome • The employer may feel the data is proprietary and confidential • It may be necessary to get data on application of learned skills (Level 3) and then derive business impact (level 4) • For a single industry it may be possible to use administrative data, industry data or industry association data which allows for comparisons/counterfactuals (ECCLI) • This is difficult! Even large companies struggle with this! • As a result much more extensive Technical Assistance needed: • Deeper engagement from Commcorp staff to assist projects and coordinators • Staff capacity and experience • Availability of more examples from different industries • Availability of more tools and templates, e.g., surveys