Balancing Evaluation and Program Needs

1. Balancing Evaluation and Program Needs REAP Symposium 2009

2. Agenda • Introduction • EYA Organizational Overview • Performance and Quality Improvement Division • PQI Charge from the EYA Leadership Team • Method: Cross-functional Teams • Areas of Focus • Balanced Scorecard/ Key Performance Indicators • Master Quality Plan and Toolkit • Totally Integrated Electronic Record • Report Functionality • Integrating Processes • Q and A “It is one of the most beautiful compensations of this life that no person can sincerely try to help another without helping oneself.”   ~Ralph Waldo Emerson

3. A Proven History Pioneering “Firsts” • 1994: One of First Private Providers of High Risk Programming for Girls in Florida. • 2008: EYA Becomes Lead Agency Provider. • 1968: First provider of an outdoor therapeutic program in Florida. • 2001: Largest private provider of conditional release programming in Florida. • 1982: First Private Provider of State Training School in Southeast. • 1996: External Review of Data and Programs and Outcomes by Clemson and Harvard. • 2007: Achieves agency-wide COA accreditation. • 1991: Funded and Implemented Early Intervention Demonstration Project (Hi-Five). • 1969: First provider of an outdoor therapeutic program for Girls in the Southeast. • 1996: First Outdoor Therapeutic Programs to be Accredited by COA. • 1985: Funded/Implemented Demonstration Project for ReEntry in Florida. • 3 • January 27, 2009

4. Eckerd Youth Alternatives • Mission: Develop and share programs that promote the well-being of children and serve at-risk youth and their families. • Vision: Ensure that each child has the opportunity to succeed.

5. EYA Values • We base our concepts upon a belief in God and the uniqueness and inherent worth of each individual. • We believe that youth always come first. • We strive to assist youth in developing their potential to achieve balanced physical, intellectual, emotional, and spiritual strength. • We believe that youth and their families are our partners in treatment. • We seek to build a culturally diverse organization that honors the cultural diversity of the youth, families and communities we serve. • We embrace innovation, flexibility, reflection, and continuous improvement. • We thrive in a culture of learning, personal development and professional growth. • We value all members of our staff and strive to provide the highest level of care and support to each. • We hold ourselves accountable for achieving superior outcomes. • We strive for excellence in all of our endeavors.

6. Continuum of Services for Youth and Families

7. Areas of Focus QUALITY • Safety • Quality Assurance • Performance Improvement& Planning • Evidence-Based Practices & Outcomes STAFF • Targeted Recruitment • Professional Development • Retention FINANCIAL • Fiscal Strength & Discipline • Accurate Analysis & Forecasting • Capital Allocation & Cash Management EXTERNAL RELATIONS • Donor Cultivation • Communications & Publications • Alumni Relations GROWTH AND TRANSFORMATION • Mission-Driven Reinvestment • Business Viability Assessment • Marketplace Analysis • Targeted Retooling & Expansion • 8 • January 27, 2009

8. Performance & Quality Improvement As a performance-based organization, we clearly define, collect, validate, analyze and communicate our outcomes to make informed decisions based on evidence

9. Cross-functional Teams • Balanced Scorecard • Master Quality Improvement Plan with Toolkit • Data Collection (Totally Integrated Electronic Record) and Report Functionality

10. Balanced Scorecard Team • The Balance Scorecard™ is a performance management tool that tracks the most important goals and initiatives at an organization. It measures financial and operational activities to ensure that they are aligned with the organizational mission, vision and strategies. • Summary • Benefit • Ensures consistency and thoughtfullness in the methodologies used to measure outcomes and provides feedback to all levels of the organization. • Tasks • Prepare list of the different scorecards • Select the measurement criteria • Identify data source, instrument, collection, analysis, reporting for each measurement • Create communication tool • Effort: Medium • Impact: High

11. Balanced Scorecard Basics • Developed by Robert Kaplan and David Norton (1990) • Based on research exploring organizational performance measurement • Communication Tool • Measurement System • Strategic Management System

12. Balanced Scorecard Perspectives • Customer: Who are our customers? What do our customers expect or demand from us? What is our value proposition in serving them? • Internal Process: What must we excel at in order to continue to add value for our customers? What are the processes we have/need to best execute our strategies? • Employee Learning: What is the level/content of employee skills needed to meet our mission/value? What information systems needs to support these processes? What organizational climate (culture) supports this success? • Financial: What are the financial perspectives that would help us understand the effectiveness and efficiency of our financial processes?

13. Balanced Scorecard Perspectives

14. Performance Measure Criteria • Linked to strategy • Easily understood • Link in a chain of cause and effect • Updated frequently • Accessible • Measurable (averages, avoiding initiatives) • Is it driving the “right” behavior?

15. Evaluation Plan

16. Sample Evaluation Plan

17. Balanced Scorecard Examples • Customer: 1) Percentage of youth completing an EYA program increasing mental health functioning by at least 10% 2) Percentage of youth completing an EYA program reporting continued positive wellbeing impact (defined by education, mental health, behavioral, including recidivism, employment, and living status.) • Internal Process: 1) Number of instances of TIER being “off-line” during normal working hours 2) Cost of fundraising expenses incurred by EYA in a fiscal year will be at least $1/$1. • Employee Learning: 1) Voluntary employee turnover will be maintained at 50% or less. 2) Percentage of participants’ managers finding an increase in productivity, initiative, confidence, or measure directly related to the subject of the training. • Financial: 1) Overall administrative overhead is less than 10%. 2) Total revenue minus total expense is at least 3% of total revenue.

18. Master QI Plan Team • A comprehensive document that establishes organizational strategies for identifying critical areas, assessing, developing and implementing quality assurance objectives and procedures in a consistent and methodical way. • Summary • Allows us to deliver top quality care to our clients, provide excellent, reliable data and information to our internal and external customers and measures outcomes for continuous quality improvement. • Benefit • Tasks • Assess current systems and tools • Cross functional needs assessment • Define requirements, resources and deliverables • Implementation strategies that include training and ongoing support. • Effort: High • Impact: High

19. Data and Reporting Team • Several key tools are used to track and report the quality monitoring and response information, including PBViews and TIER computer systems. This initiative will re-deploy these systems and train the individuals who will be using the systems. • Summary • Benefit • Appropriate tools to capture and report large amounts of data is critical in the monitoring of a diverse group of programs in multiple locations. Having effective tools will minimize the time required by staff and leadership to report and analyze quality data. • Effort: Medium • Impact: Medium • Tasks • Deploy software with clear expectations on who and when data will be entered • Engage vendor to re-deploy TIER system • Create a training plan for all users of the different systems to ensure they are comfortable using the system

20. Summary Information Transactional Details Portal/Intranet Management Business Users Everyone

21. PBViews Sample BIRT/e.Spreadsheet BIRT Report Studio 18 Canned Reports Performance “View” Briefing Books Internet Publishing Maps

22. Annual Descriptive Summary

23. Youth Outcomes • Satisfaction Survey measures perceptions of accountability, consequences, education, empathy, group, home/transition, staff • 35% youth and 14% parent respondent rate – not in parity with EYA demographics • Respondents are only from residential programs – primarily NC • 64% of youth completing an EYA program reported positive satisfaction of services • 65% of parents of youth completing an EYA program reported that their child made positive life changes • 62% of parents reported that their relationship with their child improved

24. Youth Outcomes • WJII: Measures math (calculation, math fluency, applied problems) and reading (letter-word identification, reading fluency, passage comprehension) • Average length of stay ranges from 6.2-11.3 months • Youth receive a minimum of 250 instructional minutes of math and 250 instructional minutes of reading/week • 90% youth respondent rate (Pre-Post Test) • At intake, 53% were below grade level for reading and 79% were below grade level for math • 82% of youth completing an EYA program improved reading levels an average of 1.6 grade levels • 78% of youth completing an EYA program improved math levels an average of 1.2 grade levels

25. Youth Outcomes • Matson’s Evaluation of Social Skills for Youngsters • 78% youth respondent rate (Pre-Post Test) • Average length of stay ranges from 2.7-11.3 months • Demographics are in parity with EYA population • 42% of youth completing an EYA program improved social skills by an average of 15%. • Less than 20% of youth in ReEntry or community supervision programs scored in the normal range for “appropriate” social skills. In this population, aggression and overconfidence scored high as well. • More than 80% of you in private academies and day schools scored in the normal range for “appropriate” social skills. In this population, impulsivity scored high as well.

26. Youth Outcomes • ASEBA Youth Self Report measures mental health functioning • Average length of stay ranges from 2.7-11.3 months • 82% youth respondent rate (Pre-Post Test) • Demographics are in parity with EYA population • 47% of youth completing an EYA program scored in the normal range at the beginning of the program and 72% scored in the normal range at the end of the program • Other than the commonality of clinical scores in rule breaking and externalizing behaviors, youth varied greatly in other constructs

27. Integrating into Practice • Safety: evaluating frequency and areas of vulnerability, incident cost metrics, promoting effective strategies & impact, cross-functional teams, organizational safety program • QI: educating organization, formalizing processes and systems, creating communication strategies with accountability, updating policies/procedures, refining Internal Program Reviews • PI: fine tuning performance outcomes, evaluating current methodologies and solidifying data collection, web-enabling instruments, analyzing data, interpreting and reporting results, collaborative research and data sharing,

28. Internal Program Reviews Internal Review Process • 29 • January 27, 2009

29. Next Steps • Collaboration/Partnering with other providers and researchers • Sharing methodologies and approaches • Understanding the impact of outcomes and how it influences current issues and training • Disseminating information about programs effectiveness and innovations • Designing protocols based on best evidence • Identify model programs and replicating success

30. Q and A • Opportunity is missed by most because it is dressed in overalls and looks like work. ~Thomas Edison • Maurie Lung, LMHC, LMFT • Eckerd Youth Alternatives, Inc./Life Adventures Counseling & Consulting • PO Box 10907, St. Petersburg, FL 33733 • 727-481-4577 • dmlung@lifeadventurescc.org • Greg Holeman, Chief Quality Officer • Eckerd Youth Alternatives • 100 N. Starcrest Dr., Clearwater, FL 33765 • 727-461-1236 X437 • gholeman@eckerd.org