Program Performance: Using Management Tools and Evaluation Data

1. Program Performance: Using Management Tools and Evaluation Data Assessing Performance to Drive Change Jean L. Lloyd US Administration on Aging 4th State Units on Aging Nutritionists/Administrators Conference August, 2006

2. Overview • Panel • Jean Lloyd • Bob Hornyak • Nancy Wellman • Shannon Dukes

3. Objectives • Describe context for AoA assessment of program performance • Describe AoA program performance components • Demonstrate how to use program performance data • Identify resources available from AoA

4. AoA Mission “Help elderly individuals maintain their independence and dignity in their homes and communities through integrated, high-quality, cost-effective systems of home and community-based long-term care.”

5. Context • Modernizing the Older Americans Act • Empowering consumers • Providing greater choice for high-risk individuals • Encouraging healthy lifestyles • Enhancing the long term care connection within the aging network • Medicare Modernization Act • Government Performance and Results Act (GPRA) • Office of Management and Budget (OMB) Program Assessment Rating Tool (PART)

6. AoA Responsibility Assist the network in developing tools that will allow them to efficiently and effectively assess and evaluate program performance to ensure that the impact of our programs aligns with ourmission and enhances program performance.

7. To serve older adults more effectively & efficiently To comply with GPRA & PART To meet the OAA requirements To demonstrate that OAA programs work To justify budgets To satisfy key stakeholders To plan, manage and administer programs To identify training and technical assistance needs To identify participant, program, service trends and needs To develop programs to better meet needs To identify best practices to share and replicate To inform research Why Do We Measure Performance?

8. Process Impact AoA Integrated Program Evaluation Cost-Benefit Outcome AoA Practices Integrated Program Evaluation For true transparency, all four types of evaluations are important to inform the aging network, the Executive Branch, citizens, Congress, advocacy groups, the private sector and other stakeholders.

9. GPRA • Passed by Congress, 1993 • Requires federal agencies and programs to document results • Requires federal agencies and programs to improve performance based on results • Strategic Plan http://www.aoa.gov/about/strategic/strategic.asp • Annual Performance Plans that accompany the budget justification http://www.aoa.gov/about/legbudg/performance/legbudg_performance.asp

10. Core Outcome Measures • Improve program efficiency (NAPIS/SPR,) • Improve client outcomes and quality (National Surveys, Performance Outcome Measurement Project, Program Evaluation) • Improve targeting (NAPIS/SPR, National Surveys, Performance Outcome Measurement Project, Program Evaluations)

11. PART • Required by the OMB, within the Executive Office of the President • Plays a major role in the budget process • Establishes a systematic, consistent process for rating program performance in order to make budget decisions • Emphasizes assessment criteria for program performance & management The measure of compassion is more than good intentions, it is good results. Sympathy is not enough. President George W. Bush, April, 2002

12. PARThttp://www.whitehouse.gov/omb/part/http://www.whitehouse.gov/omb/expectmore/index.htmlPARThttp://www.whitehouse.gov/omb/part/http://www.whitehouse.gov/omb/expectmore/index.html • Section Scores CY 2003: • 1 – 100% - Program purpose • 2 – 75% - Program performance measures • 3 – 100% - Program accountability • 4 – 67% - Long-term performance • Overall Score Changes • CY 2002 – 49 • CY 2003 – 81 • Score change due to new performance data, National Survey data • Overall Rating: Moderately Effective, second highest rating

13. OMB Program Assessmentof the OAA Program

14. Performance Outcomes Measures Project (POMP) • Developing and refining performance indicators for the past 7 years • Developed SUA/AAA capacity in performance measurement • Collaborative federal/state/local effort • Grantees are SUAs who are required to work with at least one AAAs; single PSA states must involve providers

15. Performance Outcomes Measures Project (POMP) • Developing core performance measurement tools • POMP 8 grantees will collaboratively be: • Validating tools that have been developed • Pilot testing statewide performance measurement methodology • Developing performance measurement toolkits for the aging network • Survey tools include: • Consumer assessment of service quality for core services • Consumer satisfaction • Consumer reported outcomes • Consumer demographic characteristics

16. Caregivers Case Management Congregate Nutrition Program Information & Assistance Assessment Senior Centers Homemaker Service Home Delivered Nutrition Program Transportation Service POMP OAA Title III Service Domains

17. Instruments Available on GPRA.Net

18. Sample POMP Successes • New York – AAAs used POMP Survey Results to support their case: • Wyoming County used POMP nutrition survey results to prove the importance of meals, with increased funds, did not have a waiting list. • Tompkins County used POMP nutrition survey results to show that clients could remain home longer with HDMs; Additional funding obtained which resulted in the expansion of the HDM routes.

19. Advanced POMP • Study the impact of OAA Services on costs, including cost avoidance • Looking at nursing home diversion and other models, including a frailty index • 10 Advanced POMP projects

20. Sample Advanced POMP SuccessPreliminary • SC – Used their results to increase state funding • Showed that participation in congregate and home delivered meals reduced emergency room visits and inpatient hospital admissions for older adults enrolled in Medicare and Medicaid • Received $2.9 M supplemental appropriation for HCBS, first new state funds in 10 years

21. Work In Progress • POMP, Advanced POMP • National Surveys • AAA Survey • Revised SPR Implementation & State Reporting Tool • Program Information Management Study • Demographic Data • Program Evaluations • Data Integration Project

22. Title IIIC and Title VI Evaluation • Applies AoA evaluation standards • Will develop a logic model for nutrition services as well as Title VI services • Will Follow CDC Program Evaluation Model • Two Phases • Design phase • Implementation phase

23. Goals for Title IIIC & Title VI Evaluation • Program Monitoring - document overall program results • Identify opportunities for program improvement • Aid program planning process • Show programs’ contributions to older adults’ independence • Assess best practices including those programs demonstrating the most effective cost-benefit outcomes and impacts

24. Status of Title IIIC and VI Evaluation 2006 • Extensive statement of work compiled • Proposals received, solicitation closed • Expect an award in the next month • Final evaluation due within 2 years • Will be the first complete evaluation of Title VI • Will build on results of the Mathematica July 1996 Title IIIC & Title VI Nutrition Evaluation

26. National SPR Data TablesCongregate and Home Delivered • Participants • Meals • Expenditures • Program income • Meal costs

27. Proportion of 60+ Served in OAA Nutrition Programs (S&MW&E)

28. Proportion of Congregate and Home Delivered Participants Served (%) In OAA Nutrition Programs [E&MW(similar proportion of older adults)]

29. Proportion of Congregate and Home Delivered Meals Served (%) In OAA Nutrition Programs [E&MW(similar proportion of older adults)]

30. Proportion of Congregate and Home Delivered Total Expenditures from OAA Funds (%) [W&W (geographic area)]

31. Proportion of Congregate and Home Delivered Total Expenditures from Program Income (%)[W,E&S (large)]

32. Comparison of Gross Cost of Meals[E,E (adjacent states) &S]

33. So, what does your state look like? How does your state compare with others? Do you measure outcomes in your state? How can you use these or other tools to help inform decisions? How does this affect program administration/implementation? Does your nutrition program need to change?