Statewide AT Program Leaders Symposium

Statewide AT Program Leaders Symposium Using Data and the Program Review Manual for Systematic Program Improvement May 17, 2011 Sara Sack--Kansas

Using Data & The Review Manual for Program Evaluation • Kansas has a “thing” about data—validity and reliability are extremely important • “Collecting good quality data plays a vital role in supplying objective information for the problems under study so that some analytical understanding of the problems and hence solutions can be obtained.” • “Making decision on the basis of poor quality data is risky and may lead to disastrous results, as the situation may be distorted and hence all subsequent analyses and decision making will rest on a shaky ground.” (Hong Kong Polytechnic University)

Using Data & The Review Manual for Program Evaluation • Data quality involves ensuring the accuracy, timeliness, completeness and consistency of data used by an organization while also making sure that everyone who uses the data has a common understanding of what the data represents. (Andrew Greenyer) • Kansas ATK staff achieve 85% reliability on 75 item coding task before being allowed to enter data into database.

Analyzing AT Data: Management • Weekly management level meeting with rotating agenda • Review grants & contract obligation & performance • AT Access Site Data Review • Collaborations & Partnerships • Communication & Public Awareness • Electronic database (and entry required by the 5th of the month for all subcontractors) makes data pull easy • Admin level reports provide data on their AT Site’s performance: completed goals by AT category, age, county, and outcomes collection within a specified date range • Super user level has access to all reports • 60:40 performance easy to obtain

ATK – 60:40 Split

Southeast AT Access SiteParsons (SKIL)

Analyzing AT Data: All Staff • All ATK staff meet every other month generally face to face (webinars in the winter) to obtain training and discuss data and performance • So….you would think we would have program evaluation pretty well covered, right?

Utilizing the Review Manual for Program Evaluation • Review process requires the program to compare 3 years of data. ATK’s data review generally compared performance to the previous year or to the same quarter of the previous year • Review process also requires programs to review their financial data for a three year period. • Looking at a three year period provided additional, important information

ATK-KEE REUTILIZATION DONATIONS

REUTILIZATION

What New Information Did We Gain From This Three Year Analysis? • In the area of Demonstration, we saw that our efforts to educate policymakers about the value of demonstration really were beginning to take hold and pay off • In Device Loan we were reminded of our efforts to “act super frugally” and invest savings into equipment purchases • AT Access Sites were reminded of the positive effects of specific Public Awareness Activities and vowed to try some of those efforts again

What New Information Did We Gain From This Three Year Analysis? • In Financial Loan of our AFP, the three year scan highlighted shifts in applications by equipment categories more clearly than just the two year scan • The funding support service that ATK provides as part of the State Financing showed major shifts in funders’ support for various technologies

What New Information Did We Gain From The Analysis of Three Years of Financial Data? • In the breakout and analysis of cost allocations trends could be identified and, as with other sections, staff recalled particularly strong efforts and activities. • In the area of Transition, the three year view of data showed how much was accomplished with very little money

Program Improvement Initiatives Based on Self-Evaluation & Program Review • Reviewers are most likely in the audience so we don’t want to tip our hand but…. • Continue to refine our online database—more automated features so staff have more time for consumers • Continue to analyze use of time—productivity is high so want to increase focus on “difficult to solve” situations • As our state moves increasingly to a “community based service model” we want to always be a part of the community solution

Program Improvement Initiatives Based on Self-Evaluation & Program Review • Conduct a detailed data analysis regarding our Public Awareness efforts and figure out if they are advancing our community based service efforts and our efforts to reach consumers that are in areas that receive fewer services • Work with the data to see if we can determine if consumers are utilizing a continuum of services

Questions? Contact: Sara Sack ssack@ku.edu

Statewide AT Program Leaders Symposium