1 / 34

Virginia Department for Aging and Rehabilitative Services

Virginia Department for Aging and Rehabilitative Services. 2014 Program Evaluation of Vocational Evaluation Services Program Evaluation Summit Louisville, Kentucky September 9, 2014. Purpose of the Evaluation. Report Requested by the Commissioner and Field Services Director on the following:

hkurtz
Download Presentation

Virginia Department for Aging and Rehabilitative Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Virginia Department for Aging and Rehabilitative Services 2014 Program Evaluationof Vocational Evaluation ServicesProgram Evaluation SummitLouisville, KentuckySeptember 9, 2014 Barbara J. Burkett, Ph.D., M.S.P.H. Division of Policy and Planning 804-726-1911

  2. Purpose of the Evaluation • Report Requested by the Commissioner and Field Services Director on the following: • Utilization of vocational evaluation services • Program’s effectiveness • Outcomes • Counselors’ satisfaction • Best practices • Other state’s use of vocational evaluation

  3. The Program Evaluation Process • Review of Internal DARS Data • Number of Clients Served • Timing of Vocational Evaluations Within The Rehabilitation Process • Demographics of Clients Receiving Evaluations • Outcomes for Clients Receiving Vocational Evaluations Compared to Those Who Do Not Receive Vocational Evaluations • Cost Metrics of Vocational Evaluations • DARS Case Audit Reviews • Literature Search • Roles and Functions of Vocational Evaluators • Review of Various State Vocational Rehabilitation Programs Use of Vocational Evaluators • Population Specific Instruments/Best Practices • Survey of DARS Counselors Regarding Current Program

  4. Internal Review of DARS Data: Methodology • Review of Internal DARS Data • Data was extracted from AWARE, the DARS case management system, and imported in to SPSS for statistical analysis. • Study Years: Federal Fiscal Years 2011, 2012 and 2013 • Vocational Evaluation Cases Identified • 2011 case notes only • 2012 case notes and vendor file codes • 2013 case notes and vendor file codes

  5. Internal Review of DARS Data: Methodology (cont.) • Case Notes • String search for every iteration of “vocational evaluation” (Example: vocational evaluation, voc. eval., v.e., evaluation, evaluation, eval,….inclusive of v* and e* with and without punctuation and capitalization) • Verified string search results by opening each case in AWARE. AND • Vendor File • Career Services • Comprehensive Evaluation • Exploratory Evaluation • Referral/Staffing/Consultation • Report Writing

  6. DARS Internal Data: Staffing 2014 • 17 Vocational Evaluation Positions • 15 Currently Filled • 2 Vacancies (Eastern, Southwest)

  7. Percentage of DARS Clients Receiving A Vocational Evaluation • Six percent of DARS clients received a vocational evaluation during the study period.

  8. Counselor Specific Use of Vocational Evaluation • There was no regional variation in requests for vocational evaluation. • There was great variation in the use of vocational evaluation by counselors. • The range in the number of vocational evaluations requested by DARS counselors during the three year time period was 1-127. • Average number of vocational evaluations requested was 8.8 per counselor per year (s.d of 8.7). • The mode was 6 requests per counselor per year.

  9. Timing of the Vocational Evaluation in the Rehabilitation Process

  10. Demographics of Clients Who Received A Vocational Evaluation at DARS • Overall for the 3 year time period, transition students were more likely than non transition students to receive a vocational evaluation.

  11. Demographics of Clients Who Received A Vocational Evaluation at DARS • More males than females received vocational evaluations.

  12. Demographics of Clients Who Received Vocational Evaluations At DARS • 43% cognitive impairment • 28.4% psychosocial impairment • 7.9% other mental impairment

  13. Outcomes for DARS Clients Receiving Vocational Evaluations-Rehabilitation Rate (Negative Finding) • The rehabilitation rate was lower for clients who received a vocational evaluation over the 3 year period.

  14. Outcomes for DARS Clients Receiving Vocational-Average Hourly Earnings at Closure (Negative Finding) • Average hourly earnings were lower for clients receiving a vocational evaluation.

  15. Outcomes for DARS Clients Receiving Vocational-Job at Closure Matches Job Goal at IPE • Clients who received a vocational evaluation were less likely to obtain a job at closure that was the same as indicated on the IPE.

  16. Outcomes for DARS Clients Receiving-Supported Employment (Positive Finding) • Clients who received vocational evaluations were less likely to be referred for supported employment services.

  17. Cost Metrics for Vocational Evaluations • The VR cost of the clients’ who received a vocational evaluation were lower than for those clients who did not receive a vocational evaluation.

  18. Case Audit Reviews • Conducted by Scott Fraley and Tim Olive Report Written by Liz Smith • Selection Criteria for Cases • Cases closed in FFY2013 that had a vocational evaluation as a planned service • Stratified random sample by region and caseload type • Total sample n=40

  19. Findings of Case Audit Review • Five (12%) of the forty cases had a vocational evaluation scheduled, but never conducted. Two of these were caused by the consumer failing to appear for the vocational evaluation. The other three reasons were not evident from the case file. • 35 Reports • Twenty (57%) had no job recommendation in the vocational evaluation report. Eight of the 20 cases with no recommendation (23%) were transition cases. • Of the fifteen cases that had a job recommendation in the vocational evaluation report: • For twelve (80%) of the cases: the vocational evaluation recommendation, the vocational goal in the IPE, and the job at case closure matched.

  20. Conclusions from Case Audit • If it is the expectation that a job recommendation occur when a vocational evaluation is conducted, then that is not occurring on a consistent basis. • When a job recommendation is made in the vocational evaluation, there is a high level of consistency between that recommendation, the IPE goal, and the consumer’s job at case closure.

  21. Survey of Virginia Vocational Evaluators(Response Rate, Number of Evaluations Conducted) • Response Rate For Survey: 85.7% n/N = 12/14 • Number of Vocational Evaluations FFY13 • x = 96.5 • range = 76-123 • Number of Vocational Evaluations April 2014 • x = 10.6 • Range 6-20 • No show rate is a problem with 2 evaluators reporting a no show rate of 46.4%

  22. Survey of Virginia Vocational Evaluators (Referrals, Time Metrics) • Referral Source • DARS evaluators • 1 report of parent/guardian • 1 client request • 1 supported employment vendor • Time From Referral to Evaluation • X =41.7 days • Range 20 – 65 • Time to Complete Written Report • X= 6.1 days • Range = 2-16

  23. Survey of Virginia Vocational Evaluators (Template, Job Recommendations) • Template Use • 50% use now • 25% would like to use • 25% would not find it helpful • Job Recommendations Made • 82% make job recommendation • 18% did not (=2)

  24. Survey of Virginia Vocational Evaluators (Barriers to Providing Quality Evaluation) • #1: Incomplete information on client at time of evaluation • #2: Client willingness to participate • #3: Scheduling problems • #4: Poor referrals • #5: Client ability to participate

  25. Survey of Virginia Vocational Evaluators (Discussion of results with counselors) • Discussion of vocational evaluation reports with DARS counselors • 55% always discuss (n=6) • 46% discuss sometimes (n=5) • 1 person did not answer this question

  26. Survey of Virginia Vocational Evaluators (Instruments/Best Practices) • Instruments • Work Behavior Inventory used by 2 evaluators • Threshold Monthly Work Evaluation: 100% not familiar with instrument • Vocational Cognitive Rating Scale: 100% not familiar with instrument • Work Capacity Evaluation used by 2 evaluators • Maryland Assessment of Social Competence: 100% not familiar • Change Assessment for Vocational Counseling (URI): 100% not familiar • IPS Fidelity Scale: 1 evaluator familiar with instrument • Supported Employment Scale: 1 evaluator familiar with this instrument

  27. Survey of Virginia Vocational Evaluators(Use of Population Based Instruments) • Population Based Instruments

  28. Review of Literature • Carolyn Graham, Ph.D. (VCU) contracted for literature review • Barbara Burkett, Ph.D. (DARS) • Best Practices: Guide to Vocational Assessment and Evaluation-4th Edition Author: Paul Power • “The State of Vocational Evaluators: A National Study” Sligar and Betters Journal of Rehabilitation Vol. 74, no. 4

  29. Review of the LiteratureOther State VR Programs • Sligar and Betters interviewed 63 agencies • 97% (n= 60) of states purchased vocational evaluation services from evaluators in private practice, psychologists, and community rehabilitation programs. • Average number of vocational evaluators by agency was 15.1 with a range of 1-72. • 65% of the respondents did not require a certification • 81% of the programs had guidelines for the vocational evaluation and the final report.

  30. Management Changes That Occurred As A Result of This Program Evaluation • The vocational evaluator positions currently unfilled in the Central Region will be filled. • The vocational evaluator position in the Southwest region will not be filled. • A new position has been created. The position will supervise the vocational evaluators, the business development managers and the self employment enterprise staff. • Duties of this position will include: • Updating the DARS Best Practices Manual • Developing improved labor market data for use by DARS • Developing requirements for report writing • Procuring new instruments identified in the literature review and conducting training

  31. Don’t Kill The Messenger: Lessons Learned Regarding the Reality of Program Evaluations

  32. Strategies to Assist the Program Evaluation Team • Clearly communicate the purpose of the program evaluation to staff involved. • Describe the role of the program evaluator. • Provide a program evaluation plan as soon as possible to the stake holders. • Include the stakeholders in the planning meetings if possible • Present the findings to those involved and allow for input before final evaluation report. • Don’t take comments personally. This is the nature of the job.

  33. Where Can I Find This Presentation and Additional Information on the Virginia Study? • www.decisionswithdata.com • Blog

  34. Statistical Methodology • Null Hypothesis (Ho) and Alternative Hypothesis (Ha) Stated • Population Distribution Determined For Variables • Probability plot • Normal Distribution Non-normal Distribution or small sample size • Parametric or Non-Parametric Test Chosen • Test of Difference Between Proportions • Z-test: parametric • Chi-Square Test: non-parametric • Test of Difference Between Means • T-test: parametric • Kolomogorov-Smirnov Test: non-parametric • See Appendices of Program Evaluation for statistical formulas

More Related