Opportunities for Postsecondary Success Project Center for Community Partnerships Department of Occupational Therapy Co

1. Opportunities for Postsecondary Success ProjectCenter for Community Partnerships Department of Occupational TherapyColorado State University Goal Attainment Scaling AHEAD 2012 Conference 7/11/12 Cathy Schelly, M.Ed., OTR Assistant Professor Director, Center for Community Partnerships Patti Davies, PhD., OTR Professor Research Director

2. Opportunities for Postsecondary Success (OPS) • Implementation site: Center for Community Partnerships (CCP) • OPS Students: • CSU & Front Range Community College students with Asperger’s/ASD, TBI, ID • Transitioning youth with ID from local school district (partnership with Project Search) • Young adults with ID seeking community employment

3. Establishing measurable goals with participants: Goal Attainment Scaling

4. Goal Attainment Scaling • Goal attainment scaling (GAS): an individualized approach for measuring the achievement of goals (King et al., 1999). • Originally developed to assess adults in a community mental health setting, but has since been applied to numerous practice areas, including education, health, and social work (MacKay, Somerville, & Lundie, 1996).

5. Original goal attainment scaling method • The original GAS scale uses a 5-point scale, ranging from -2 to +2, with zero representing the expected level of performance after intervention.

6. Alternative methods

7. OPS Scale

8. OPS Goal Attainment Scaling GOAL: Implement effective study strategies.

9. Applying GAS in practice • Identify the overall objective. • Identify the specific problem area(s) that the individual wants/needs to address. • Identify the behaviors or events that will indicate improvement in each area. • Determine the methodology to collect the desired information. • Identify outcomes for scaling. • Determine the individual’s current status (baseline). • Determine how progress will be documented.

10. What is the overall goal? • Setting your expected level of outcome or the desired goal: • For example, you and the student may agree that developing self-advocacy skills is the overall program goal. • Document the student’s current status (baseline). • Establish incremental changes in behavior from baseline to depict improvements or ‘losing ground’ as student pursues goal.

11. What are the specific problem areas? • Prioritize areas of concern and formulate SMART goals. • Prioritized goals are given a weight, which is used to convert the scale score into a standard score.

12. What are SMART goals? • Specific= a specific goal has a much greater chance of being achieved. It is the who, what, where, when, which, and why of the goal. • Measurable=establish concrete criteria for measuring progress. • Attainable=help client’ set goals that are meaningful to them and that they want to achieve. • Relevant=a goal must represent an objective toward which you are both willing and able to work. • Time-specific=a goal should be grounded within a time-frame.

13. Tips for scaling goals… Select the expected level of performance. Identify the least favorable outcome and the most favorable outcome. Identify the intermediate levels of performance. Set SMART goals. Develop scale with equally interval levels.

14. Important considerations Where is my client’s current level of performance (baseline)? When/how often should I follow-up to determine and document client’s change in status?

15. What behaviors and events will indicate improvement? Create a scale that will be useful for measuring changes in performance. For the student working on self-advocacy skills, this may involve creating a scale based on the number of times the student attends his/her professor’s office hours. For student who is wanting to make friends in the residence hall (RH), this may involve creating a scale based on the number of RH activities he participates in.

16. GAS Goal: Recognize and address issues that result in lack of persistence spring semester.

17. GAS Goal: Implement study strategies.

18. GAS Goal: Self-advocacy for learning accommodation needs.

19. GAS Goal: Completes new tasks when assigned jobs are finished.

20. Process we use • Develop relationship with student. • Connect student with mentor. • Gather information re: needs, wants of student, and areas where there are challenges (assessment process). • Develop semester goal with student – review with team to ensure that goal is SMART. • Enter goal into GAS database. • Begin goal attainment activities, documenting incremental changes. • At end of semester, regroup and establish new goal(s) for next semester.

21. CCP Assessment Process • Functional, community-referenced process • Identify strengths, challenges, barriers in all domains • Identify support needs • Formulate student/participant driven goal(s)

22. Challenges/Support Needs

23. Assessment process is ongoing • Assessment is the critical starting point that paves the way to successful planning, educational mentoring, and provision of necessary supports that facilitate success. • Functional assessment continues after the student is working towards goals to continually guide planning, solve problems, improve academic performance and, ultimately, persistence, retention and graduation. • Same guidelines apply for students pursuing employment/careers.

24. Canadian Occupational Performance Measure (COPM) • Under consideration, to complement GAS. • Assesses individual’s perceived occupational performance in life areas. • Process: • Conduct semi-structured interview to identify wants/needs of client. • Based on wants/needs, create goals. • Supplement interview information through direct observation, environmental assessment, or standardized tests.

25. Benefits of COPM • Measures client’s perceived progress in goal attainment between pre-COPM, intervention and post-COPM (designed as outcome measure). • Easy to administer ~20 minutes. • Demonstrated reliability and validity. • Similar to GAS, useful with a wide variety of clients in many different settings. • Can weight and prioritize goals (again, consistent with GAS).

26. Audience Discussion • What type of assessment process(es) are you using? • What are the advantages of your assessment process? • What are the disadvantages, if any, of your assessment process? • What population do you serve?

27. Practice session on writing goals • Review GAS worksheets. • Develop goals as per worksheet, OR, for student you are working with. • Connect goals to domain areas. • Develop SMART goals. • Discussion • Actual goals that we established with our students/participants

28. How will data be collected? • What process will your group implement to collect and analyze data? • How will information be collected and who will collect it? • Role of the Transition Coordinator • Role of the Mentee • Role of the Mentor

29. Our GAS Research • Exciting results from pilot study fall 2011 and spring 2012 • Noted progress in following areas: • Academics • Self-advocacy • Socialization • Persistence • Retention • Graduation • Employment

30. Preliminary Data • Participants - total 23 • 12 students at CSU • 5 students at FRCC • 6 students enrolled in Project Search • Number of goals written – total 43 • 15 in Fall semester 2011 • 28 in Spring semester 2012

31. Students enrolled in Project Search • 6 students • Goals written for a 3 month internship • 8 goals written • 1.3 goals on average per student • 83% of the students made steady improvement on their goals during their internships • 16% of the students displayed a decline in their GAS score at the end of the internship • 100% of students were hired after completing their internships

32. Students in College (FRCC) and University (CSU) • 17 students • Goals written for a semester • 35 goals written • 1.8 goals on average per student • 47% of the students made steady improvement on their goals across the semester • 24% of the students displayed a decline in their GAS score at the end of the semester

33. Changes in Goal Attainment Scores Before and After Intervention

34. Changes in Goal Attainment Scores Before and After Intervention

35. ANOVA Statistic: F(1,20) = 136.17, p < .0005 Effect Size: • Partial η² = .87

36. Converting Goal Attainment Scales Scores to T Scores • Sum the GAS score for the goal(s) • If a participant only has one goal the sum will be the score for that one goal. • If a participant has more than one goal the score for each goal will be added together. • Convert the summed score to a T Score • Formula in Kiresuk & Sherman (1968) • More precision • Tables in Kiresuk, Smith, & Cardillo (1994) • Less precision

37. Advantages of Using T Scores • Helps to equate scores across participants when they have different number of goals. • Provides variability to pre-intervention scores which is necessary for conducting statistical analyses.

38. Goals can be Weighted • To reflect • Relative value of each goal based on intervention outcomes • Relative importance of each goal based on participants’ perspectives • Relative to the difficulty of the goal • Weights do not need to sum to a particular fixed total • Examples • 1, 2, 3 or 2, 4, 6 or 10, 20, 30 • Equal weighting of goals lose little information

39. Goal Attainment Scaling as an Intervention Tool • Weighting and selecting goals based on client’s desires • Setting goals in collaboration with client • Using improvement on GAS scores as motivation for the client to work hard • Can be used in combination with the COPM • Determine goal areas • Weight goals

40. Goal Attainment Scaling as an Research Tool • The evaluation and establishment of goals may be conducted by someone different than the person providing the intervention. • Follow-up assessment on goals may be conducted by someone other than the person providing the intervention. • Consider comparing intervention programs or using a control group. • Consider using other evaluation tools in addition to GAS.

41. Goal: University Graduation Success!

42. Goal: College Graduation Success!

43. Advantages of using GAS with the OPS project • Provides a quantitative measure of outcome. • Can be used to compare a student’s progress over time. • Can be used to compare performances across clients in the same program but who have different skills, challenges (Ottenbacher & Cusick, 1989). • Allows for a collaborative approach-working with the student to set realistic goals. • Provides flexibility for measuring diverse outcomes (Brown, 2009).

44. Limitations of GAS • Scaling can be a time-consuming process. • Biases can occur in goal setting, scaling, and rating. • Temptation to modify goals throughout the course of intervention. NOTE: Good goals are really hard to write!!

45. References • King, G.A., McDougal, J., Palisano, R.J., Gritzan, J., & Tucker, M.A. (1999). Goal attainment scaling: its use in evaluating pediatric therapy programs. Physical and Occupational Therapy in Pediatrics, p. 31-52. • Kiresuk, T. & Sherman, R. (1968). Goal attainment scaling: A general method of evaluating comprehensive mental health programs. Community Mental Health Journal, 4 (6), 443-453. • Kiresuk, T.J., Smith, A., & Cardillo, J.E. (1994). Goal Attainment Scaling: Application, Theory, & Measurement. Hilldale, N.J.: Lawrence Erlbaum Associates. • Mailloux, Z., May-Benson, T.A., Summers, C.A., Miller, L.J., Brett-Green, B., Burke, J.P., et al. (2007). The Issue Is-Goal Attainment Scaling as a measure of meaningful outcomes for children with sensory integration disorders. American Journal of Occupational Therapy, 61, 254-259. • McKay, G., Somerville, W., & Lundie, J. (1996). Reflections on goal attainment scaling (GAS): cautionary notes and proposals for development. Education Research, 38,2, 161-172. • Ottenbacher, K.J. & Cusick, A. (1990) Goal attainment scaling as a method of clinical service evaluation. American Journal of Occupational Therapy, 44, 6, 519-525. • Steenbeek, D., Ketelaar, M., Galama, K., & Gorter, J.W. (2007). Goal attainment scaling in pediatric rehabilitation: a critical review of the literature. Dev Med Child Neurol, 49, 550-556. • Turner-Stokes, L. & Williams, H. (2010). Goal attainment scaling: a direct comparison of alternative rating methods. Journal of Clinical Rehabilitation, 24, 66-73.