The Transition Assessment and Goal Generator

1. The Transition Assessment and Goal Generator Jim Martin, Amber mcConnell, & Jennifer Burnes University of Oklahoma Zarrow Center Recommended APA-Style Citation Martin, J., McConnell, A., & Burnes, J. (2014). The transition assessment and goal generator. Presented at the 9th Oklahoma Transition Institute, Norman, OK.

2. Thanks to IES and National Center for Special Education Research TAGG Developed with a Grant from the National Center for Special Education Research and OU Zarrow Center Funds

3. Quality Transition Education ProducesGreater Outcomes Man on horse wearing armor holding a transition flag bravely traveling into the future. Another person on a stick horse going in another direction. Transition Education Begins With Transition Assessment

4. Need a Transition Assessment to Identify What Students Need to Learn for a Better Transition Into Adult Life • What do I need to learn now • To do the job I want after graduating from high school? • To learn where I want to further my education after graduating from high school? • Results will facilitate writing I-13 compliant IEP and transition planning discussions

5. TAGG Overview • Purpose • Assess non-academic skills associated with post-school employment and further education • To provide lists of student strengths, needs, a written summary, and annual transition goals matched to common core standards to facilitate writing I-13 compliant IEPs • Designed to Assess • Secondary-aged students with mild to moderate disabilities who plan to be competitively employed and/or enrolled in higher education after graduation, their educators, and parents

6. Overview Continued • Versions • 3 versions: student, family, and professional • Format • On-line written English • May be printed and taken by hand, but item scores must be entered into website to produce results • In next few months TAGG versions in Spanish, Mandarin, and other languages will be added • Users may choose to listen to audio or watch ASL videos for each instruction and item

7. Overview Continued Reading Levels Professional 10.4 Family 5.7 Student 4.8

8. TAGG Results Profile • The TAGG Will Provide Results to Copy and Paste Into IEP and To Use for Transition Planning Discussions • Graphic results profile by constructs • Written summary • Listing of strengths • Listing of needs • Annual transition goals matched to Common Core Standards

9. Development of TAGG Items • TAGG items derived from research studies that identified behaviors of former high school students with disabilities engaged in post-high school employment and/or further education • The research team initially used the research studies to develop • 10 construct definitions • Items developed from constructs • 15 or so iterative TAGG versions were created before testing began

10. Initial Structure: Ten Initial Constructs Knowledge of strengths and limitations Actions related to strengths and limitations Disability awareness Employment Goal setting and attainment Persistence Proactive involvement Self-advocacy Supports Utilization of resources

11. First Effort to Establish TAGG Structure • Users from multiple states completed the initial test-version TAGG • 349 high school students with disabilities • 271 family members • 39 professionals • Applied various factor analyses statistics • Went from 10 constructs to 8 • Went from 75 items to 34 • Construct structure confirmed by two more year-long studies

12. Professional and Family Constructs After FA Stayed Strengths and Limitation Disability Awareness Persistence Interacting with Others Goal Setting and Attainment Employment Student Involvement in IEP Support Community Dropped Actions Related to Strengths and Limitation Utilization of Resources TAGG-P: (c2=1043.62, df=499, RMSEA=.058, CFI=.92, TLI=.91, RMSR=.0597) TAGG-F:(c2=862.74, df=499, RMSEA=.057, CFI=.91, TLI=.90, RMSR=.058)

13. Student Version Constructs After FA After FA Constructs Strengths and Limitations & Support Community Disability Awareness Persistence Student Involvement in IEP Interacting with Others Goal Setting and Attainment Employment Dropped Constructs Actions Related to Strengths and Limitation Utilization of Resources Combined Constructs Strengths and Limitations Support Community TAGG-S: (c2=819.00, df=505, RMSEA=.047, CFI=.89, TLI=.88, RMSR=.064)

14. Numbers of Participants to Validate TAGG Across Three Years Table depicting 139 professionals, 847 family members, and 1,537 students.

15. Final TAGG Constructs Eight Constructs Defined: See TAGG Fact Sheet

16. Strengths and Limitations • Express personal strengths • Describe personal limitations • Explain academic situations where assistance is needed • Describe academic situations where success is experienced

17. Disability Awareness • Report accurate information regarding one’s own disability • Express types of accommodations needed for success • View a disability as only one aspect of life • Explain the special education services that one receives to others

18. Persistence • Understand the importance of putting forth continued effort in school • Work toward a goal until it is accomplished • Utilize multiple strategies to stay on task • Continue to work toward a goal after facing adversity

19. Interacting with Others • Maintain at least one good friend • Successfully participate in small groups to complete projects • Successfully participate in community organizations such as sport clubs and social groups • Successfully interact with peers, teachers, and other adults

20. Goal Setting and Attainment • Participate in the systematic learning of goal setting and attainment • Understand of the importance of setting and striving for goals • Set post-school goals that match skills and interests • Attain at least one transition goal

21. Employment • Express the desire for a job that matches career interests • Demonstrate job readiness skills, such as being on time, completing work as assigned, and working cooperatively • Successfully participate in a career technology or job-training program • Obtain a paid job

22. Student Involvement in the IEP • Identify effective and ineffective accommodations • Request additional accommodations when encountered with one that is ineffective • Discuss post-school goals with the IEP team • Actively lead one’s IEP meeting

23. Support Community • Recognize the difference between individuals who provide a positive source of support from those who do not • Identify situations when positive support people are needed • Use help from positive support people only when needed and necessary • Maintain a support network by showing appreciation or reciprocity

24. Summary of Psychometric Findings See TAGG Technical Manual Available at OU’s Zarrow Center TAGG web site for more information and updates as more validity evidence becomes available.

25. Internal Reliability • Generally a score between .7 and .8 is considered “good” • Each TAGG version has great overall internal consistency and satisfactory subscale consistency (ranging from α =. 89 to α =. 95) • TAGG Internal Reliability Measures are . . .

26. Test-Retest Reliability • Scores of .7 or higher represent good or satisfactory test-retest reliability • 14 weeks after the first TAGG was completed, same users completed the TAGG again. • A large correlation was found between the first and the second administration • .80 for professional TAGG • .70 for family TAGG • .70 for student TAGG • The TAGG has what type of test-retest reliability . . . . .

27. Measurement Invariance • The three TAGG versions are appropriate for students regardless of: • Students’ time in general education classes • Number of transition education classes completed • High School grade level • Disability category • This means • TAGG results do not need to be adjusted for the above factors • TAGG users believe TAGG items are of equal value

28. Fairness Validity Evidence: Gender • Do differences exist by gender? • No overall difference by gender on TAGG-P, TAGG- F, or TAGG-S • Some construct differences exist across all versions. For Instance: • On TAGG-S, females rated themselves higher on student involvement than males

29. Fairness Validity Evidence: Disability Category • Do differences exist by disability category? • No overall significance difference by disability category • Due to sample size, only students with Emotional Disturbance, Intellectual Disabilities, Autism, Specific Learning Disabilities, and Other Health Impairments were included in this analysis. • In general, students with ED and Autism scored lower on the three TAGG versions than students who had SLD or OHI disability categories. • In general, students who had SLD scored the highest on the vast majority of constructs on all three TAGG versions.

30. Fairness Validity Evidence: SES Effect • Free/reduced lunch eligibility • No significant differences for construct scores on TAGG-P or TAGG-S. Only small differences for TAGG-F scores. • Family employment • No significant differences for construct scores • Family education • Significant differences due to family education were found on the TAGG-F scores. • Almost all of the differences occurred between students whose family member was at the highest level of education (Master’s/Ph.D./other professional degree) and those at the lowest level of education (less than HS).

31. How Close Are Students, Professionals, and Family TAGG Scores? • How closely do the different TAGG versions assess the same student? • Medium correlations across Parent, Educator, and Student versions when assessing the same student. • This is considered excellent for this type of assessment

32. Concurrent Validity Studies • How Close Do the TAGG and AIR Self-Determination Assessment Match? • Medium Correlation • This implies the TAGG addresses some self-determination skills and assesses other skills, too. • What we would want in a concurrent validity study.

33. Scoring Using IRT Item Response Theory

34. Advantages to Using IRT • Advantages of IRT include • The ability to scale different item types • Provides a common metric for scales with different number of items • Weights items differentially by their validity for assessing the construct of interest

35. We Used a Four-Step IRT Algorithm Placed each scale onto a common score metric Projected item characteristics (e.g. item difficulty) onto the scale score metric Conducted a within-student comparison of scale scores across constructs to determine relative strengths and weaknesses Conducted a within-construct comparison of a student’s scale score to item responses (e.g. difficulty) to generate appropriate goals for identified weaknesses

36. Stanine Scores The stanine transformation places scale scores into one of nine categories. Each of the nine categories has a width corresponding to a half of a standard deviation on the normal curve, with the mean lying at the center of the stanine scores (i.e., score of 5)

37. Scale Score to Stanine Conversion for Reporting Purposes Normal cure with 3.5 to 6.5 stanines marked as in the average range. Students placing in the green area are in the average range

38. Overall Score The overall scale score is a weighted combination of all items

39. Future TAGG Development

40. More To Come • Now Conducting a second round of follow-up studies of former high school students with disabilities who took the TAGG • We will determine the relations between their high school TAGG profile and post-school outcomes • This will predict constructs associated with positive employment or further education outcomes • Results will be included into the TAGG profile to prioritize importance to skills that once learned will be most likely associated with positive post-school outcomes

41. Final TAGG Details • Minimal Cost • $3 per set (Professional, Student, Family versions) • Pay with Purchase Order or Credit Card • All funds handled by the OU Office of Technology Development • TAGG profiles saved for 7 years • Data kept on OU high-speed secure cloud servers • Purchased credits may be transferred to other registered TAGG users • Unused credits refunded for one-year after purchase

42. TAGG Location • The OU Zarrow Center’s Web Page http://zarrowcenter.ou.edu • The TAGG Section of the ZC Web Page https://tagg.ou.edu/tagg

43. Sample TAGG Screen Shots The On-Line TAGG and Result Profile

44. Screen shot of TAGG Professional version with instructions and 2 questions, scored on a 1 to 5 scale.

45. Disability Awareness Profile Bar graph of disability awareness scores for the student, parent, and teacher users. The X axis ranges from 0 to 9, with the teacher, parent, and student scores at 6, 6, and 7, respectively.

46. Combined Score Profile Bar graph of combined score for student, parent, and teacher with a 0 to 9 score. The student, parent and teacher scores are 5, 6, and 5, respectively.

47. Greatest and Relative Strengths

48. Areas of Greatest and Relative Need

49. Summary Statement for IEP Chad Bailey’s skills were assessed using the TAGG, a norm-referenced assessment with research-based items known to be associated with post-school employment and education. Compared to similar students, Chad’s scores are average. Results indicate greatest strengths are in the areas of Goal Setting and Attainment. Chad’s relative strengths include Disability Awareness and Student Involvement in the IEP. Greatest needs are in the area of Strengths and Limitations, with Employment being a relative need.

50. Suggested Annual Transition Goals To prepare for success in employment, the student will write an essay describing three situations where the student used his or her strengths with 90% grammar and context accuracy by the end of the essay writing unit.