Effective Assessment of TA Training

1. Effective Assessment of TA Training TA Training Community of Practice Silvia Bartolic March 30, 2017

2. Introductions

3. Why Is TA Training Important? • What interests do the following groups have in the TA Training program?: • Undergraduate students • Teaching assistants • Faculty members

4. Program Goals • Think individually and identify 2-3 goals of your TA Training program • In pairs, discuss how these goals relate to the needs of each of these groups: students, teaching assistants, faculty. • Can you identify any gaps?

5. How can we evaluate? • Focus groups/Interviews • Journaling • Surveys • Peer-Review • Grades/learning outcomes? • Other?

6. Focus Groups and Interviews • Strengths: • Purposeful Interaction • Access to meaning-making processes • Participant-driven and inductive • Efficient and low-cost • Limitations: • Moderator training • Researcher control • Data limitations • Practical considerations • Other Considerations: • Number of participants per group • Number of overall focus groups • Usually code with qualitative software such as NVivo

7. SOAR Model(Used for end of term feedback group session) • Strengths: What were the best parts of the program? • Opportunities: What opportunities exist for the program? • Aspirations: What do we most want to accomplish with the program? • Results: How do we recognize success?

8. Journaling • Strengths: • Helps participants synthesize and apply ideas • Opportunity to provide feedback on journals • Can observe in-depth meaning making (i.e.: how and to what extent are TA Training concepts resonating with TAs?) • Flexibility (can be done during TA Training, after TA Training, after TA leads a session) • Can assess improvement over time if doing multiple journals • Limitations: • Needs to be built into TA Training program and/or have incentives • Analysis can be time consuming • Can be challenging to summarize • Other Considerations: • Rubrics and aggregation

9. Sample Journal Evaluation Rubric Reflective practitioner: Clarity: The language is clear and expressive. The reader can create a mental picture of the situation being described. Abstract concepts are explained accurately. Explanation of concepts makes sense to an uninformed reader. Relevance: The learning experience being reflected upon is relevant and meaningful to student and course learning goals. Analysis: The reflection moves beyond simple description of the experience to an analysis of how the experience contributed to student understanding of self, others, and/or course concepts. Interconnections: The reflection demonstrates connections between the experience and material from other courses; past experience; and/or personal goals. Self-criticism: The reflection demonstrates ability of the student to question their own biases, stereotypes, preconceptions, and/or assumptions and define new modes of thinking as a result. Aware practitioner: Clarity: Minor, infrequent lapses in clarity and accuracy. Relevance: The learning experience being reflected upon is relevant and meaningful to student and course learning goals. Analysis: The reflection demonstrates student attempts to analyze the experience but analysis lacks depth. Interconnections: The reflection demonstrates connections between the experience and material from other courses; past experience; and/or personal goals. Self-criticism: The reflection demonstrates ability of the student to question their own biases, stereotypes, preconceptions. Reflection novice: Clarity: There are frequent lapses in clarity and accuracy. Relevance: Student makes attempts to demonstrate relevance, but the relevance is unclear to the reader. Analysis: Student makes attempts at applying the learning experience to understanding of self, others, and/or course concepts but fails to demonstrate depth of analysis. Interconnections: There is little to no attempt to demonstrate connections between the learning experience and previous other personal and/or learning experiences. Self-criticism: There is some attempt at self-criticism, but the self-reflection fails to demonstrate a new awareness of personal biases, etc. Developed by Steven Jones, Coordinator, Office of Service Learning, IUPUI

10. Surveys Strengths: • Confidential/anonymous • Large amounts of data quickly/cost effectively • Get an ‘average’ view • Low/no interviewer bias Limitations: • Lack of detail/specific examples • Social desirability /understanding biases • Low response rate Other considerations: • Sampling issues • Online/face-to-face/telephone/mail • Incentives

11. Peer-evaluation (formative) Strengths: • A teaching and learning experience • Less intimidating than faculty evaluation Limitations: • Requires additional training • May cause fear/self-consciousness Other considerations: • Summative evaluation? (for teaching portfolio) • Use of video (self reflection)

12. Sample Question • Theme: Marking & Student Feedback • Sample focus group questions: • How did the TA Training program prepare you for grading challenges? • Can you tell us about a time when you drew upon a grading strategy from the TA Training? • Sample survey question: • On a scale from 1 to 5 (1 = not at all prepared; 5 = very prepared), how prepared did you feel for grading challenges after the TA Training program?

13. Other Evaluation Method Considerations • Complementarity between methods • Limitations in TA training retention over time (Rodriguez, 1983) • Any methods we haven’t mentioned?

14. Activity: Brainstorming Questions • What questions could we ask the following groups to evaluate TA training? • Faculty members • Teaching assistants • Undergraduate students

15. I Have data – now what? • How is TA training assessment in your program currently analyzed and responded to? • Overall program? • Specific workshops? • What else could/should be included? • How can the information be used effectively?

16. Program Evaluation Templates (CTLT) • Individual Workshop Review Template: http://blogs.ubc.ca/tatrainingworkshops/files/2016/05/Survey.TA_.-Training.Workshop.Individual.docx • Workshop Strategies Template: http://blogs.ubc.ca/tatrainingworkshops/files/2016/05/Survey.TA_.Trainig.Workshop.Strategies.docx • Overall TA Training Evaluation Template: http://blogs.ubc.ca/tatrainingworkshops/files/2016/05/Survey.TA_.Training.-Overall.docx

17. Works Cited & Useful Resources • Auberbach, C. F, & Silverstein, L. B. (2003). Qualitative data: An introduction to coding and analysis. New York: New York University Press. • Babbie, E. & Banaquisto, L. (2010). Fundamentals of Social Research (3rd ed.). Toronto, ON: Nelson Education Ltd. • Blair, J., Czaja, R. F., & Blair, E. A. (2013). Designing surveys: A guide to decisions and procedures. Thousand Oaks, CA: Sage Inc. • Kitzinger, J. (1994). The methodology of focus groups: The importance of interaction between research participants. Sociology of Health & Illness, 16 (1), 103-121. • Krueger, R. (1988). Focus Groups. Newbury Park, CA: Sage • McLafferty, I. (2004). Focus group interviews as a data collecting strategy. Journal of Advanced Nursing, 48 (2), 187-194. • Morgan, D. (1997). Focus Groups as Qualitative Research (2nd ed.). Thousand Oaks, CA: Sage Publications. • Nardi, P. M. (2006). Doing survey research: A guide to quantitative methods. 2nd Edition. Boston: Pearson Education Inc. • Strauss, A., & Corbin, J. (1998). Basics of qualitative research. Thousands Oaks, CA: Sage Publications.