Principles of Effective Evaluation

1. Principles of Effective Evaluation Prof Keithia Wilson Academic Leader Student Success & Retention

2. Importance of Evaluation • Results in evidence-based practice • Helps us to identify the most effective & strategic interventions (make a difference or add value) • Leads to sustainable & enjoyable practice • Provides accountability to School & Group leadership • Contributes to scholarship of the FYE

3. Kirkpatrick (1994) Model of Evaluation - Overview 4 levels of outcome evaluation – • Level 1 – Reaction • Level 2 – Learning • Level 3 – Behaviour • Level 4 – Results

4. Level 1 - Reaction Measures participant’s reactions to the training program, including: • reactions to the overall program (outcomes) e.g., * “To what extent did you find the Orientation program useful?” • reactions to specific components of the program (processes) e.g., * What aspects of Orientation did you MOST APPRECIATE and found USEFUL? * What aspects of Orientation did you LEAST APPRECIATE and feel is MOST IN NEED OF IMPROVEMENT?

5. Level 1 - Reaction • Important to include closed-ended items including rating scales e.g., How effective was the Orientation program overall in helping you to make a transition to university? not at all effective very effective 1 2 3 4 5 6 7 How useful was the initial information session on the predictors of success in first year? not at all useful very useful 1 2 3 4 5 6 7 • As well as open-ended items What else would you have liked included in the Orientation program today?

6. Level 1 - Reaction Some FYE examples – • Starting@Griffith Survey questions on Orientation (retrospective student survey) • School specific Orientation Program evaluation at the end of O-Day & the Common Time (ongoing Orientation Program) – student surveys, focus groups • Peer Mentoring Program final evaluation by mentors & mentees

7. Level 2 - Learning Measures what participants have learned from involvement in the program • What is measured needs to relate to what was covered in the program viz. learning objectives • Typically covers knowledge, skills, or attitudes • Needs to include both rating scales & open-ended questions • Can include self-report & tests of actual knowledge

8. Level 2 - Learning Some examples – • Student self-report * I feel confident that I know what to do to be successful academically this semester not at all confident very confident 1 2 3 4 5 6 7 * What have you learned about the transitions you will need to make as a commencing student in order to be successful? 2. Student knowledge * I know the name of my First Year Advisor yes no (circle one) * I know the difference between a lecture and a tutorial (circle one) Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree

9. Level 2 - Learning Some FYE examples – • Starting@Griffith Survey data evaluates learning outcomes in terms of knowledge & attitudes (e.g., knowing where to go to get help; perceptions of good teaching) • Outcome evaluation for Peer Mentees (What are your 3 most important learnings from your participation in the Peer Mentoring Program?) • First-Assessment-First-Feedback amber alert intervention (What have you learned as a result of your participation?)

10. Level 3 - Behaviour Measures the transfer of knowledge, skills & attitudes from the training context to in vivo or real-life contexts (the classroom & study). • Evaluate both before & after the program if practical • Use survey, focus groups, interviews with students, mentors, staff (convenors, sessionals) • Starting@Griffith Survey (Did you attend Orientation? How much time are you spending on study?)

11. Level 3 - Behaviour Some FYE examples – • Starting@Griffith survey data • Attendance at E-Day, O-Day & Common Time/ASP Program • Attendance at lectures & tutes • Assessment submission rates in courses

12. Level 4 - Results Measures “return-on-investment”, or the extent to which the program has produced results. Some FYE examples include – • Hard student outcomes such as - * pass rates in courses for assessment items * GPA in semester 1, 2 & FY overall * student retention into year 2 • Soft student outcomes such as – * student satisfaction

13. Evaluation Key question – What difference do our FYO&E interventions (e.g., Orientation, Peer Mentoring, Attendance, First Assessment Intervention etc.) make overall? Program/Intervention1 Reactions2 Learning 3 Behaviour 4 Overall Productivity/Result Program/InterventionFeelKnowDowhat is the result or difference overall?