Exploring the Practice of Assessment in First Year

1. Exploring the Practice of Assessment in First Year Suzanne Guerin – School of Psychology Jonathan McNulty – School of Medicine & Medical Science Michael Staunton – School of History & Archives

2. Introduction • Assessment is central to teaching and learning practice and policy (Goos, Gannaway & Hughes, 2011) • Evidence of increased awareness and variation • Continues to be an area for improvement (Craddock & Mathias, 2009) • Developments can create challenges for key stakeholders (Brew, Riley & Walta, 2009) • Little research into provision of assessment, choice of methods and factors that influence practice (Craddock & Mathias, 2009)

3. Aim and Justification for Study • Aim of this study is to explore the practice of assessment in first year teaching • Underpinned by centrality of assessment, esp. in first year, in UCD’s education strategy; One of the key aims of which is: ‘To foster early and lasting student engagement’, which includes : ‘A review and reform of the structure, outcomes, assessment and remediation strategies for first year, and in particular the first semester, to support the transition from second- to third-level and to adapt to the different needs of different students; The further development of approaches to engage and support students, especially in their first year, including small group learning, peer-mentoring, academic advice and mentoring, specific supports for the development of transferable skills and information literacy, and general welfare supports‘.

4. Justification • We need to examine practice against UCD Teaching & Learning assessment principles (O’Neill & Noonan, 2011): • regular, low stakes assessment with opportunity for feedback on their progress • in-class self and/or peer review • collaborative learning (peer & group work, project work) • redesign of learning sequence, blended learning opportunities • Active/task-based learning with more authentic assessments (subject/discipline identity) • Awareness of student work-load

5. Research Questions • What factors currently contribute to assessment practice in first year modules? • What supports and obstacles are evident in assessment practice? • To what extent are assessment practices aligned with the UCD assessment principles?

6. Overarching design • Module review (Autumn 2011-Spring 2012) 2. Survey of staff (Autumn 2012) 3. Case studies (Spring-Summer 2013)

7. Phase 1: Module Review • Aim: to establish the types of assessment used in first year modules and key patterns • Examined institutional data provided in Module Descriptors, which contain the core information provided to students • Content analysis (Downe‐Wamboldt, 1992) of information from 2011-2012 • 627 first year modules, drawn from 36 schools in the Colleges of Arts and Celtic Studies, Business and Law, Engineering and Architecture, Health Sciences, Human Sciences, and Science

8. Module review: Key Findings • Average module size = 149 (Range 10-647) • Average number of assessments = 2.5 (Range 0-8) • Only 11.8% of modules included reference to assessment in the module descriptor text

9. Module review: Assessment Type

10. Module review: Assessment Timing

11. Phase 2: Designing a Survey of Staff • Aim of survey is to examine practice of assessment in more detail • Series of key stakeholder interviews conducted to inform design of the survey • Seven individuals (four male, three female) from administration, academic and management positions • Thematic analysis (Braun & Clarke, 2006) used to identify main themes

12. Phase 2: Interview Findings • Recognised importance of assessment – as both a positive and negative influence: “I think the assessment can be a hindrance [on engagement]. On the other hand, of course, it keeps the guys who are not genuinely excited going because they know they are going to be assessed.”

13. Phase 2: Elements of Assessment

14. Discussion: module review and interviews • Average number of assessments lower than previously identified and fewer modules had three or more assessments (O’Neill & Noonan, 2011). • Variation in methods reflects Craddock & Mathias (2009) though clear emphasis on exams. • Interviews identified a broad range of influencing assessment choices reflects previous research (Harris & James, 2006; Havnes & McDowell, 2008). • Patterns suggest limited impact of the UCD assessment principles (O’Neill & Noonan, 2011).

15. Staff survey: online questionnaire • Survey of all staff involved in delivery of first year modules • Survey examines: • Current assessment practice in first year modules • Related workload for staff and students • Familiarity with and attitudes towards the UCD Assessment Principles • Factors influencing assessment practice

16. Staff Survey: Methodology • Online survey of staff • Targets staff listed as coordinating Level 1 UG modules in 2011-2012 and 2012-2013 • Initial email to Heads of School and Heads of Teaching & Learning in all schools • Invitation email sent by Prof. Bairbre Redmond to stress institutional importance of the survey • Follow up email after two weeks (due week of 26/11/12)

17. Staff Survey: Preliminary Analysis • 52 responses to date, 61.5% completed • 44% male, 44% female, 12% missing data • Years teaching in UCD: • Range 1-35 yrs, Average = 10.9 (SD = 7.35) • Number of Stage 1 UG modules coordinated • Range 1- 5, Mode = 1 • Number of Stage 1 UG modules taught • Range 1- 5, Mode = 1 • Teaching & Learning Responsibilities: 65.4%

18. What College are you based in?

19. Assessment Types Examined • Annotated bibliography • Book review • Case study • Clinical: Objective structured exam • Clinical: Practical assessment • Data-based project • Debates or discussion • Demonstration • Design project • Essay • Exam: Essay-based • Exam: Multiple choice • Exam: Mini-tests • Exam: Oral • Exam: Short answer questions • Exam: Take home exam • Group work • Lab report • Learning contracts • Learning journal • Library research assignment • Make a video • Peer assessment • Performance piece • Portfolio • Presentation: Poster • Presentation: Oral • Problem scenario • Report • Role play • Self evaluation • Work-based problem

20. Types of Assessment Used

21. Module review: Assessment Type

22. Types of Assessment Used • Evidence of group assignments, self- and peer- assessment, active / task-based assessments, learning journals, etc. • Assessment FOR learning

23. Additional Questions on Assessment • What proportion of the module does the assessment account for? • How useful are the methods of assessment? • What is the associated workload for staff and for students? • Definition and use of formative assessment. • Awareness and views of UCD assessment principles • Factors influencing assessment practice • Views on strengths and limitations of current module descriptor system

24. Phase 3: Case studies: sampling criteria • Stratified sampling: • Module size • Number of assessments above and below the mean • Traditional v non-traditional forms of assessments • Involvement in T & L e.g. 4 large modules, half with assessments above the mean, half below 4 small modules, half with assessments above the mean • one case study per block (n=8)

25. Case studies: data collection • Documentary material e.g. module guides • Student feedback (via module coordinators) • Focus groups with students • Interviews with module coordinators Data analysis: • anonymous cross case analysis (public), • underpinned by individual case analysis (private)

26. Thank you • Suzanne Guerin – suzanne.guerin@ucd.ie • Jonathan McNulty – jonathan.mcnulty@ucd.ie • Michael Staunton – michael.staunton@ucd.ie UCD Fellowships funded under the Strategic Innovation Fund II