Academic literacy assessment, skills and support in a professional MSc programme

1. Academic literacy assessment, skills and support in a professional MSc programme Dr Ellen Hurst

2. Context • Professional MSc programmes • 2 programmes, run concurrently in block ‘contact weeks’ during vacations • Students are often full-time employed in industry, seeking MSc for professional development • Degree culminates in a 60-credit ‘mini dissertation’ (high standards)

3. Aims & Methodology • Action research project 2008- present • Aimed at developing academic literacy assessment and support • The project so far has featured two (interrelated) subsections: • Part 1: Analysis of entry examination ‘introduction to research’ and development of essay-based replacement exam • Part 2: Development of writing workshops intervention

4. Literature that informed the design of the research • New Literacy Studies (Street); Bourdieu; Liebowitz • Definition of Academic Literacy (Liebowitz 2010): • Reading and writing • Numeracy • Digital literacy • Information literacy • Discourse • And language as in English, IsiXhosa or Afrikaans • Also Genres (Hyland)

5. Part 1 • AEHE assignment: Analysis of entry examinationfor ‘Introduction to Research’ self-study course • Diagnostic exam intended to assess the 'research writing skills' of students entering on to the programme to see if they are prepared enough to succeed in the programmes/ mini-dissertation • Prior to 2009: MCQ test (course materials: students receive a CD with information on academic writing/ referencing etc. when they are accepted onto the programmes) • 2009: developed an essay-based exam

6. MCQ test Which sentence, (a) or (b) from each of the following pairs would you expect to find in an academic journal? • It is not possible to draw any firm conclusions from the research. • Therefore, the research has been sort of inconclusive.

7. Percentage pass rates for MCQ 2008

8. Essay-based exam Write an essay summarising the three articles and compare and contrast the authors’ views. • Please focus on: • Common themes and discussions in the articles • Points of difference/ divergence in the articles • The summary should develop your own argument or viewpoint on the articles by citing from them, either in support of an author’s viewpoint, or to critique/ contest their arguments.

9. Percentage pass rates for essay 2009

10. Analysis • Comparison of 2009 exam with assignment grades of cohort – lack of correlation between exam marks & assignment success • (see handout)

11. Findings • The problem identified: • the MCQ test was badly formulated to test academic literacy skills (validity of the assessment) • Even following the development of a new exam format, the score of the students on the test did not relate to their success on the course (some students repeatedly fail the intro to research exam, but do well in all their courses, leading to it being a barrier to completion) • Furthermore, the ‘academic literacy’ of students was not supported outside the material provided for this exam, whether or not students scored well (Lack of utilisation of the entry exam for literacy development purposes)

12. Part 2 • Development of an intervention to support students’ academic literacy and research skills with a series of workshops • Lucia Thesen from CHED (Postgraduate Writing Project), the Programme Conveners and a number of other interested lecturers from EBE (Peter Wilkinson) met in 2008 to discuss support for research students • I offered to trial a pilot series in CEM –programme governance had also called for literacy interventions, and I was at that time convening both the ‘intro to research’ exam and the research methodology module • We designed a series of workshops intended to provide an embedded academic literacy component which supported students in the run-up to the dissertation; but also to improve written assignments • Started the series of workshops with guest presenters from CHED (Kevin Williams, Clement Chihota, Lucia Thesen)

13. Results of workshops • 17/18 who responded in 2011 found the workshops useful • Equally useful for assignments & mini dissertation • Referencing & reading were the most popular workshops (but this may have been due to timing of evaluation/ number of attendees)

14. Additional requirements? Is there anything missing you think we should cover? • How about personal feedback for individual module assignments and writing • It would have been helpful if the 'intro to research' was included in the presentations and we then wrote the exam at a later stage of the program. I failed it twice, but am getting good marks for the assignments!! • Rather have block sessions in first course week than small session every time/ module. Know all info necessary before need to start assignment. • At the last module it would be helpful that a complete pack of all sessions from 1st year is provided for students (either in class, mail or vula) • The content and intention of the writing skills is fine but particularly useless if there is no feedback on the students submissions

15. Some limitations • Difficulty in objectively identifying impact – due to different admission processes, different students, different judgements of writing ‘quality’ by examiners etc • Workshops don’t currently carry credits • Difficulty thus far in managing the PS workshops due to guest lecturers, outside conveners etc. • Workshops have been hard to sustain – due to practicalities (organising them to fit both programmes), the reliance on a single convenor (me) and the perception by some staff that they detract from ‘core’ teaching time

16. Future of intervention? • Future of intervention: currently designing a workbook (OER grant) containing 5 workshops – to use as replacement text for entry examination • Will be reinforced by follow up ‘workshops’ during the course and/or literacy assessment of course assignments (online platform/ feedback?) • Need to bind intro to research exam to workshops: entry examination, along with the research methods, was taken over by the course convener last year, but next year I’ll be involved in setting & assessing the exam as literacy expert, which could lead to formative opportunities (and possible allocation of credits to workshops/ literacy assessments further into the programmes)

17. Final thoughts • The demands we put on postgraduate students in terms of literacy, genres, using sources and so on are often outside their previous experience • If we require certain standards of them, we need to offer them support • English additional language students are often the most affected by the genre requirements • Supporting our postgraduates should be just as much an Academic Development issue as supporting undergraduates (and often gets overlooked)