Decisions, decisions … Surveying the first-year experience Mantz Yorke Lancaster University mantzyorke@mantzyorke.plus.c

1. Decisions, decisions … Surveying the first-year experience Mantz Yorke Lancaster University mantzyorke@mantzyorke.plus.com EFYE Conference, Middlesbrough, 26 April 2006

2. Importance The first year is critical for student success UK ‘loses’ by the beginning of the second year: ~ 8% of young entrants to FT degree programmes ~ 15% of mature entrants to FT degree programmes The proportion ‘lost’ varied with subject area and institution. Significant influences are students’ background characteristics Combined programmes, computing and engineering have the highest incidences of non-continuation

3. Causes of non-continuation • Are multi-faceted • Four broad areas: • Flawed decision-making re entry to HE • Experience of the programme and institution • Difficulty in coping with the academic demand • Adventitious events • Institutions can exert most influence on (2)

4. Institutional research • Concept is well-established in the US • Termed ‘institutional self-study’ by Watson & Maddison • IR activities are often dispersed in UK HEIs • IR deals with reflective practice at the level of the HEI • IR’s main aims are • to assist an HEI’s understanding of its performance • to provide information for enhancement and accountability

5. Institutional researchers • Analyse databases • Conduct investigations of various kinds • (US studies of retention tend to be heavily quantitative) • Report to institutional managers • Share expertise and findings with the IR community

6. Analysis of existing data: an example • Context • A university faculty suffering excessive student attrition • Need for staff to review practice • Data available • Ratings from post-module questionnaires • Failure rates in assessments • Entry qualifications (quite a lot of missing data) • Analyses • Post-module questionnaire data • Attrition data set against entry qualifications

7. Based on ~3200 responses Based on ~2150 responses

9. Worse D1 B E1 E2 X A* C1 D3 D2 C* A C2 E3 B* F Better

10. The HE Academy FYE study • Purposes • To provide the sector with data that • is informative • can be used as a baseline for comparison with future • studies, particularly in a context of ‘top-up’ fees • can be used comparatively, within and across both • subject areas and institutions, to inform both policy • development and quality enhancement activity

11. Challenges • Coverage of a range of • subjects • institutions • aspects of the first-year experience • with • an instrument of tolerable length • acceptable technical robustness • reasonable response rates

12. Sampling (Institutions as in early 2005) New universities Colleges Old universities Allied to Med Bio Sci Psychology Computer Sc Eng & Tech Social Studs Bus & Admin Humanities Creative A&D

13. Questionnaire - 1 • When should the FYE survey be run? 3 options: • Early in the autumn • February/March • End of first year • Time needed to negotiate with institutions

14. Questionnaire - 2 Wide range of aspects of the first-year experience Statements with 5-point agree/disagree scale • some negatively expressed items, where judged politic • scrambled order of presentation Informed by a range of previous instruments Completion on paper in class time, not in own time / web Relatively limited number of items (but >> NSS) •no strong tradition re lengthy surveys (contrast Australia, US) • limited time for survey • reliance on patience and goodwill of institutions, students

15. Administrative matters (not trivial!) Need for a relatively relaxed timetable for survey Preparedness to tolerate local variation in admin Anticipating matters such as • students taking joint/combined programmes • students who are present solely because of options • part-time students who are present Piloting Coding of OMR forms Management of paperwork, generally

16. Bias Judged Questionnaire items: content low Questionnaire items: structure low Sampling of HEIs low Admin of survey in HEIs [ 9 – 68% of nominal ] Students absent ? Actual student non-response low ‘Yea-saying’ low

17. Responses (as at 7 April 2006) Sent out to HEIs c20,000 Distributed in HEIs Far fewer, but not yet known Returned 6,644 Blank 291 Jocular/offensive 5 ‘Yea-sayers’ (?) 4 Usable 6,344 Completion rate of returns 95%

18. Caution! • Returns from one HEI have yet to be included • Data from ‘stragglers’ have yet to be received • The data that follow are highly provisional • They are presented in very gross terms • Sensitive sub-analyses are essential • PLEASE NOTE • Because the data that follow are provisional, they • are not for quotation

19. Some demographics (valid %) Age: 18-21: 75% Over 21: 25% Gender: Male: 40% Female: 60% Ethnicity: White: 80% Family background: Managerial/prof: 40% Other known 40% First in family to enter HE: 45% Previous HE experience: 32% Considered withdrawing: 30% - proportion of whom wishing to switch: 38%

20. Some interim findings Family background: no difference re experience in HEI 1st in family in HE: ditto Older students: more motivated more positive relationship with staff Gender: females more motivated, engaged Knowing a lot about programme: better scores all round Knowing a lot about HEI: ditto More time on private study: positive across the board

21. Considered withdrawal?

22. To scale, or not? • Feedback Corr coeff • Mean (a) (b) • (a) … has been prompt 3.07 • (b) … has helped me in my learning 3.46 .56 • Detailed comments received 3.18 .43 .53 • Scale (a,b,c) 3.24 • Reliability of scale (Cronbach alpha) = 0.75

23. Concluding comments • No study, whether qualitative or quantitative, is ever • perfect • Compromises are dictated by a range of considerations, • including • Type(s) of question being asked • Time and resources available • Acceptability of approach • Ethical issues • Institutional practicalities • Always a need to be able to explain the methodological • choices made, and the biases that these might incur