Conscientiousness as an indicator of Degree Success in Non-Traditional Students

1. Conscientiousness as an indicator of Degree Success in Non-Traditional Students Catherine Marshall Centre Director Presented by Dr Mary Dodd Learning and Teaching Leader Foundation Centre Durham University

2. Perhaps the greatest challenge yet to face the educational system of most nations is how best to address issues of equity and excellence within a context of diversity • (Cushner, 1994)

3. Foundation Programmes • Foundation Programmes were recently highlighted in the Milburn report as an initiative for Higher Education Institutions to engage in for improving social mobility • Widening participation activities involve risk for both the student and the institution • Identifying factors that can predict student success could be useful in recruitment and retention

4. Many Different UK Foundation Programme Models • Home students - remediation programmes for students who have studied to A Level, but not achieved a high enough level – deepening participation • Home students – widening participation programmes – offering an alternative route for students who have not studied at A level • International students – programmes which offer a bridging year for those whose national education system does not reach A level standard • International students – programmes which focus solely on Language Provision

5. Durham Foundation Programme • Combination of widening participation for mature, non-traditional home students and international students who need a bridging year • Durham also has a Pre-sessional programme run by the English Language Centre which offers year-long English Provision

6. Home Students International Students From many different countries Between 17 and 20 Have school leaver’s certificate High socioeconomic background • Predominantly local • Over the age of 21 • Do not have the requirements for immediate entry to their chosen degree • Low socio-economic background

7. Foundation Centre teaching • Focus is on holistic approach • International education – synergy between home and international students invaluable • International students want to learn alongside home students • Local home students benefit from sharing experiences with a diverse student group

8. Widening Participation • The Foundation Centre contributes to the University’s current widening participation target for home students • Mature students: FC is a major contributor to the 19.3% of mature students in the University • State School Entrants: all home students are from state schools • LPN and social class: 60% of Foundation students in the last 2 years had household income below £25K

9. Numbers of home and international students Number of students Year of study

10. Percentage Mature Students Percentage of Mature Students Year of Study

11. Percentage and Number of Local Students Local Students Year of study

12. Percentage Students with non-A level qualifications

13. Ratio of female to male Percentage of Students Year of Study

14. Assessment • Follows University regulations • One-time-only submission and no negotiation of marks • Internal examination with some coursework • One resit opportunity

15. Outcomes • There are direct progression routes to every department at Durham University. • Students must pass the whole programme with 50% in every module

16. Success rate • Typically the successful completion rate for all students is between 75% and 80% • The progression rate to Durham university departments for students is between 65% and 70% • Of those who do not progress to Durham some withdraw from the course, and some transfer to other universities

17. Percentage of Foundation Students passing, failing and withdrawing from the first year of their degree

18. Measuring Potential • For traditional undergraduates there is a heavy reliance on A level grades, but there is no consistent academic measure for non-traditional students • Ensuring that students who progress will succeed is important • for the student, particularly with fees of £9,000 • the institution, given the higher rate of non-completion by non-traditional students • and subsequent students, the pathway needs to be shown to be successful if it is to be maintained

19. Design of the Study • This particular study was based on 234 cases of students who studied on the Foundation Programme between 2004 and 2007 and progressed to degrees at Durham University • Data was collated on the students’ age, sex, origin, qualifications prior to entry, programme of study, average score on the Foundation Programme, their attendance percentage and timely submission of assignments and year 1, year 2 and degree outcomes.

20. Calculating a ‘Conscientiousness Index’ • Analysis of attendance showed that students who attended fewer than 80% of classes were much less likely to pass the programme. • The data on attendance percentage and percentage of timely submission of assignments were combined to create a Conscientiousness Index (CI). • This follows work by the Medical Education Research Group at Durham University who have correlated a measure on Conscientiousness to perceptions of professionalism (McLachlan et al., 2009) and performance in both skills and knowledge exams (Kelly et al., 2012).

21. CI in Foundation Attendance at all teaching sessions is compulsory • 18 hours formal teaching time per week • 6 compulsory sessions lectures, workshops, practical sessions • over 21 weeks • CI component calculated as % attendance of all possible sessions Assignment submission • On average 12 summative assignments during the year • CI component calculated as % assignments submitted on time

22. Qualitative Comparative Analysis • Proposed by Charles Ragin • Boolean Algebra • Formalises the logic of qualitative analysis • Possible to bring the logic and empirical intensity of qualitative approaches to studies that embrace more than a handful of cases -- research situations that normally call for the use of variable-oriented, quantitative methods. • Consider various inputs and analyse their effects on the output

23. Truth Tables • The following fictitious example is used to help describe how the process of QCA works. The fictitious data considered the output of whether students were successful in an English Language module, based on the inputs of GCSE standard ability, good attendance and reading set texts. • QCA uses binary data i.e. 1 and 0 only, where 1 indicates the presence of a factor and 0 indicates its absence. In order to recode data into this system, a specific cut off point needs to be identified for continuous data and categorical data with more than one category can be shown with a range of binary variables. In the case of GCSE standard, the membership of the groups is likely to be either yes or no and simply dichotomised. For good attendance, however, there would need to be a decision made about what percentage of attendance was considered “good”.

24. Truth Tables • This data is then presented in a truth table. A truth table is a way of representing the raw data into all the possible combinations of values of the independent variables. Each row of the truth table represents one logical combination of values and is then assigned a 1 or a 0 depending on whether a particular output is present. The truth table also gives the number of cases with that particular set of conditions and in some instances there may be rows with no cases.

25. A Ability to read and write to GCSE standard B Good attendance C Reading set texts

26. Necessity and Sufficiency • Traditional statistical methods are usually based in significance based on variables being both necessary and sufficient – focussing on one pathway to the outcome • QCA allows analysis of necessity and sufficiency separately – can identify several pathways to the outcome

27. Necessity and Sufficiency • Any condition which must be present to produce a particular outcome is considered necessary i.e. if within all the cases with a particular condition there is a subset with the outcome, that condition is considered necessary. • Any condition that will always produce the outcome is considered sufficient i.e. if within all the cases with an outcome there is a subset with a particular condition, that condition is considered sufficient.

28. Necessity and Sufficiency

29. Necessary but not sufficient • In the example given, condition A – the ability to read and write to GCSE standard is a necessary condition, because all combinations of the conditions which have the outcome include the condition A • It is not sufficient on its own, as there needs to be at least one other condition present, either class attendance or reading the set texts for a successful outcome

30. A Ability to read and write to GCSE standard B Good attendance C Reading set texts

31. Quasi-Sufficiency, Quasi-necessity • QCA originally designed for use with small datasets, e.g. Ragin’s hypothetical example (1987 pp 90) has a data set of 22 countries. • In this study, the data set is larger and is made up of individual people rather than countries. In this case relationships of perfect sufficiency or necessity are unlikely and the real world situation is much more likely to involve quasi-sufficiency or quasi-necessity (Cooper, 2005). • In the example below the first case shows complete sufficiency, with all members of the subset demonstrating the outcome, whereas in the second case 87.5% of the subset demonstrate the outcome, but that means that 12.5% of them do not, but it is a high enough percentage to be considered quasi-sufficient.

32. Quasi-sufficiency

33. Creating a solution to the truth table • Using quasi-sufficiency means that there needs to be an element of judgement in deciding which of the configurations of cases will be included in the solution to the truth table which identifies which pathways will produce the outcome • Any configurations with complete consistency with sufficiency i.e. a consistency score of 1.000000 are included • For quasi-sufficient configurations the normal threshold is 0.8 although any large discrepancies in consistencies between adjacent rows can be taken into account as long as the threshold is no lower than 0.7

34. Results – Honours Degree The output analysed was achieving an Honours Degree The inputs considered were • Age • Sex • Programme of study • Qualification • Students’ origin • Performance on the Foundation Programme • Conscientiousness at Foundation level

36. Solution for the Outcome Achieving an Honours Degree solution coverage: 0.372881 solution consistency: 0.868421

37. Necessary Conditions • All four solutions have the element of high conscientiousness as part of their configurations and this is the only element that is found in all four pathways. Consequently a • An analysis of necessity for high conscientiousness for the outcome of achieving an honours degree was shown to be approaching necessity at 83.0%. Analysis of Necessary Conditions for the outcome variable: honsdeg

38. Good Honours Degree Many universities use good honours degrees as a key performance indicator, so as well as whether students achieved an honours degree, the outcome of achieving either a first class or a 2:1 degree was analysed to elucidate whether any configurations of inputs were sufficient for that outcome

39. Truth Table for Good Honours Degree

40. Solution for Achieving a Good Honours Degree • Only one pathway was found to be sufficient QCA Output for the Outcome Achieving a Good Honours Degree

41. Necessary Conditions Analysis of necessary conditions was conducted for all four inputs • High conscientiousness is approaching necessity at 0.927 • Doing well on the Foundation Programme is also quite high at 0.841 • Traditional qualifications are not a necessary condition at all for the outcome of achieving a good honours degree at 0.280 • at 0.71, being female is just on the borderline of being considered necessary - it may be that being female is associated with having high conscientiousness

42. Potential Uses: Selection? IT may be possible to use conscientiousness as a selection too e.g. the medical team found strong correlation with the NEO-P-IR inventory and their measure of conscientiousness However, issues with whether conscientiousness is remediable

43. Other Potential uses: • Formative • Used for feedback to student • To identify students who need greater support for retention and success • Summative • Used to determine progression. Can identify outliers and major changes. Can be used to explore ‘failure to remediate’ • Informative • if the data is retained and cohort study conducted might be used to inform organisation of potential issues which may occur in the future

44. Practicality • Objective • Can largely be done by admin staff • Does not require a ‘decision to report’ • Runs continuously all year • Requires registers to be taken • Requires centralisation of information previously gathered but not collated

45. Extending it to other groups • In use by medical and dental schools • Being developed by surgeons, anaesthetists, clinical psychologists and general practitioners • Rolled out to paramedics and physiotherapists • Being explored for teachers in training For further questions, please contact: Catherine Marshall c.a.marshall@dur.ac.uk

46. ∂ Let the journey begin …