Towards a culture of data collection and analysis in mathematics support centres

1. Towards a culture of data collection and analysis in mathematics support centres Professor Tony CroftDirector, Mathematics Education Centre, Loughborough Universitywww.sigma-cetl.ac.uk

2. Loughborough & the University Established as a technical institute in 1909. 1966 awarded its charter as Loughborough University of Technology 1996 renamedLoughborough University Today – one of the best Universities in the UK

5. A centre for excellence in university wide mathematics and statistics support

7. Overview • Growth in activity • Difficulties of getting good and convincing data • What has been written about this ? • Measuring and modelling success

8. background

9. background • “higher education has little option but to accommodate to the students emerging from the current GCE [ie pre-university schooling] process” Smith, 2004, Section 4.39,

10. Continuing growth in UK activity Surveys: Demand:When sigma announced a competition for funding to develop new or existing centres, 14 high quality bids were received. Sigma has now helped to establish new centres at Growth of centres

11. Around the world Australia:there are 39 universities, of which 32 have some form of Mathematics & Statistics Learning Support.(Helen MacGillivray, QUT, 2008) Ireland:13 institutions with mathematics support provision are reported in the audit of Gill Donoghue & Johnson (CEMTL 2008) Growth of centres

12. Around the world Switzerland’s first….. Opened September 2008 Growth of centres

13. The Loughborough Centre • video

14. Evaluation, accountability • So with huge and rapid growth of mathematics support activity it is right and proper that we ask…Are our efforts worthwhile, and how would we know ?Can we justify this activity ? • What data is it possible to collect ? • Is trying to collect data worth the effort ? • How can we interpret the data ? • Modelling Growth of centres

15. Difficulties of getting data: • Drop-in activities were often started by concerned teachers who found a few hours in the week to offer additional support to students. • Staff, if they were paid at all, were paid to tutor not to gather data, and certainly not to analyse it. • Very little long term strategy – ‘cinderella’ service wherein staff rarely knew if they would continue to be employed from one year to the next. • Departments often reluctant to give out data. • Cooperation of departments is not always forthcoming. • Difficulties of accessing the data. • Moving goal posts from year to year. • There are many factors which influence students’ success in a mathematics module:curriculum change; recruitment and selection criteria change; staff change; students change!; even things like time of day change! • There may be lots of students seeking help, but they come from many different courses, with very different needs, so it can be difficult to compare Difficulties of getting convincing data

16. Resource Issues / financial constraints • Almost all centres were established on very modest budgets and their purpose was to be up-front helping students. • “more than half of centres [in the Australian survey] do not have a full-time person employed”. • Many staff are sessional only (term only). • Short term funding. • Often staff recruited to maths support work do not have any experience of data collection and analysis, and are attracted to the work so that they can help students with their maths. Difficulties of getting convincing data

17. What has been written on Evaluation ? • [Mathematics Learning Support] needs sufficient security to attract, train and retain staff, and to play its part in the ongoing and longitudinal data collection and analysis that should be an integral part of its contribution to the university. All universities should ensure that such data collection and analysis are undertaken and performed correctly to provide vital information for university academic management. However, as reported, few of the facilities currently have the resources to undertake this important work.Helen MacGillivray (QUT, Australia 2008) • In the audit of Gill et al…measures of success are listed, but [Carlow] there is no definitive way of measuring the effectiveness…messiness of the data.. [DCU].…LSU’s should have a research and development function that is adequately recognised for funding (ITT Dublin) What has already been written ?

18. Measuring and modelling success • I will describe both soft and hard measures. What has already been written ?

19. Measures of success – soft measures • Growth in the number of centres • Usage Data – attendance records, return visits, year of study, department, problem • Student Feedback Questionnaires – biased and usually positive • Module Feedback Questionnaires – reach a broader group of students, but often little is said about maths support. • External Review Measures of success

20. Usage Data • Student visits Measures of success

21. Usage Data • Students • Visits • Departments Measures of success

22. External Review • The Institutional Audit report of 2004 records: • “The audit team identified the following areas as being good practice in the context of the University: the work of the Mathematics Education Centre...” • “Having discussed the work of the MEC with members of staff across the University, the team came to the view that [its] contributions to the University's resources for staff development, and their work more generally, constituted a feature of good practice.” • http://www.qaa.ac.uk/reviews/reports/institutional/loughborough2004/Rg063loughboroughUni.pdf The Institutional Audit report of 2008 records: sigma CETL has its origins in the Mathematics Education Centre but has widened its concern from the teaching of mathematics to engineers to include support for mathematics education across the University. The ready accessibility of useful help was praised by both undergraduate and postgraduate students that met the audit team. Other students described the benefits of the support rooms and associated equipment. Postgraduate students were appreciative of the one-to-one help and individual study programmes provided for them by the Centre.http://www.qaa.ac.uk/reviews/reports/institutional/Loughborough08/RG378Loughborough.pdf Measures of success

23. NSS • National Student Survey 2008 • Comments relating to “maths support” • Department: Mathematical Sciences:2 positive comments referring to MLSC (unspecified) • Department: EconomicsTwo positive comments citing “excellent maths support” (one) and “superb support in the form of the maths support centre” (one) • Department: Electronic and Electrical Eng:One positive comment: “the uni is more than equipped to help and bring me up to speed with my maths skills that I may have previously missed or forgotten”. Measures of success

24. Modelling success– hard measures • Studies with more substance are beginning to emerge which might inform methodology of data collection and analysis: • Dowling & Nolan • Patel • Croft & Pell • Lee, Harrison, Pell & Robinson • Parsons • MacGillivray Measures of success

25. Modelling success– hard measures • What sort of things might we count ? • % of support centre users who pass, against % of non-users who pass. • Mean exam score of users against mean exam score of non-users. • Pass rates of at-risk students who use the centre against those at risk that don’t. • Module grades of users. • Number of users who just scrape a pass. • Examine failing students to see if they attended the centre. • Retention rates longitudinally Measures of success

26. Robinson Measures of success

27. Bamforth et al (2007) Measures of success

28. Bamforth et al (2007) Measures of success

29. Bamforth et al (2007) Measures of success

30. Dowling & Nolan (DCU) • Published in CETL MSOR 2006 Conference Proceedings • Effectiveness of the Maths Learning Centre measured by usage statistics, feedback including interviews. • Model:pass rates of “at risk” students who did / did not visit the centre were compared over two years. Measures of success

31. Measures of success

32. Measures of success

33. Pell & Croft • Anecdotally:Some of the students most in need of support fail to access it at all.Many of those who make good use of the centres would pass anyway. • Analysed five first year engineering maths modules in 2004-5.644 students. • Data obtained on their module grade, number of visits to the centre.No access to entry grades. Measures of success

34. Pell & Croft • 21 students attended >=2 and achieved grade D • 63 out of 74 failing students did not attend more than once Measures of success

35. Pell & Croft • 35% of those achieving A* sought maths help more than once • 15% of E Grade Students and 15% of F Grade students sought help more than once • fail grade students, in addition to having ability problems have attitudinal problems Measures of success

36. National Audit Office 2007…. • Staying the course: the retention of students in higher education. • “A university’s approach to retention should be a positive one and it should provide students with opportunities to improve their grades rather than just addressing gaps in their knowledge” • The data from Pell would suggest the support centres are already doing this rather well!

37. There are some things you just can’t measure…. • I just wanted to share my good news that after five years of study at Loughborough I managed to gain a first class honours in Product Design and Manufacture. I believe that without the hours you dedicated to the maths learning support centre I would not have been able to pass the maths modules on my foundation year and first year. Your support, patience and encouragement were invaluable when it came to a subject that I had little confidence in when I first arrived at Loughborough. Kathryn