Measuring Learning Outcomes in UK Mathematics Education Projects

1. Measuring important learning outcomes in the context of two linked UK (mathematics education) projects: From instrument development, to measure validation and statistical modeling Maria Pampaka (The University of Manchester) Trondheim, February 2011 TLRP: “Keeping open the door to mathematically demanding F&HE programmes” (2006 – 2008) TransMaths: “Mathematics learning, identity and educational practice: the transition into Higher Education” (2008-2010)

2. Outline of Presentation Some background to the projects Instrument Development Measure construction and validation A measure of pedagogy A measure of Mathematics Self Efficacy A modeling approach to respond to the research questions

3. TLRP: “Keeping open the door to mathematically demanding F&HE programmes” (2006 – 2008) TransMaths: “Mathematics learning, identity and educational practice: the transition into Higher Education” (2008-2010) • Funded by ESRC (Economic and Social Research Council) • The University of Manchester, School of Education • The team: Lead Principal Investigator: Prof Julian Williams Other PIs: • Laura Black • Pauline Davis • Graeme Hutcheson • Brigit Pepin • Geoff Wake Researchers: Paul Hernandez-Martinez Maria Pampaka

4. Educational System in UK (England)

5. Aims of the Projects TLRP:To understand how cultures of learning and teaching can support learners in ways that help them widen and extend participation in mathematically demanding courses in Further and Higher Education (F&HE) • AS Mathematics Vs AS Use of Mathematics TransMaths: To understand how 6th Form and Further Education (pre-university) students can acquire a mathematical disposition and identity that supports their engagement with mathematics in 6fFE and in Higher Education (HE) • Focus on Mathematically demanding courses in HE (‘control’ : non mathematically demanding, e.g. Medicine and Education) • Mixed methodology: longitudinal case studies, interviews and surveys

6. DP1 Sept-Nov 2006 DP2 Apr-June 2007 Teachers’ Survey DP3 Sept-Dec 2007 TLRP Research Design: The Survey in the general framework Classroom practices Programme effectiveness Learner identities March 06 Questionnaire design Pilot case studies Sept 06 (i) initial interviews (i) initial questionnaire Case studies in UoM and traditional AS (ii) interviews round 2 (ii) post test June 07 (iii) follow-up interviews Sept 07 Follow up case studies (iii) delayed post test Dec 07

7. RQ1: How do different mathematics educational practices found in pre-university/university transition interact with social, cultural and historical factors to influence students’ (a) learning outcomes, and (c) decisions in relation to learning and using mathematics? RQ2: How are these practices mediated by different educational systems (their pedagogies, policies, technologies, assessment frameworks, institutional conditions and initiatives)? The Surveys within the TransMaths Project and the Relevant Research Questions

8. Analytical Framework Instrument Development Measures’ Construction and Validation (Rasch Model) Model Building (Multiple Regression, GLM)

9. The teacher’s Survey [TLRP] • Section A: Background and class information

10. The TLRP Instrument (Student Questionnaire) • Section A: Background Information • Section B: [Disposition to enter HE and study mathematically demanding subjects] • Section C: Mathematics Self-Efficacy Instrument

11. Section A: Background Information • Name, college, date, date of birth • Address and telephone number (for follow up survey/interview) • Gender • Course (UoM or AS Maths) • Previous math qualification (GCSE grade and tier) • University attended by close family • Language of first choice • Education Maintenance Allowance (EMA)

12. Section B: Dispositions… Disposition to go to HE Intention to go to University? • Expectations: family, friends, teachers Disposition to continue with mathematically demanding courses in HE • Intention to study more maths after this course? • Amount of mathematics in preferred option • Importance of amount of mathematics of course in decision • Feelings about future study involving maths • Preferred type of maths (familiar, new) University: what course? 

13. Using Mathematics: Self Efficacy A “pure” item: You are asked to rate how confident you are that you will be able to solve each problem, without actually doing the problem, using a scale from 1(=not confident at all) to 4(= very confident)

14. Using Mathematics: Self Efficacy An “applied” item:

15. The TransMaths Student Questionnaire Section A: Background Information University, Course/Programme Previous math qualifications Ethnicity, gender, country of origin, language Proxies of socio economic background Special educational needs A series of instruments about different aspects of the transition to HE…

16. The TransMaths ‘instruments’

17. Analytical Framework Instrument Development Measures’ Construction and Validation (Rasch Model) Model Building (Multiple Regression, GLM)

18. Constructing the measures: Measurement methodology ‘Theoretically’: Rasch Analysis Partial Credit Model Rating Scale Model ‘In practice’ – the tools: FACETS and Quest Software [Winsteps more user friendly] Interpreting Results: Fit Statistics (to ensure unidimensional measures) Differential Item Functioning for ‘subject’ groups Person-Item maps for hierarchy

19. Example 1: Measuring Mathematics Self Efficacy…some background • Self-efficacy (SE) beliefs “involve peoples’ capabilities to organise and execute courses of action required to produce given attainments” and perceived self-efficacy “is a judgment of one’s ability to organise and execute given types of performances…” (Bandura 1997, p. 3) • "a situational or problem-specific assessment of an individual's confidence in her or his ability to successfully perform or accomplish a particular maths task or problem" (Hackett & Betz, 1989, p. 262)

20. Background – Why Mathematics Self Efficacy? • ‘Important in students’ decision making (sometimes more than actual test scores) • Positive influence on students’ academic choices, effort and persistence, and choices in careers related to maths and science. • How to measure? • Contextualised questions • TLRP project: a 30 item instrument for pre-university students

21. Example 1: Measuring Mathematics Self Efficacy [at the transition to university] Instrument measuring students’ confidence in different mathematical areas ( 10 items): • Calculating/estimating • Using ration and proportion • Manipulating algebraic expressions • Proofs/proving • Problem solving • Modelling real situations • Using basic calculus (differentiation/integration) • Using complex calculus (differential equations / multiple integrals) • Using statistics • Using complex numbers

22. Measuring Mathematics Self EfficacyAn example “applied” item An example Item …

23. Methods and Sample • 10 items • 4 point Likert Scale (for frequency) • Sample:1630 students • Rasch Rating Scale Model

24. Results [1] – Checking Validity One measure? Item Fit Statistics t check for the assumption of unidimensionality

25. Items more relevant to AS/A2 Maths context More difficult for non maths students Results [2] – Checking validity Differences among student groups Differential Item Functioning

26. Results [3] Two separate measures Multidimensional Scaling?

27. Constructing the measures Example 2: The Teacher Survey (TLRP) • ‘28 item survey to teachers • 5 point Likert Scale (for frequency) • Sample:110 cases from current project • Rasch Rating Scale Model

28. Constructing the measures – Validity [Unidimensionality - Fit] B6: I encourage students to work more slowly B24: I cover only the important ideas in a topic

29. Constructing the measures:A measure of ‘pedagogical style’

30. I tend to follow the textbook closely Students (don’t) discuss their ideas I encourage students to work more quickly I teach each topic separately I tell students which questions to tackle I know exactly what maths the lesson will contain Students (don’t) invent their own methods Constructing the measures:A measure of ‘pedagogical style’

31. Validation supported by qualitative data “…I do tend to teach to the syllabus now…If it’s not on I don’t teach it. … but I do tend to say this is going to be on the exam…” “…. from the teachers that I’ve met and talked to… it seems to me that one of the big differences is, I mean I don’t sort of use textbooks… [ ]…I want to get students to think about the math, I want students to understand, I want students to connect ideas together, to see all those things that go together and I don’t think a text book did that…[ ]. “It’s old fashion methods, there’s a bit of input from me at the front and then I try to get them working, practicing questions as quickly as possible…” “… there’s a sense that I’ve achieved the purpose…I’ve found out what they’ve come with and what they haven’t come with so…we can work with that now”

32. The most challenging measure… Transitional Experiences

33. Analytical Framework Instrument Development Measures’ Construction and Validation (Rasch Model) Model Building (Multiple Regression, GLM)

34. From measures to GLM Modeling - TLRP • Variables • Outcome of AS Maths (Grade, or Dropout) • Background Variables • Disposition Measures at each DP • Disposition to go into HE (HEdisp) • Disposition to study mathematically demanding subjects in HE (MHEdisp) • Maths Self Efficacy (MSE: overall, pure, applied) • A score of ‘pedagogy’ based on teacher’s survey

35. The TLRP Sample • Longitudinal design • DP1: 1792 • DP2: 1082 • DP3: 608 • Resolution for some outcome variables (e.g. AS outcome) • Phone survey, School’s databases, Other databases

36. Results [1]: Math Dropouts • Percentages of dropouts by course and previous attainment • Effect Plots for a logistic regression model of dropout

37. Our modeling framework (TransMaths)

38. An example Model (TransMaths) We hypothesized that: Outcome of Year 1 (at University)= Entry Qualification + Dispositions + Transitional experiences + Background Variables

39. The resulting Linear Regression Model Positive effect: Positive transitional experience, previous maths qualification Negative effect: Course, Low Participation Neighbourhood, Mathematics Self Efficacy

41. Just a short summary This is the quantitative aspect of mixed method project which also includes case studies and interviews The multi-step methodology described helps us to create and validate our measures and then… Use them to model (GLM- regression modeling) in order to respond to the research questions we originally set… For instance What influences students successful transitions between various stages of education (e.g. to University)? How can we predict students’ progress at university) ?

42. Thank you Q & A