Student Surveys and the Student Learning Experience

1. Student Surveys and the Student Learning Experience • Michael ProsserHigher Education Academy

2. Academy Mission • The Academy’s mission is to help institutions, discipline groups and all staff to provide the best possible learning experience for their students.

3. Presentation based upon: • Over 20 years research into the student learning experience in higher education in the United Kingdom, Australia, Sweden and Hong Kong • Experience in Australia with the Course Experience (Perceptions) Questionnaire • The interpretation of the results of student surveys is not value or theory free: • Interpretations in terms of student satisfaction and rankings • Interpretations in terms of student perceptions relating to student learning

4. Overview of the student learning perspective Figure 1: Model of Student Learning

5. STUDENT APPROACHES TO LEARNING • Surface Approach • Intention to reproduce • - rote memorise information needed for assessment • - failure to distinguish principles from examples • - treat tasks as external impositions • - focus on discrete elements without integration • Deep Approach • Intention to understand • meaningfully memorise information for later use • - relate new ideas to previous knowledge • - relate concepts to everyday experiences • relate evidence to conclusions 

6. Approach to learning represents the relationship between the student and the task or subject Approaches vary between subjects Approaches vary between tasks within subjects Approaches not stable but variable Surface and deep approaches describe a fundamental variation in the way students approach learning – they do not describe the rich detail in the way students approach their learning

7. STUDENTS’ PERCEPTIONS OF THE LEARNING CONTEXT • Research by Entwistle and Ramsden (1983) using interviews and questionnaires identified a number of student perceptions relating to the way they approached their studies • Student perceptions of: • Quality of teaching – including quality of feedback • (NSS: Teaching, Assessment and Feedback, Academic Support)) • Clearness of goals of course and standards of assessment • (NSS: Assessment and feedback) • Workload so high that it was not possible to understand everything • Assessment measuring reproduction and not understanding • were found to relate to how they approach their studies and to learning outcomes (exam results and other indicators)

8. RELATIONSHIP BETWEEN PERCEPTIONS AND APPROACHES Study of 8837 students in 51 first year subjects around Australia Amongst the data collected were students responses to: contextualised Ramsden's Course Experience Questionnaire and a contextualised Biggs Study Process Questionnaire 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

9. Approaches to Study Surface Approach 32. Although I generally remember facts and details, I find it difficult to fit them together into an overall picture 35. The best way for me to understand what technical terms mean is to remember the textbook definitions Deep Approach 28. I try to relate ideas in this subject to those in other subjects, wherever possible 34. In trying to understand new ideas, I often try to relate them to real life situations to which they might apply. Biggs Study Process Questionnaire

10. Student Perceptions Of Learning Context Good Teaching 15. The staff made a real effort to understand difficulties students might be having with their work. Clear Goals and Standards 1. It was always easy to know the standard of work expected 6. I usually had a clear idea of where I was going and what was expected of me in this subject. Appropriate Workload 25. The sheer volume of work in this subject meant that it couldn't all be thoroughly comprehended (-). Appropriate Assessment 8. To do well in this subject, all you really need is a good memory (-). Ramsden’s Course Experience Questionnaire

11. Analysis conducted using: • Individual student as unit of analysis across all first year subjects • Individual student as unit of analysis within first year subjects subjects • First year subjects as unit of analysis

12. Factor Analysis of Perceptions of T & L Context and Approach to Study - individual _________________________________________________________________ Scale Factors ____________________________ 12 _________________________________________________________________ Perceptions of Context Good teaching .80 Clear Goals and Standards .67 Appropriate Workload -.69 Appropriate Assessment -.65 Approach to Study Surface Approach .81 Deep Approach .73 _________________________________________________________________ Principal Components, Varimax Rotation, n=8837

13. At individual student level: • 1.   A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback) • 2.   A surface approach is associated with perceptions that the workload is too high and assessment tests reproduction • That is, variation in individual students’ perceptions of the learning context across all subjects is associated with the approaches to study (and learning outcome) • 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

14. Factor Analysis of Perceptions of T & L Context and Approach to Study – within first year medical subjects in different universities _________________________________________________________________ Scale Subject A Subject B Subject C ________________ ________ 121 2 1 2 _________________________________________________________________ Perceptions of Context Good teaching .84.77 .79 Clear Goals and Standards.68 .70 .69 Appropriate Workload -.73 -.68 -.69 Appropriate Assessment -.70 -.60 -.60 Approach to Study Surface Approach .75 .84 .80 Deep Approach .62 .64 .72 __________________________________________________________________

15. Within each subject: • 1.   A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback) • 2.   A surface approach is associated with perceptions that the workload is too high and assessment tests reproduction • That is, variation in students’ perceptions of the learning context within subjects is associated with the approaches to study within subjects – within subject variation in perception is not measurement error. • 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

16. Factor Analysis of Perceptions of T & L Context and Approach to Study - subject _________________________________________________________________ Scale Factors __________________________ 12 _________________________________________________________________ Perceptions of Context Good teaching .81 -.34 Clear Goals and Standards .78 Appropriate Workload -.75 Appropriate Assessment -.87 Approach to Study Surface Approach .91 Deep Approach .82 _________________________________________________________________ Principal Components, Variamax Rotation, n=51

17. At subject level: • 1.   A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback) • 2.   A surface approach is associated with perceptions that the teaching is not so good, workload is too high and assessment tests reproduction • That is, variation in students’ perceptions of the learning context between subjects is associated with the approaches to study • 1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

18. Example for student learning in first year mathematics • In a study of first year university mathematics students in a large research intensive university, analyses of short open ended written statements by 236 students identified: • Conceptions of Mathematics: • Fragmented Conceptions • Maths as numbers, rules and formulae • Maths as numbers etc with applications to problems • Cohesive Conceptions • C. Maths as a way of thinking • D. Maths as a way of thinking for complex problem solving • E. Maths provides insights for understanding the world

19. Fragmented Conceptions • Maths as numbers, rules and formulae • Maths is the study of numbers, and the application of various methods to changing numbers • Maths as numbers etc with applications to problems • Maths is the study of numbers and their applications in other subjects and the physical world

20. Cohesive Conceptions • Maths as a way of thinking • Maths is the study of logic. Numbers and symbols are used to study life in a systematic perspective and requires the mind to think in a logical and often precise manner • Maths as a way of thinking for complex problem solving • Maths is an abstract reasoning process which can be utilized to explore and solve problems • Maths provides insights for understanding the world • Techniques for thinking about observable, physical phenomena in a quantitative way and also for thinking more abstractly with little or no relation to the directly observable universe.

21. Approaches to Studying Mathematics: • Surface approach – focus on reproducing • Learning by rote memorisation – to reproduce knowledge and procedures • Learning by doing lots of examples – to reproduce knowledge and procedures • Deep approach – focus on understanding • Learning by doing lots of examples – to understand theory and concepts • Learning by doing difficult problems – to understand and to relate to other knowledge • Learning by doing difficult problems and studying theory to look for situations that theory may apply.

22. Approaches to Studying Mathematics: • Surface approach – focus on reproducing • Learning by rote memorisation – to reproduce knowledge and procedures • I liked calculus because I could remember formulas which is how I used to study. I would rote learn all the formulas and summarize all my theoretical notes • Learning by doing lots of examples – to reproduce knowledge and procedures • The way I go about studying for mathematics is by doing lots of examples and questions. Firstly I would study the notes and learn the formulas, then I put all of that to use by doing heaps of examples

23. Deep approach – focus on understanding • Learning by doing difficult problems – to understand and to relate to other knowledge • After listening to explanation of how a particular maths works the most essential features a repetition to develop speed (this usually consists of boring menial tasks) and an equal component of very difficult problems which require a great deal of thought to explore that area and its various properties and their consequences • Learning by doing difficult problems and studying theory to look for situations that theory may apply. • Read the relevant theory and try to get on the same “wavelength” as the person who actually discovered it. Before I attempt any problems I try to think where you can use the concept: i.e. what the concept was invented for. Then I attempt problems (on my own).

24. Relationship between Conception of Mathematics and Approach to Studying Mathematics  ____________________________________________________________ Conception Approach Total _________________ Surface Deep ____________________________________________________________ Fragmented (A & B) 179 17 196 Cohesive (C, D, E) 4 36 40 Total 183 53 236 ____________________________________________________________ Chi-square=126, p<.001 Phi=.57 ES=2.1 Crawford, K., Gordon, S., Nicholas, J. and Prosser, M. (1998) Qualitatively different experiences of learning mathematics at university. Learning and Instruction, 8, 455-468.

25. Defining Items from the Conceptions of Mathematics Questionnaire

26. Cluster Analysis of Prior Experiences and Understandings, Perceptions and Approaches and Post Experiences and Understandings Subscales

27. ______________________________________________________________________________________________________________________________________________________ Variables Mean Surface Mean Deep (n=147) (n=127) ___________________________________________________________________ Prior Experiences and Understandings Prior Fragmented Conception of Mathematics 0.31 -0.46 Prior Cohesive Conception of Mathematics -0.39 0.45 Prior Surface Orientation 0.34 -0.44 Prior Deep Orientation -0.56 0.59 Prior Academic Ranking (TER) -0.29 0.35 Perceptions and Approaches Good Teaching -0.37 0.37 Clear Goals and Standards -0.23 0.24 Appropriate Workload -0.30 0.32 Appropriate Assessment -0.12 0.12 Surface Approach 0.44 -0.45 Deep Approach -0.54 0.57 Post Experiences and Understanding Post Fragmented Conception 0.34 -0.41 Post Cohesive Conception -0.27 0.27 Achievement (Final Mark in Mathematics) -0.34 0.40 ___________________________________________________________________

28. Conclusions • The Table shows clear relations between prior experiences, perceptions of teaching and approaches and post experience and understanding. • The analysis identified 2 groups of students • In particular, those students: • with more fragmented and less cohesive conceptions on entry, describe having adopted more of a surface approach to studying at high school and less of a deep approach, • perceive the teaching to be poorer, to adopt more of a surface and less of a deep approach to studying at university, and • have more fragmented and less cohesive conceptions after studying and poorer achievement • The opposite was found for the other group

29. Overview of the student learning perspective Figure 1: Model of Student Learning

30. Substantial amount of research linking results on surveys such as the NSS with student approaches to study and learning outcomes Student perceptions are a function of both their prior experiences and understandings and the course design and teaching Do not improve student satisfaction by focusing on satisfaction - the individual items or scales – need to better understand why they responded the way they Spread of results – proportions responding in certain ways – better than mean of responses Substantial disciplinary variation – unreasonable to compare between disciplines Can only expect small effect sizes in changes in scores over time - .2 of a SD – if over 3 - 5 years changes of the order of .1 to .2 points (3.5 to 3.6 or 3.7).

31. Focus on the student perception rather than satisfaction – better understand their perceptions – why they respond the way they have • Comprehensive and aligned set of student evaluation instrument designed and interpreted in terms of student perceptions • Institutions quality assurance and academic development and CPD activities aligned with the underlying model • Treat data as indicators requiring further investigation – focus groups, open ended responses etc

32. Academy Response • Conferences relating to student experiences • Workshops and seminars to interested HEI’S and groups of staff • Work with interested institutions on ways to improve their students’ experiences as indicated by the NSS • Suggesting ways of further developing and / or improving future surveys • Review of dissemination site • Development of a bank of optional items, a selection of which individual institutions may wish to add to their NSS survey

33. Correlation matrix of NSS – institutional