The importance of context in understanding teaching and learning: reflections on thirty five years of pedagogic research

1. The importance of context in understanding teaching and learning: reflections on thirty five years of pedagogic research Graham Gibbs

2. For every complex question there is a simple answer – and it’s wrong. H.L. Mencken

3. Context is always as relevant as concept Terry Olsen

4. For me context is the key - from that comes the understanding of everything. Kenneth Noland

5. To generalise is to be an idiot. William Blake

6. “Dimensions of quality” • Proposed Higher Education Council will specify the information institutions have to provide to prospective students to indicate the level of quality of the course they are interested in. • Attempt to generalise about what aspects of teaching influence learning gains, so as to specify PIs to justify varied fee levels • Possible to identify a range of valid indicators • ...but few indicators apply equally well to all contexts

7. The phenomenon... Evidence that student engagement (as measured by the NSSE) predicts learning gains, appears to apply to all kinds of institutions, except non-selective mid-west US colleges. What is going on here?

8. The phenomenon There is a substantial relationship between the level of student engagement and their learning gains, except for students in the top quartile in terms of past educational achievement. What is going on here?

9. The phenomenon... There is a substantial negative correlation between the emphasis an institution places on its teaching and on its research, except for a small number of rich US Liberal Arts colleges. What is going on here?

10. The phenomenon... At the University of Oxford there is no relationship between a department’s research performance and the performance of its students at Bachelors' level, but there is at PhD level. What is going on here?

11. The phenomenon Among the top universities for teaching in the UK, as measured by the NSS, are the Open University and the University of Oxford. They have almost nothing in common. What is going on here?

12. The phenomenon ... Perry, R.P. and Smart, J.C. (eds.) (1997) Effective Teaching in Higher Education: Research and Practice. Bill McKeachie“Good teaching matters and we know what it is” The Open University use none of the practices identified by McKeachie and at Oxford they are rarely ‘salient’.

13. “When you don’t understand something it is usually because you have made an assumption that you should not have made” FerencMarton Improving Student Learning Symposium, 1993 ‘Teaching Tips’ used to tell its readers, in a chapter about course design, how many lectures would need to be planned....

14. Common assumptions • Findings from one context apply to all other contexts • Theories or explanations that have explanatory power in one context will also have explanatory power in all other contexts • Teaching methods that work in one context will work in all other contexts

15. How should engineering be taught? • Imperial, MIT, Trondheim: the ‘standard’ model • UPC Barcelona, Lulea, Oxford • Maastricht • Open University • Queensland • Aalborg Who is ‘right’? Can anyone use any of these models and it still work well?

16. The application of ‘systematic course design’, starting from ‘objectives’ • Wide assumption that systematic course design is ‘a good thing’ • Oxford does not use anything resembling systematic course design • All (but the OU) of the radical approaches to Engineering education start from their model, not from objectives or systematic design • ...what is salient is the process, and its rationale, not the goals

17. Salient phenomena in large classes • ‘Class size’ phenomenon • Number of students in the room: ratings of teachers • Number of students in the room: social dynamics • Number of students in the room: sharing facilities • Cohort size: • access to learning resources • alienation and social breakdown • more surface approach - type of assessment used • Number of students in the smallest class students attend • Open University cohorts of >10,000, tutor groups of <20

18. Salient phenomena in effective departments • Vast list of ‘characteristics of effective academic leaders’ • Long list of phenomena evident in effective teaching departments, many of which relate to leadership • Some excellent departments display almost all of these characteristics, but some display almost none. • Completely different patterns of leadership: • where the department had addressed a seriousproblem through strategy or where they were maintaining excellence through culture and values • in different disciplines with different patterns of work and decision making: Philosophy vs Chemistry • in different organisational cultures: collegial, bureaucratic, corporate, entrepreneurial

19. Salient phenomena in the research-teaching relationship • Not ‘teaching competence’ (Marsh, SEEQ) • Relationship between learning resources and engagement/effort • Closeness of teaching/learning organisation, identity • Orientation of students towards academia

20. Salient phenomena in the way assessment impacts on learning • Underlying theory: ‘conditions under which assessment supports student learning’ • Empirical evidence: audits, AEQ, focus groups • Huge variation in assessment contexts

21. Assessment environments differ widely • Differences between Bachelors’ programmes • % marks from exams: 3% - 100% • number of times work marked: 11 - 95 • number of times formative-only assessment: 0- 134 • number of hours of oral feedback: 36 mins – 68 hours • no. of words of feedback on assignments: <2,000 - >15,000

22. Students experience of assessment environments differs widely

23. Salient phenomena in the way assessment impacts on learning • Underlying theory: ‘conditions under which assessment supports student learning’ • Empirical evidence: audits, AEQ, focus groups • Huge variation in assessment contexts • Pattern of ‘tidy’ relationships if viewed from far enough away and averaged across contexts, in a research study ...these relationships do not support use of systematic course design

24. Is systematic course design a ‘good thing’? When there is a high level of specification of learning outcomes and a high level of alignment of goals, assessment and criteria, students’ experience is characterised by: • Less coverage of the syllabus • Experience of less and poorer quality feedback • Less use of feedback • Less experience of ‘appropriate assessment’ • Less experience of ‘clear goals and standards’ • Less deep approach

25. Salient phenomena in the way assessment impacts on learning • Underlying theory: ‘conditions under which assessment supports student learning’ • Empirical evidence: audits, AEQ, focus groups • Huge variation in assessment contexts • Pattern of ‘tidy’ relationships if viewed from far enough away and averaged across contexts in a research study ...these relationships do not support use of systematic course design • Messy patterns, and very varied interventions, once local phenomena identified during a development project

26. Context-specific salient assessment phenomena • Lamentable teacher employment practices leading to incoherent standards • Wild innovation leading to excessive diversity of assessment • Mad quality regulations leading to hidden ‘illegal’ practices • External professional requirements at odds with internal assessment regulations, and academic culture at odds with professional culture • Total separation of teaching from assessment, and formative from summative assessment • Culture of ‘striving for excellence’ (rather than ‘sufficing’ or ‘all being equal’) (Beaty: ‘orientation’) • Emphasis on few, final, exams, but revision periods that allow time for integration, and exam questions demand integration

27. Salient phenomena in the assessment of groups • ‘social loafing’ is assumed to be inevitable • huge literature reporting it in every contexts you can think of • huge literature on how to avoid it • growing literature on it not being a problem • ...when there is a healthy learning milieu and sophisticated, collaborative students • ...common phenomenon of poor outcomes of innovation when teachers and students are unfamiliar with new methods

28. Observations • If a theoretical model, or an empirical prediction based on it, is not born out, it may simply be that it is not salient in that context – but it might still be very useful in other contexts. • Need to understand the context and try and spot what is salient before choosing which theory to use. • ...so need to be theoretically eclectic. • What is salient is often social rather than psychological • It is just as important to be familiar with varied contexts as it is to be conversant with varied theoretical frameworks • If you rely on one framework you will be wrong and irrelevant most of the time. • Not possible to simply ‘add up’ frameworks into a single framework encompassing everything.

29. Advice to the ISSOTL movement and to ‘scholars of teaching’ • Be more eclectic • Read some Sociology and Humanistic Psychology • Become more familiar with contexts • Report contexts in your papers • Be guarded in your generalisations • Try and spot the assumptions you have made that you should not have made...