Quick win or slow burn? Modelling UK HE CAA uptake

1. Quick win or slow burn?Modelling UK HE CAA uptake Bill Warburton University of Southampton

2. Agenda • Why isn’t CAA uptake higher? • Quick win / slow burn • Dual-path theory of CAA uptake • Models of CAA uptake • What does ‘success’ really mean? • Applications of dual-path theory

3. Why isn’t uptake higher? • Background to the research • Academics seemed to avoid CAA even where it appeared to be fit for purpose • Wanted to understand why this was • Three year research project including national survey and targeted interviews • Output was a grounded theory of CAA uptake

4. Quick wins? • Pressure on tutors (Gibbs et al. 2000 etc.) • Worsening student/staff ratios • Pressure for more frequent monitoring of student progress • Students more demanding • RAE • Productivity pressures  utilitarian approach  less favourable outcomes  peers discouraged

5. Quick wins? … when the email came round about the [CAA] disaster… some of those colleagues… just went non-linear… how can we possibly have… taken on something which under the most fundamentally obvious things that it had to work under, it fails at the first hurdle? (Tutor AmO5M007)

6. Slow burn? • Tutors aiming primarily for pedagogical improvements incurred less risk • Often managed by Learning Technologists • Characterised by a staged approach: • Incremental growth in stakes (and risk) • Lower cumulative risk levels • Modest productivity gains

7. Dual-path theory of CAA uptake Context Tutor’s learning & teaching practice Interaction Ad-hoc practice discourages colleagues HIGHER-RISK ZONE Risk mitigating measures taken Negative feedback loop Conditions Tutor’s priority is for productivity gains Consequence: Higher-risk trajectory TUTOR DECIDES TO USE CAA Perceived risk level Conditions Tutor’s priority is for learning & teaching gains Consequence: Lower-risk trajectory Positive feedback loop Interaction Structured practice encourages uptake amongst peers LOWER-RISK ZONE

8. Dual-path theory of CAA uptake Tutor’s experience mitigates risk Strategic policy & strategy Learning & teaching infrastructure Tutor’s learning & teaching practice Institutional validation of existing good practice Procedural risk mitigating measures taken by LTs Ad-hoc practice discourages peers Negative feedback loop Higher-risk trajectory Tutor’s priority is for productivity gains TUTOR DECIDES TO USE CAA Perceived risk level Prioritise learning & teaching gains Lower-risk trajectory Positive feedback loop Structured practice encourages uptake amongst peers Physical risk mitigation measures taken by IT depts Strategic funding & commitment Physical Infrastructure Strategic resourcing

9. Strategic cultural conditions • Pressures for greater productivity • Drive for online learning- MLEs, VLEs • Effective strategy to manage change • Informed long-term top-down commitment • Clear institutional L&T strategy inc. QA of CAA Models of CAA uptake – Concentric shell • Infrastructure cultural conditions • Effective CAA Best Practice guide • Central effective support for CAA • Shared question banking • Proven cost/benefit gains Strategic consequence SMT demonstrates institutional commitment to tutors& LTs Strategic cultural interactionswith infrastructure SMT gives credit for CAA initiatives & allows time to develop CAA • Tutor cultural conditions • Tutors not conscripted • Tutors’ traditional autonomy threatened • Tutors have realistic expectations of CAA • Tutors believe CAA drives pedagogic development Individual consequence 1 Tutor decides to use CAA • Infrastructure interactions with tutors • LTs manage complexity of CAA for tutors • LTs & peers provide tutors with subject-specific exemplars • LTs & tutors work together in teams to create & review tests • CAA champions encourage tutors • Enthusiastic students encourage tutors Individual consequence 2 Tutor’s CAA trajectory is low risk • Tutor operational conditions • Tutors acquired a good grasp of assessment • Tutors early experience of CAA was good • Tutors are innovators and early adopters • Tutors acquired requisite IT skills Infrastructure consequence LTs & tutors reassured that CAA is ‘safe’ & fit for purpose Strategic operational interactions with infrastructure SMT provides adequate resources for effective support & training • Infrastructure operational conditions • CAA system is easily used • Adequate N/W & W/S infrastructure • CAA system is stable, reliable, resilient • CAA system is effectively interoperable • CAA system is flexible, fit-for-purpose • CAA system is secure & centrally supported • CAA system is accessible to all students • Strategic operational conditions • Secure central financial resources

10. High stakes Models of CAA uptake - Tutor Trajectories Linear Institutional Linear tutor Cut short Ad hoc Twist or stick Disastrous Low stakes, small scale Low stakes, large scale Planning Piloting Embedded End of life

11. 1. Ad hocdisseminationof CAA practice at department level by individual tutors 2. Coordinated disseminationof CAA practice facilitated by L&T specialists 3. Coordinated proceduralcontrol by LTs 5. Coordinated strategy for CAA uptake approved by SMT 7. Calls for improved assessment practice from funding bodies and other central organisations Strategy L&T practices Tutor’s propensities Tutors’ CAA trajectories Tutor’s experiences Physical infrastructure Resourcing 4. Coordinated physicalrisk mitigation by central L&T specialists 6. Coordinated resourcing provided through senior management Time Models of CAA uptake – ‘Arctic Roll’ Model 1

12. 1. Poor publicity from tutorsat department level 2. Ineffective disseminationof CAA practice facilitated by L&T specialists 3. Ineffective proceduralrisk mitigation by L&T specialists 5. Inertia of SMT in coordinating a strategy for CAA uptake 7. Widely published concerns about ‘dumbing down’ in national press Strategy L&T practices Tutor’s propensities Tutors’ CAA trajectories Tutor’s experiences Physical infrastructure Resourcing 4. Fragmentary approach to physicalrisk mitigation by L&T specialists 6. Resources withheld by senior management Time Models of CAA uptake – ‘Arctic Roll’ Model 2

13. Tutors? Whose notion of success? What does ‘success’ mean? Learning technologists? QA staff? Senior management? Individual tutors? Faculty/department? Success on what scale? Institutional? Range of subject use (Hum/Qual/Quant) Range of use (diag/form/sum) Range of items types Number of users ‘Width’ of practice (in terms of scale) Number of tests taken Level of integration with corporate MIS systems e.g. MLE Metrics of success on an institutional scale… ‘Embeddedness’ i.e. is taken for granted & has become ‘invisible’ and is (for some well-informed specialists) the key metric of success ‘Width of dissemination’ Has the application been taken up by other institutions?

14. Applying dual-path theory • At institutional level • Institution-specific factors populate models • Reveal likely patterns of institutional uptake • At the level of individual tutors • risk propensity predicts CAA trajectory type • could be used to direct resources where they might be used most effectively to mitigate risky practice • Clear descriptors facilitate common references for characterising uptake

15. That’s All, Folks! Questions? • Do you see any important gaps? • Does the Dual Path Risk Mitigation idea ring true?