T he business case for academic development

1. The business case for academic development A symposium hosted by: Linda Creanor, Glasgow Caledonian Cathy Gunn, Auckland Neil Lent, Edinburgh Keith Smythe, Napier

2. Does evidence align with stakeholder priorities? • Why ask the question? • Experience shows it’s hard to answer • Different forms serve different purposes Report on activity Evidence to inform practice

3. Report on activity • Accountability • Progress on strategic initiatives • Show return on investment • Raise awareness

4. Inform practice • Educational research described by one author as ‘alchemy’ (Sloman 2012) • Literature notes poor links between theory and practice in learning technology research • Scholarship of teaching / learning design research is fine in context.. not generally

5. Evaluation: Understanding cause and effect Dr Neil Lent Institute for Academic Development, University of Edinburgh

6. Not always easy, for example: The Quality Enhancement Framework for learning and teaching in Scottish higher education (QEF)

7. Principles underlying the QEF • Collegiality • Consensual development = greater ownership • Alignment of aspirations to specific actions not general exhortation • A relatively ‘light touch’ is most likely to yield improvement

8. ‘Impact’ • Problem of certainty • Difficulty of establishing lines of determination • Diagnostic function: does it work, does it fit our values and vision of enhancement • Alignment of values and practices not measurement of cause and effect

9. Culture(s) of enhancement • Culture as practice and discourse • Away from what is towards an aspiration towards what could be • Review and reflection embedded and internalised way of life • Multiple stakeholders • Risk-taking • Sharing good practice

10. How can we evaluate cultural change in a system? • Build a model of cultural values for a system: who is there? what they do? what they use? • Look for alignment of practices and values observed with those associated with the desired culture • Think of activity in terms of loose frameworks and relationships between people and subsystems

11. The case for Blended Learning: shaping the evidence

12. Context Glasgow Caledonian University: • Post-1992 with a strong widening participation agenda • Blended learning integral to vision • Restructured in 2011 (again…) Academic Development & Blended Learning • Recent separation of development and research • Blended Learning team of 3 in Centre for Learning Enhancement & Academic Development (GCU LEAD) • 9 Learning Technologists in 3 Schools

13. Academic Development & Distributive Leadership CPD framework influenced by Australian Faculty Scholars Distributive Leadership Model (Lefoe, 2010) “... a distribution of power within the sociocultural context of universities, and a sharing of knowledge, of practice and reflection through collegiality.” (Lefoe et al, 2007:5)

14. Reporting on Blended Learning Strategic developments over previous year • Average no. accesses per student BLENDED LEARNING REPORT to Academic Policy Committee & Senate Usage trends for blogs, wikis, podcasting etc. VLE usage stats • %-age of modules with ≤ 12 accesses per student Analytics to capture broader access trends School comparisons

15. Caveats…. “The ‘headline’ data should be interpreted with some degree of caution.” “… [they] do not present a comprehensive picture of blended learning activity at GCU ....” “They do not tell us what kind of learning activities students are engaged in online, or how interactive these might be.” (Blended Learning Report, 2012)

16. Includes UK Sector Comparison UCISA Biennial Survey of Technology Enhanced Learning • For example: web dependent, web enabled, online • BUT how accurate are these figures? NSS and ISB scores for teaching and technology use • ‘teaching on my course’, ‘access to resources’ (NSS) • ‘virtual learning’ (ISB) • Useful info mainly from students’ open comments • Most recently the focus was on staff engagement (or lack thereof) and availability of online resources rather than technical issues.

17. Positive Outcomes • Blended Learning Roadmap updated annually • Recommendations made & priorities agreed • Agreed actions shared among - • Blended Learning Team • School Learning Technologists • Information Services • Library Reinforces the importance of strategic direction and central co-ordination

18. Issues What?An over-reliance on quantitative rather than qualitative evidence Who?Collaboration, shared responsibility and local ownership – embedding distributive leadership (Creanor, 2012 – in press) Why? Highlighting that central co-ordination and expertise remain key to strategic impact How?Balancing facts & figures with scholarly activity, robust evaluation, case studies ……???

19. Making the case for taught staff provision: evaluating the personal and strategic impact of two institutional programmes Dr Keith Smyth Office of the Vice Principal (Academic) Edinburgh Napier University

20. Three framing propositions • Pg Certificate Learning and Teaching (or equivalent) occupy a central position within many of our institutions, often linked to appointment, promotion and LTA strategy • Our approach to ‘evaluating’ taught provision for staff is often limited to standard post-module and post-course measures (standard surveys, numbers of completers) that provide little insight into the impact on the individual or their wider, developing role within the institution • The current debate on the worth of Pg Certs is demanding a richer evaluation of their role and value (Stefani, 2011)

21. Current context at Edinburgh Napier • Post 1992 teaching-led institution. Office of the Vice Principal (Academic)is responsible for institutional LTA strategy, and runs two taught programmes for staff. • Pg Cert Teaching and Learning in Higher Education. HEA/SEDA/NMC accredited. Running for 15 years. All new lecturers must complete within two years. Blended delivery. • Pg Cert/Dip/MSc Blended and Online Education. Fully online, applied and practice-based. SEDA accredited, and linked to new institutional TEL strategy and an initiative to establish accredited online educators in every School.

22. Key challenges • The programme teams know, anecdotally and through standard methods of evaluation, that the programmes are well received and seen as relevant to practice and wider institutional aims. We also see some evidence of impact through other means, e.g. Teaching Fellow applications. • However, we do not really know about the range of ways in which the programmes are impacting practice on a personal and institutional level and do not have a strong qualitative evidence base from which to illustrate impact and effectiveness over the medium and longer terms… • ..and justify the institutions resourcing of the programmes

23. How we propose to address this • Institutional investigation ‘A study of the effectiveness of Edinburgh Napier’s Academic Practice Programmes’ • Three months, mixed-methods with a strong emphasis on qualitative dimensions of impact or ‘Illuminating the link between developing staff as individuals and the strategic LTA development of the institution’ (Gosling, 2008) • Current and former students over last ten years for Pg Cert and last 5 years for the MSc BOE

24. General issues to be addressed • Impact on LTA practice and student experience • Recognition of LTA practice internally and externally (e.g. internal LTA Awards, discipline-specific LTA awards) • Leadership and promotion (supporting colleagues, pathways to Teaching Fellow/STF, programme and subject leader, leading school or university LTA initiatives) • ‘Stories’ which evidence individual playing central role in LTA strategy implementation locally or across institution • Publication and scholarship relating to Pg Cert/MSc projects

25. Issues in TEL practice to be addressed • Nature and impact of the work being undertaken as a ‘local champion’ through sponsored completion of Pg Cert BOE • Evidencing own engagement with new TEL strategy in design and delivery of own course provision • Scholarship and dissemination in TEL • Recognition of good practice (including through shortlisting for recent TEL related LTA awards, or external recognition)

26. What we propose to do with this • Produce ‘impact evaluation’ report for University LTA committee to include institutional and local analyses • Create case studies to contribute to LTA Resource Bank • Identify enhancements to the programmes • Disseminate findings of the evaluation, including the approach taken and data collection instruments used, to allow others to undertake similar evaluations • Forward plan periodic re-running of evaluation on smaller scale (perhaps tied into programme review cycle)

27. Tracking the invisible: impact evaluation for elearning development Dr Cathy Gunn Centre for Academic Development, The University of Auckland

28. Evidence to inform practice • A case study exploring evaluation challenges Formative = input to design Summative = impact on practice Overt elements Covert elements

29. Evaluation framework and example

30. Outcomes for staff and students • ALL students able to acquire necessary skills • Design concept & web elements reused across subjects led by ‘new experts’ • Time saved, autonomy promoted… • But…impact of elearning development role is hard to quantify

31. If critical factors are largely invisible… … how can we report them?

32. Summary and discussion • Neil – cause and effect or align activity with values and vision? • Linda – over-reliance on quantitative data which paints a partial picture • Keith – impact evaluation and case studies • Cathy – invisibility & risk when all runs smoothly AD that is deemed unnecessary

33. Discuss… • Are we: • Asking the right questions? • Gathering the right information? • Talking the language of our audiences?

34. We’d like to share your views • In an ideal world, what kind of culture and practice would we want in this area? What constraints & opportunities exist in the less than ideal world we currently inhabit? Most importantly – what positive moves can we make towards the ideal?

35. Creanor, L. (2012 – in press), Raising the profile: an institutional case study of embedding scholarship and innovation through distributive leadership, Innovations in Learning & Teaching International Gosling, D.(2008) Educational Development in the United Kingdom http://www.hedg.ac.uk/documents/HEDG_Report_final.pdf Gunn, C., & Donald, C. (2010). Tracking the Invisible: An eLearning Group’s Approach to Evaluation In L. Stefani (Ed.), The Effectiveness of Academic Development (pp. 133-142). Routledge. Gunn, C., & Steel, C. (2012). Linking Theory to Practice Learning Technology Research. Research in Learning Technology, 20(2) Lefoe G., Smigiel H., Parrish D. (2007) Enhancing HE through leadership capacity development, in Enhancing Theory and Scholarship, Proceedings of the 30th HERDSA Annual Conference, Adelaide, Australia, 8-11 July http://ro.uow.edu.au/asdpapers/56/ Lefoe, G. (2010) Creating the Future: Changing Culture Through Leadership Capacity Development, in Ehlers, U-D & Schneckenberg, D. (Eds), Changing Cultures in Higher Education, part 1, 189-204, Springer Verlag, Berlin, Heidelberg Stefani, L. (Ed) (2011) Evaluating the Effectiveness of Academic Development: principles and practice. Routledge