Evaluation of the SFC e-Learning Transformation Programme

1. Evaluation of the SFC e-Learning Transformation Programme Veronica Adamson Jane Plenderleith GlenaffricLtd

2. Nature of Transformation Transformational change will require a conscious and deliberate decision made by (one or more) institutions to do something differently, in a systematic way, across the whole institution, on a defined timescale of two or more years

3. Key Principles • The process of change will mean that certain aspects of learning and teaching are conducted in a new way • The process of change is consistent with, and embedded in, institutional strategies, and is not a peripheral process driven solely by the possibility of external funding • The intended outcome is sustainable, and is expected to result in long-term change in activities beyond the period of external funding • The process will yield measurable benefits to the institution and its learners

4. Key Influences • Center for Academic Transformation at Rensselaer Polytechnic Institute • Its mission is ‘to serve as a source of expertise and support for those in higher education who wish to take advantage of the capabilities of information technology to transform their academic practices’ • MIT Transformational Model • CITSCAPES Development Tool • Becta Matrix

5. Pew Grant Programme • Supported by an $8.8 million grant from the Pew Charitable Trusts, the Program in Course Redesign was created in April 1999 • Program is supporting thirty colleges and universities in their efforts to redesign instruction using technology to achieve quality enhancements as well as cost savings • Preliminary results show that all institutions reduced costs by about 40% on average, with a range of 20% to 84%

6. Program in Course Re-Design • Continuum from fully F2F to fully online • Five design models identified: • Supplementalretains basic course structure but add technology-based activities • Replacementreduction in class meeting time • Emporiumbest time to learn is when students want to not when tutor wants to teach • Fully onlinestaff perform tasks alone – translated to online but labour intensive • Buffetall of the above treat students the same but could use menu based on student needs with varying learning styles

7. Becta Matrix Self-assessment Tool • Developing an e-Learning Strategy • College Strategic Plan

8. e-Learning Environments Study Listed below are functions of processes, services and systems that support learning and teaching. To what extent will these systems, services and processes be developed or integrated in your institution by September 2005, and to what extent do you want them to develop in the future?

11. Evaluation Model

12. Evaluation Aims • Assess the extent to which the projects met their aims and objectives and implemented new models of delivery in FE and HE institutions • Provide constructive feedback to the Council’s executive and to project teams on the progress of the projects • Assess the extent to which the programme as a whole (including the arrangements for monitoring and co-ordination) met its objectives • Inform future Council policy on e-learning

13. Evaluation Key Tasks • Develop an evaluation framework/methodology, present this to the projects and undertake the evaluation of the programme • Review progress by the six projects and identify emerging issues • Report to the SFC executive on key emerging issues and to make recommendations (formative feedback) for maximising the impact of the programme in the short and the long term • Report to the SFC on the effectiveness of the arrangements for programme monitoring and co-ordination, and dissemination.

14. Project Visits • Core project documentation (eg project initiation document and project plan) • Project management arrangements (strategic and operational) • Communication strategy and reporting • Project evaluation plans and baseline data gathering • Cultural/organisational issues • Project support needs

15. Project Visits CeLLSUniversity of Dundee, Tuesday 17 May e-ConstructionGlasgow Metropolitan College, Wednesday 18 May BlendEdReid Kerr College, Paisley, Wednesday 1 June (am) ISLE,University of Paisley, Wednesday 1 June (pm) Re-engineering Assessment PracticesUniversity of Strathclyde, Glasgow, Thursday 2 June TESEP,Napier University, Edinburgh, Friday 3 June

16. Evaluation Framework • Programme overview and context • Purpose of evaluation • Evaluation objectives • Methodology • Key informants and other stakeholders • Evaluation outputs and outcomes • Logframe

17. Modified Logframe

18. Project Support Needs • What skills, knowledge and experience are required in the project team in order to complete the project successfully? • Does the project team currently have these skills, knowledge and experience? • What kinds of external advisory support will you need to complete the project successfully? • What form would you like this advice and support to take? • How will you review your project team development and advisory support needs?

19. History of the Future Exercise • Logic Models - tool to describe the effectiveness of a project • Logic Maps - simplified version for articulating implicit theories of change • Distinction between outputs and outcomes • Surfacing assumptions allows the project team to identify differences and reach a consensus

20. History of the Future Exercise • Imagine that your project is completed and that it has succeeded in all of its goals. • You are being interviewed tomorrow on a local radio show to talk about the project. • In order to avoid fluffing your words on live radio the questions have been sent to you in advance so that you can jot down your main points.

21. What to do On the form note: • What did your project actually do • Who’s going to benefit from this • Why is it important • How did you help to make the project a success You have 10 minutes

22. Now share with your team • Form a group with the others in your team • Share your interview notes • Identify any key differences • Briefly feedback to everyone - any surprises?