A Practical Corporate E-Learning Effectiveness Model

1. A Practical Corporate E-Learning Effectiveness Model Lim Kin Chew SIM University 23 Feb 2012 8th International Conference on E-Learning for a Knowledge-Based Society Bangkok, Thailand

2. Quotation - 1 “Although recent attention has increased e-learning evaluation, the current research base for evaluating e-learning is inadequate … Due to the initial cost of implementing e-learning programs, it is important to conduct evaluation studies.” (American Society for Training and Development, 2001).

3. Quotation - 2 “Despite the ample support and enthusiasm for virtual models of learning from both the corporate and educational arenas, the effectiveness of e-Learning is by no means a sure thing. Successful e-Learning programmes require specific conditions and constant monitoring and a lot more research has yet to be done before an acceptable universal model can be developed.” (Kar-tin Lee, Hong Kong Institute of Education, Malaysian Online Journal of Instructional Technology, August 2005, ISSN: 1823-1144, Vol. 2, No.2, pp 61-71)

4. Purpose of this Paper • Share experiences on how an actual evaluation of a large-scale e-learning project was carried out • Add to the research base on e-learning effectiveness

5. Agenda • Why do we need a practical corporate e-learning effectiveness model? • Different approaches to evaluation models • Case study on an e-learning implementation • QPICC model • Lessons learnt • Conclusion

6. Different Approaches to E-learning Evaluation (Source: Graham Attwell (ed.). (2006). Evaluating E-learning - A Guide to the Evaluation of E-learning. Evaluate Europe Handbook Series Volume 2. ISSN 1610-0875.)

7. Why do we need a practical e-learning effectiveness model? • E-learning effectiveness is difficult to carry out. • Even if it is carried out, it is usually done in a rather ad hoc manner. • E-learning evaluation takes up time, resources, manpower, cost and it may end up inconclusive • E-learning evaluation should be done by experienced e-learning practitioners • E-learning evaluation depends very much on what the user client wants to evaluate or measure. The consultant guides him along.

8. Purpose of Project • To evaluate the effectiveness of the e-learning efforts in facilitating the training functions of ABC • Based on the evaluation outcomes, the client wants appropriate recommendations on e-learning capabilities so as to further enhance instructional delivery in the ABC training.

9. Case Study on E-learning Evaluation • ABC (name changed because of confidentiality) is a government body in charge of training personnel for home security. • They have been using e-learning since 2006. • Most of the training has been supplementary, i.e. used to support the classroom face-to-face sessions. • Investments have been made to set up three LMSs • Management now wanted to find out how effective has the e-learning been in training the personnel • Client has several departments. Only three departments were involved in the evaluation project.

10. Objectives & Methodologies - 2 • To evaluate the effectiveness of ABC e-learning training platform in the following areas: • qualify for the pedagogical underpinnings considered for e-learning modules; • measure the quality of e-learning course wares; • measure the performance output by benchmarking the e-learning platform against a set of pre-defined criteria; • determine the adequacy of existing e-learning infrastructure and to validate congruency in these capabilities with their instructional intent; • identify trainer and training designer’s competency gaps in e-learning facilitation and design based on the evaluation outcome.

11. Proposed E-learning Quality & Effectiveness Model for ABC

12. How the Project was organized Team Leader Team UniSIM Team ABC • Participative approach • Expertise transfer • Joint review of results, observations and conclusions • PLAN – DO – CHECK – ACT (PDCA)

13. Evaluation tools used in Project

14. Sample size of Trainees

15. Project Schedule on E-learning Evaluation for ABC

16. E-learning Development Cycle 14 competencies defined 2 competencies defined 9 competencies defined 4 competencies defined 3 competencies defined

17. Effectiveness of E-learning How effective has the e-learning been in the ABC training?

18. Overall Key Findings of the Evaluation - 1 Management Level • Lack of a short-term or long-term plan for e-learning (vision, goals, targets, timeline, etc) • Unsure of the position of e-learning in the overall training curriculum Pedagogical area • Lack of a consistent explanation or definition of e-learning as used in the context of the ABC environment • Have a narrow interpretation that e-learning means delivering courseware or content-based learning

19. Overall Key Findings of the Evaluation - 2 • E-learning is used in a “reinforcement” mode • All the courseware titles (except the scenario-based video courses) that have been evaluated are based on passive learning. Other types like e-Test system are used in Dept A and other Departments have been mentioned but we were not shown. • Need to deploy appropriate instructional strategies (e.g. discussion, lecture, self-directed learning, mentorship, small group work, project, collaborative learning, case study, forum) for effective training using e-learning. Passive learning is based very much on memory work. Need to be aware of paradigms like constructivism and social constructivism to engage the learners. • Trainers need to know how to track trainee’s e-learning progress • Need for a proper assessment of the performance of the trainees in e-learning and then to measure the transfer to his job performance

20. Overall Key Findings of the Evaluation - 3 Organizational Culture towards E-learning • Need sharing of best practices amongst staff members • Need to build up e-learning training resources for e-learning training staff • Need to have an E-learning Outsourcing Development Methodology Awareness and competency levels of e-learning stakeholders • Need to raise awareness on the following areas: • What is e-learning? • What e-learning can do and cannot do? • Why ABC is pushing for e-learning? • Need to move to a more competency-based model to train staff and assign tasks and responsibilities

21. Overall Findings of the Evaluation - 4 Overall Need to have an ABC centre which coordinates the following: • Monitor and raise the e-learning competencies of the e-learning personnel. • Maintain the quality of the courseware or other e-learning activities • ABC can also position itself as the central body for the following activities: • e-learning awareness and promotional activities • Identify foundational e-learning competencies for e-learning personnel in the ABC Training schools • work with external organizations to get the best deals in any tender i.e. courses, infrastructure requirements etc.

22. Scoring Scheme

23. Overall Effectiveness Model

24. % of E-learning in Training Curriculum (Note the low percentages of e-learning component in the training curriculum.)

25. Lessons Learnt - 1 • E-learning evaluation is not straightforward. No set formula or set template or set model to use. • Based very much on what the client wants to measure. • The percentages of e-learning in the overall training curriculum in the three departments are 1.4, 1.2 and 3.2 respectively. Is it worth your while to do the evaluation? • Most of the feedback are feedback gathered from the trainees, supervisors, focus group discussions. These are all very qualitative. • Job performance is the hardest area to determine.

26. Lessons Learnt - 2 • Plan to measure and measure what you have planned. E.g. Quality of courseware, job performance. • Difficult to measure what you have not planned to measure. • Senior Management may have other views about e-learning. Good to have their buy-in.

27. What is E-learning? • E-learning is the use of Internet and digital technologies to create experiences that educate our fellow human beings • E-technologies do not change how human beings learn but remove constraints • E-learning is not just Web-casting lecture, and not training materials dumped online Professor Tapio Varis Unesco Chair in global e-Learning University of Tampere

28. E-Learning Self-directed learning Collaborative learning Peer learning Passive learning Instructor-led Pedagogy LMS Internet Intranet Mobile CD-ROMs Highly structured Open Contents Test-based Knowledge-based Learning objects Rich multimedia Technology Contents

29. End of Presentation Lim Kin Chew kclim@unisim.edu.sg