Maximizing Learning Outcomes from Web-Based Training: A Meta-Analysis

1. Maximizing Learning Outcomes from Web-Based Training: A Meta-Analysis Dr. Traci Sitzmann Advanced Distributed Learning Co-Laboratory

2. ADL Technical Center (Johnsontown, Pennsylvania) Collaborates with all the ADL Co-Labs to develop and validate ADL concepts, technologies and utilities ADL Co-Lab (Alexandria, Virginia) Serves as the Initiative's central organization for guiding, coordinating and integrating the operations of the ADL Network Job Performance Technology Center (Alexandria, Virginia) Acts as a catalyst for the advancement of research, development and implementation of dynamic capability-based job performance technology solutions across the Department of Defense. Workforce ADL Co-Lab (Memphis, Tennessee) Facilitates the development and integration of ADL technologies in industry to enhance the learning and training of the workforce of the future Joint ADL Co-Lab (Orlando, Florida) Promotes collaborative development of ADL prototypes and ADL systems acquisitions, primarily among the US Department of Defense components. ADL Co-Lab Network Academic ADL Co-Lab (Madison, Wisconsin) Serves as the focal point for academia in promoting high quality, reusable content for distributed learning • We are collaborating with • several universities on evaluations: • Colorado State • Cornell University • University of Iowa • UCLA

3. ARE WE: Providing access to the highest quality education and training, tailored to individual needs, delivered cost effectively, anywhere and anytime? Realizing the ADL Vision Web-based Learning Home Field Online Gaming Distributed Simulation School In Transit Embedded Training Content Repositories Office

4. Purpose of Current Research • Compare Web-based instruction (WBI) to classroom instruction (CI) in terms of: • Its effectiveness for teaching declarative and procedural knowledge • Student reactions to training • Examine moderators of the effectiveness of WBI

5. Method • 2 researchers independently coded 96 articles • Types of studies included: • 65 published studies • 18 dissertations • 13 unpublished studies • Note: Send relevant studies to traci.sitzmann.ctr@adlnet.gov

6. Participant Demographics • 19,331 students • Types of courses • 113 undergraduate courses • 29 graduate courses • 26 corporate training courses • Average age = 24 years

7. Analyses • Hedges and Olkin’s (1985) procedure → mean corrected d effect d > 0 indicates WBI is more effective than CI d = 0 indicates WBI and CI are equally effective d< 0indicates CI is more effective than WBI • Subgroup procedure → test for categorical moderators

8. Meta-Analytic Notional Example

9. Effectiveness of WBI Declarative knowledge d = .15* • WBI is 6% more effective than CI for teaching declarative knowledge Procedural knowledge d = -.07 • WBI and CI are equally effective for teaching procedural knowledge Satisfaction with training d = .00 • Trainees are equally satisfied with WBI and CI

10. How can I design more effective Web-based training courses? 8 moderator variables: • Similarity of instructional methods • Experimental design • Learner control • Human interaction • Practice • Feedback • Population • Age • Length

11. Meta-Analytic Notional Example

12. Does the quality of research matter? Media are “mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes change in our nutrition.” -- Clark (1983) • Instructional methods include lecture, practice, examples, discussion, videos, etc. • Delivery media include CI, computer-assisted instruction, WBI, etc.

13. Similarity of instructional methods H1: WBI and CI will be equally effective for teaching declarative knowledge when similar instructional methods are used to deliver the two courses. • Same instructional methods d = .04 • Different instructional methods d = .29 • Results support Clark’s theory

14. Experimental Design H2: WBI relative to CI will be equally effective when trainees are randomly assigned to courses. • Experimental designs d = -.26 • Quasi-experimental designs d = .18 • Results support Clark’s position that media effects in single study research are largely spurious

15. Learner Control Q1: Will the level of learner control moderate the extent to which trainees learn declarative knowledge from WBI relative to CI? • Low learner control d = .07 • High learner control d = .30 • Trainees learn more when they have control in Web-based courses

16. Human Interaction H3: Relative to CI, trainees will learn more declarative knowledge in classes with higher levels of human interaction than classes where there is little human interaction during WBI. • Low human interaction d = .19 • High human interaction d = .18 • Hypothesis 3 not supported

17. Practice H4: Relative to CI, trainees will learn more declarative knowledge when they practice during WBI. • Only WBI incorporated practice d = .31 • WBI and CI incorporated practice d = .16 • Only CI incorporated practice d = -.27 • Neither WBI or CI incorporated practice d = -.25 • Practice is beneficial during both WBI and CI, supporting hypothesis 4

18. Feedback H5: Relative to CI, trainees will learn more declarative knowledge when they receive feedback during WBI. • Only WBI incorporated feedback d = .33 • WBI and CI incorporated feedback d = .16 • Only CI incorporated feedback d = -.27 • Neither WBI or CI incorporated d = .08 feedback • Feedback is beneficial during both WBI and CI, supporting hypothesis 5

19. Population & Age Q2: Will the population (student v. employees) moderate the effectiveness of WBI relative to CI for teaching declarative knowledge, after controlling for the age of trainees? • Age accounted for a significant 44.2% of the variance in declarative knowledge • Effectiveness of the delivery media did not differ for student and employee samples, after controlling for the age of trainees

20. Length of Training Q3: Does the length of training moderate learning declarative knowledge from WBI relative to CI? • Number of days spent in training correlated .33 with the declarative knowledge effect size • Relative to CI, Web-based trainees gain more declarative knowledge as the length of the class increases

21. Conclusions • WBI and CI are equally effective for teaching declarative knowledge when the same instructional methods are used • Overall WBI and CI are equally effective for teaching procedural knowledge • Trainees are equally satisfied with the two delivery media

22. Designing More Effective Web-based Training Courses • Provide trainees with control during training • Online communication should be synchronous rather than asynchronous • Incorporate a variety of instructional methods • Require active involvement of trainees • Provide an Internet skills course for trainees lacking technical skills • Trainees should practice the training material • Provide trainees with feedback throughout the course