LEMMA: Learning Environment in Multilevel Modelling and Applications

1. LEMMA: Learning Environment in Multilevel Modelling and Applications Fiona Steele London School of Economics & Political Science Director NCRM LEMMA node, University of Bristol http://www.bristol.ac.uk/cmm/learning/online-course/index.html

2. LEMMA Training • Capacity building in the analysis of data with complex structure • Ultimate goal is to move learners to “take-off”, i.e. conducting and publishing multilevel analyses • Different modes of delivery • Face-to-face workshops • Web-based materials in a virtual learning environment

3. LEMMA e-learning: First steps • Originally planned to host searchable repository of materials • BUT … • Written by different authors using different styles and notation • Not clear how to tag and organise materials • Concluded that we needed new materials organised as a course • Coherent structure – provide ladder for novices • Ensure consistent terminology and notation → LEMMA online course launched April 2008

4. Lessons from Face-to-Face Training • Learners need motivation and time • Best motivated are those with multilevel data and research questions • Learners often do not possess prerequisites for multilevel modelling • In practical sessions, learners tend to focus on mechanics of using software rather than interpretation

5. LEMMA online course: principles • Accessible to anyone with basic statistics training • Modules to have two integrated components: concepts and practical • Facility for learner’s self-evaluation (quizzes) • Collect data to evaluate materials • Design materials so they can be modified by other trainers

6. A typical module • Concepts • Approx 40-50 pages, split into lessons • Illustrative examples based on a single dataset • Draw links between fitted model equations, graphs of predictions and verbal interpretation • No reference to software • Practical • MLwiN, Stata and R (SPSS coming soon) • Detailed analysis of one dataset (different to that used in Concepts) • Quizzes

8. Current modules (1) • Using quantitative data in research • Introduction to quantitative data analysis • Multiple regression • Multilevel structures and classifications • Introduction to multilevel modelling • Regression models for binary responses • Multilevel models for binary responses • Multilevel modelling in practice Note: Modules 1-3 and 6 not on multilevel modelling

9. Current modules (2) • Single-level and multilevel models for ordinal responses • Single-level and multilevel models for nominal responses • Three-level multilevel models • Cross-classified multilevel models • Multiple membership multilevel models • Missing data • (coming soon) Multilevel models for repeated measures

10. Choice of e-learning application • In 2005 choice was between Moodle and Blackboard • Moodle chosen because: • Free and open-source • Excellent on-line community support • More easily customised than Blackboard

11. Growth in number of registered users Launched April 2008 Now ~14k users

12. Intended use of materials* *All figures correct as of 16/6/14

13. Users by sector

14. Users by country 72% users from outside the UK

15. Users by primary discipline

16. User familiarity with statistical methods

17. User familiarity with mathematical statistics

18. Evaluation of LEMMA online course • Survey of UK users conducted by NCRM in 2013 • 264 responses received • Full report at http://eprints.ncrm.ac.uk/3261/1/MOLEY_NCRM_Impact_2011-2013.pdf

19. Summary statistics on usage • Median time spent on course is 10 hours (SD = 14.8) • ~ 65% had at least partially completed modules on Introduction to Quantitative Research and Multiple Regression (Modules 1-3) • So important to provide introductory materials as preparation for multilevel modelling modules Source: ESRC NCRM survey

20. Reasons for registration

21. Combining with face-to-face training “The course enabled me to get better value from face-to-face training, by providing in-depth preparatory reading. Together with the face-to-face training, I improved my knowledge of MLM to a sufficient degree to engage in detailed discussions about analytical methods with academic collaborators.” “…I’d like to take further classes in this, but I need the online course first so I’m prepared for the further courses.” Respondents to NCRM survey. Online course modules are recommended reading for LEMMA face-to-face courses.

22. Impact of LEMMA online course Source: ESRC NCRM survey

23. Resource considerations • Writing online materials is resource intensive • Materials unsupported so need to be clear and thorough • Ensure consistency of style and notation across modules and cross-reference previous modules • Writing quizzes and feedback especially time-consuming • Wide range of technical expertise required • Installation and maintenance of Moodle • Programming skills for customising Moodle • Web design • Database management • Ongoing technical support and trouble-shooting

24. Future developments • New materials in pipeline • SPSS practicals for selected modules • Module on longitudinal data analysis • Convert materials to interactive e-books • New ESRC project led by William Browne

25. Thanks • Funding for 3 phases of the NCRM LEMMA node, 2005-2015 • Technical team • SachaBrostoff • Hilary Browne • Christopher Charlton • Hugh Garner • Authors